Rhizamonitor
live

Run

Fix Run drivers burn CPU while

done ship-change

Steps

  1. build/builddone3h 19m

    Nov 5 13:09 · 4 of 4 leaves passed

  2. build/renderdone1s

    Nov 5 13:09 · exit 0

  3. build/checkdone3s

    Nov 5 13:09 · exit 0

  4. review/codedone7s

    Nov 5 13:09 · exit 0

  5. review/reviewdone55m 46s

    Nov 5 14:05 · 5 of 5 checks passed

  6. review/renderdone1s

    Nov 5 14:05 · exit 0

  7. render-prdone2s

    Nov 5 14:05 · exit 0

  8. open-prdone9s

    Nov 5 14:05 · exit 0

  9. wait-prdone35m 50s

    Nov 5 14:41 · held after 67 checks

  10. pr-greenfail3s

    Nov 5 14:41 · exit 1 PR #304 at 4a3cdfd: failed: check; failing tests: TestScripts/check, TestScripts/stacked-cycle

  11. ci-triagedone1s

    Nov 5 14:41 · exit 0

  12. ci-fixdone1m 38s

    Nov 5 14:43 · said done and until held

  13. render-prdone1s

    Nov 5 14:43 · exit 0

  14. open-prdone8s

    Nov 5 14:43 · exit 0

  15. wait-prdone35m 3s

    Nov 5 15:18 · held after 66 checks

  16. pr-greenfail3s

    Nov 5 15:18 · exit 1 PR #304 at 3b8fc46: failed: check; failing tests: TestCatchUpBudget

  17. ci-triagedone2s

    Nov 5 15:18 · exit 0

  18. ci-fixdone4m 8s

    Nov 5 15:22 · said done and until held

  19. render-prdone1s

    Nov 5 15:22 · exit 0

  20. open-prdone8s

    Nov 5 15:22 · exit 0

  21. wait-prdone38m 51s

    Nov 5 16:01 · held after 73 checks

  22. pr-greenfail3s

    Nov 5 16:01 · exit 1 PR #304 at e12d7be: failed: check; failing tests: TestMakeTestTimeout

  23. ci-triagedone1s

    Nov 5 16:01 · exit 0

  24. ci-fixdone3m 41s

    Nov 5 16:05 · said done and until held

  25. render-prdone1s

    Nov 5 16:05 · exit 0

  26. open-prfail5s

    Nov 5 16:05 · exit 1 rhiza: git push: remote: fatal error in commit_refs To github.com:dphaener/rhiza.git ! [remote rejected] cs/fix-run-drivers-burn-cpu-while -> cs/fix-run-drivers-burn-cpu-while (failure) error: failed to push some refs to 'github.com:dphaener/rhiza.git'

  27. open-prfail57s

    Nov 5 16:06 · exit 1 rhiza: gh pr edit 304 --body-file /home/rhiza/tmp/rhiza-pr-body-3067625931.md: GraphQL: Something went wrong while executing your query on 2026-11-05T16:06:20Z. Please include `C4DE:337E4C:1BB578:1DE70D:6AECA97C` when reporting this issue.

  28. open-prfail1m 45s

    Nov 5 16:08 · exit 1 rhiza: gh pr edit 304 --body-file /home/rhiza/tmp/rhiza-pr-body-2907540501.md: GraphQL: Something went wrong while executing your query on 2026-11-05T16:08:05Z. Please include `E22C:3F668F:1CCC78:1F03D7:6AECA9E3` when reporting this issue.

  29. open-prdone10m 39s

    Nov 5 16:18 · exit 0

  30. wait-prdone8m 39s

    Nov 5 16:27 · held after 17 checks

  31. pr-greendone3s

    Nov 5 16:27 · exit 0

  32. signoffdone0s

    Nov 5 16:27 · approved by rule autonomous

  33. rebasedone8s

    Nov 5 16:27 · exit 0

  34. wait-rebaseddone14m 41s

    Nov 5 16:42 · held after 28 checks

  35. rebased-greendone2s

    Nov 5 16:42 · exit 0

  36. landdone33s

    Nov 5 16:42 · exit 0

  37. wait-landeddone13m 31s

    Nov 5 16:56 · held after 26 checks

  38. landed-greendone3s

    Nov 5 16:56 · exit 0

  39. rebase-mergedone2s

    Nov 5 16:56 · exit 0

  40. wait-rebased-mergedone3s

    Nov 5 16:56 · held after 1 checks

  41. rebased-merge-greendone2s

    Nov 5 16:56 · exit 0

  42. mergedone6s

    Nov 5 16:56 · exit 0

  43. wait-maindone2s

    Nov 5 16:56 · held after 1 checks

  44. main-greendone2s

    Nov 5 16:56 · exit 0

  45. close-outdone8s

    Nov 5 16:56 · exit 0

  46. shipdone17s

    Nov 5 16:57 · exit 0

Review

compliance passed

No findings.

quality passed

No findings.

qa passed

  • minor The new budget tests TestReadBudgetApply (internal/api/write_budget_test.go) and TestCatchUpBudget (internal/index/refold_test.go) do not match BUDGET_TESTS = ^(TestReadBudget|TestBudget|TestGeneratedModel)$, so make check runs them in the parallel non-budget pass beside every other package, not in the serial -p 1 budget pass under the flock. TestCatchUpBudget has no quiettest.Wait either. Under CI load their medians (125 ms of 200, 34 ms of 50 measured here) may flake. Consider adding them to BUDGET_TESTS. Makefile:35

scope passed

No findings.

model passed

  • nit The leaf names apply and add, set, link, retire, mv, but runWriteOn opens the write index for every write it runs when IndexPath is set (unlink and the other single writes too). The code does more than the model says; the statement could say 'every write through runWriteOn'. internal/api/write.go:189
  • nit apply-validates-from-index is a write-path capability placed under fast-reads, whose statement is 'Read commands answer in under 50 ms'. It fits loosely; a write-side parent would describe it better. .rhiza/rendered/capability/rhiza/views/fast-reads.md:11
  • nit The driver keeps answers in memory across polls (revalidated by per-change-set Signature) with a 50 ms poll debounce and a 2 s settle after its own write. Neither the leaf nor model-reads-go-through-the-index records the debounce/settle rule, and the decision's 'no in-memory fold kept across commands' holds only because a driver is one process. Worth a sentence in the model; behaviour itself is fine. internal/api/run_model_cache.go:212

Log

log · tail
  - attempt 1 builder, session 81e1aba2-8ceb-4752-a378-01a1ab2326ec: pass (said done)
  - attempt 1 verifier, session e4d9e831-b53a-404b-9572-01a1ab3d8a51: fail (verifier found 2 problem(s))
  - attempt 2 builder, session f638fe45-8c6c-4df4-8462-01a1ab53b525: pass (said done)
  - attempt 2 verifier, session 998f373d-3e79-48fa-b020-01a1ab685887: fail (verifier found 1 problem(s))
  - attempt 3 builder, session 8f01539a-32be-46d2-be4c-01a1ab8117bb: pass (said done)
  - attempt 3 verifier, session a5e353eb-b5c3-4f42-8110-01a1ab9b2b4d: pass (Evidence written by the verifier)
  - attempt 1 finding f1: Acceptance names `go test -trimpath ./internal/api -run TestReadBudget` as timing change show, but the change show timing lives in a separate TestChangeShowBudget that `-run TestReadBudget` does not match: `go test -trimpath ./internal/api -run 'TestReadBudget$' -v` prints no 'change show' line (count 0). Move the timing into TestReadBudget (e.g. as part of its loop or a t.Run subtest) so the acceptance command runs it.
  - attempt 1 finding f2: `go test -trimpath ./internal/api/...` (acceptance command) FAILED: TestChangeShowBudget 'change show accepted after an unrelated entry median over 50ms on every attempt' (attempt 3/3 median 91.3ms, max 182ms). Run alone (`-run 'TestReadBudget|TestChangeShowBudget'`) it passed, but only on retries: accepted fresh 62ms then 37ms, accepted after an unrelated entry 52ms, 53ms, then 46ms on the third and last attempt. So the 50ms budget for change show of the accepted change set is not met reliably. The functional tests pass: TestChangeShowIndexedMatchesFold, TestChangeShowIndexedErrors, TestShipCheckSettledReadsNoLog, `-short ./internal/api/...`, ./internal/index/..., and the four named testscripts.
  - attempt 2 finding f1: Acceptance `go test -trimpath -count=1 ./internal/api -run 'TestReadBudget$'` failed 3 of 4 runs at f55faac8, each time on change show over the 50ms median on all 3 attempts: run 1 'accepted after an unrelated entry' 74/79/70ms; run 2 'accepted fresh' 56/79/68ms; run 4 'draft after an unrelated entry' 92/65/57ms. Run 3 passed with medians 30-44ms. The machine was contended for all four runs (load 5-8 on 4 cores, another builder's go test running). Under the same load, TestReadBudget's existing reads (show/ls/tree/what-if, 4-47ms) stayed within budget, with only roadmap what-if once needing a retry. Change show sits at about 30-45ms even on good attempts, so it has too little headroom to meet the 50ms median reliably, and the timing moved into TestReadBudget now fails that test. Everything else passes: TestChangeShowIndexedMatchesFold (zero log reads, notes, after an unrelated entry), TestChangeShowIndexedErrors, TestShipCheckSettledReadsNoLog, `-short ./internal/api/... ./internal/index/...`, and the four named testscripts.
- apply-validates-from-index (step build/build): passed: attempt 2: verifier passed
  - attempt 1 builder, session 82402066-6da8-4ca9-a09d-01a1ab3db176: pass (said done)
  - attempt 1 verifier, session f462defb-bdec-431f-903b-01a1ab538fb7: fail (verifier found 2 problem(s))
  - attempt 2 builder, session 3dc30362-7b4c-4459-92c6-01a1ab687e3d: pass (said done)
  - attempt 2 verifier, session 766eb5c6-3c4c-4d2a-ac8e-01a1ab80f5f6: pass (Evidence written by the verifier)
  - attempt 1 finding f1: Writes through the index are not accepted or refused as the full fold does. On the indexedLog fixture (internal/api/change_indexed_test.go) in the draft change set: retire ssd (a node the change set created, which has child nvme), then `set {"ref":"nvme","fields":["statement=x"]}` or `mv {"ref":"nvme","parent":"store"}`. The folded path (IndexPath empty) accepts both, rewriting nvme's node.create op. The indexed path refuses both with `error 4: no node matches "nvme"`. I found this with a probe test in a scratch copy of 0ca42cfb that compared dry-run answers step by step. The leaf's property test TestWritesThroughIndexMatchFold passes only because its generator steers around this case: the comments on its mv and retire cases say 'the what-if view drops the child, which the interim model keeps'. So the acceptance point 'every write accepted or refused, with the same problems, through the index as through the full fold' does not hold. Fix the divergence, or get it decided, and then remove the generator's exclusions.
  - attempt 1 finding f2: The acceptance asks for 'a property test over generated logs and writes', but TestWritesThroughIndexMatchFold generates only writes. Every seed runs over the same fixed indexedLog fixture, and the seed changes only the write sequence. No log shape is generated: not the state of other change sets, not landed versus in-flight dependencies, not conflicts. So the equivalence is tested on one hand-built log.
- run-driver-idle-poll-cheap (step build/build): passed: attempt 1: verifier passed
  - attempt 1 builder, session 727185aa-0c98-4ba0-8110-01a1ab9b4dcf: pass (said done)
  - attempt 1 verifier, session d39564a9-aa7c-41fc-8354-01a1abc8f2d4: pass (Evidence written by the verifier)
- change-show-ship-check-from-index (step review/review): open
  - attempt 2 fixer in cycle 2, session 4476136a-e99f-41c5-9a32-01a1abedc79d: pass (said done)
  - attempt 2 verifier in cycle 2, session ab9b4f3b-61e0-4edb-9a70-01a1abef6e40: pass (Evidence written by the verifier)
- run-driver-idle-poll-cheap (step review/review): open
  - attempt 2 fixer in cycle 2, session 4a0f87f3-a0be-4cef-a5ab-01a1abef8cce: pass (said done)
  - attempt 2 verifier in cycle 2, session 00b28496-a233-48bc-ad58-01a1abf8d947: pass (Evidence written by the verifier)

## Checks

- compliance (step review/review): passed after 1 cycle
  - cycle 1: passed, check session 63f7ff28-ae0c-416f-993d-01a1abd45206 pass (said done)
  - round 1 (cycle 1): no findings
- quality (step review/review): passed after 2 cycles
  - cycle 1: failed, check session c4e29b63-e3f0-4777-951e-01a1abd47251 pass (said done), critic session 231ce413-b3b0-466c-b200-01a1abec6084 pass (said done)
  - cycle 2: passed, check session 1cf31d49-e743-45c4-a02a-01a1ac0252ec pass (said done)
  - round 1 (cycle 1): 3 findings, 2 confirmed
    - f1 major at internal/api/change_indexed.go:36 on change-show-ship-check-from-index: change show over the index builds its view from the strict what-if overlay (x.WhatIfOverlay), which drops a node whose create no longer applies; the interim model keeps it, and write_indexed.go's own comment names exactly this difference. So change show no longer gives the same answers as before. Repro in a scratch repo: `add capability top --record`; `change new Edge`; `add capability early`; `add capability late`; `mv early --parent late`; then `change show --json`. The merge-base binary lists leaves [early]; this branch lists leaves [late], because `early` is gone from the view and `late` looks childless. Runs build leaves, so the run would build the parent and skip the real leaf. TestChangeShowIndexedMatchesFold uses one hand-made log and misses this; the change needs a generated-log property test like TestWritesThroughIndexMatchFold, or the same writeLayer view the write path uses. — confirmed: Reproduced in a scratch repo with the exact steps (add capability top --record; change new Edge; add capability early; add capability late; mv early --parent late). The merge-base binary's change show --json gives leaves [early]; this branch's gives [late]. newIndexModel (change_indexed.go:36) uses the strict x.WhatIfOverlay, which write_indexed.go:145-148 itself says drops a node whose create no longer applies (a move under a node created later). The interim model keeps that node, so change show answers differ, and runs build leaves from it.
    - f2 major at internal/api/run_leaves.go:436 on run-driver-idle-poll-cheap: A verifier wait's Evidence poll is cached under the zero change set, so any entry anywhere in the log drops the cached answer and the next poll recomputes it. leafEvidence needs w.changeSets(), and through the index that is FoldedOf over every change set, gob-decoding each one with its history. On this repo (2617 change sets) one FoldedOf of all sets took 200-300 ms wall on parallel decode workers, so each recompute costs more CPU than that. While agents write, this happens on almost every poll, which is the CPU burn this change set exists to remove. nodeCreator only needs the change sets that created the Evidence nodes linked to the leaf (their history), so it could read FoldedOf of those alone and key the cache by the run's change set. driverPollBudget polls with leaf "" and never times Evidence, so the budget test cannot see this. — confirmed: Evidence (run_leaves.go:436) calls readModel with cs = id.ID{}. revalidate (run_model_cache.go) drops every kept value whose cs is zero whenever the stamp moves, so any entry anywhere in the log forces a recompute. The recompute calls w.changeSets() -> writeIndex.changeSets -> FoldedOf(all set IDs), which gob-decodes every stored folded change set with its history. That breaks the leaf's statement that a poll after an unrelated entry decodes and folds only that change set. driverPollBudget polls with no Evidence, so the budget test cannot catch it.
    - f3 minor at internal/api/run_model_cache_test.go:245 on run-driver-idle-poll-cheap: The 'after an unrelated entry' CPU budget opens the index twice from another reader before each measured poll, so the driver's own catch-up of the appended entry is never measured. The acceptance asks for the CPU of a poll after one unrelated appended entry. In production the driver's Refresh pays that catch-up, unless another reader got there first.
  - round 2 (cycle 2): no findings
- qa (step review/review): passed after 1 cycle
  - cycle 1: passed, check session edef540b-2d98-4fe4-8cf2-01a1abd497e3 pass (said done)
  - round 1 (cycle 1): 1 finding, 0 confirmed
    - f1 minor at Makefile:35 on apply-validates-from-index: The new budget tests TestReadBudgetApply (internal/api/write_budget_test.go) and TestCatchUpBudget (internal/index/refold_test.go) do not match BUDGET_TESTS = ^(TestReadBudget|TestBudget|TestGeneratedModel)$, so make check runs them in the parallel non-budget pass beside every other package, not in the serial -p 1 budget pass under the flock. TestCatchUpBudget has no quiettest.Wait either. Under CI load their medians (125 ms of 200, 34 ms of 50 measured here) may flake. Consider adding them to BUDGET_TESTS.
- scope (step review/review): passed after 1 cycle
  - cycle 1: passed, check session 3a64832e-a628-432d-84b8-01a1abd4c652 pass (said done)
  - round 1 (cycle 1): no findings
- model (step review/review): passed after 1 cycle
  - cycle 1: passed, check session fc8f14b8-db83-4a9d-9e4c-01a1abd4fccf pass (said done)
  - round 1 (cycle 1): 3 findings, 0 confirmed
    - f1 nit at internal/api/write.go:189 on apply-validates-from-index: The leaf names apply and add, set, link, retire, mv, but runWriteOn opens the write index for every write it runs when IndexPath is set (unlink and the other single writes too). The code does more than the model says; the statement could say 'every write through runWriteOn'.
    - f2 nit at .rhiza/rendered/capability/rhiza/views/fast-reads.md:11 on apply-validates-from-index: apply-validates-from-index is a write-path capability placed under fast-reads, whose statement is 'Read commands answer in under 50 ms'. It fits loosely; a write-side parent would describe it better.
    - f3 nit at internal/api/run_model_cache.go:212 on run-driver-idle-poll-cheap: The driver keeps answers in memory across polls (revalidated by per-change-set Signature) with a 50 ms poll debounce and a 2 s settle after its own write. Neither the leaf nor model-reads-go-through-the-index records the debounce/settle rule, and the decision's 'no in-memory fold kept across commands' holds only because a driver is one process. Worth a sentence in the model; behaviour itself is fine.

## Code checks

- step review/code: pass

## Sign-offs

- signoff (visit 1): approved by rule autonomous

## Retries

- 2026-11-05T16:04:09Z retried at ci-fix, after: step "ci-fix" went over max_visits 2
- 2026-11-05T16:06:14Z retried at open-pr, after: step "open-pr" ended with outcome "fail"
- 2026-11-05T16:07:59Z retried at open-pr, after: step "open-pr" ended with outcome "fail"
- 2026-11-05T16:18:39Z retried at open-pr, after: step "open-pr" ended with outcome "fail"

## Sessions

- build/build, session e316e405-325d-4d41-a0f9-01a1ab1d53c8: tokens input 30, output 16648, cache creation 73288, cache read 1071295; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session debd4e8b-de3a-4cf8-819f-01a1ab21465a: tokens input 22, output 3567, cache creation 39059, cache read 639535; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session 81e1aba2-8ceb-4752-a378-01a1ab2326ec: tokens input 136, output 51126, cache creation 170150, cache read 10006483; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session e4d9e831-b53a-404b-9572-01a1ab3d8a51: tokens input 30, output 4157, cache creation 46217, cache read 812266; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session 82402066-6da8-4ca9-a09d-01a1ab3db176: tokens input 94, output 39200, cache creation 150100, cache read 6298876; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session f462defb-bdec-431f-903b-01a1ab538fb7: tokens input 24, output 5845, cache creation 46401, cache read 733804; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session f638fe45-8c6c-4df4-8462-01a1ab53b525: tokens input 82, output 26797, cache creation 98773, cache read 3875817; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session 998f373d-3e79-48fa-b020-01a1ab685887: tokens input 38, output 5604, cache creation 46873, cache read 1037761; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session 3dc30362-7b4c-4459-92c6-01a1ab687e3d: tokens input 80, output 25583, cache creation 114043, cache read 3918391; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session 766eb5c6-3c4c-4d2a-ac8e-01a1ab80f5f6: tokens input 36, output 7428, cache creation 54748, cache read 1202941; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session 8f01539a-32be-46d2-be4c-01a1ab8117bb: tokens input 110, output 33152, cache creation 121044, cache read 5829534; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session a5e353eb-b5c3-4f42-8110-01a1ab9b2b4d: tokens input 24, output 4052, cache creation 37926, cache read 680227; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session 727185aa-0c98-4ba0-8110-01a1ab9b4dcf: tokens input 146, output 64323, cache creation 176130, cache read 10371593; helper tokens input 0, output 0, cache creation 0, cache read 0
- build/build, session d39564a9-aa7c-41fc-8354-01a1abc8f2d4: tokens input 18, output 4113, cache creation 44490, cache read 531433; helper tokens input 0, output 0, cache creation 0, cache read 0
- review/review, session 63f7ff28-ae0c-416f-993d-01a1abd45206: tokens input 40, output 5245, cache creation 65494, cache read 1365245; helper tokens input 0, output 0, cache creation 0, cache read 0
- review/review, session c4e29b63-e3f0-4777-951e-01a1abd47251: tokens input 54, output 13792, cache creation 87203, cache read 2442432; helper tokens input 0, output 0, cache creation 0, cache read 0
- review/review, session edef540b-2d98-4fe4-8cf2-01a1abd497e3: tokens input 84, output 23229, cache creation 80017, cache read 3294154; helper tokens input 0, output 0, cache creation 0, cache read 0
- review/review, session 3a64832e-a628-432d-84b8-01a1abd4c652: tokens input 14, output 1828, cache creation 41444, cache read 393303; helper tokens input 0, output 0, cache creation 0, cache read 0
- review/review, session fc8f14b8-db83-4a9d-9e4c-01a1abd4fccf: tokens input 46, output 8775, cache creation 72940, cache read 1847360; helper tokens input 0, output 0, cache creation 0, cache read 0
- review/review, session 231ce413-b3b0-466c-b200-01a1abec6084: tokens input 26, output 5028, cache creation 40154, cache read 727378; helper tokens input 0, output 0, cache creation 0, cache read 0
- review/review, session 4476136a-e99f-41c5-9a32-01a1abedc79d: tokens input 20, output 5126, cache creation 46884, cache read 637286; helper tokens input 0, output 0, cache creation 0, cache read 0
- review/review, session ab9b4f3b-61e0-4edb-9a70-01a1abef6e40: tokens input 44, output 6118, cache creation 39655, cache read 1161307; helper tokens input 0, output 0, cache creation 0, cache read 0
- review/review, session 4a0f87f3-a0be-4cef-a5ab-01a1abef8cce: tokens input 50, output 10156, cache creation 61404, cache read 1746441; helper tokens input 0, output 0, cache creation 0, cache read 0
- review/review, session 00b28496-a233-48bc-ad58-01a1abf8d947: tokens input 20, output 3657, cache creation 34353, cache read 540998; helper tokens input 0, output 0, cache creation 0, cache read 0
- review/review, session 1cf31d49-e743-45c4-a02a-01a1ac0252ec: tokens input 24, output 4544, cache creation 49005, cache read 785101; helper tokens input 0, output 0, cache creation 0, cache read 0
- ci-fix, session d18e38c9-98d1-4c21-8571-01a1ac2869f9: tokens input 26, output 4685, cache creation 37586, cache read 720220; helper tokens input 0, output 0, cache creation 0, cache read 0
- ci-fix, session 7ddac7db-34fa-4a8b-b53a-01a1ac4a3441: tokens input 20, output 3313, cache creation 28893, cache read 510135; helper tokens input 0, output 0, cache creation 0, cache read 0
- ci-fix, session 02639b67-1552-4d91-9e64-01a1ac7418f6: tokens input 14, output 2091, cache creation 22739, cache read 326443; helper tokens input 0, output 0, cache creation 0, cache read 0

Totals: tokens input 1352, output 389182, cache creation 1927013, cache read 63507759; helper tokens input 0, output 0, cache creation 0, cache read 0