Ask a coding agent to do the same non-trivial thing twice, and you get two different implementations. Both fine. Both defensible. Denis had been noticing this for months and doing absolutely nothing about it, because with a single run there is nothing you can do — one run is an anecdote. You read the diff, you decide you like it or you don't, and the actual question never comes up.
The question being: what would it have done the other nine times?
You can't see that at n=1. And n=2 doesn't really help either, because two runs is a comparison, and what he wanted was a distribution — where the model lands reliably, and where it is quietly flipping a coin on his behalf.
So he spent a day building a framework that runs one task ten times in parallel, keeps every run isolated from the others, and then shows what the ten did differently. It is called multirun, and it lives here:
Note: this post is a direct continuation of Denis's "Preview Environments" and "Agentic Workflows: Branched Environments" posts. Same core trick, pointed somewhere new. If the words "database branching" mean nothing to you yet, the second one explains them properly, so it doesn't get repeated here.
Introduction
He built it out of curiosity. What came out the other end were two uses worth recommending to somebody else, so those go up front, and the machinery comes afterwards.
The first one is evals. Denis had never written one. Not properly, not against his own prompts. Every time the subject came up it sounded like infrastructure — a harness, a judge model, a golden dataset, some score to chase — and it went onto the list of things he would do when he had a spare week. Well… the spare week never arrived.
Running the same task ten times and reading where the ten answers agree is an eval. A crude one, but a real one, and there is no judge model or benchmark anywhere in this post. The ten runs in here converged completely on the schema and then split exactly 5/5 on two separate decisions, and those splits are the result. They name the sentences his task file was missing. He did not have to guess at them, and nothing had to be built to find them beyond a loop that runs the task more than once.
The second one he has been calling results branching, which is a straight lift from database branching, and he is keeping it. It is for the days when you are about to build the thing you will live with for years — the tenancy model, the auth boundary, the schema that will have three years of rows in it before you get another shot at it. The normal habit is to ask the agent, read what comes back, and either take it or argue with it. That is one sample out of a distribution you never get to see.
So don't take the first answer. Fan the same task out ten ways, let them run without seeing each other, and read the spread before you commit to one. The fan-out in this post took fifty minutes of wall time and about thirty-two dollars at API rates, which is not nothing, but it is a great deal less than what a wrong ON DELETE costs you eighteen months later.
Neither of those needs much cleverness, honestly. What they need is ten runs that cannot touch each other, and that is the part that is actually hard.
The isolation is the whole design problem, and it splits into two halves.
Ten agents editing one repository is nonsense. That half is easy — git has had worktrees for years, every run gets its own, done.
Ten agents sharing one database is worse than nonsense. That half was hard for a long time, and it stopped being hard the moment the database learned how to branch.
That is really the entire reason this project became buildable in a day instead of a quarter.
TL;DR
multirun — point it at a repository at a specific sha, give it a task, and it fans out N isolated runs of Claude Code, capturing everything each one did.
Every run gets a git worktree and its own Postgres branch, both off the same seeded parent. Diffing a run's branch back against that parent is how you see what the agent did to the data, not just to the files.
First real fan-out: ten runs, same task, same model,
max_parallel: 3, on his k3s cluster. 10/10 finished, 20/20 checks passed, 50 minutes wall, 930 turns, zero leaked branches.All ten reached the same schema shape. Then they split 5/5 on whether deleting a team deletes its data, 5/5 on the join table's primary key, and 8/2 on whether the new column is allowed to be NULL.
Convergence on structure, coin flips on judgment. That's the finding — and there was no way to see it without running the thing ten times.
Two uses fell out of it. It is the cheapest eval you can run — where the ten answers split is exactly where your prompt was ambiguous — and it is a way to branch a decision you will live with for years, instead of taking the agent's first answer for it.
What a run is
The whole framework fits on one line:
run = (source, task, variant) -> artifacts
Source is a repository at a specific sha, plus an optional database seed. Never vendored, always pinned — if the ten runs don't start from a byte-identical tree, then comparing them means nothing at all.
Task is a markdown prompt and its checks, versioned as files, so that an old task can be re-run against a newer source later.
Variant is whatever differs per run: the model, a prompt variation, or nothing at all. That last case is the one he actually cares about, because when nothing differs, the only variable left is the model itself.
Everything else in the system — the tables, the dashboard, the comparison writeups — is just a view over artifacts.
Why branching, and not copying
Ten runs need ten databases, and they all need to begin life identical.
The obvious approach is to copy one database ten times, and the obvious approach is bad. A copy costs you the full size of your data every single time, it takes minutes, and afterwards you own ten real databases that you have to remember to delete. At 30 MB that is merely annoying. At anything resembling production data, you will not do it twice.
Branching is not copying. Because Neon separates compute from storage, a branch is a new logical pointer into the same storage, and only the pages that actually diverge ever get written — nothing is duplicated up front. He went through the mechanics at length in Agentic Workflows: Branched Environments, so here it is just the number that matters: a branch taken off this bench's parent while this post was being written was ready in 1.38 seconds.
That parent holds about 30 MB. He has not tried this at 300 GB, so take it as reasoning and not as a measurement.. but the whole point of copy-on-write is that the number shouldn't move much, because nothing is being copied.
The part he actually wanted
Provisioning speed is nice. The diff is the reason.
Every run branches off the same parent, so when a run finishes its branch gets diffed back against that parent, and that gives two things: the schema delta (what DDL it emitted) and the data delta (rows added, changed, deleted, per table).
A transcript tells you what the agent said it did. The branch diff is what it did. And ten of them are directly comparable, because all ten are measured against the same origin.
The tradeoff is that branches are real objects with a real bill attached. His plan includes ten of them and charges per branch per month after that, and a leaked branch is exactly the sort of thing nobody notices for a quarter. So teardown is not a finally block somewhere in the code — it is a first-class state with its own row in the results database, and every run passes through it whether it succeeded, failed, or timed out:
PENDING -> PROVISIONING -> SPAWNED -> RUNNING -> DIFFING
-> TEARDOWN -> DONE | FAILED
Notice that DIFFING comes before TEARDOWN. The diff needs the branch alive, and the branch needs to die. Getting those two in the wrong order is how you end up with ten orphans and no data.
One config file describes an entire fan-out. global: holds the defaults and runs: is a list of per-run overrides — and for this fan-out every single run is identical, which is exactly the point.
Putting it in the cluster
Locally, all of this is a loop. On his k3s cluster it becomes a controller.
core is a plain Deployment. No CRD, no operator boilerplate — he did not want to write a reconciler for this, and does not think it needs one. It watches a ConfigMap inbox that multirun publish drops a fan-out config into, merges global + per-run config for each run, renders that into a per-run ConfigMap, and spawns exactly one Kubernetes Job. Run state lives in a Postgres results database rather than in etcd, because he wants run history queryable in SQL months from now, not kubectl get output that ages out of existence.
The runner itself is deliberately stupid. It reads /config/run.yaml, does the run, writes artifacts to a shared PVC. It never talks to the Kubernetes API, never talks to the Neon API, and never imports orchestrator code. Config in, artifacts out is the entire contract — and that is what keeps a failed pod debuggable, since you can hold a whole run in your head from one YAML file and one directory.
The agent runs through Anthropic's Agent SDK on his Claude subscription, rather than the raw API. Two things follow from that, and both bit him. Subscription credentials inside a pod rotate, so a stale Secret fails auth with nothing obvious in the logs. And ten simultaneous agents on one subscription hit rate limits, which is why max_parallel defaults to 3. Ten branches does not mean ten at once.
The part he would rather leave out
This was running on the cluster, and working, before it was under ArgoCD.
The manifests existed. The registration PR was open. And he had kubectl apply'd them "as a bridge" to unblock the fan-out. Well… the bridge became the deploy. Everything on that cluster is GitOps-managed, and this one thing was a hand-placed object pretending to be one, which is arguably worse than not deploying it at all, because it looks completely fine right up until the moment someone syncs.
So it got adopted properly: an Application, the manifests in git, secrets as SealedSecrets. The adoption rolled nothing — zero drift between what he had applied by hand and what git said, so the fan-out that was running at the time never even noticed.
The lesson he took from it is smaller than "use GitOps", and more useful. He had verified that the core pod is running, when the thing he was actually claiming was the app is deployed. Those are two different sentences, and he checked the easy one.
After adoption. The bridge he'd applied by hand and the manifests in git were byte-identical, so nothing rolled.
Ten agents, one task
The task he picked is the most boring migration in software, and he picked it for exactly that reason. Take an app that was built single-user, and add teams to it. Every SaaS does this eventually. There is no clever answer waiting at the end.
The prompt is a short markdown file, and the important thing about it is what it refuses to specify:
This app was built single-user: every row that matters hangs off one user id, and nothing in the schema knows what a team is. Add teams.
- A user can create a team, belong to teams, and act within a team context.
- Data that was per-user and should now be shareable within a team is reachable by teammates; data that should stay personal stays personal. Deciding which is which is part of the task.
- The rows that already exist keep working: an existing user signs in and loses nothing.
- Access control still holds afterwards — a member of team A can not reach team B's data through any path you added.
How you model it — tenant column, join table, something else — is your call. Make the schema change as a real migration, wire the app code to it, and say at the end what you chose and what it costs.
The source is a Next.js + Drizzle + Neon template of his, pinned at one sha. The seeded parent database holds 3 users, 3 profiles and 4 assets — deliberately tiny, so that he could read the data diff with his own eyes. Two checks, both dumb: the app still builds, and something with "team" in the name exists in the schema. The grading happens in the diffs, not in the checks.
Ten runs. All sonnet. 10/10 done, 20/20 checks passed, 50 minutes end to end at three at a time. Between 77 and 117 turns each, 462k output tokens and 63 million cache reads across the ten, 143 files touched, and every single one of the ten branches released afterwards.
Per-run effort out of the trace data — flip the measure, the ten bars stay put.
Where they agreed
Every single run built the same thing. A teams table, a team_members join table, a team_id column on assets.
Not one of them did tenant-column-only. Not one did anything exotic. All ten decided that assets are the shareable thing and profiles stay personal, without ever being told which was which. All ten kept every pre-existing row alive. And not one of the ten reached for row-level security in the database — access control stayed in application code, ten times out of ten.
He expected them to scatter. They converged hard.
Where they didn't
Does deleting a team delete its data? Five runs said
SET NULL, so the assets survive and fall back to personal. Five saidCASCADE, so the assets die along with the team. That is a user-visible product decision that nobody asked the agent to make, and it came out exactly 5/5.What is the join table's primary key? Five went composite
(team_id, user_id). Five went surrogateidwith a unique constraint on the pair. Another coin flip.Is the new column allowed to be NULL? Eight runs made
assets.team_idnullable — NULL means personal, existing rows need no touching, the migration stays pure DDL. Two runs refused the null entirely:NOT NULL, apersonalflag on teams, and a data migration that mints a personal team for every existing user and moves their assets into it. Same sentence, two genuinely different philosophies of tenancy.
There is smaller scatter underneath all that, too. Six used a Postgres enum for the member role, three used text with a CHECK, one left it as bare text with a default. Six gave teams a unique slug, four didn't. Six added a current-team pointer to profiles, and then split between two different names for it.
Ten runs is not enough to pin any of these ratios down. A 5/5 at n=10 is something a true 70/30 would produce often enough that you can't rule it out, so the number to take from this isn't the 50/50 — it's that there's a split there at all, on a decision the task file assumed had one answer. Getting the real ratio means more runs, which the bench can do since it's just a config value, but he hasn't run them.
All seven decisions, sorted by how many of the ten agreed. This is the real comparison data out of the run diffs — hover a segment for the run numbers.
One oddity, and it's a good one. Run 06's database is the only one of the ten with actual rows sitting in teams and team_members — one team, two members. No migration inserted those, which means that agent tested its own feature through the running app and then simply left the test data behind. The only reason anyone knows is that the data diff caught it. Nothing in the transcript would have sent him looking.
And the failure
Because it's the more useful half, honestly.
His first attempt at this fan-out was 10 out of 10 failed, in under ten minutes. The runner image had no openssh-client, so cloning the private template over SSH died in the init container. Every time.
The smoke test never caught it, because smoke clones a public repository over HTTPS — which is a completely different code path from the one every real run takes. One line in a Dockerfile.
Teardown held, though. All ten of those branches were released anyway, which is the only reason he could shrug at it and just re-run.
Watching it happen
Ten agents working in parallel inside pods you cannot see into is an unpleasant feeling. So the last piece is a read-only dashboard: instances, runs side by side, per-run detail with the phase timings, the check results, the branch diff, and the artifacts.
The plan had been to change the runner for the live view — stream each turn to a file on the shared PVC as it happens. Then he went to implement it and found the runner had been doing exactly that since day one. Every agent message, appended to trace.jsonl, flushed per message, sitting on the PVC right next to the artifacts.
So the dashboard just tails that file over a websocket, which means live and replay are the same wire. A run that finished four hours ago plays back through the identical code path as one happening right now, and no pod logs are involved anywhere, so it still works long after the pod is gone.
Every fan-out this bench has run, including the one that failed 10/10.
The ten runs of add-teams. Notice how tight the effort is — whatever the variance is, it is not effort variance.
One run: phase timings, both checks, and what it did to the data.
The cost readout deserves a note. The SDK reports what each run would have cost at API rates — $2.49 to $4.39 per run, $32.25 for all ten. None of it was actually billed, since this runs on his subscription. But it is the number he uses to think about what a fan-out is worth, and at roughly thirty dollars, "what would it have done the other nine times" stops being a rhetorical question.
Closing
The thing he is keeping from this: variance is a property of the model and the prompt, not noise.
Ten identical runs did not produce ten random results. They produced total agreement on the schema, and two exact 5/5 splits on decisions that nobody asked them to make. Both halves of that are actionable. The agreement tells him the model has a strong prior about how tenancy gets modelled in Postgres, and he can stop reviewing that part quite so hard. The splits tell him precisely which sentences his task file was missing.
And once you can measure a split, you can try to close it — write the prompt more precisely, run ten more, see whether 5/5 turns into 9/1. That is an eval. It took a loop and a config file, there is no judge model in it anywhere, and until this week he had no way to run one against his own prompts at all.
The other half is the one he would actually go and use on Monday. Pick the next change you would genuinely regret getting wrong — the tenancy model, the auth boundary, the schema that will outlive the code sitting on top of it — and don't take the agent's first answer for it. Run it ten times. Read the spread. Where all ten agree you can move quickly, because that is the model's prior and it is usually the dull correct thing. Where they split, there is a decision sitting there and it was always yours to make; the agent has just been making it for you quietly, one sample at a time.
He has been calling that results branching. It is the same mechanism as the eval — you are only pointing it at one decision instead of at the whole prompt.
He keeps coming back to what he wrote in the branched environments post — that the pre-AI techniques are what make the agentic ones actually work. This is the same thing again, just one layer up. Worktrees, database branching, GitOps, teardown as a state machine.. none of that is new, and none of it is about AI. It is all boring infrastructure practice. But it is the only reason ten agents can run at once without stepping on each other, and it's what turns "the agent did something" into a measurement.
He started this wanting to know whether he could trust the model. He still doesn't really have an answer to that. What he has instead is a list of the places where he hadn't decided something and it decided for him. That has turned out to be more useful.
Thank you for reading :)
After-post notes
_N1: The bench, the fan-out and the numbers in this post are all real — the config that produced them is in the repo, and you can run the same fan-out yourself.
N2: If this post inspired you to think, feel free to email Denis and start a discussion! He loves talking technology, and even so if you are, too, actively living in "the arena"._
You can find everything about Denis on denislavgavrilov.com/about
Thank you for reading and have a good one!
