Before You Ration Anything, Price It

The way I see it — most of the worst engineering decisions I’ve made (or almost made) started with a number that looked urgent, and ended with a measurement I should have done first.

Today a post hit my feed that is essentially that whole category of mistake, compressed into a single afternoon. An engineer on Kaggle checked their accelerator quota and found 108,511.735 seconds used against an allowance of 108,000 seconds — the dial sitting at 100.47%. Four competitions were active on that account, and all of them had pushed a submission within the same 25-minute window earlier that morning. The obvious interpretation was that one shared pool was being ripped apart by a four-way fight for GPU time, and whichever notebook’s submission hit first was eating the week.

So they built a priority table. Rank 1, rank 2, rank 3, do-not-run. A clean allocation policy. As it happens, the table had no numbers in it — that was the first warning sign, and nobody noticed.

Here’s the part that should make every engineer who works with metered resources sit up a little straighter: real demand for the pool turned out to be 13%. Not 100.47%. Not four competitions fighting over scraps. One competition doing real work, and a checkbox that said “Gpu” on a training run that never once touched a GPU.

I want to walk through what actually happened, because the error isn’t the one you think it is, and the pattern behind it will show up in your own infrastructure about once a quarter.

The table with no numbers

A priority table answers one question: who goes first. It does not answer what does a turn cost. The engineer had ordered four competitions by importance but had never measured how much a session from each one actually drew against the quota pool they were rationing. In hindsight, that’s a strange thing to skip — you can’t allocate a budget you haven’t priced.

The absence of any unit-price number in a document whose entire job was allocation was itself the signal that nobody had priced anything yet. But the 100.47% line triggered a fast cognitive jump: pool is overallocated, I need to ration. So the immediate output was a policy, not a measurement.

I’ve done exactly this myself. I see a CI budget spike and reach for a rule that caps how many staging environments each team can run, without ever looking at what a single staging deploy actually costs against the line item that scared me. The instinct is powerful: a number appears > 100%, you clamp down. But clamping down without pricing the things you’re clamping is how you ration an imaginary scarcity while the real cost driver — an idle managed cluster, a mislabeled run, an unmetered path — sits there laughing.

Three measurements

Before the priority table was ever shipped, a quick check was run. The engineer measured what was actually hitting the personal quota on that account, and three things fell apart at once.

1. Rescoring doesn’t touch personal quota. Code competitions on this platform re-run your submitted notebook on the host side to score it. The assumption was that this re-run bills against your personal quota. But one competition had already confirmed: personal quota read the identical 108,511.735s before and after a full scoring cycle — bit for bit. Another competition confirmed it independently. So, at least on this account in late July 2026, the only thing that spent personal quota was a new commit. The whole allocation table’s unit — “how many submissions can we afford” — was pricing something that cost nothing.

2. A “GPU” label bills wall clock whether or not a GPU is used. A kernel-log audit on one competition turned up 1,043 seconds of runtime with zero occurrences of device: GPU anywhere in the log. The training path was hardcoded to CPU, the model defaulted to CPU, and yet the kernel’s metadata said machine_shape: "Gpu". The quota meter charged the full 1,043 seconds against the accelerator pool anyway. Flip one field to enable_gpu:false and, on this account, CPU-only re-runs become unmetered. The scarce resource wasn’t GPU time — it was a label.

3. The billing multiplier isn’t 1x. A direct measurement found that 1.5 wall-clock hours of work consumed 2.92 hours of quota — a 1.94x multiplier, scaled to the accelerator count on that machine shape. So the week’s usable budget wasn’t “108,000 seconds” in any intuitive sense. Divided by the multiplier, it was closer to 30 quota-hours, or roughly 15.5 hours of actual wall-clock work. A much smaller number than the raw figure suggests, and one that had never been converted before.

Once you have those three, you can build an actual unit-price table instead of a priority ranking:

• Smoke commit: ~0.4h
• Local probe: ~2.4h
• Local eval (120-task): ~10.2h
• Submission rescoring: 0 (competition compute, not personal)
• CPU-only kernel: 0 (unmetered)

Two of the five operations that the priority table was supposedly rationing cost nothing at all.

Real demand: 13%

With unit prices in hand, the engineer profiled what each competition on the account was actually running. Four were active, one was scaffolded but hadn’t started work.

• A program-synthesis competition (LLM + test-time training, using PyTorch, Triton/CUDA): 11,520s
• An agent-security competition (pure Python search, zero machine-learning imports): commit time only, approaching 0
• A tabular geoscience competition (numpy/pandas, CPU-only auxiliary model): 0
• A game-agent competition (C tree search + LightGBM on CPU, zero torch/tf/jax imports): 0
• Not yet started: 0

Sum it: roughly 11,520 of 108,000 seconds, about 10.7%. Padding every estimate generously, the ceiling was 13%. One competition, out of five, was doing anything that touched an accelerator at all. The 100.47% reading wasn’t four competitions fighting over a shared pool. It was one competition’s mislabeled kernel burning wall clock against an 87%-empty pool.

Had the priority table shipped as written, it would have spent real effort rationing a pool that was almost entirely free, while the actual cause — a checkbox that said “Gpu” on a run that never used one — sat unfixed. Why? Because the table never asked what anything cost.

The same error, three times, three different answers

The post’s author notes that this wasn’t the first time in the same project that the resource carrying the name of the bottleneck turned out not to be the actual bottleneck. It was the third, and each correct answer was different:

• Early on: Believed scarce — submission slots (94 of them). Actually scarce — quota (a submission is 25x cheaper in quota and its result 14x more precise than a local eval). Found by measuring unit price directly.
• July 31: Believed scarce — quota. Actually scarce — commits (rescoring leaves personal quota bit-identical before and after). Found by comparing quota before and after a scoring cycle.
• A separate competition: Believed scarce — GPU time. Actually scarce — per-decision CPU budget (0.3–30s used against a 600s ceiling, 99.93% of it sitting unused). Found by instrumenting actual usage.

Three different names for “the thing we’re rationing.” Three different real answers underneath. The pattern across all three rows is identical: treat the name of a resource as the bottleneck, and never price the operations charged against it.

It’s a close cousin to the mistake of relying on a quality gate whose own measurement noise exceeds what it’s guarding, but it’s distinct: that one measured the wrong way; this one never measured at all before deciding how to ration.

And because the answer changed each time, “what’s scarce this time” has to be asked fresh at every phase. Carrying forward last cycle’s answer as if it were a constant is exactly how the next misdiagnosis gets seeded.

The subagent made the same mistake one level up

Here’s a detail that I find equally instructive — and a little unsettling. The engineer parallelized the profiling work across five subagents, one per competition. One of those subagents reported the quota required for its competition as 140,000 seconds — a number larger than the entire weekly pool on its own. The cause? It had double-counted submission rescoring as personal-quota spend, the very same wrong assumption that started the whole investigation. The real figure was 11,520 seconds. Its report was off by 12x.

Now here’s the part worth sitting with: the correct number was already written down, in that same competition’s own report, which the subagent cited as evidence in its output. It read the file. It quoted from the file. It never applied what the file said to its own arithmetic.

This isn’t a story about a model being unreliable in some general sense. It’s the same failure I’d just made myself, one level up: citing a source doesn’t guarantee the citation gets used, and a measurement you delegate needs to be checked against its own primary source before it goes into a summary — not trusted because it came with a citation attached.

I’ve watched junior engineers do this with cloud cost analyses. I’ve watched senior engineers do it with capacity planning spreadsheets. I’ve definitely done it myself — pulling a number out of a dashboard, quoting it in a document, and never verifying whether the calculation behind it matches what the dashboard was actually measuring. The presence of a citation creates a false sense of rigor that’s often worse than no citation at all, because the second you quote something, you stop questioning it.

The part that transfers

None of this is really about Kaggle GPU quota. It’s about any metered resource you’ve decided is scarce before checking what’s actually charged against it:

• A cloud bill line item that spikes and spawns a new policy about staging environments
• CI minutes that feel scarce and trigger a rule saying “only run these jobs on main”
• An API rate limit that looks tight, so you build a queuing system before measuring whether real traffic ever comes close to it
• Database connections, seat licenses, build concurrency slots — anything where the name of the resource sounds like the bottleneck

Two rules come out of this, and I’ve started writing them on a sticky note near my monitor:

Before you believe “X is scarce,” measure the unit price of the operations you’ve attributed to X. The resource’s name doesn’t set the bottleneck; its measured price does.

Don’t build the allocation policy before you’ve measured the price. A rationing table built on an unmeasured cost doesn’t fix the underlying bug — it makes the bug permanent, because a policy that exists reads as a problem that’s already being managed. On a cloud bill, before you write the rule that caps how many staging environments each team may run, price one staging deploy, one CI job, and one flaky-test rerun against the invoice line that scared you. If it turns out the line is dominated by an idle managed cluster nobody scheduled work onto, the cap you were about to write would have taxed the wrong people and left the cluster running.

Two more rules, for the reverse:

• Check whether a path that looks free is actually being billed. And check the reverse just as hard: a path that looks expensive might already be free.
• Re-ask “what’s scarce this time” at every phase, on every project. The answer moves. Treating last time’s answer as a constant is how the next misdiagnosis gets seeded.

And for anyone working with automated analysis or “AI” assistants: a measurement you delegate goes into a summary only after you’ve checked it against its own source. A citation is not the same claim as a correct conclusion drawn from it. If the number you’re quoting exists in a primary source you can inspect, inspect it.

Where this breaks

An essay telling people to measure before they ration owes its own falsifiers. The original author offered three, and I’ll echo them because they’re sharp:

• If any of the concurrent competitions turns on real GPU training — an offline model, say — the “13% of the pool” figure stops holding. The unit-price table stays valid; only the allocation recalculation changes.
• The billing behavior itself (that rescoring left personal quota untouched) is platform policy, measured on one account in one window. Platform policy can change without notice. That bit-identical check needs to be re-run periodically, not treated as a fact fixed at measurement time — otherwise this essay becomes the source of the next misdiagnosis instead of the fix for the last one.
• The 1.94x wall-clock multiplier is a function of the accelerator count on that specific machine shape. It doesn’t transfer to a different shape without re-measuring.

And the meta-falsifier: the next time someone sees a dial at 100%, the instinct to build a priority table won’t have gone anywhere. The only defense is a personal rule that you don’t build the table until you’ve priced the operations. The number that looks like a crisis is usually just the most recent number you looked at — and the measurement you didn’t do yet is what’s actually scarce.