Why your LLM app gets 429s even when you're under the rate limit

You ship a feature that calls an LLM. Traffic is modest — nowhere near the requests-per-minute ceiling on your plan. Then the 429s start.

You check the dashboard. Request count is comfortable. You add a concurrency cap. The 429s get less frequent but never stop. You add exponential backoff. Throughput drops and the 429s still don't stop. Somewhere around the third hour you start wondering whether the provider's limits are just wrong.

They aren't. You're metering the wrong resource.

You're not behind one rate limiter. You're behind three

Every classic rate limiting tutorial — including, until recently, ours — teaches you to think in one dimension: requests per unit time. One bucket, one refill rate, one verdict.

LLM APIs don't work that way. A request is not the unit of cost, because two requests can differ in cost by three orders of magnitude. A "hi" costs a few tokens. A RAG call stuffing forty retrieved chunks into context costs tens of thousands. Charging both one unit from the same bucket would be like a warehouse counting trucks and ignoring what's inside them.

So providers meter several dimensions at once. Anthropic's Messages API enforces requests per minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM), each as a separate bucket. OpenAI enforces RPM, TPM, and daily variants RPD and TPD, plus images per minute for image models.

The verdict is the tightest bucket. Not the average, not the one you're watching.

Two dimensions with headroom and one without still returns a 429. If your dashboard graphs request count, this failure is invisible to you.

This alone explains a lot of mystery 429s. But it isn't the interesting part.

The interesting part: you're billed for a number you don't know yet

Here's what makes LLM rate limiting genuinely different from every rate limiter in the textbook.

A token bucket assumes cost is known at admission time. A request arrives, it costs one token, you check the bucket, you decide. The entire algorithm rests on knowing the price before you commit.

With an LLM call you don't. You know your input size — you can count that locally before sending. But the output size is decided by the model, one token at a time, over the next several seconds. The true cost of the call is only knowable after it has finished streaming.

So the provider has to guess, and how it guesses is a real, load-bearing implementation detail that differs by vendor:

	OpenAI	Anthropic
Input accounting	Estimated from character count	Estimated at request start, adjusted during the request
Output accounting	`max_tokens` is charged up front	Metered in real time as tokens are produced
Does `max_tokens` cost you?	Yes — the limit is calculated as the maximum of `max_tokens` and the estimated request size	No — `max_tokens` does not factor into OTPM
Cached input	Counts	`cache_read_input_tokens` do not count toward ITPM on most models

Read that third row twice, because it is the single most expensive line of config in most LLM codebases.

On OpenAI, setting max_tokens: 4096 "just to be safe" on a call that reliably returns 200 tokens means you are charged 4096 against your TPM every single time. Twenty times over. Your dashboard shows real token consumption — small — while your rate limiter is being drained by reservations you never used. You will get 429s at roughly 5% of your apparent capacity, and nothing in your own metrics will explain why.

On Anthropic the same parameter is free, and the lever that matters instead is prompt caching: with an 80% cache hit rate against a 2M ITPM limit, you can push about 10M input tokens per minute, because cache reads don't count against the bucket.

Same two-line diff. Opposite outcomes. This is why "best practices for LLM rate limiting" articles that don't name a provider are close to useless.

Admission has to commit capacity before the price is known. Everything hard about LLM rate limiting follows from this one gap.

The pattern that fixes it: reserve pessimistically, reconcile on completion

Once you see the problem as "commit before you know the price," the solution is the one databases have used for decades. Reserve, then settle.

Estimate the cost before sending. Count input tokens locally — tiktoken for OpenAI, the token counting endpoint for Anthropic. Add a pessimistic output estimate, ideally a high percentile of your observed history for that prompt type rather than a constant.
Admit only if every bucket has room. Check RPM and the token buckets against the reservation. If any is short, don't send. A request you send into an exhausted bucket doesn't just fail — on many providers it still consumes quota on the way down, so blind retrying digs the hole deeper.
Hold the reservation while the call is in flight. This is the step people skip. Ten concurrent calls of unknown size are ten unpriced liabilities against a shared budget.
Reconcile when the response completes. Read the real usage off the response and release the difference back into the bucket.
Correct from the provider's own headers. Your local model will drift. Every response carries the truth: x-ratelimit-remaining-tokens on OpenAI, anthropic-ratelimit-input-tokens-remaining and anthropic-ratelimit-output-tokens-remaining on Anthropic. Snap your local bucket to the server's number when they disagree.

async def call(self, prompt: str, max_out: int) -> Response:
    est_in = count_tokens(prompt)
    est_out = self.p95_output_for(prompt_kind(prompt))  # not a constant
 
    # Admission: every dimension, or we wait. `reserve` blocks until the
    # continuously-refilling buckets can cover the whole estimate at once.
    async with self.limiter.reserve(requests=1, inp=est_in, out=est_out) as lease:
        resp = await self.client.messages.create(...)
 
        # Settle against reality — the estimate was always a guess.
        lease.settle(inp=resp.usage.input_tokens, out=resp.usage.output_tokens)
 
        # And trust the provider over ourselves.
        self.limiter.sync(resp.headers)
        return resp

The async with matters. If the call throws halfway through a stream, the lease has to release or you leak capacity until restart — a slow strangle that looks exactly like a memory leak on a throughput graph.

Watch the difference

Two clients, one identical request stream — same arrivals, same token costs, same seed. Both queue work they can't send yet, so neither gets to look good by quietly dropping load. The only difference is what each one checks before it sends.

The left client does what most codebases do on the first pass: it caps requests per minute and sends. The right one runs the two-bucket reserve-and-reconcile loop above. Hit Run, then switch the workload to RAG and watch the request buckets stay almost full on both sides while only one of them stays clean.

The hatched band on the right panel is live reserved capacity — tokens held for calls that are still streaming. Watch it expand as calls go out and collapse as each response settles. That band is the thing a requests-only limiter has no representation for, and it is where all the 429s come from.

One result there is worth sitting with, because it's the opposite of what people expect. The two clients finish roughly the same number of calls. Token capacity is the binding constraint, and no admission policy invents more of it. What changes is that the left client converts part of that same workload into errors — failures that still burn a request slot, still cost latency, still surface to a user, and still trigger retries that make the next minute worse. The two-bucket client does the same amount of real work and simply waits instead of failing.

That's the honest pitch for admission control generally: it rarely makes a saturated system faster. It makes it degrade in a way you can reason about.

Why your backoff is making it worse

Now the second failure, the one that turns a bad minute into a bad afternoon.

You add exponential backoff: on 429, wait 1s, then 2s, then 4s. Every instance does this. Which means every instance that failed at roughly the same moment retries at roughly the same moment. The bucket refills, the entire fleet slams it simultaneously, most of them 429 again, and now they're synchronized even harder for the next round.

You've built a retry storm. Load arrives in coordinated spikes instead of a smooth stream, so peak demand massively overshoots capacity even when average demand is comfortably under it.

Identical retry volume in both panels. The only change is when each client wakes up — which is the whole difference between recovering and staying down.

Three corrections, in order of how much they buy you:

Honour retry-after. Both providers send it on a 429. It is not a hint; it is the provider telling you exactly when capacity returns. Retrying earlier is guaranteed to fail and, on a token-bucket provider, spends quota confirming that. Your own exponential curve is a guess about a number you were just handed.

Add full jitter. Not delay ± 10%. Full jitter — sleep(random(0, delay)) — which is what actually decorrelates a fleet. This is the single cheapest fix in the whole post and it's usually a one-line change.

Cap the backoff at one refill period. A token bucket that refills continuously over a minute is fully replenished within a minute, no matter how long you sit there. Backing off to 64 seconds is pointless; backing off to 512 seconds means you've taken a five-minute outage over a five-second problem. Cap at roughly one bucket-refill and stop.

There's a fourth situation worth naming because it looks like a bug in the provider: Anthropic also applies acceleration limits, so a sharp jump in usage can return 429s even when you're inside your posted per-minute limits. If you're launching a backfill, ramp into it rather than opening the throttle at once.

The part that gets harder with more than one worker

Everything above assumes one process holds the bucket. Scale to twenty workers and the limit is still org-wide, so a local bucket per worker means you're running twenty limiters against one budget — each one correct, all of them collectively wrong.

The options, roughly in order of how much they cost you:

Static partitioning. Each worker gets 1/N of the quota. Trivially correct, and wasteful exactly when it hurts: idle workers hoard capacity while busy ones throttle.
A shared bucket in Redis. One atomic Lua script does check-reserve-and-settle across the fleet. Correct, and adds a round trip to every call — usually fine next to a multi-second LLM call.
A gateway. Route everything through one proxy that owns the buckets. Best behaviour, and it becomes a component you have to run.

The trade-off is the classic one, and it's the same shape as the sliding window versus fixed window decision: precision costs coordination. What's specific to LLMs is that the reservation is held across a multi-second call, so a shared bucket carries state that a request-counting limiter never has to.

Why this shows up in interviews now

"Design a rate limiter" has been a standard system design question for a decade, and the expected answer has been the same for a decade: token bucket, maybe sliding window, discuss the trade-off, done.

That answer is now incomplete, and the follow-up is starting to appear in loops at companies building on model APIs: what if the cost of a request isn't known until after you serve it?

It's a genuinely good question because it's not solvable by recalling an algorithm. It forces you to reason about admission control under uncertainty, reservation and settlement, local estimates converging on authoritative state, and how retry policy interacts with a shared budget. The token bucket is still the right primitive. It just doesn't survive contact with the problem unmodified — and being able to say precisely which assumption breaks and how you'd patch it is what separates a memorised answer from an engineering one.

If you want the underlying primitives with the same treatment, the rate limiting topic runs token bucket, leaky bucket, fixed window, sliding window, and sliding log side by side against one shared workload — the same "identical stream, different policy" setup as the simulation above.

The short version

You are behind several limiters at once, and the tightest one decides. Graph tokens, not requests.
Cost is unknown at admission. Reserve pessimistically, reconcile against real usage, and trust the provider's headers over your local model.
max_tokens is charged against your TPM on OpenAI and is free against OTPM on Anthropic. Check which one you're on before you tune anything.
Honour retry-after, add full jitter, cap the backoff at one refill period.
Prompt caching raises effective throughput on Anthropic, because cache reads mostly don't count toward ITPM.

The general lesson is older than any of this. A rate limiter is only as good as its cost model, and when the cost model is a guess, the interesting engineering is in how you correct the guess — not in which algorithm you picked.