Scale your AI system for real traffic

When traffic grows, an AI system fails in three predictable ways: it hits a provider rate limit, it piles up work faster than it can finish, and individual calls get slow enough to time out. The fixes are four patterns you can layer in order: batch what is not urgent, queue what is, cache what repeats, and degrade gracefully when you still run out of room.

This guide is about the system around the model, not the model call itself. It assumes you already have a working feature, like the summariser in an online store, and now ten times the users want it at once.

What breaks when an AI system gets real traffic?

A single request looks instant in a demo. Multiply it by a thousand concurrent users and three limits show up at the edges.

• Rate limits. Providers cap how fast you can call them. The Claude API enforces three separate ceilings per model: requests per minute (RPMRequests per minute, the cap on how many API calls you can make in a 60-second window), input tokens per minute (ITPMInput tokens per minute, the cap on how much prompt text you can send per 60 seconds), and output tokens per minute (OTPMOutput tokens per minute, the cap on how much text the model can generate for you per 60 seconds). Cross any one of them and you get an HTTP 429 response.
• Backpressure. If work arrives faster than you finish it, something has to hold the overflow. With no queue, the overflow lands on the user as a timeout or a crash.
• Slow calls. A long generation can take many seconds. Hold an HTTP connection open the whole time and your server runs out of connections long before it runs out of CPU.

The rest of this guide is the four patterns that turn that failure picture into a system that bends instead of breaking.

Which rate limit am I hitting?

You cannot fix a limit you have not identified. The 429 response tells you which one tripped, and the response carries a retry-after header with the number of seconds to wait before trying again. Honour that header rather than guessing.

The Claude API replenishes capacity with a token bucketA rate-limit method where capacity refills continuously over time instead of resetting all at once on a fixed schedule algorithm. Your allowance refills continuously up to your maximum, rather than resetting on a fixed clock. The practical effect: a 60 RPM limit behaves closer to 1 request per second, so a sudden burst can trip the limit even when your per-minute average looks fine. Smooth your traffic and you get more headroom for free.

One detail changes how you design everything downstream: on most Claude models, tokens read from the cache do not count toward your ITPM limit. So caching is not only a cost lever, it is a throughput lever. More on that below, and in the dedicated prompt caching guide.

Pattern 1: batch the work that can wait

Not all work is interactive. Re-summarising every product description overnight, scoring a backlog of support tickets, or running an evaluation set are all jobs where the answer is needed eventually, not now. Batching trades latency for throughput and cost.

The Claude API offers a Message Batches API for exactly this. You submit many requests as one asynchronous batch and poll for results. Batch requests run at 50% of the standard cost. Most batches finish in under an hour, and a batch has a 24-hour processing window: you can collect results when every request completes or after 24 hours, whichever comes first.

The rule of thumb: if no human is waiting on the result this second, batch it. You take pressure off your per-minute rate limits and cut the bill in half. Drive this from your own backend on a schedule. You can ask Claude Code to scaffold the submit-and-poll job for you.

Pattern 2: queue the work that cannot

Interactive work cannot wait until tomorrow, but it can wait a few seconds. A queue is the buffer that absorbs a burst so your model calls go out at a steady rate instead of all at once.

The shape is always the same. Your app accepts the request, drops a job on a queue, and returns immediately with a job id. A separate pool of workers pulls jobs at a controlled rate and calls the model. The user polls the job id or gets notified when it is done.

A queue buys you four things at once:

• Backpressure handling. The queue holds the overflow instead of your web server. Bursts flatten into a steady stream.
• Concurrency control. You tune how many workers run, which sets how fast you call the model. Keep that rate just under your RPM and token limits.Common gotchaSizing workers against RPM alone is a classic trap: a few workers sending long prompts can blow past your ITPM token ceiling while sitting well under the request cap, and you get 429'd anyway.Cap concurrency against whichever limit you hit first, which for big prompts is usually tokens, not requests.
• Retries with backoff. When a call returns 429 or fails, the worker waits and retries instead of dropping the job. Exponential backoffWaiting progressively longer between retries, often with a small random jitter, so a struggling service is not hammered means each retry waits longer than the last, with a little random jitter so all your workers do not retry in lockstep.
• Decoupling. Slow model calls no longer hold an HTTP connection open. The user got their job id in milliseconds.

When a worker hits a 429, the correct move is to read the retry-after header, sleep for that long, and put the job back. That single behaviour is the difference between a queue that recovers from a spike and one that melts down trying to push through it.Common gotchaA fixed backoff with no jitter makes a fleet worse, not better: every worker trips the same 429 at once, sleeps the same amount, and retries in lockstep, so the burst that broke you reforms a few seconds later.Add a small random jitter to each wait so workers spread out, and back off per worker rather than pausing the whole pool.

Pattern 3: cache what repeats

The fastest call is the one you never make. Two kinds of caching matter, and they solve different problems.

Response caching. If users ask the same thing, store the answer keyed on the input and serve it on a repeat. A product FAQ or a "summarise this fixed document" feature can have a high hit rate. This removes the call entirely, so it saves latency, cost, and rate-limit budget all at once. Set a sensible expiry so answers do not go stale, and only cache work that is deterministic enough to reuse.

Prompt caching. Most calls are not identical, but they share a large, stable prefix: a long system prompt, a big reference document, your tool definitions. Prompt caching stores that prefix on the provider side so repeat calls skip reprocessing it. As noted above, on most Claude models cached input tokens do not count toward your ITPM limit, so a high cache hit rate raises your effective throughput well past the headline number. The full mechanics, breakpoints, and TTLs are in prompt caching.

Order your prompt so the stable parts come first and the variable parts last. That maximises how much of each request hits the cache. The same ordering discipline shows up in context engineering, and it pays off twice here.

Pattern 4: degrade gracefully

You will still hit moments where every lever is pulled and demand exceeds capacity. At that point, aim to degrade gracefully: never hand the user a blank error.

Graceful degradation means having a defined fallback for each failure, ranked from best to worst:

• Smaller model. If the primary model is saturated or slow, route to a faster, cheaper one. A slightly weaker answer beats no answer.
• Cached or partial result. Serve a recent cached answer, or the part of the work that did finish, with a note that it is partial.
• Non-AI fallback. Fall back to keyword search, a templated response, or a static help page when the AI path is fully unavailable.
• Honest queueing. As a last resort, accept the job, tell the user it is in progress, and notify them when it is ready.

Two guardrails make degradation safe. Set a timeout on every model call so a slow response cannot pin a worker forever, and trip the fallback when the timeout fires. Add a circuit breakerA guard that stops sending requests to a failing dependency for a cooldown period, then tests with a trickle before fully reopening so that once a dependency is clearly failing, you stop hammering it for a short cooldown and serve fallbacks instead, then test with a trickle of traffic before reopening. A circuit breaker turns a slow cascade into a fast, bounded failure.

How do the four patterns fit together?

The four patterns stack. Each one shrinks the load that reaches the next.

Read it left to right. Batching removes non-urgent work from the live path. Caching removes repeat work. Queuing smooths what remains to a rate your limits allow. Graceful degradation catches whatever still overflows. Most systems do not need all four on day one, so add them in that order as load grows.

What should I check before scaling to production?

Run this before you call a system production-ready under load.

• You handle 429 by reading retry-after and backing off, not by retrying immediately.
• Non-urgent jobs go through a batch path, not the live request path.
• A queue sits between intake and your model calls, with worker concurrency tuned under your rate limits.
• Repeated inputs are response-cached, and long stable prefixes use prompt caching.
• Every model call has a timeout and a defined fallback.
• You log enough to see which limit you hit and how often.

You do not have to write the plumbing by hand. Describe your traffic shape and your limits to Claude Code and have it scaffold the queue, the worker loop, and the retry logic, then review the result against this checklist. The same discipline that makes a single agent reliable, covered in test and debug your agent, is what keeps a fleet of them responsive once real traffic arrives.