Problem RestatementProblem
Anthropic asked: you need to make a large number of HTTP requests (say 100,000 calls to an external API) as fast as possible, but the API has rate limits (e.g., 500 requests/second) and sometimes returns errors or responds slowly. How do you design the client? What trade-offs exist between speed, limits and reliability?
Why the Naive Way Is Slow
One request at a time: if each takes 200 ms (mostly waiting on the network), 100,000 requests take 5.5 hours. The CPU is idle almost all the time. We're limited by latency, not work.
Little's Law (explained simply): throughput ≈ requests in flight ÷ latency. With 200 ms latency, 100 concurrent requests give ~500 requests/sec. So concurrency is the main lever.Deep Dive — 100,000 requests against a rate limitDeep dive
The API allows 500 requests a second. The goal is to finish as fast as that budget permits without being throttled or banned.
A sequential loop
Send a request, wait for the response, send the next.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
L["for url in urls: get(url)"] --> ONE["One request in flight"]
ONE --> RTT["200 ms round trip each"]
RTT --> TOT["100,000 x 200 ms = 5.5 hours"]
TOT --> IDLE["Using 5 of the 500 requests/sec allowed"]The process spends virtually all its time waiting on the network, and the rate limit is nowhere near the constraint — the concurrency of one is.
Fire them all concurrently
Launch every request at once, or a thread per request.
Now the limit is hit immediately and the API returns 429s. Naive retries make it worse, because the retries are also unbounded. Threads add their own ceiling: 100,000 of them is far more than a process can hold, and the memory and context switching cost dwarfs the work.
Async I/O, with a concurrency cap and a token bucket
Two different limits, doing two different jobs:
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
Q["Work queue - 100K URLs"] --> RL["Token bucket - 500 req/s"]
RL --> SEM["Semaphore - e.g. 100 in flight"]
SEM --> POOL["HTTP client - connection pool, keep-alive, HTTP/2"]
POOL --> API["External API"]
API -->|"429 / 5xx / timeout"| RETRY["Backoff with jitter, honour Retry-After"]
RETRY --> Q
API -->|"200"| OUT["Results"]- The token bucket enforces the API's rate, smoothing bursts so the average stays inside the allowance instead of sprinting into a 429 and stalling.
- The semaphore caps in-flight requests, which is a different constraint: it bounds memory, sockets and the damage a slow endpoint can do. Rate and concurrency are related by latency — 500/s at 200 ms means about 100 in flight — so setting only one leaves the other unmanaged.
- Async I/O makes hundreds of concurrent requests cheap. These tasks are almost entirely waiting, so an event loop handles them in one process where threads would not.
- Reuse connections. Keep-alive and HTTP/2 remove a TCP and TLS handshake per request, which at this volume is a large fraction of the total time.
- Jitter the backoff, or every retry from a throttled batch returns in unison and throttles again.
Then measure before tuning further: if the CPU is saturated the bottleneck is parsing, not the network; if latency rises as concurrency does, the server is the limit and adding more in-flight requests makes it slower, not faster.
Code Sketch (Python asyncio)
import asyncio, random, time
import aiohttp
class TokenBucket:
def __init__(self, rate, burst):
self.rate, self.tokens, self.burst, self.t = rate, burst, burst, time.monotonic()
self.lock = asyncio.Lock()
async def take(self):
async with self.lock:
while True:
now = time.monotonic()
self.tokens = min(self.burst, self.tokens + (now - self.t) * self.rate); self.t = now
if self.tokens >= 1:
self.tokens -= 1; return
await asyncio.sleep((1 - self.tokens) / self.rate)
async def fetch_all(urls, rate=500, concurrency=100, max_tries=4):
bucket, sem, results = TokenBucket(rate, burst=rate), asyncio.Semaphore(concurrency), {}
timeout = aiohttp.ClientTimeout(total=10)
async with aiohttp.ClientSession(timeout=timeout, connector=aiohttp.TCPConnector(limit=concurrency)) as s:
async def one(url):
for attempt in range(max_tries):
await bucket.take()
async with sem:
try:
async with s.get(url) as r:
if r.status == 200:
results[url] = await r.json(); return
if r.status not in (429, 500, 502, 503, 504):
results[url] = ("error", r.status); return
wait = float(r.headers.get("Retry-After", 0))
except (aiohttp.ClientError, asyncio.TimeoutError):
wait = 0
await asyncio.sleep(max(wait, (2 ** attempt) * 0.1 + random.random() * 0.1))
results[url] = ("error", "retries exhausted")
await asyncio.gather(*(one(u) for u in urls))
return resultsMeasure to Find the Bottleneck
- If throughput is stuck below the rate limit, check latency per request and concurrency (Little's Law).
- If you're at the rate limit, only batching or a higher quota helps.
- If CPU is maxed (JSON parsing), add processes.
- If bandwidth is maxed (large responses), compress (gzip) or request fewer fields.
Wrap-UpWrap-up
Replace sequential calls with bounded concurrency (async I/O and a semaphore sized to about rate × latency), throttle with a client-side token bucket that honors Retry-After, reuse connections with keep-alive or HTTP/2, set timeouts, and retry only retryable errors with exponential backoff and jitter. Use batch endpoints when available, and measure latency, rate, CPU and bandwidth to find the real bottleneck.