Building a Token Bucket Rate Limiter in Python

Every API you have ever used has one of these. The idea is simple enough to fit in thirty lines, and the thirty lines teach you more about rate limiting than any dashboard does. Here is the version I keep reaching for when a small service needs to stop hammering a downstream API.

The idea

Picture a bucket that holds tokens. A request costs one token. The bucket refills at a fixed rate, say two tokens a second, and it can never hold more than its capacity. A request that finds a token goes through. A request that finds an empty bucket gets a 429.

Capacity controls bursts. Rate controls the long-run average. A bucket with capacity 5 and a refill of 2 per second lets five requests through at once, then settles into two a second. That split is the whole reason the token bucket beat the fixed window for my use: real traffic arrives in bursts, and a limiter that rejects the first burst is a limiter that breaks logins every Monday morning.

The code

import time

class TokenBucket:
    def __init__(self, rate, capacity):
        self.rate = rate            # tokens added per second
        self.capacity = capacity    # max tokens the bucket holds
        self.tokens = capacity      # start full
        self.updated = time.monotonic()

    def allow(self):
        now = time.monotonic()
        self.tokens = min(self.capacity,
                          self.tokens + (now - self.updated) * self.rate)
        self.updated = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

Two details matter. The refill is lazy: tokens are only added when a request arrives, computed from the time since the last check. Nothing runs in the background, no thread, no timer. And the clock is time.monotonic(), not wall time, so a system clock adjustment cannot dump a bucketful of tokens into the limiter or starve it.

A real run

I fired twelve requests, 0.2 seconds apart, at a bucket with rate 2 and capacity 5:

request  1: ALLOWED
request  2: ALLOWED
request  3: ALLOWED
request  4: ALLOWED
request  5: ALLOWED
request  6: ALLOWED
request  7: ALLOWED
request  8: BLOCKED (429)
request  9: ALLOWED
request 10: BLOCKED (429)
request 11: ALLOWED
request 12: BLOCKED (429)

The first five are the burst, spent straight from the full bucket. Requests 6 and 7 slip through because the bucket refilled while they waited: 0.2 seconds at 2 tokens a second is 0.4 tokens per gap, so roughly one fresh token lands every half second. After that the pattern locks in: allow, block, allow, block. Two per second, no more, for as long as the load keeps coming.

That steady state is the part worth staring at. The limiter does not care how many requests arrive. It admits exactly the configured rate and turns the rest away. Your job is only to pick the rate honestly.

Where this breaks

This class lives in one process, in memory. Restart the process and the bucket starts full, which hands every restart a free burst. Run two workers and each has its own bucket, so your real limit is twice what you configured. Neither problem shows up in a test with one script and one process. Both show up the first week in production.

The usual fix is to move the token count into Redis and do the refill check inside a Lua script, so the check and the decrement happen atomically. Same math, shared state. The in-memory version stays useful for exactly one thing: a single-process script, a CLI tool, or a test client that needs to stop flooding something. Know which one you are building.

Pick the numbers from real traffic, not from a blog post. Look at your busiest minute, decide how much of a burst is legitimate, set capacity to that, and set the rate to what the downstream service can actually sustain. Then watch the 429s. If real users hit them, the rate is wrong, not the users.