Module 1 · Throughput under limits · Lesson 2
Concurrency and flow control
Size how many model calls to keep in flight with Little's law, build a token-aware limiter in async Python, share it across workers, adapt it to what the provider tells you, and keep one customer's backlog from starving the others.
By the end of this lesson you can
- Compute the concurrency needed to use a rate limit fully with Little's law, and explain why more concurrency past that point only adds queueing
- Implement an async limiter that combines a concurrency cap with token buckets denominated in tokens, and reconciles estimates with actual usage
- Share a limiter across many workers with an atomic Redis token bucket, and explain the trade-off against static partitioning
- Adapt the concurrency limit to 429s and latency, and explain the link to TCP congestion control
- Schedule work fairly across customers with token-weighted deficit round robin, and quantify the effect on waiting times