Module 2 · Latency, resilience and capacity · Lesson 6

Response caching and failover

Serve some answers without calling the model, exactly or by similarity, and know why similarity caching is unsafe for case-specific questions. Then keep the fleet running through provider overloads and regional incidents with Claude on several platforms, without a cold-cache stampede on the fallback.

75 min 1 video 3 exercises 7-question quiz Answers

Setup for the exercises:Install Claude Code Python with uv

By the end of this lesson you can

Design exact-match response cache keys that include every input that changes the answer, including document versions and tenant
Explain how semantic caches decide hits, and quantify their false-positive cost as a classifier
Identify why semantic caching is dangerous for case-specific work, and name the uses where it is acceptable
Compare Claude on the Claude API, Amazon Bedrock and Google Cloud as failover targets on quotas, endpoints, residency and features
Plan failover so that the fallback's quota survives the cold-cache surge, with warm standby, stickiness and shedding

Two different questions come up once the fleet is busy. First: some calls are repeats, so why pay for them again? A crashed ingestion job re-extracts the same TA6 property information form; three fee earners ask the assistant what the firm's ID-check policy is. Second: the provider will have a bad day, so where does the traffic go? Both are about serving a request without the primary path, and both have a trap. For caching, the trap is an answer that was right for a different question. For failover, it's a fallback that collapses under the very surge it was meant to absorb.

Response caching and failover

Response caching and failover

Response caching and failover

Sign in to keep reading.

Sign in to keep reading.