Module 2 · Optimising and operating prompts · Lesson 6
Prompt registries and online tests
Where prompts live and how a new version reaches production - git, registries such as Langfuse, Braintrust and PromptLayer, labels, caching and rollback - and how to run a case-level online test whose result you can trust, with sample-ratio checks and no peeking.
By the end of this lesson you can
- Choose between prompts in git, a prompt registry or a hybrid, and define what a prompt version must contain
- Describe how labels, client-side caching, fallbacks and rollback work in current registries, and the failure modes each introduces
- Implement deterministic case-level assignment and link every output to the prompt version that produced it
- Detect a sample ratio mismatch with a chi-square test, and explain why daily peeking at a fixed-horizon test inflates false positives
- Choose between an A/B test, a blind paired review and offline gates when case volume is low