Module 2 · Signals from real use · Lesson 5
Understanding usage at scale, privately
Find out what thousands of conversations are about and where an assistant fails without anyone reading them, using facets, clustering, aggregation thresholds and audits, and validate the pipeline on synthetic data with planted topics and personal data.
By the end of this lesson you can
- Build a pipeline of facet extraction, embedding, clustering and cluster description over conversation logs, with model-written summaries that omit personal data
- Apply privacy layers (summaries without personal data, minimum unique-account and conversation thresholds, a cluster audit) and compute what differential privacy's noise does to small counts
- Validate the pipeline on synthetic conversations with a planted topic mix and planted personal data, measuring topic recovery and leakage at every stage
- Turn cluster-level outcome rates into new eval tasks and review strata without copying raw conversations