Module 2 · Signals from real use · Lesson 4
Feedback from users
Instrument thumbs, edits, retries and complaints so every signal links to a trace and a denominator, measure each signal's validity against expert labels, and correct a feedback-based rate for who chose to give feedback.
Setup for the exercises:Install Claude CodePython with uvTrace LLM calls with OpenTelemetry and Langfuse
By the end of this lesson you can
- Design a feedback event schema that links explicit signals, implicit signals and the exposures behind them to the trace, the rated answer and the prompt version
- Measure a signal's validity the way you would a judge's, with TPR and TNR against expert labels from a sample designed for that signal
- Compute a naive and a stratum-reweighted feedback rate, with its effective sample size, and state what reweighting cannot correct
- Explain how optimising for user approval produces sycophancy, and choose counter-metrics that users cannot move by being pleased