Measure a semantic cache as a classifier

Treat every cache hit as a prediction that "the stored answer is right for this query", label pairs with the truth, and measure. The dataset is what makes the measurement worth anything, so most of the effort goes into categories that each test one failure: paraphrases (should hit), the same question on another matter, negations, tenure swaps, date changes (should all miss), and firm FAQ paraphrases (should hit, within the firm). The decision that matters most is to report results by category: the overall precision hides the fact that the "different" categories fail almost completely and for different reasons.