When is speculation worth it?

Speculation trades tokens for latency, and tokens are not free even when you aren't paying per token: they take capacity in the same OTPM bucket as every other call. So the model has two sides. The saving is the probability that a guess is right times the time it saves. The cost is extra load on the bucket, which raises the queueing wait of every call, through the $\rho/(1-\rho)$ curve. The decision that matters most is to compute the cost for the whole fleet, not only for the turn that speculates, because that is where speculation stops paying first.