Isotonic calibration fits flat regions: runs of scores that receive one probability. A flat region costs a decision only when the cases it pools fall on both sides of the decision's threshold, and a ranking only when its ties are broken at random. Using an exact known-risk simulation and a pre-registered benchmark, the paper finds flat regions cheap for cost-threshold decisions: choosing a calibrator by validation payoff did no better than choosing it by Brier score. For allocation, random tie-breaking costs up to 1.7 successes per 1,000 cases, which breaking ties by the original score recovers exactly.
Cite: Gaurav Sood. "When Pooling Costs a Decision: Probability Calibration with Calibre." https://gsood.com/research/calibration-where-it-counts-cost-and-data-informed-isotonic-regression/
BibTeX:
@misc{soodcalibration,
author = {Gaurav Sood},
title = {{When Pooling Costs a Decision: Probability Calibration with Calibre}},
note = {Working paper},
url = {https://gsood.com/research/calibration-where-it-counts-cost-and-data-informed-isotonic-regression/}
}