LLM-as-judge systems produce numerical scores, but downstream decisions often require pairwise comparisons: which response is better? Converting scores to win rates requires calibration—the mapping from score differences to win probabilities. We develop calibrated estimators for pairwise win rates from cardinal LLM judge scores, accounting for judge miscalibration and non-transitivity in preferences.
Cite: Gaurav Sood. "From Scores to Signs: Pairwise Win-Rate Estimation with Calibrated LLM Judges." https://gsood.com/research/from-scores-to-signs-pairwise-win-rate-estimation-with-calibrated-llm-judges/
BibTeX:
@misc{soodfrom,
author = {Gaurav Sood},
title = {{From Scores to Signs: Pairwise Win-Rate Estimation with Calibrated LLM Judges}},
note = {Working paper},
url = {https://gsood.com/research/from-scores-to-signs-pairwise-win-rate-estimation-with-calibrated-llm-judges/}
}