Calibration
How well a model's stated confidence tracks actual correctness. LLMs are typically over-confident, meaning they often present incorrect information with high certainty.
Calibration refers to the alignment between a model's predicted probability scores and its empirical accuracy. In practice, it measures whether the model's confidence levels reflect its true error rate. Most LLMs suffer from poor calibration, exhibiting a systematic bias toward high-confidence outputs even when the generated content is factually incorrect.
Calibration is the statistical correspondence between a model's assigned confidence scores and the observed frequency of correctness, typically evaluated via reliability diagrams or Expected Calibration Error (ECE). A model is perfectly calibrated if, for any confidence score p, the empirical accuracy equals p. LLMs frequently demonstrate over-confidence, where the predicted probability distribution is skewed toward high-certainty tokens despite a lack of corresponding ground-truth accuracy.
evolution
- 1982 · historyIntroduction of Brier Score
Glenn Brier introduced a scoring rule to measure the accuracy of probabilistic predictions, establishing the foundation for calibration assessment.
- 2017 · historyModern Calibration Analysis
Guo et al. published 'On Calibration of Modern Neural Networks,' demonstrating that deep learning models are poorly calibrated and proposing Temperature Scaling as a solution.
- 2020 · historyLLM Calibration Research
Researchers began systematically documenting that large language models exhibit significant overconfidence, often failing to align their output probabilities with actual correctness.
- 2023 · historyInstruction-Tuned Calibration
Studies showed that while RLHF improves helpfulness, it often exacerbates overconfidence, leading to new efforts in calibrating LLMs through post-hoc techniques and verbalized confidence scores.