calibration-error

A calibration error metric quantifies the discrepancy between a model's predicted probabilities and the actual observed frequencies of outcomes. It is computed by binning predictions into probability intervals, computing the accuracy (fraction of positives) within each bin, and measuring the gap between predicted and observed values across bins. Common formulations include Expected Calibration Error (ECE), a weighted average gap across bins, and Maximum Calibration Error (MCE), the largest single-bin gap. The metric operates as a diagnostic for probabilistic models, identifying whether a model's confidence aligns with empirical correctness — a well-calibrated model's predictions at 70% should correspond to roughly 70% actual occurrence. Persistence: formalized through machine learning evaluation frameworks, deployed in model validation pipelines, and tracked as a standard metric in model cards and evaluation reports across AI/ML practice. [formal: error_calibratio | substrate: behavior | horizon: a moment | explicit: yes | epoch: 0.01]