Concept

Probability Calibration Curves

Probability Calibration Curves are Machine Learning concepts. First implementations are in the build queue: the write-up leads, the indicators follow.

What are probability calibration curves?

A probability calibration curve, often called a reliability diagram, tests whether a model's stated probabilities mean what they say. Predictions are grouped into bins by predicted probability, and for each bin the observed frequency of the positive outcome is plotted against the average predicted probability. A perfectly calibrated model lies on the diagonal: among all bars where it said 70 percent, the event happened about 70 percent of the time. Curves bowing away from the diagonal reveal systematic overconfidence or underconfidence.

Traders care because probabilities are only useful for sizing and thresholding if they are honest. Many models rank outcomes well while producing distorted probabilities: Bayesian classifiers tend toward overconfident extremes, and boosted ensembles often compress predictions away from 0 and 1. A miscalibrated 80 percent is fine for ranking setups but dangerous when fed into position sizing. Calibration curves diagnose the distortion, and methods such as logistic signal calibration (Platt scaling) or isotonic regression correct it by remapping raw scores to calibrated probabilities on held-out data.

How it's calculated

The curve is built by binning predictions and comparing predicted to observed frequencies. The Brier score summarizes overall probabilistic accuracy.

bin_k = set of predictions with p_hat in interval k

confidence_k = mean(p_hat) over bin_k

accuracy_k = fraction of positive outcomes in bin_k

curve: plot accuracy_k against confidence_k for all k

Brier = (1/N) * sum((p_hat_i - y_i)^2)

ECE = sum((n_k / N) * abs(accuracy_k - confidence_k))

p_hat_i: model's predicted probability for observation i

y_i: realized outcome for observation i (1 if the event occurred, 0 otherwise)

n_k: number of predictions in bin k

N: total number of predictions

k: bin index (10 equal-width bins is a common default)

ECE: expected calibration error, a weighted average gap between the curve and the diagonal

Bin count is a judgment call: few bins hide structure, many bins make each estimate noisy.

Curves must be computed on data not used to fit or calibrate the model, or they flatter the model.

How traders use it

Auditing a signal model before sizing: if the curve shows 90 percent predictions winning only 60 percent of the time, thresholds and stake sizes built on the raw probabilities are wrong and need recalibration first.
Choosing a recalibration method: a smooth S-shaped distortion suggests Platt scaling; irregular, non-monotonic-looking bins with enough data suggest isotonic regression, which is more flexible but needs more observations.
Comparing models on more than accuracy: two models with equal hit rates can have very different curves, and the better-calibrated one is more useful for expectancy math and risk budgeting.
Monitoring drift: recomputing the curve on recent live predictions shows whether calibration achieved in backtesting has decayed as the market regime shifted.
Respecting the limits: with a few hundred trades the per-bin frequencies carry wide error bars, so small wiggles in the curve are noise, not structure.

Calibration curves vs. related tools

Logistic Signal Calibration: The curve is the diagnostic; logistic (Platt) calibration is one of the treatments, fitting a sigmoid that remaps raw scores toward the diagonal. You read the curve first, then decide whether and how to correct.

Sigmoid and Softmax Transforms: These transforms squash scores into the 0-to-1 range, which makes outputs look like probabilities but does nothing to make them honest. Only comparison against realized frequencies establishes calibration.

Related concepts · Features & pipeline

Feature Engineering3 Logistic Signal Calibration3 PCA1 Label Definition & Prediction Horizon1 Online/incremental Learning1 Feature Selection0 Train/validation Discipline0 Model Overfitting0

Concept family

Machine Learning

32 concepts mapped · 32 in the Library

Probability Calibration Curves FAQ

What does a curve below the diagonal mean?

The model is overconfident in that region: it predicts higher probabilities than the events justify. Above the diagonal means underconfidence. Many trading models are overconfident at the extremes and roughly calibrated near 50 percent.

How many trades do I need for a meaningful curve?

More than most traders have. With 10 bins you want at least a few dozen observations per bin, so several hundred predictions is a practical minimum, and even then the tails of the curve remain noisy.

Does good calibration mean the model is profitable?

No. A model that always predicts the base rate is perfectly calibrated and completely uninformative. Calibration measures honesty of probabilities, not their sharpness; a useful model needs both.

Should I calibrate on training data?

Never. Calibration fitted or evaluated on training data inherits the model's memorized fit and reports a curve closer to the diagonal than live performance will show. Use a held-out window or out-of-fold predictions.

Build Probability Calibration Curves your way.

Quant writes, tests, and refines it with you — then it runs on LuxAlgo charting or ports to TradingView.