Forecasting and Calibration
Gabriele Farina
MIT 6.7980 · Topics in Multiagent Learning · Fall 2026
The important picture for today
over
over
forecaster over
on , utilities in
Legend: Each arrow turns the algorithm in its starting box into the algorithm in its ending box using the optimization primitive written on the arrow.
Forecasting: What does a 70% chance of rain mean?
A forecaster reports before seeing the weather: means rain, and means no rain.
It should rain on about 70% of those days.
Repeat this check for every probability the forecaster uses.
The above condition is called calibration.
How calibration became an online learning problem
1977 · Murphy & Winkler study whether weather forecasters’ probabilities agree with observed frequencies.
1982 · Dawid develops calibration as a criterion for probability forecasts.
1985 · Oakes and Dawid show that every deterministic forecasting rule fails calibration on some outcome sequence.
1998 · Foster & Vohra construct randomized forecasts that are calibrated on every outcome sequence.
Murphy & Winkler (1977) · Dawid (1982) · Oakes (1985) · Foster & Vohra (1998).
More formally: Expected calibration error (ECE)
Consider a finite grid of forecasts, and
Overprediction in one forecast bucket cannot cancel underprediction in another.
Impossibility when Nature can use the forecast adversarially
It is impossible to control ECE against an adversarial process. After seeing , Nature makes it rain when and keeps the day dry otherwise:
Forecasts below underestimate rain by more than . Forecasts at least overestimate it by at least .
What changes if Nature sees the forecast distribution, but not the forecast drawn from it?
Oakes (1985), Dawid (1985); Foster & Vohra (1998), introduction.
Calibration without a weather model
Using past outcomes and private randomization, a forecaster can make its long-run calibration error arbitrarily small on every outcome sequence.
The algorithm needs no atmospheric measurements or meteorological model. It also works when the process generating the outcomes changes over time.
Is this a positive or a negative result?
Intuition: a forecaster can “cook the books”
Simplified example: imagine that the calibration error was measured as
There exists a simple way of ensuring the above quantity . How?
Multicalibration protocol
1. Observe , the information available before forecasting.
2. Announce a distribution over forecasts .
3. Nature chooses the outcome .
4. Observe and update the learner.
Nature sees , but must choose before seeing the sampled forecast. The expectations below average over this draw.
Both forecasts and outcomes lie in a compact convex set . We assume the tests are bounded and measurable.
Write the error for a general test
A test measures the error . It gives a weight for scalar errors and a direction for vector errors:
If contains whenever it contains , the bound also controls . We can include the zero test.
ECE is a special case of multicalibration
Ignore the context and choose a separate sign for each forecast value:
For the observed forecasts,
Choose each sign to make that bucket’s accumulated error nonnegative.
From forecasts to decisions
As it turns out, multicalibration has a flavor similar to -regret minimization. We will see how both can be reduced to
(regret minimization) + (a nonlinear optimization primitive)Recall Φ-regret
Choose in a compact convex set and earn . Compare this with playing , for each fixed deviation .
External regret
for a fixed action .
Swap regret
On a simplex, for a column-stochastic matrix .
We want .
First route: Gordon-Greenwald-Marks
over
over
forecaster over
on , utilities in
Ask the learner for , play , then return the utility
Gordon, Greenwald & Marks (2008). This version uses exact fixed points. We assume an efficient fixed-point oracle.
The fixed point makes the two regrets equal
over
over
forecaster over
on , utilities in
Fix any comparator . The external regret against it is
Learn calibrated forecasts
over
over
forecaster over
on , utilities in
The no-regret learner suggests which test to pay attention to. We then try to “neutralize” it.
Expected variational inequalities
Given , find a distribution over satisfying
The operator may be discontinuous. A distribution over forecasts can satisfy the bound.
We assume an EVI solver. Its cost and accuracy depend on the domain and how we can access .
An algorithm for multicalibration
Run an external-regret learner on (or its convex hull).
1. Observe and ask the learner for .
2. Solve an -EVI for ; output .
3. Observe and send the linear utility below to the learner.
The EVI inequality applies to the realized .
Calibration is bounded by external regret
over
over
forecaster over
on , utilities in
Fix . Add and subtract the learner’s scores:
Apply in each round.
From multicalibration to Φ-regret
over
over
forecaster over
on , utilities in
... we might as well best respond to the prediction.
Best respond to the forecast
Forecast the next utility vector . Choose a best response:
Draw
Play
Observe
Let be the distribution of for . Use a fixed rule to break ties.
Announce before Nature chooses , and keep the sampled action hidden. Taking the mean action can change regret for nonlinear deviations.
Which multicalibration tests to use?
For each , the test is the change it makes to our chosen action:
With context-dependent deviations, use .
Bound Φ-regret by calibration error
over
over
forecaster over
on , utilities in
(Last step because )
Since is a best response and , the forecast term is nonpositive.
Combine the two forecasting reductions
over
over
forecaster over
on , utilities in
Two routes from external regret
| Ingredient | Gordon-Greenwald-Marks | Forecasting route |
|---|---|---|
| Learn over | Valid deviations | Tests |
| Optimization step | Fixed point of | EVI for |
| Play | The fixed point | Best response |
| Regret bound | Exact identity | Upper bound + EVI error |
No geometry of endomorphisms needs to be taken into account.
Summary
over
over
forecaster over
on , utilities in