u/sportshackai

Title: Held-out calibration on 4,570 MLB starts: Brier 0.2375 vs 0.2806 baseline — looking for criticism of the validation design

I've been working on a probabilistic starting-pitcher strikeout model and wanted to post the validation methodology rather than predictions.

The entire 2025 MLB season was withheld from model fitting and used as the out-of-sample test set: 4,570 starts the model had never seen.

The main results:

Brier score: 0.2375
Naive/base-rate Brier: 0.2806
Expected Calibration Error: 0.0049
Mean PIT: 0.4977
Holdout: 4,570 starts

The piece I'm most interested in is calibration rather than raw classification accuracy.

For the reliability table, predicted probabilities were grouped into probability bands and compared against observed frequencies.

For example:

50–55% predicted → 50.9% observed
55–60% → 55.5%
60–65% → 59.3%
65–70% → 63.3%
70%+ → 70.1%

I'm deliberately trying to avoid the usual sports-model trap of saying something like "the model was 64% accurate" without establishing what was predicted, at what probability, or against what baseline.

The validation gate was designed around a few requirements:

1. No lookahead. Inputs/fitting procedures can only use information that would have existed before the game being predicted.
2. Full-season holdout. 2025 was excluded from training rather than randomly splitting games across seasons.
3. Brier against a baseline. A Brier score in isolation isn't particularly informative, so I'm comparing it against a naive probability baseline on the identical events.
4. Reliability/calibration. If the model assigns a group of events ~60%, they should occur roughly 60% of the time.
5. PIT diagnostics. I'm looking at where actual outcomes fall within the predicted distributions, rather than only checking the mean prediction.

One thing I've tried to be careful about is separating calibration from usefulness. A model can be beautifully calibrated by staying close to the base rate and still contain very little information. Conversely, a sharp model can have useful discrimination while being overconfident.

So rather than asking whether these numbers are "good," I'd be interested in how people here would try to break this validation design.

A few questions I'm considering:

• Is the naive/base-rate Brier benchmark the right primary baseline, or would you want additional benchmarks?
• Would you report Brier decomposition into reliability, resolution and uncertainty?
• Would you bootstrap confidence intervals around Brier/Brier skill rather than report point estimates?
• For calibration, would you prefer adaptive/equal-count bins over fixed probability bands?
• What would you use to test whether the apparent calibration survives season-to-season distribution shift?
• Are PIT + reliability + Brier redundant here, or do you think all three earn their place?
• What failure mode would you look for first if you were reviewing this?

I'm much more interested in finding where the methodology is weak than in defending the headline number.

Would appreciate any criticism from people who have done probabilistic baseball forecasting.