I built a next pitch prediction bot and scored it against real, live pitches for a month (50,760 pitches). Here is what it learned about pitcher predictability, plus a "deception" metric I threw out.
I built a model that predicts the type of the next pitch (four seam, sinker, cutter, changeup, curve, slider, sweeper, splitter, knuckle) before it is thrown, wired it to live games, and have been scoring every prediction against what actually got thrown. This post is the ture, honest version of what came out, including the parts that did not work, because I think the negatives are more interesting than the headline number if i'm honest.
TLDR
- Live accuracy over 50,760 scored pitches: 42.8% top-1, 85.7% top-3. Offline, on a frozen test set, it beats a strong pitcher and count conditioned baseline by +6.7 percentage points (38.9% to 45.6%). Live is lower than offline, which is what should happen.
- Predictability is a repertoire trait, not a talent gap. Elite arms show up at both ends of the predictability scale.
- The model's single biggest failure mode: when it is wrong, it guesses fastball.
- I built a metric that looked like it measured pitcher deception. It is very reliable. Then I tested it against real batter outcomes and it predicted nothing, so I threw the interpretation out.
1. How good is it, honestly
42.8% top-1 sounds low until you compare it to the right baseline. The naive "always guess his most common pitch" is easy to beat.... the real baseline is each pitcher's own count - conditioned mix, smoothed. Beating that by ~7 points offline means the model is reading context, not just base rates. Top-3 at ~86% is the more useful number for most real uses: even when the exact call misses, the pitch is almost always in the top three.
The live number moves around by day, mostly because of *who pitched*, not model quality. I compute a composition adjusted expected accuracy (each day's pitches weighted by each pitcher's own history) and the actual line tracks it closely. A recent dip is a harder slate, not decay (Page/Hinkley drift detector - no alarm).
2. Predictable is not the same as good
This is the finding I care about most. Rank pitchers by how often the model nails the exact next pitch and you get elite arms scattered top to bottom:
Kenley Jansen and Tim Hill are basically a coin you can call in advance. Max Fried, Yoshinobu Yamamoto, and Tarik Skubal are near the bottom. All of them are good!!! Chris Sale has a low arsenal entropy and Skubal a high! Both are, quite obviously, aces. So predictability and quality are uncorrelated, and you should never fold a "predictability" number into a "how good is he" number. Predictable is a description of the arsenal, not a criticism of the pitcher.
3. When the model is wrong, it guesses fastball
Every offspeed and breaking pitch's single most common wrong prediction is a four-seam fastball:
Part of this is the model over defaulting to the majority class, and part is genuine label fuzz (Statcast's four-seam / sinker / cutter boundary is noisy). When I collapse those three into one "fastball family," a big chunk of the apparent error disappears, but not all of it, so the overcall is real, not just a labeling artifact.
4. The metric I built, liked, and killed
I wanted to measure pitcher deception: is a pitcher harder to read than his raw pitch mix diversity alone would predict? I built a residual (actual readability minus what mix entropy predicts). It looked great! :
- It is reliable: split-half correlation of 0.92, and it survives controlling for role and count context.
- It produces a sensible leaderboard (some guys read easier than their mix implies, some harder).
Then I did the step most people skip. I correlated it against things batters actually produce, controlling for stuff quality:
Nothing. No relationship with whiff rate, called strike rate, chase rate, or run prevention. So the residual is a reliable, stable property of how my model reads a pitcher, with no demonstrated connection to deception. Most likely it is a model blind spot, not really a hitter relevant trait. Reliability is not validity. I kept it as an internal diagnostic and built no "deception index" on top of it, which was hard to do because the story was so good.
5. What does not improve it
I ran four honest ablations trying to push accuracy up: pitch movement features, within at bat sequencing plus tunneling, an online ingame adaptation layer, and batter side features. All null or actively harmful tbh. The takeaway is that the model is near its ceiling for pitcher side features, the signal lives in tendency and rate features (how often he throws X in this count, what he threw last, pitcher and batter identity), and those are already saturated. The online adaptation experiment did surface one real thing: pitchers negatively autocorrelate (after a fastball streak they are more likely to change), which the static transition rates already capture.
6. The boring parts that make me trust it to the best of my knowledge....
Leak-safe temporal splits with a hard assertion against future leakage, probability calibration (overall ECE ~0.01; the 3-0 count is the known weak spot at ~0.05 and I am fixing it), a live trust score built from historical per-(count, pitch type) precision, drift detection on the composition adjusted residual, and empirical Bayes shrinkage on the per pitcher numbers so small samples do not lie.
Limitations
- Live (42.8%) trails offline (45.6%), as expected from distribution shift and new pitchers.
- The 3-0 count is genuinely miscalibrated (rare, small sample, overconfident) and is the one production issue I would flag.
- This is pitch type, not location, and the swing/whiff model is secondary.
- The "deception" residual failed external validation, so please do not read the readability numbers as a deception measure.
Happy to answer questions on any of it, and genuinely curious what this sub would test next. My thought is that any further accuracy is a dead end and the interesting frontier is measurement (what predictability correlates with, if anything), but I have been very, very wrong before within this project.
Also for fun - my dashboard screen shots.
Also included write up