Day 4 results
A few days ago I changed the validation threshold so it scales with the number of times a pair has been searched, instead of using one fixed threshold for every pair.
The change was based on a formula, but I had not checked whether the formula matched an empirical null distribution.
I tested it using SPY 1h’s actual search count: 124 attempts and 20 shuffled null runs.
The measured 95th-percentile null value was 0.9956. The deployed formula predicted 0.8862.
Using the deployed threshold, 13 of 20 shuffled runs exceeded it. The threshold was therefore allowing noise to pass much more often than intended.
The problem was the assumed distribution. The formula treated the metric as unbounded, even though the metric is capped at 1. The curve shape was wrong.
I replaced it with a bounded version, but I’m treating that as a temporary correction. The better approach is probably to measure the null distribution at the actual search scale instead of extrapolating from a small sample.
Twenty shuffled runs are not enough to estimate a precise 95th percentile, so I’m treating this as a diagnostic, not a final statistical result.
I also ran a frozen-model comparison. I stopped one model from retraining or being replaced and compared it with the normal system over the same four-day paper-trading window.
The normal system finished about 1.4 points ahead on the tracked comparison metric. That is a small sample and not evidence of a durable edge.
I’m interested in feedback on:
- whether 20 shuffled runs is enough for an initial diagnostic
- whether the null should be generated at the full search scale
- whether the frozen-model comparison is fair
- whether the bounded correction is statistically appropriate
The main result is that my original threshold fix was not properly validated. The null test exposed that.