u/dadumdiss

Okay so I've been building this system where we train thousands of models from different families (gbm, mlp,etc) and backtest them vigorously and then track their signals to create a leaderboard where the idea is no matter the market condition, there's gonna be a model doing good last couple of hours so we can choose a good performing one for the last X time period. The best models so far have been pretty consistent and consistently outperforming SPY and bitcoin buy and hold. think my Al is lying to me but have told it multiple times in new sessions across coding agents to audit the maths and it says it's good. think the Al are lying to me. Can you help me prove it please?

Edit: detailed methodology at superquacks.com/methodolody

A few days ago I changed the validation threshold so it scales with the number of times a pair has been searched, instead of using one fixed threshold for every pair.

The change was based on a formula, but I had not checked whether the formula matched an empirical null distribution.

I tested it using SPY 1h’s actual search count: 124 attempts and 20 shuffled null runs.

The measured 95th-percentile null value was 0.9956. The deployed formula predicted 0.8862.

Using the deployed threshold, 13 of 20 shuffled runs exceeded it. The threshold was therefore allowing noise to pass much more often than intended.

The problem was the assumed distribution. The formula treated the metric as unbounded, even though the metric is capped at 1. The curve shape was wrong.

I replaced it with a bounded version, but I’m treating that as a temporary correction. The better approach is probably to measure the null distribution at the actual search scale instead of extrapolating from a small sample.

Twenty shuffled runs are not enough to estimate a precise 95th percentile, so I’m treating this as a diagnostic, not a final statistical result.

I also ran a frozen-model comparison. I stopped one model from retraining or being replaced and compared it with the normal system over the same four-day paper-trading window.

The normal system finished about 1.4 points ahead on the tracked comparison metric. That is a small sample and not evidence of a durable edge.

I’m interested in feedback on:

• whether 20 shuffled runs is enough for an initial diagnostic
• whether the null should be generated at the full search scale
• whether the frozen-model comparison is fair
• whether the bounded correction is statistically appropriate

The main result is that my original threshold fix was not properly validated. The null test exposed that.

I posted about the system here yesterday (https://www.reddit.com/r/algotrading/comments/1vfzlo8/i_built_a_247_system_that_trains_backtests_and/) and got real feedback (champion selections as a rolling max, shuffle-test idea, and the tenure)
u/Many-Pick5066 and u/Effective_Manager273, thank you!

I did what you suggested and dug into it. I fixed the market window and hyperparameters completely and only varied the random seed across 10 fits: consistency_score (min/max of first-half vs second-half Sharpe, our stability check) swung from 0.00 to 0.71, std=0.21, on identical data. Only 1 of 10 seeds would've cleared the gate. GBM and logistic came back std=0.00. Fixed it by scoring off the median of 5 seeds instead of one. Live model behavior is untouched.

results seem stable so far. algotrading rules say no PL porn so not gonna bore you with that. detailed results published.

Where do you think this system would break? What would success look like to you? Looking to hear your opinions.

I’ve been building this solo for the past couple of months. I’m sharing it here because this sub is good at tearing apart backtest-only claims, and I’d rather find out where I’m fooling myself now.

The system continuously tests three types of strategies:

• Classical ML models
• TimesFM with a trainable head
• LLM-written, rule-based strategies

They compete across SPY and BTC on six timeframes. Every model has to pass the same walk-forward, consistency, and minimum-trade-count gates before it can become a “champion.” Champions are then re-verified on a rolling basis and replaced if they stop qualifying.

I’m not leading with a Sharpe ratio because a raw Sharpe over a short window can look impressive while saying very little. Even buy-and-hold SPY can annualize to a great-looking Sharpe over the right window. The dashboard therefore shows every strategy’s Sharpe beside buy-and-hold over the exact same period.

The part I think is most useful is the forward-only paper-trading ledger. It’s completely separate from the backtests. Positions open and close using live signals and live prices across three execution tiers:

• Realtime
• Actually delayed by one minute
• Tighter, institutional-style fees and slippage

The point is to see how execution quality changes the result instead of hiding everything inside one assumed friction number.

The ledger is only one day old, so none of this is statistically meaningful yet. But here’s what it currently shows:

• SPY on the 15-minute and 1-hour timeframes is holding up so far, both before and after fees and slippage.
• SPY on the 5-minute timeframe is net negative, even before friction. That’s a real problem I haven’t solved.
• BTC has not produced a single model that passes every gate on any timeframe. More than half of the actual attempts fail directional accuracy outright—they’re worse than a coin flip. My current read is that the feature set has no meaningful BTC edge at these timeframes, not that I need to loosen the gates.
• Every current champion was promoted within the past 24 hours. None has earned any real trust yet, and the dashboard labels them “too new to judge.”

I’m not selling anything. I’m looking for criticism of the methodology and for people to share the results with. (see daily results via orbitquantapp.com before i post here and giving away 100 lifetime accesses if it ends up working)

Where would you expect a system like this to be lying to itself? What evidence would you need before considering it trustworthy enough for real money?

https://preview.redd.it/h3x191um5ihh1.png?width=1897&format=png&auto=webp&s=09247b7686649e5633cfa9593a58214388143e06