u/espressodoppioo

Building my bot and doing a lot of backtesting these days.

I had a breakout system that looked bulletproof on BTC: +66%, profit factor 3.2, 13% max drawdown, profitable in 6 of 9 walk-forward windows. So far, so great.

But, it only fired about 8 trades a year. At that frequency a single-asset walk-forward can't tell a real edge from getting lucky. The sample is just too small, no matter how you slice the windows.

So I froze the exact config, no re-tuning, and ran it on ETH and SOL.

• BTC: +66%
• ETH: -11%
• SOL: -60%, with a 0% win rate.

Also tried different parameters, but no parameter set rescued the other two. It was fit to BTC.

Might still be something "real" that only happens on BTC. But more likely just overfitting.

In contrast, my market-neutral funding carry pays +6.7 / +6.6 / +5.6% on BTC, ETH, SOL. Neatly aligned, what a real structural edge should looks like, boring and the same everywhere.

If your edge is low frequency and only tested on one asset, you don't know it's real yet. You know it fit one history. Might still make you money.

Do you cross-validate across instruments, or is single-asset walk-forward enough for you?

Inspired by another post here, I wanted to share my experience so far.

Been building an algorithmic crypto trading bot with AI for a few months now, also some real money, and I want to share the honest version, because this sub looks like it leans toward 'AI finds the edge' and my experience was pretty much the opposite.

I tested a lot of entry signals - models to predict next-move direction, pattern classifiers, the works. Almost every one looked great in-sample. Some so much, that I thought my dream of getting rich quick would actually work out ;) But then they died the moment I ran proper validation (walk-forward, a multiple-testing correction, checking it on assets it wasn't tuned on). Single-asset price prediction in crypto is close to noise, and AI/ML is extremely good at fitting noise and calling it signal. The prettier the backtest, the more suspicious I got.

The things that actually survived testing were rarely anything based on classic indicators. I rather found structural edges any spreadsheet could describe: a market-neutral funding-carry trade (collect the funding that perps pay), and a hedged volatility-selling strategy. They get paid whether the market goes up or down.

Where AI helped me a lot and will for all time coming is the engineering around it. It wrote and reviewed the plumbing far faster than I would have had. Atomic entry+stop orders, a stop-loss that re-asserts every tick, position reconciliation, logging. The stuff that actually keeps a real-money bot from blowing up. I feel this part is not being talked about enough. It really is the boring part, but here AI is a giant multiplier for productivity.

To a smaller degree AI is also helpful in generating ideas, especially for topics that are new to you.

AI is for me an essential part of building my trading bot. But Im not using it to actually predict price. And maybe its just me, but I have a hard time trusting its predictions after my own backtests.

Curious where people here have actually gotten AI to add value - prediction, execution, research, or the plumbing? And has anyone gotten an ML price signal to survive real out-of-sample?

Everyone repeats it: don't take the signal raw, wait for confirmation. I wanted to check whether that actually helps my numbers, so I measured one specific version of it.

What I actually tested (this matters): the divergence signal family on BTC, across thousands of setups, where 'confirmation' just means waiting a fixed number of bars after the raw signal before entering. That is only ONE way to read the word.

Signed mean return by how long I waited:

0 bars: -7.4 bps

3 bars: -31.7 bps

10 bars: -39.5 bps

30 bars: +2.7 bps

On this setup, waiting didn't filter out the bad trades. It mostly gave back the move I was trying to catch, until the effect washed out into noise around 30 bars.

I have to say, this is one signal family, on one asset, with one narrow definition of confirmation. It says nothing about waiting for a candle close, a break of structure, a retest, a volume spike, or multi-timeframe alignment. There are almost certainly setups where some form of confirmation is necessary. I just couldn't confirm the benefit in the exact thing I measured. The funny thing is, this is exactly what was claimed by some "strategy guru".

So I'm not claiming 'confirmation is useless.' I'm saying: when I tried to verify this particular version of the advice, the numbers didn't back it up. I may check other forms of confirmations and also on other signals.

My learning (again): Only trust your own backtests. Check everything yourself before putting in any real money.

Has confirmation actually improved your backtested numbers, and how did you define it?

Divergence trading is everywhere in crypto: price makes a higher high, RSI makes a lower high, 'momentum is fading, short it.' I wanted to actually know if it works (and if), so I tested it about as hard as I could.

8 variants (regular + hidden, bull + bear, RSI + volume), 6 timeframes from 5m to 1D, 4 confirmation delays, walk-forward over 6+ years of BTC, significance from permutation tests. About 2,200 configurations total. Sounds a lot, combining those adds up real quick.

Pure chance would hand you roughly 220 'significant' looking results at p < 0.10. I found 4. At p < 0.05: zero.

The prettiest one (+1,990 bps on the daily) was n=8 signals, p=0.31. That is the exact cherry someone picks to sell you a course. I mean, could this be a fantastic edge? Sure it could. But I would not trust it and never ever put real money in a result like this.

My best guess why: the divergence isn't the signal, the pivot is. If you systematically enter at local highs and lows, your odds are structurally worse than random.

To be fair, this only kills divergence as a standalone entry. As one factor inside a specific regime it might add something, I haven't tested that. And also, there might be other ways to bring the idea of divergences into code.

Has anyone actually walk-forwarded divergences and gotten a different answer? Genuinely curious.

Saw another "90% win rate" screenshot today and wanted to share something that took me way too long to learn.

Win rate tells you how often you're right. It tells you nothing about how much you lose when you're wrong. Those are two completely different questions, and the second one is the one that blows up accounts. (mine as well in the past)

A 90% win rate is basically the fingerprint of mean reversion (and selling options): you collect a pile of small wins, then hand it all back in one move that quietly eats a month, if you are not careful. The number isn't lying, it's just answering a question nobody should be asking.

Concrete example from my own testing: I had a system winning ~63% of trades that still bled money, because the average loser was 2.5x the average winner. Looked great on the win-rate line, lost money on the equity curve.

My trading journey so far was mostly pain. But at least I stopped losing money.

What I actually check now before trusting a strategy:

- average win vs average loss (the payoff ratio)

- the single worst trade and the worst drawdown

- trade count (a 90% win rate on 10 trades is just noise)

If someone shows you a win rate without the loss distribution, they're showing you the half of the picture that looks good.

Curious how the discretionary folks here think about it: do you track your payoff ratio, or mostly go by feel on the exits?

For the last few months I've been building Botty, a small algorithmic trading bot that runs real money on crypto. But the actual project isn't "make a profitable bot." Well, that was the idea, get rich quick 😀 Now it's documenting, in public, every idea that doesn't work and why. Turns out that's most of them don't work.

Quick example: I had 20 strategies that looked great in backtests. Then I ran a proper statistical test that corrects for how many things you tried. 0 of 20 survived. That stung, but it was the most useful day of the whole project.

The thing nobody tells you about a project like this: 90% of the work is unglamorous plumbing and killing your own favorite ideas. An overnight job segfaulted and wiped a week of results. I once wrote 1.3TB to disk by accident because of a dumb loop. The "AI co-builder" sped things up but did not skip the boring part.

I'm writing it all up as I go, dead strategies included, because every build-in-public account I followed only ever showed the wins and that always felt a little dishonest.

Question for the people building something technical here: how do you decide what's worth sharing when most of the real work is failure? Do you post the dead ends, or only the milestones?