u/Finance__broski

some of you might've seen my build-to-kill post from a few days back. in the 4 days since, the system got "finished". multi sleeve, everything wired, and the final backtest spat out: ₹25L becomes ~2.7Cr median over 15 years, worst drawdown −31%. i'm not gonna lie, i stared at that number for a while and started mentally spending money.

problem is it's family capital. my dad's money, one shot, no refill. so before touching anything i decided to spend a day being the meanest reviewer my system could face. not "double checking". actively trying to prove my own work is garbage. wrote down everything that died, posting it here because honestly some of these hurt and i'd rather you find them in your systems before your first bad week does.

1. my hedge could not physically exist. the tail protection was shorting a basket of ~41 stock futures, and at my capital that whole basket got about ₹4-5L of notional. one stock futures lot is ₹5-15L. so the backtest had been "holding" a hedge for years that my account literally cannot hold even one leg of. every drawdown number i was proud of assumed protection that doesn't exist at my size. i genuinely sat there for a minute after this one.

2. found a 2.4x sizing mismatch between my model and my live code. the projections sized the options sleeve as a fraction of capital. the execution code sized it as a margin budget. two different conventions, both looked reasonable, nobody ever compared them. reconciling constant by constant showed live would've run 2.4x the short-vol risk that every single tail number assumed. no error message. every report looked normal. this would've only announced itself in a crash, at 2.4x the size i "planned" for.

3. one of my sleeves is literally illegal. currency futures leg, decent sharpe, backtested clean. turns out RBI barred retail from exchange traded currency derivatives without underlying exposure back in 2024. no data feed tells you this. your parquet files don't care what the rulebook says. felt like an idiot googling it after weeks of "research".

4. look-ahead was hiding in my liquidity filter of all places. i filtered tradeable stocks by fill rate computed over the full history, which means the filter knew in 2015 which names would still be liquid in 2024. that alone was +1.6pp/yr of completely fake alpha. everyone checks their signals for look-ahead. nobody checks the boring plumbing. it's always in the plumbing.

5. my crash kill-switch was validated on a proxy that flattered it 2x. tested on an index proxy: cuts max dd from −25%, looked great. rebuilt on the actual portfolio daily: real drawdown is −44% and the kill-switch cuts way less of it, at double the cost. if you validated something on a proxy you validated nothing.

6. and the part that actually humbled me, the errors ran the other way too. my slippage haircut, which i thought was the responsible conservative choice, turned out ~15x too harsh once i actually measured effective spreads from 23M rows of 1-min options data instead of assuming. and my tax model charged flat 30% on f&o... at my actual income slab the real rate is basically 0 (87A rebate, if you trade f&o small and don't know this, go look it up today). so i wasn't being conservative. i just didn't know, in both directions.

where it landed: honest numbers are roughly half the wealth and nearly double the drawdown of the pretty version. then measurement gave some back legitimately. it now runs at lower leverage than the optimizer wanted, on paper only, and it has to earn its way to real money.

the thing i actually kept from all this is a stupid boring document where every input is tagged: measured / estimated / assumed / no-way-to-know, with how much the answer moves if it's wrong. before the audit i had ~40 inputs and could honestly account for maybe 15. the scary column isn't what's wrong. it's what's unmeasured.

running count since i started: ~50 ideas tested, ~45 dead. the survivors are boring and staying that way.

happy to go deep on any of these in comments. can also paste the audit checklist i now run on everything if anyone wants it, fair warning it's deeply unpleasant to use on your own work. pretty sure that's the point.

quick one, since sharpe is the number everyone quotes and almost nobody computes carefully. these are the ways i've watched it get inflated, including in my own stuff before i caught it.

1. computed on in-market periods only. if your strategy sits in cash half the time and you only compute sharpe over the days it's actually in a trade, you've thrown out all the flat/zero-vol periods and the number looks way better than reality. quick tell: someone shows a 4+ sharpe but a -15% drawdown and the two don't reconcile — the sharpe's almost certainly on active periods only while the drawdown's from the full curve. compute it on the whole equity curve or it's not real.

2. too few trades. a sharpe of 3 on 60 trades means basically nothing, the error bars are enormous — you can't tell it apart from a sharpe of 1. it needs a lot of independent bets before it's trustworthy. low trade count + high sharpe = luck until proven otherwise.

3. daily marking (options/derivatives especially). mark a book daily with smooth marks (stale quotes, wide spreads, illiquid strikes) and your measured vol comes out artificially low, which pumps the sharpe. marking on real holding periods usually knocks it down. this exact thing inflated my own options book before i switched to honest marking.

4. naive annualization. annualizing by sqrt(252) assumes your returns are independent. trend/momentum strategies, or anything with stale/serially-correlated marks, break that — and the naive annualization overstates it. if your returns autocorrelate, your annual sharpe is too high.

none of it's exotic, it's just that sharpe is one number hiding a lot of assumptions, and the honest version is almost always lower than the one in the pitch deck. full curve, net of costs, enough trades to mean something, marked honestly — then believe it.
curious what other ways people have seen it gamed, feels like there's a bunch.

ok so this is gonna be long, fair warning.

i'm 21, did a BBA, not a quant by any stretch. for the last few months i've been building a systematic book for indian markets, equity momentum + an options vol-premium sleeve + a couple of overlays. and the thing i need to say upfront before anyone gets excited: i have not traded a single rupee of it live. it's 100% backtest. so please read this as "stuff i learned trying" and not "stuff that made money." honestly i'd rather get torn apart in the comments than by the market later.

anyway. the stuff i actually learned, in case it saves someone else the months.

i went in thinking it was about finding clever strategies. it's not. it's about data.

like 80% of my time went into just making the data not lie to art because it's boring but it's the whole game. couple ofexamples that humbled me early:

- my first universe quietly dropped delisted companies. so obviously every backtest looked amazing, i was only ever testing the survivors lol. had to rebuild the whole thing keeping dead names in.

- point in time. you have to stamp every fundamental with the date it was actually announced, not the quarter it belongs to. sounds obvious, very easy to get wrong (more on that later, it bit me hard).

- renames. stuff like MUNDRAPORT becoming ADANIPORTS. if you just glue the price series together you get fake gaps. took me a while to even notice.

if you're starting out my honest advice is assume your data is wrong and your job is to prove it. that alone is most of the work.

the mindset that helped most: try to KILL your idea, not prove it.

i kept a chunk of out-of-sample data locked away and didn't let myself look at it till the very end. it caught me twice on things that looked beautiful

in-sample and fell apart the second they hit fresh data.

here's the graveyard, which is honestly the most useful part of this post:

- breakouts / 52w highs — basically momentum wearing a hat, and worse on risk

- short term reversal — dead for me in indian equities

- low vol as its own alpha — only useful as a tilt inside something bigger, useless alone

- dispersion, multi-asset trend on indices/fx — nothing

- MCX commodities positional — weak, and it didn't even help in crashes which was the only reason i wanted it

- OI buildup, raw IV change, rollover %, promoter stake changes, weekly momentum — all noise once i tested them properly

like 90% of what i tried died. and i've come to think that's correct? the default result of an honest test should be death. if everything you test works, your test is broken.

what survived is small and even the "real" stuff is annoying.

a couple of things held up out of sample, i'm gonna stay vague on the specifics (keeping a bit of powder dry, no offence). but even the survivors didn't let me feel smart. the options sleeve for example has a tail risk that you genuinely cannot hedge away without also killing the return. i tried all the "free crash protection" ideas. there's no such thing. futures bleed you dry in calm times. in the end sizing was the onlyclean lever i had on drawdown, which is a boring answer but it's the true one.

also the numbers lied to me, in my favour obviously.

stuff that quietly inflated my results until i caught it:

- marking the option book daily makes the sharpe look way better than it has any right to

- slippage on small/midcaps is real money and i was ignoring it

- TAX. high turnover means 20% STCG and i never see anyone include this. it can eat the entire edge. include it or your backtest is fiction.

after fixing survivorship + costs + tax my "great" CAGR came way down. still ok, but a very different conversation than the screenshot i almost got excited about.

the part that actually stung: i went back recently and audited everything as harshly as i could, and i found a look-ahead bug in my own code. one date field was saved as text instead of an actual date, and somewhere deep in a function it got mixed with real dates, the sorting silently broke, and the whole thing ended up showing each month the company's future financials.

an "edge" i'd credited to a fundamentals filter, like +1.6% a year, turned out to be entirely that bug. fixed the date parsing, re-ran it clean, and the edge just... wasn't there. gone.

what got me wasn't that i made a coding mistake, it's that it was invisible. it wasn't in the logic, it was in the plumbing, and it looked completely

fine. you really do fool yourself by default. (and yeah, ngl, his stuff fast, and it'll hand you a clean looking functionwith a look-ahead bug baked in and zero warning. the code is easy now. the skepticism is the actual job.)

so where i've landed:

- clean data > clever strategy, every single time

- assume you're wrong, keep a holdout you don't touch

- if you don't model costs and tax you're lying to yourself

- hedging isn't free, size your risk instead

- the market is mostly efficient and the few real edges are small and a pain to actually capture

- and since i've made exactly ₹0 from any of this, staying humble isn't a choice lol

that's where it is. a book i believe in on paper and haven't risked a rupee on. if you've actually taken something like this live here i'd really love toknow what broke in the gap between backtest and reality, that's the part i'm most scared of. and if you see a hole in any of the above pls be brutal, that's literally why i'm posting.

When a backtest comes back good, the default is to believe it. That's the one that bites you in

forward testing. Two better moves:

1. Find where it breaks. A plain earnings screen I ran backtested at 31.6%/yr — looked like edge. ~4 pp/yr of it was data artifacts: the universe silently dropped delisted names (survivorship — never held the losers), and it ranked on earnings keyed to quarter-end instead of the ~42-days-laterannouncement date (look-ahead). Honest data, same strategy: 27.4%. Still good, but 4 points lighter, and now I knew which 4 and why.

2. Find what made it great. Why is HDFCBANK the #1 weight in the NIFTY 50 when it's not the biggest by market cap? ~100% public float, no promoter. TCS is bigger by mcap but sits lower — Tata holds ~72%, so most isn't tradable float. If your attribution doesn't decompose by float, you're crediting the wrong driver.

The principle: greatness you can't attribute is luck until proven otherwise. A result you can't explain — good or bad — you can't trust or repeat.

So when it looks great, get suspicious in the useful way: break it, or explain it. Believing it is the only wrong answer.

What's the best "too-good result that was actually a data artifact" you've caught?

I built a survivorship-free, point-in-time NSE/BSE dataset (delisted names kept; fundamentals stamped with their real announcement date) and ran a single earnings-yield sort four ways. Same stocks, same strategy — only the data honesty changes:

- Naive (survivor-only + earnings assumed known at quarter-end): 31.6%/yr

- Honest (delisted names kept + earnings known only on announce date): 27.4%/yr

- Gap: +4.1 pp/yr of pure data bias.

~+4.0 pp/yr of that is look-ahead — using earnings the day the quarter ended instead of when they were announced (median 42 days later). If your screener aligns fundamentals to period-end, you're trading on numbers that weren't public yet.

Survivorship looks tiny inside a value screen (it rarely holds the value-traps that die), but on a broad equal-weight basket it's real: +2.5 pp/yr, from holding 378 names that delisted on their actual dates (DHFL, Jet, RCOM…) instead of dropping them.

Vanilla on purpose — the point is the data, not the alpha. 2019–24 for the fundamentals run; 25 bps/side; top-500 liquidity universe (not index constituents).

Check your own: (1) does your universe hold any name that delisted in your window? no → survivorship bias. (2) fundamentals keyed to period-end or announce date? period-end → look-ahead.

Free real-data sample + the runnable notebook — links in the comments. What's everyone using for clean point-in-time Indian data?

Free sample + the runnable notebook: https://github.com/Finance-broski/pit-data-sample