I’m trying to build a system to evaluate alpha-generating hypotheses, and I’d appreciate some guidance on how to do this rigorously.
The setup is: I receive a detailed JSON file containing
- a hypothesis
- an expected chain of reactions driving the thesis
- affected tickers with expected directional moves
- a time horizon for the hypothesis
- supporting evidence
The challenge is figuring out how to evaluate and filter these hypotheses, especially since they’re generated by an LLM and likely include a lot of noise and false positives.
So far, I’ve been considering a few approaches:
- Monte Carlo simulations on individual tickers
- Regime-based factor regression to test how similar conditions performed historically
I also thought about backtesting, but I’m struggling with how to apply it properly. Many hypotheses are based on new information or events that haven’t occurred before, so there’s no clear historical analog. That makes it unclear how to backtest scenarios driven by novel news or forward-looking narratives.
Overall, I’m unsure which techniques are actually appropriate here and which ones might just introduce noise or false confidence.
How would you approach building a robust evaluation pipeline for this kind of problem? Any frameworks, methods, or pitfalls to be aware of would be really helpful.