If changing the broker kills the strategy, was there ever really an edge?
I’ve been thinking about adding execution portability to the robustness checks I use before trusting an algo, but I’m not convinced it’s actually a fair requirement.
Basic problem is pretty simple
I can keep the signal logic, sizing and exits completely unchanged and still get a very different system once I change the assumptions around execution.
So instead of only stress testing parameters and market regimes, I’m considering testing the execution layer separately.
Something like this
Baseline
- normal spread assumption
- normal commission model
- current slippage assumption
- normal fill probability
| Stress test A | spread 25% worse | |||
|---|---|---|---|---|
| Stress test B | spread 50% worse | |||
| Stress test C | normal spread | 1 tick adverse slippage on entry and exit | ||
| Stress test D | spread 25% worse | some adverse slippage | small fill delay | a percentage of limit orders left unfilled |
I’d then compare more than just final P/L
The metrics I’m thinking about are
- expectancy per trade
- profit factor
- Sharpe
- max drawdown
- fill rate
- percentage of gross edge lost to execution
- trade count after missed fills
Numbers above are just hypothetical stress parameters. I’m more interested in the framework than those exact thresholds.
Here’s where I’m getting stuck.
Say a strategy has a PF around 1.30 under the baseline execution model. A modest increase in spread brings it close to 1.10. Add a little adverse slippage and it drops below breakeven.
There are two completely different ways I can read that.
The first is that the strategy was fragile from the start. If a small change in trading costs wipes out the edge, I probably shouldn’t trust the backtest much. Second is that this is asking the strategy to survive something it was never designed for. A short-horizon system can have a real edge that only exists with a specific fee structure, spread, order type or execution venue. In that case the execution environment isn’t noise around the strategy. It’s part of the strategy.
That second case is what makes broker portability questionable to me as a robustness test. I wouldn’t expect a market-making system to keep the same economics after changing the fee structure. I also wouldn’t expect a tight scalper to behave the same after doubling its spread assumptions. At some point I’m no longer stress testing the same business logic. I’m changing the conditions that created the edge in the first place.
But giving the execution model zero robustness testing seems just as bad.
Backtest that only works at one exact spread and one exact slippage assumption feels way too brittle for live trading.
So I’m leaning toward measuring an execution degradation curve instead of using a simple pass or fail rule.
Basically I want to know how quickly the edge decays as execution gets worse.
A gradual decay seems a lot healthier to me than a strategy that falls off a cliff after one extra tick.
What I don’t know is where that becomes unacceptable.
For people running systems live, how do you handle this?
Do you expect an algo to remain profitable across moderately worse execution assumptions, or are you fine with an edge being tightly tied to one broker, venue or fee structure as long as those conditions are stable?
And if you stress test execution, what do you actually perturb: spread, commissions, latency, fill probability, slippage distributions, or all of them together?