u/Dear-Doughnut-1013

want to test my order handling, partial fills, rejections, all the stuff that breaks, before i risk capital. but every broker sandbox ive tried just returns success for everything. useless.

does anyone have a sandbox or uat that actually behaves like the real thing. tired of finding bugs in production.

genuinely asking. my options strategy backtests great but live is always worse and i cant figure out why. assumed my strategy was decaying but ive tweaked it a dozen times and the gap stays.

is this just normal or am i missing something obvious. what actually causes the backtest to live gap for options.

6 months running langsmith for our langchain agent (langchain 0.3.x, langgraph 0.2.x, claude sonnet 4.5). deepeval for the offline eval side. the combo works for what it works for, but there's a structural eval gap i haven't figured out how to close.

the gap: langsmith dataset eval runs against a labeled set of ~600 examples. pass rate is 87% which sounds fine. prod failure rate on actual user traffic is ~14% (measured by user thumbs-down + manual review of a stratified sample). so our offline eval is failing to predict our online failures by a meaningful margin.

audited 100 failing prod traces last sprint. ~40% of them were on input patterns that don't exist in our offline eval set at all. we built the eval set 4 months ago from then-current user behavior. user intent distribution shifted (new product surfaces opened up, different tool combinations got more popular, phrasing patterns evolved) and our eval set didn't shift with it.

what i actually want: continuous eval that pulls samples from prod traces, runs them through the rubrics, auto-promotes the failing ones to the eval set, and surfaces emerging failure clusters. confident AI does this and looks promising. langsmith dataset has the right primitives (you can add prod traces to datasets via the API) but the workflow is manual and the eval-against-production-trace pattern is clunky at our volume (~12k traces/day).

specifically trying to figure out:

1. tools that actually do continuous eval on prod traces well at >10k/day volume. langsmith manual workflow is too painful. confident AI looks built for this. anyone else?

2. PII/redaction story. prod traces contain user data we can't ship to a third party without redaction. langsmith handles this with their enterprise on-prem option but downstream eval tools mostly assume cleartext.

3. auto-promotion strategy. my current thinking: use a judge to grade the trace, auto-add to eval set if (a) judge says failed and (b) trace is significantly different from existing eval cases (embedding distance > threshold). human review pass weekly to catch judge errors.

anyone running this at non-toy volume? particularly interested in how you handle the eval-set-drift problem (auto-promotion biased toward judge's blind spots) and the input distribution shift detection (do you measure it explicitly or react to it after the fact?).

badly need extra money for financial expenses. also, please recommend any legit side hustles. feel free to message me.

Any filipinos here?