Login

r/mltraders

How I Built a Real-Time Nifty 50 Forecast Accuracy Engine — And What It Taught Me- self service tool for intraday trader
▲ 13 r/mltraders+11 crossposts

How I Built a Real-Time Nifty 50 Forecast Accuracy Engine — And What It Taught Me- self service tool for intraday trader

Most market forecasters have the same problem.

They post a forecast in the morning. The market closes. They move on.

Nobody measures. Nobody improves.

I decided to change that.

The Problem With "I Was Right"

After years of analyzing Nifty 50 intraday movements, I realized something uncomfortable.

I could look at my forecast at 3:30 PM and say "I got the direction right." But that told me almost nothing useful.

Was I right at 9:15 AM or only after 2:00 PM? Was my model 10 minutes early or 10 minutes late? Did I get the morning session right but miss the afternoon? Was Model A better than Model B today — and by how much?

These questions had no answers. Until I built something to answer them automatically.

What I Built

A real-time Nifty 50 forecast accuracy engine that runs , updates every minute during market hours, and computes 30 different metrics automatically.

It looks like a standard chart. But under the hood it is doing something most trading tools don't do — comparing forecast shape against live market data, minute by minute, all day long.

Here is what it tracks:

Correlation metrics:

  • Full day Pearson correlation
  • Last 60, 30, 15 and 5 minute rolling windows
  • Best matching 30-minute window of the day
  • Worst matching 30-minute window of the day

Direction accuracy:

  • Overall up/down direction match percentage
  • Up move accuracy separately
  • Down move accuracy separately
  • Longest correct direction streak
  • Current streak at any moment

Magnitude accuracy:

  • Average error per bar in points
  • Percentage of bars within 5, 10 and 20 points
  • Maximum error (worst single minute)

Time shift detection:

  • Is the forecast running early or late vs actual?
  • By how many minutes?
  • At what shift does correlation peak?

Session analysis:

  • Morning session match (9:15 to 12:00)
  • Afternoon session match (12:00 to 15:30)

Trend accuracy:

  • Did forecast predict the right day direction?
  • Did it catch the peak within 30 minutes?
  • Did it catch the trough within 30 minutes?
  • How close was the forecast high vs actual high?
  • How close was the forecast low vs actual low?
  • End of day accuracy

Overall:

  • Composite weighted score
  • Automatic ranking when running multiple models

The Discovery That Changed Everything

The most surprising metric was time shift.

For weeks my correlation scores looked decent — around 65 to 70 percent. I thought that was reasonable. Then I added time shift detection.

It showed my model was consistently running 10 to 15 minutes ahead of the actual market.

The forecast shape was correct. The timing was off.

Once I knew that, I could account for it. Within two weeks my full day correlation jumped from 68 percent to 81 percent — not because my model got better, but because I finally understood how it was wrong.

You cannot fix what you cannot measure.

Running Multiple Models

The second insight came from comparing models side by side.

I run three different forecast approaches each morning. Before this tool I would look at them visually and pick the one that "felt" most reasonable.

Now I have a comparison table. Every metric. Every model. Automatically ranked.

Some days Model A wins on correlation but Model B wins on direction accuracy. Some days one model nails the morning session while another gets the afternoon right.

The table shows exactly where each model is strong and where it falls apart. That is information you cannot get from looking at lines on a chart.

The chart itself has full interactions — hover tooltips, crosshair, zoom, pan, timeframe switching from 1 minute to 30 minutes, moving averages. What the Hover Shows

When you move your cursor over the chart you see:

  • Exact time label
  • Live Nifty value at that minute (change from open)
  • Each forecast model value at that minute
  • Difference between actual and forecast in points

In the analysis table every cell highlights the best performer in green. You can see at a glance which model is winning, which metric each model leads, and what the composite score is right now.

What This Is Not

This is not a trading system. It does not give buy or sell signals.

It is a measurement and improvement tool. Its job is to tell me honestly how accurate my forecast was today — in 30 different ways — so I can understand my model better and improve it over time.

The goal is not to be right every day. The goal is to understand exactly how and when and why I am wrong, so the model gets better over time.

What Is Next

will update and have real time from Monday or whatever possible at earliest

The Bigger Point

Anyone can post a forecast. Very few people measure it rigorously.

If you are serious about market forecasting — intraday or otherwise — you need a measurement system as rigorous as your forecasting system.

Otherwise you are flying blind and calling it analysis.

Build the feedback loop. Measure everything. Improve systematically.

That is how forecasting becomes a skill rather than a guess.

*I publish daily Nifty 50 intraday forecasts along with real-time accuracy tracking. Follow for updates on methodology, results and the ongoing development of this tool.*They post a forecast in the morning. The market closes. They move on.

Nobody measures. Nobody improves.

I decided to change that.

The Problem With "I Was Right"

After years of analyzing Nifty 50 intraday movements, I realized something uncomfortable.

I could look at my forecast at 3:30 PM and say "I got the direction right." But that told me almost nothing useful.

Was I right at 9:15 AM or only after 2:00 PM? Was my model 10 minutes early or 10 minutes late? Did I get the morning session right but miss the afternoon? Was Model A better than Model B today — and by how much?

These questions had no answers. Until I built something to answer them automatically.

What I Built

A real-time Nifty 50 forecast accuracy engine that runs , updates every minute during market hours, and computes 30 different metrics automatically.

It looks like a standard chart. But under the hood it is doing something most trading tools don't do — comparing forecast shape against live market data, minute by minute, all day long.

Here is what it tracks:

Correlation metrics:

  • Full day Pearson correlation
  • Last 60, 30, 15 and 5 minute rolling windows
  • Best matching 30-minute window of the day
  • Worst matching 30-minute window of the day

Direction accuracy:

  • Overall up/down direction match percentage
  • Up move accuracy separately
  • Down move accuracy separately
  • Longest correct direction streak
  • Current streak at any moment

Magnitude accuracy:

  • Average error per bar in points
  • Percentage of bars within 5, 10 and 20 points
  • Maximum error (worst single minute)

Time shift detection:

  • Is the forecast running early or late vs actual?
  • By how many minutes?
  • At what shift does correlation peak?

Session analysis:

  • Morning session match (9:15 to 12:00)
  • Afternoon session match (12:00 to 15:30)

Trend accuracy:

  • Did forecast predict the right day direction?
  • Did it catch the peak within 30 minutes?
  • Did it catch the trough within 30 minutes?
  • How close was the forecast high vs actual high?
  • How close was the forecast low vs actual low?
  • End of day accuracy

Overall:

  • Composite weighted score
  • Automatic ranking when running multiple models

The Discovery That Changed Everything

The most surprising metric was time shift.

For weeks my correlation scores looked decent — around 65 to 70 percent. I thought that was reasonable. Then I added time shift detection.

It showed my model was consistently running 10 to 15 minutes ahead of the actual market.

The forecast shape was correct. The timing was off.

Once I knew that, I could account for it. Within two weeks my full day correlation jumped from 68 percent to 81 percent — not because my model got better, but because I finally understood how it was wrong.

You cannot fix what you cannot measure.

Running Multiple Models

The second insight came from comparing models side by side.

I run three different forecast approaches each morning. Before this tool I would look at them visually and pick the one that "felt" most reasonable.

Now I have a comparison table. Every metric. Every model. Automatically ranked.

Some days Model A wins on correlation but Model B wins on direction accuracy. Some days one model nails the morning session while another gets the afternoon right.

The table shows exactly where each model is strong and where it falls apart. That is information you cannot get from looking at lines on a chart.

The chart itself has full interactions — hover tooltips, crosshair, zoom, pan, timeframe switching from 1 minute to 30 minutes, moving averages. What the Hover Shows

When you move your cursor over the chart you see:

  • Exact time label
  • Live Nifty value at that minute (change from open)
  • Each forecast model value at that minute
  • Difference between actual and forecast in points

In the analysis table every cell highlights the best performer in green. You can see at a glance which model is winning, which metric each model leads, and what the composite score is right now.

What This Is Not

This is not a trading system. It does not give buy or sell signals.

It is a measurement and improvement tool. Its job is to tell me honestly how accurate my forecast was today — in 30 different ways — so I can understand my model better and improve it over time.

The goal is not to be right every day. The goal is to understand exactly how and when and why I am wrong, so the model gets better over time.

What Is Next

will update and have real time from Monday or whatever possible at earliest

The Bigger Point

Anyone can post a forecast. Very few people measure it rigorously.

If you are serious about market forecasting — intraday or otherwise — you need a measurement system as rigorous as your forecasting system.

Otherwise you are flying blind and calling it analysis.

Build the feedback loop. Measure everything. Improve systematically.

That is how forecasting becomes a skill rather than a guess.

I publish daily Nifty 50 intraday forecasts along with real-time accuracy tracking. Follow for updates on methodology, results and the ongoing development of this tool.

u/Potential_Leek_4814 — 13 hours ago
▲ 4 r/mltraders+3 crossposts

implemented the Logarithmic Market Scoring Rule (LMSR) from scratch

Been digging into prediction markets and ended up implementing LMSR (Logarithmic Market Scoring Rule) in Python.

It’s the mechanism that turns trades into prices, and I wanted to see it working end-to-end instead of just reading the math.

Repo if anyone wants to poke it: https://github.com/mwaleedta/lmsr-pricing-engine

Open to feedback or ideas for extensions (simulation, arbitrage, multi-market setups, etc.)

u/assassin9163 — 22 hours ago
▲ 12 r/mltraders+6 crossposts

Great AI tool for retail investors

Tracking every recommendation my AI pipeline makes — here's the current win rate across sectors

Been running ProspectAI autonomously across multiple sectors.

Here's what's in positive territory right now:

UTILITIES
• D — rec. May 17 | entry $61.73 → now $68.24 (+10.55%) ✅
• CEG — rec. May 17 | entry $267.20 → now $286.94 (+7.39%) ✅

HEALTHCARE
• LLY — rec. May 1 | entry $963.33 → now $1,043.26 (+8.30%) ✅

CONSUMER DISCRETIONARY
• MAR — rec. May 13 | entry $350.23 → now $370.22 (+5.71%) ✅

SEMICONDUCTORS
• AMD — rec. May 19 | entry $420.99 → now $444.73 (+5.64%) ✅

Every entry zone and trigger price was generated autonomously by the pipeline — no manual intervention.

The pipeline runs: Reddit sentiment → Technical analysis → Fundamental analysis → Adversarial critic → Final strategy.

All recommendations tracked live 👇
https://prospect-ai.moisesprat.dev

u/Downtown_Extension_6 — 1 day ago
▲ 6 r/mltraders

suspiciously high OOS sharpe on an RL pairs strategy, tried to kill it and couldn't. roast my setup.

been building an RL agent that trades a cointegrated pair. walk forward, 14 out of sample folds, average OOS sharpe came out to 3.45. that's high enough that my default assumption is i broke something, so before i get excited i want people who've done this to tell me what i'm missing.

setup:

  • PPO agent, three actions: long the spread, short the spread, flat
  • trained on the in-sample window of each fold, scored only on the held-out window right after it
  • 14 folds, non-overlapping, roughly 8 years of daily data
  • entry and exit are the agent's call, not a fixed z-score band
  • costs modeled at 5 bps per side including slippage
  • fixed position sizing
  • around 40 trades per fold, so it isn't one lucky trade carrying the whole thing

stuff i've already checked: features only use data up to time t, no future info in the state, folds don't overlap so nothing leaks across them, and costs aren't zero. still holds up.

what i keep coming back to:

  • maybe 3.45 is just what a clean cointegrated pair gives you right until the relationship breaks, and it dies the moment the spread decoheres
  • maybe i'm overfitting the pair selection itself across folds
  • maybe the reward is quietly leaking something i haven't spotted

code's on my profile if you want to tear it apart. genuinely trying to find the flaw, not flex a number. where would you look first?

reddit.com
u/lexicalmaze — 3 days ago
▲ 0 r/mltraders

Built a Volatility Regime ML Predictor scoring 0.90+ OOS AUC across Equities & Precious Metals. Looking for methodology and blindspots

Hey everyone,

I've been researching in my free time and come across some research paper and start developing supervised machine learning pipeline designed to predict Vol-Regime (Expansion vs. Compression) rather than directional price. The objective is to utilize this as a master regime filter: when the model predicts high probability of variance expansion, it would utilize is trend model logic and reduces sizing; during variance compression, it toggles to mean-reversion logic and scales up.

I’ve been extremely paranoid about lookahead bias and data leakage. I wanted to present my exact methodology here and ask if the community sees any glaring blindspots or hidden leakage I might have missed

The Setup:

  • Data: 10 years of Equities (~62k obs) and 17 years of Gold (~99k obs, to test cross-asset generalization with zero hyperparameter changes).
  • Features: High-frequency volatility estimators (Garman-Klass, Rogers-Satchell), structural variance (RiskMetrics, HAR), and microstructure flow (volume acceleration, VWAP dev).
  • Target (Label): 4-period forward Garman-Klass realized variance. I use a strict multi-period embargo before the forward window starts to sever the t vs t+1 boundary overlap artifact.
  • Validation: Custom 3-Zone Purged Walk-Forward Splitter (Train -> 40-bar Embargo -> Validation -> 40-bar Embargo -> OOS Test). 312 sequential folds tested.

The result:

  • Equities (15-month unseen holdout): LightGBM AUC: 0.9526 (Linear LR Baseline: 0.9501)
  • Gold (24-month unseen holdout): LightGBM AUC: 0.9081 (Linear LR Baseline: 0.8894)

Question:

  1. An OOS AUC of 0.95 on equities usually screams leakage, but since I am predicting variance (which clusters mathematically) rather than price direction, is an AUC in the 0.90s actually realistic?
  2. Any specific statistical stress-tests you'd recommend before taking this live?

https://preview.redd.it/fv4vre7cwrah1.png?width=2977&format=png&auto=webp&s=ae4e3688f50193a3f711d5fe3ea86965cb031c8f

Gold Final Holdout: 0.90+ AUC across 17 years of OOS
reddit.com
u/International_Net716 — 4 days ago
▲ 10 r/mltraders+3 crossposts

I used Moomoo AI + my own Codex stock bot to analyse CIMB for a month

https://preview.redd.it/w4cxszllejah1.jpg?width=589&format=pjpg&auto=webp&s=099b02463ff5cd99c19b8f1d69c0c59c5a0dac63

https://preview.redd.it/mwym00mlejah1.jpg?width=589&format=pjpg&auto=webp&s=6245b1f0dcba9e12cc4ed5c95c8dc861a50dbd31

https://preview.redd.it/ogyngzllejah1.jpg?width=589&format=pjpg&auto=webp&s=6d959079421321269fdd3261b4791013701c2ff3

https://preview.redd.it/iwz5uxllejah1.jpg?width=589&format=pjpg&auto=webp&s=b7666ceff32d5df33e6d968b31852d1dc0e4488d

I used to think that if you understand forex, you should be able to understand stocks too.

After trying it for a while, I think that is partly true — but only partly.

Forex trained me to look at price action, risk management, momentum, and entry timing. But with stocks, I realised I need one extra layer: understanding the actual company behind the chart.

For example, if I want to trade or invest in CIMB, I don’t just want to know whether the chart looks bullish or bearish. I also want to know:

- Is the company financially healthy?
- Is the valuation reasonable?
- What are analysts expecting?
- What are the risks?
- Is the current price still a good entry, or did I already miss it?

(You can refer picture 2-3)

The problem is that doing all this manually through Google, annual reports, news, and analyst summaries takes time. By the time I finish reading, sometimes the entry point is already gone.

Recently I tried using Moomoo AI to speed up this research process. I used CIMB as an example. The AI gave me a company summary, valuation data, dividend yield, analyst target, and even a daily technical brief on the stock.

What I liked is that it didn’t just show a chart. It helped connect the chart with company-level information.

At the same time, I have also been testing my own AI bot built with Codex. I created a personal stock analysis skill and used CIMB as one of the test cases for about a month.

For this short test, my securities P/L ratio was +1.10%.

Obviously, this is not a serious long-term result yet. One month is too short, and +1.10% does not prove that the system works. But it did make me think that AI can be useful as a research assistant, especially for filtering information faster.

My current view:

AI should not decide trades for us.
But AI can help us research faster, compare data faster, and avoid entering blindly.

For me, the best workflow is:

  1. Use AI to summarise the company
  2. Check valuation and financial health
  3. Review technical signals
  4. Compare with my own trading plan
  5. Only enter if the risk/reward still makes sense

Curious to hear from others here:
Do you use AI tools for Malaysian stock research?
And for those who moved from forex to stocks, what was the biggest adjustment for you?

Not financial advice. Just sharing my own experiment and learning process.

#moomoo #cimb u/moomoo_official $CIMB

reddit.com
u/ImpressionCultural36 — 5 days ago
▲ 24 r/mltraders+1 crossposts

For Anyone Looking for Financial Data APIs

While working on investing, analytics, and data-driven projects, I’ve spent time evaluating different financial APIs to understand their strengths, limitations, and practical use cases. I put together this short list to save others some time if they’re researching data sources for trading tools, dashboards, backtesting, or general market analysis. It’s a straightforward overview meant to be useful, not promotional.

Financial APIs worth checking out:

Mboum API – Time series data and technical indicators
- Price: Free tier available, premium plans start around $9.95/month
- Free tier: Yes

EODHD API – Historical market data and fundamentals
- Price: Free tier (20 requests/day), paid plans start around $17.99/month
- Free tier: Yes

Alpha Vantage – Time series data and technical indicators
- Price: Free tier available, premium plans start around $29.99/month
- Free tier: Yes

SteadyAPI – Time series data and technical indicators
- Price: Free tier available, premium plans start around $14.95/month
- Free tier: Yes

Yahoo Finance (via yfinance) – Lightweight data access for Python projects
- Price: Free (unofficial API)
- Free tier: Yes

Polygon.io – Real-time and historical US market data
- Price: Free tier available, paid plans start around $29/month
- Free tier: Yes

Alpaca Markets – Trading API with market data and paper trading
- Price: Free for data and trading API access
- Free tier: Yes

Finnhub – Market news, sentiment, fundamentals, and crypto data
- Price: Free tier available, paid plans start around $50/month
- Free tier: Yes

u/Real_Grapefruit_5570 — 8 days ago
▲ 9 r/mltraders+8 crossposts

Data ingestion and avoiding lookahead bias is a massive headache, so I built an open-source CLI agent to automate my backtesting setup.

It takes a plain-English strategy idea, generates validated Python using your own LLM key, and runs a historical backtest.

I just added Binance support today.

My biggest challenge right now is the automated safety checks—it currently scans the AST for lookahead flaws before executing.

The tool is free and open source locally at finnyai.tech, with an optional $10/mo tier for managed hosting.

If anyone here builds automated validation for strategy code, how do you handle edge cases and LLM data hallucinations?

u/Awkward_Weather5721 — 9 days ago
▲ 28 r/mltraders+1 crossposts

I succeeded at creating a profitable Trading Algorithm.

This took months, a 24/7 fan ontop of my computer, and scolding by my parents. But i think i actually succeeded. The foward test results have been extremely well over the last 2 weeks of deployment so far.

u/skypick112 — 13 days ago
▲ 3 r/mltraders+1 crossposts

Tool/Platform Recommendation for someone with Python/Tensorflow background

Hi, I'm coming from a Python/Numpy/Tensorflow ML background and looking to trade crypto. Is Freqtrade the right tool for me? I understand it is mainly using Pytorch, how easy is it to use Tensorflow instead or should I go ahead and move to Pytorch?

reddit.com
u/bolodski — 12 days ago
▲ 4 r/mltraders+2 crossposts

Results of the first 12 months of my trading AI

It's been 12 months since I tested my trading AI in real conditions. It makes 10 predictions a day based on current events, and the results are rather impressive for a start:

-61% accuracy

-151/230 positive days

-Average P&L of +0.74% per day

-The biggest loss is -3.11%

-Total cumulative return of +424% without leverage

My AI gives predictions every day about the open/closed variation of 10 actions (no matter which ones but at sp100).

I haven't tried to add trading algo, we're just open/close, evenly distributed capital.

I think I'm holding something, the system is holding the road in real conditions on IBKR

This is currently a personal project. I've been a computer science student and AI enthusiast for over 10 years. My goal is to become a quantitative trading researcher for an investment fund, or to create one.

I launched a website for my AI, where predictions are displayed daily, with the ability to receive them via email or REST API. If you're interested, send me a private message

Don't hesitate to give your opinion!

reddit.com
u/orakle12 — 10 days ago
▲ 3 r/mltraders+2 crossposts

I built a Directional Arbitrage Bot for Polymarket crypto markets in Rust — here's the architecture

Hey everyone,

Sharing a new bot I've been building — it's a directional arbitrage bot for binary prediction markets (BTC/ETH/SOL Up/Down on Polymarket). Written in Rust for the latency requirements this strategy demands.

The core idea

Pure arbitrage on prediction markets is straightforward — if Up + Down < $1, you buy both and lock in the difference. Safe, but the upside is capped at the spread.

This bot starts from that arbitrage structure but adds a directional tilt. If the model identifies that one side has additional edge beyond the arb, it skews the position — buying more of the stronger side and less of the hedge side. The result is:

  • Arbitrage base = structural protection
  • Directional tilt = additional EV when the model has conviction

Why this matters in short crypto markets

BTC/ETH/SOL 5-15 minute markets on Polymarket are interesting because the underlying asset can move sharply while Polymarket's CLOB reprices one side with a delay. That lag is the exploitable window. By the time the order book catches up, a pure arb bot has already exited. A directional bot can hold the stronger side through the reprice and capture the full move.

Architecture (Rust)

// Core position structure
struct Position {
    arb_base: f64,      // guaranteed arb fill on both sides
    directional_tilt: f64,  // extra exposure on favored side
    hedge_ratio: f64,   // partial hedge on weaker side
    edge_threshold: f64, // minimum model edge to tilt
}

Key design decisions:

  • Limit orders only — never market orders. Paying the spread kills the arb base immediately
  • Tilt gated by edge threshold — the model has to show meaningful conviction before skewing the position. No tilt = pure arb fallback
  • Hedge ratio is dynamic — scales down as directional confidence increases, never goes to zero
  • Rust for execution — async order management with Tokio, minimal GC pauses, deterministic latency on the quoting loop

Position logic flow

1. Scan CLOB for Up + Down &lt; $1 (arb opportunity exists)
2. Run edge model on both sides
3. If edge delta &gt; threshold → tilt toward stronger side
4. Place limit orders: full size on strong side, hedge_ratio on weak side
5. Monitor fill state and adjust tilt if market moves before fill

EV breakdown

Scenario Pure Arb Directional Arb
Model correct Spread only Spread + directional gain
Model wrong Spread only Spread - tilt loss (partial hedge limits damage)
No fill Zero Zero

The hedge means a wrong directional call doesn't blow up the position — it just reduces the arb profit. The floor is always the arb spread minus fees.

Current limitations

  • Edge model is still rule-based — pattern recognition on price movement sequences. Not ML yet
  • Liquidity on some markets is thin enough that the tilt size is constrained by available depth
  • Correlated markets (BTC and ETH moving together) need net exposure tracking across both, not per-market limits

Why Rust

Polymarket's CLOB API has enough latency variance that the quoting loop needs to be tight. Rust gives deterministic async performance without the GC pauses you'd get in Node or JVM. The order state machine is also complex enough that Rust's ownership model catches a lot of bugs at compile time that would be runtime errors elsewhere.

GitHub: https://github.com/HarrierOnChain/Prediction-Markets-Trading-Bot-Toolkits

Happy to discuss the edge model, position sizing logic, or the Rust async architecture. Also curious if anyone has solved the correlated market exposure problem cleanly.

reddit.com
u/guischulhick — 13 days ago
▲ 2 r/mltraders+1 crossposts

I tried processing 10 Million HFT ticks/day on a Free 1GB RAM Cloud Server. It crashed immediately. Here’s how I rebuilt it to use just 125MB of RAM.

Hey everyone,

I’ve been working on a personal project to build a zero-lag High-Frequency Trading (HFT) ingestion pipeline. I wanted to see if I could handle institutional-level data velocity on the absolute cheapest hardware possible: an "Always Free" 1GB RAM micro-instance.

The Failure 💥 Initially, I spun up a Go websocket client + QuestDB. The moment the market opened (peaking at 1,000+ ticks/sec), QuestDB chewed through all 1GB of RAM, and the server locked into an Out-Of-Memory (OOM) kernel panic death loop. I couldn't even SSH in.

The Re-Architecture 🛠️ I had to drop heavy time-series databases and optimize every single layer. Here is what I built:

  1. Go Ingestion: Bypassed JSON parsing entirely. Wrote a Go collector to unpack Level-5 Binary byte-streams directly from the exchange.
  2. The Shock Absorber: Dropped Kafka. Used NATS for Pub/Sub. It handles the firehose backpressure perfectly and idles at just 20MB RAM.
  3. Stream Processing (Python): Instead of querying a DB, I wrote an in-memory state machine. It computes 14 ML features live (like MTF candle sizes and Anti-Spoofing Orderbook Imbalances) without any disk I/O.
  4. Direct Parquet Writing: Built a micro-batcher using PyArrow. Once 5,000 ticks buffer in RAM, it flushes directly to a compressed .parquet file.

The Result 🚀 The entire pipeline runs without dropping a single tick, compressing 10 Million+ daily data points into ML-ready datasets, all while maxing out at 125MB RAM. I also pipe the NATS stream directly to a React frontend via WebSockets for a live, zero-database UI.

I wrote a detailed technical deep-dive covering the whole architecture on Medium if anyone is dealing with similar compute constraints: 🔗 [hft]

You can also see the live dashboard streaming here (Best viewed during Indian Market Hours 9:15 AM - 3:30 PM IST): 🔗 http://92.4.70.154:8080/god_mode.html

Would love to hear if anyone has tackled similar memory constraints or has suggestions to improve this architecture!

https://preview.redd.it/4m6gdd8cd09h1.png?width=2846&format=png&auto=webp&s=f1b50ffc00ca70ba63b44d29a0c0fc6c00d5562f

u/Interview-Cracker-AI — 13 days ago