u/Interview-Cracker-AI

Hey everyone,

I’ve been working on a personal project to build a zero-lag High-Frequency Trading (HFT) ingestion pipeline. I wanted to see if I could handle institutional-level data velocity on the absolute cheapest hardware possible: an "Always Free" 1GB RAM micro-instance.

The Failure 💥 Initially, I spun up a Go websocket client + QuestDB. The moment the market opened (peaking at 1,000+ ticks/sec), QuestDB chewed through all 1GB of RAM, and the server locked into an Out-Of-Memory (OOM) kernel panic death loop. I couldn't even SSH in.

The Re-Architecture 🛠️ I had to drop heavy time-series databases and optimize every single layer. Here is what I built:

1. Go Ingestion: Bypassed JSON parsing entirely. Wrote a Go collector to unpack Level-5 Binary byte-streams directly from the exchange.
2. The Shock Absorber: Dropped Kafka. Used NATS for Pub/Sub. It handles the firehose backpressure perfectly and idles at just 20MB RAM.
3. Stream Processing (Python): Instead of querying a DB, I wrote an in-memory state machine. It computes 14 ML features live (like MTF candle sizes and Anti-Spoofing Orderbook Imbalances) without any disk I/O.
4. Direct Parquet Writing: Built a micro-batcher using PyArrow. Once 5,000 ticks buffer in RAM, it flushes directly to a compressed .parquet file.

The Result 🚀 The entire pipeline runs without dropping a single tick, compressing 10 Million+ daily data points into ML-ready datasets, all while maxing out at 125MB RAM. I also pipe the NATS stream directly to a React frontend via WebSockets for a live, zero-database UI.

I wrote a detailed technical deep-dive covering the whole architecture on Medium if anyone is dealing with similar compute constraints: 🔗 [hft]

You can also see the live dashboard streaming here (Best viewed during Indian Market Hours 9:15 AM - 3:30 PM IST): 🔗 http://92.4.70.154:8080/god_mode.html

Would love to hear if anyone has tackled similar memory constraints or has suggestions to improve this architecture!

https://preview.redd.it/4m6gdd8cd09h1.png?width=2846&format=png&auto=webp&s=f1b50ffc00ca70ba63b44d29a0c0fc6c00d5562f

Hey everyone,

I’ve been working on a personal project to build a zero-lag High-Frequency Trading (HFT) ingestion pipeline. I wanted to see if I could handle institutional-level data velocity on the absolute cheapest hardware possible: an "Always Free" 1GB RAM micro-instance.

The Failure 💥 Initially, I spun up a Go websocket client + QuestDB. The moment the market opened (peaking at 1,000+ ticks/sec), QuestDB chewed through all 1GB of RAM, and the server locked into an Out-Of-Memory (OOM) kernel panic death loop. I couldn't even SSH in.

The Re-Architecture 🛠️ I had to drop heavy time-series databases and optimize every single layer. Here is what I built:

1. Go Ingestion: Bypassed JSON parsing entirely. Wrote a Go collector to unpack Level-5 Binary byte-streams directly from the exchange.
2. The Shock Absorber: Dropped Kafka. Used NATS for Pub/Sub. It handles the firehose backpressure perfectly and idles at just 20MB RAM.
3. Stream Processing (Python): Instead of querying a DB, I wrote an in-memory state machine. It computes 14 ML features live (like MTF candle sizes and Anti-Spoofing Orderbook Imbalances) without any disk I/O.
4. Direct Parquet Writing: Built a micro-batcher using PyArrow. Once 5,000 ticks buffer in RAM, it flushes directly to a compressed .parquet file.

The Result 🚀 The entire pipeline runs without dropping a single tick, compressing 10 Million+ daily data points into ML-ready datasets, all while maxing out at 125MB RAM. I also pipe the NATS stream directly to a React frontend via WebSockets for a live, zero-database UI.

I wrote a detailed technical deep-dive covering the whole architecture on Medium if anyone is dealing with similar compute constraints: 🔗 [hft]

You can also see the live dashboard streaming here (Best viewed during Indian Market Hours 9:15 AM - 3:30 PM IST): 🔗 http://92.4.70.154:8080/god_mode.html

Would love to hear if anyone has tackled similar memory constraints or has suggestions to improve this architecture!

https://preview.redd.it/4m6gdd8cd09h1.png?width=2846&format=png&auto=webp&s=f1b50ffc00ca70ba63b44d29a0c0fc6c00d5562f