Websocket died silently and my bot kept trading a frozen book for hours. How do you detect stale feeds?
A little bit contex I`m running a copy/automation setup across a few venues. Signals come in bot manages positions from there.
One night a position just sits wrong for hours and I cant work out why the bot isnt doing anything about it. Went through the strategy logic, the signal side, all looked normal. Wasnt until I checked the raw feed timestamps that it clicked, the market data websocket on one of the venues had stopped. No error and the connection still showed open on the client, there was just nothing coming down it.
So as far as the bot knew everything was live and it sat there managing a book that had been frozen for hours while the actual market moved on.
That venue runs localtrade JSON-RPC over a websocket and the book updates carry a sequence number. Which is handy for a dropped message the number jumps and you know you missed one.
Except that does nothing for this case. A dead feed doesnt skip a number, it just stops sending, so theres no gap for the check to catch. What actually catches it is a timeout, the sequence number is useless when nothings arriving.
Copytrade made it worse.
A mirror keeps sizing off the master account and once that state quietly goes stale the mirror is acting on prices that stopped updating and slowly drifts from where it should be. Nothing errors, you just notice later that the two accounts dont line up.
Fix was to stop trusting the socket. Stale data is a hard fault now if the feed hasnt ticked inside a timeout the bot treats it as lying and flattens rather than guessing. Socket being open doesnt mean the feed is alive anymore, the data has to be recent or it bails.
Right now I`m leaning towards a separate watchdog that just tracks the last update timestamp per venue, outside the trading loop, and trips if anything goes stale past a threshold.
Feels a bit crude though. Is there something cleaner people actually run for this in prod, a heartbeat trick, some smarter liveness check? Mostly interested in multi venue setups where one leg can stop updating while the rest look totally fine.