Despite being cautious about LLMs, we still fucked up
I am one of the few peoples in software engineering who already were aware of the tech industry enshittification before AI trend even began. I worked my first job at a startup and the signs it beings a crazy market was visible from the start, people care for code velocity more than quality. I left my job and did not apply to any other workplace because I was burned out. Then the whole AI news froze the hiring everywhere. This pushed me towards working on my own low budget software consultancy rather than relying on market to be sane again. But me and my co-founders did not find it worth while after 6 months and had to pivot because of widespread AI psychosis hurting our margins and there is no money for innovation unless it can be turned into a SaaS product.
We pivoted into financial investment algorithms and research as means of making money for short term and have grown into a team of 5 people now. Each of us from a different background in science ( I do most of the code and they being from academic side of finance, mathematics and physics ) do theory and analysis of financial instruments.
We are subscribed to Ed since last year I think and regularly follow what he has to say. But our AI usage is not zero and here where our problem began. We are paying for our AI usage through OpenRouter which charges on a per million token basis, this is simply because we want zero data retention ( which is opt in smh ). Hence all frontier models are simply not feasible for us to pay for every month. Mostly sticking to open-weight models like deepseek v4 flash/pro and kimi k2.6 which are well under $3-4 / M.
We were using AI to fix obvious code paths, debugging functions in our main strategy and creating analysis on market data that we were gathering. Writing long python notebooks by hand is very time consuming and AIs were giving us quite a productivity boost. Their reasoning seemed quite fine for the tasks we have.
This is where I think it clouded my judgment. Because we were up against a deadline we never set up any pipeline or rule-book for checking bias. So whenever we hit a strange result, we came up with all sorts of explanations except the obvious one - that something was wrong in the code or in our assumptions about market data.
Six months into development, while I was manually refactoring parts of it, I fixed a tiny bit of code. It was in a simple data loader utility that pulled bars data from our database into the model. Before it included one extra minute of data (a <= where it should have been <), it introduces a look-ahead bias and we weren't aware of it for the past 5 months and considered it as a positive result and ran with it. This was catastrophic because all of our assumptions and work were invalidated in an instant. This loader was generated by an AI, which we copied into our program on day one. Although the code was reviewed when it was added, it's the conjunction of this code with rest of our system was not explored by anyone.
Although this may look like that the same mistake can be made by a human. It wouldn't have happened if it was being worked on without AI help. Our main codebase is really small. AI's have contextualized this part numerous times, every time I have asked it to generate a new analysis notebook or something, it goes and reads the same data loader code. It has reasoned about our code base directly when I asked it to find any obvious bugs. It loves dissecting the complicated paths but never picked up this simple error. Any human would have caught this way way sooner if they had some time debugging and reading the code thoroughly, and took time to debug and check the output.
Now that we are snapped out of this AI psychosis it is really clear it's not just about this one error. AI changes the way you think. The worse part is that the output it gives now works 90/100 times. But just because it "works" does not mean it's the right thing to use. It's worse than gambling, it's like winning every time you are betting but there is a guy puncturing holes in your back and draining your blood out to offer it to the devil.