Analysts and AI Overfitting
As I'm saying this out loud and typing it, I know it's a personnel question rather than consulting one. BUT, typing out to gauge maybe other's experiences. In what ways are generational differences presenting themselves with modern tools?
Small boutique niche firm. We've steadily moved up market to compete with multiple larger firms in our space. We are still nimble and move on things quickly compared to some peers.
Within some parameters, we've given one of our younger senior analysts practically open road on a few AI tools for data cataloguing and analysis.
Frankly, what they've built is immensely impressive, never had access to this kind of data before, and will completely change our deliverables and market value.
Problem is surfaced however, is their reliance and overfitting on the analysis. The competing sentiments are "The data says X" and we are having to say consistently data is great, but just because an AI ingested 5+ years of information doesn't mean it knows everything.
Their tendency is the desire to put anything quantifiable into a report, and we are saying that TMI can be harmful and it's up the the advisor on how information is presented.
The sad part . . . multiple people internally have expressed that so much time spent chatting with Claude is having an impact on their critical thinking skills, and while super productive, it's coming at a cost of over confidence or extreme scrutiny of AI output. Weird times folks.