Genie is so Dumb and I am tired of Pretending Otherwise
Hey Guys, Data Engineer with 3.2 YOE here,
I have been working on and off on databricks for multiple projects across domains like Retail, logistics and Now Pharma.
In my current role, I focus on creating end to end pipeline with different data sources which often get refreshed on a bi-weekly basis for pharma related data
The client is extremely demanding and has no technical expertise to gauge how much time it really takes to build and maintain something so complex and therefore expect the team to use Claude teams license and also Genie
The problem with Genie is that, I can not trust it, It will not listen to my instructions even after having a separate instructions.md which I update on a daily basis, in fact I update this after each session is complete, and no, I do not use AI to update this, every change request that the client and the client's team has goes through my own words of instruction updates, I have an entire markdown file which has contexts for multiple islands of projects, workspace folders and notebooks that I have to maintain and transition to devops team.
I also have created 4 different skill family for Genie, in relation to
- migration,
- devops,
- maintainence and
- review.
And it is so disgustingly bad at handling all this context, that it fails me across all 4 areas.
- I have seen it lie to my face multiple times,
- Casting columns as null,
- altering data types without asking consent,
- never following my work stream and process in a sequential format
- Having set the wrong notebook paths in a task even though i explicitly tell it exactly what to do.
It often confuses streams of works and ends up mixing so many things that remove this knot and fuck up itself costs me so much time!!!
- It is very poor in adhering to the domain specific instructions that we provide
- More than 3 requests in a chat, and it is practically useless
- And branching of chats is the most useless feature that they have introduced.
I have created multiple diagnosis queries to check the work of this agent, and it will straight up lie to you, so much, so convincingly, that you know it knows all the biases you have and it even goes a step ahead and just narrates a story that you can provide to your team in the stand-up.
My only concern is, after all this bullshit, my team, of nearly 6 (some of them have never worked in databricks or delta table environments like this) was charged $990 dollars in the month of august, for this shit output??? Have some shame Databricks, fix your worthless product or remove that feature or stop charging so much if you are beta testing in actual prod.
Today, I am writing this post because of something that I caught live, that pushed me to the edge.
Something that is critical, production level issue, which If I was not paying attention while merging could have been a huge disaster, and mind you the pipelines I make are client facing, what is even more distrubing is the fact that it will just make so many unnecessary changes to a simple query or a pyspark function just enough so that the tests are passed.
(So basically it does not want to get caught, and makes a mistake so that we can prompt it again to fix this mistake and then Databricks can charge us more, this looks like a dark pattern to me)
In your preview (prima-facia), everything is good, the tests are passing, the job is running, but holy!!!,
- it was casting 3 columns which are essential for downstream processes as null.
- It boldly suggests that we remove those 3 columns, and or cast them as null. How is this a solution Databricks? The AI is supposed to have more context than me because it is a agent made specifically for Databricks Environment Correct?? That is how it is sold?? And upon spending just 2 mins, I realized that the fix that it is suggesting will blow up critical information that the client team should see in the app because it is not even there in the first place, and all of this is because it can not read the correct notebooks, even if you tag it.
- It had set a retired notebook's path in the task and very boldly claiming that everything is good.
My concern is, let us say that there is a junior who is good at SQL, but has never maintained projects like this in this scale, he/she would be overwhelmed and won't even be able to find the mistakes that the agent is stating, because unlike GPT 6 or Opus 5.5 Genie does not openly claim that it does not know something,
it does not ask for more context,
it does not state that something is missing,
it just patches with some assumption that we don't even provide in the first place.
My advice to any Juniors out there, Stop using Genie, Completely.
The product is not ready, it is not good.
