u/InevitableClassic261

One thing I find very interesting in Data + AI right now is that the most valuable use cases are starting to look less flashy and more useful.

For a while, a lot of the conversation felt centered around model size, hype, and what looked impressive in demos. But in actual work, the solutions that seem to matter most are much simpler and more practical. Things like helping support teams understand issue spikes faster, helping retail teams spot waste risk earlier, helping operations teams detect bottlenecks sooner, or helping business users ask better questions on top of trusted data.

That shift feels important to me.

It feels like Data + AI is moving from “look what this model can do” toward “look what this system can help people do better.” And honestly, I think that is where the real value begins.

What makes this even more interesting is that it also raises the value of good data engineering. Because when AI starts getting used for real decisions, data quality, governance, freshness, and trust matter even more. A smart layer on top of weak data still creates weak outcomes. So in a way, the rise of AI is also making the fundamentals more important, not less.

I think the next strong wave of Data + AI will not come only from bigger models. It will come from better integration with real workflows, better use of trusted enterprise data, and smaller useful systems that reduce friction for real teams.

Curious if others are seeing the same thing.

What Data + AI use case feels genuinely useful to you right now, not just impressive?

I have been looking through this year’s Data + AI Summit details, and honestly, it feels like one of those events that can be genuinely useful if you work anywhere around data, analytics, or AI.

It is happening June 15 to 18 in San Francisco, with a virtual option as well, so it is not limited only to people who can travel. Databricks is also positioning it as a large event with 800+ sessions, keynotes, training, and a lot of technical content across data, analytics, and AI.

What makes events like this valuable to me is not just product announcements. It is the chance to hear how real teams are building things, what is working, what is changing, and where the industry is actually moving. The Summit site says it brings together practitioners, leaders, and builders from more than 160 countries, which usually means you get a good mix of technical depth and bigger-picture direction.

I also think this kind of event is especially useful right now because data and AI work is getting more connected. It is no longer only about one narrow skill. Data engineering, governance, analytics, apps, and AI are all starting to touch each other more. A conference that covers that wider picture can help people step back from daily work and see the bigger direction more clearly. Databricks’ official agenda page says the event is built around four days of learning across data, AI, and LLM technologies.

Another practical reason to attend is networking. The official FAQ says the conference will be held across Moscone North, West, and South in San Francisco, with many nearby activities and partner events, so it sounds like the kind of week where a lot of side conversations and useful connections happen outside the sessions too.

So my take is simple: if you are serious about growing in data engineering, analytics, machine learning, apps, or AI this year, this seems like a solid event to consider. Even if someone cannot attend in person, the virtual option makes it more accessible.

Curious if others here are planning to attend, either in San Francisco or virtually. What are you most hoping to learn from this year’s Summit?

I see many people asking where to start with Databricks, especially if they come from SQL, ETL, BI, or general data engineering backgrounds. I felt the same at one point because there is no shortage of content, but that is also the problem. Too many videos, too many disconnected examples, and too much jumping between basics and advanced topics without a clear flow.

What helped me most was keeping the learning path simple. First understand the platform at a basic level. Then work on SQL and PySpark with small examples. After that, move into Delta Lake, jobs, transformations, and practical data engineering patterns. If the order is wrong, learning feels heavy. If the order is right, things start making sense much faster.

One thing I liked recently is that BricksNotes seems to take a more practical approach instead of making Databricks feel bigger than it needs to be. It feels more focused on learning by doing, which is honestly what most people need. Not just theory, not just marketing-style content, but actual step by step learning that helps you build confidence.

That said, I still think the best approach is to combine a few things. Use official Databricks docs to understand features clearly. Use community posts to learn from real users. And use structured practical resources for hands-on learning so you are not always figuring out what to study next.

If you are just starting, my honest suggestion is this. Do not try to learn everything at once. Start small. Practice daily. Repeat the same concepts until they feel natural. Databricks becomes much easier when you stop chasing everything and start building real understanding one step at a time.

Would love to know what actually helped others here learn Databricks in a practical way.

https://preview.redd.it/q5955gvl2d1h1.png?width=1672&format=png&auto=webp&s=59224d5f06571bc3fe5d1c239f429782b09b3669

One update in the new Databricks exam guide really caught my attention. The structure has moved from 5 domains to 7 domains, and I honestly think that is a good change.

What stood out to me most is the addition of CI/CD and Troubleshooting. That feels much closer to real-world data engineering. In actual work, it is never only about writing code or knowing a few concepts. A lot of the learning comes from understanding how changes move across environments, how things break, and how to fix them when they do.

I like this update because it feels more practical. It gives the impression that the exam is not just testing theory, but also looking at skills that matter in day-to-day engineering work. Anyone can study the happy path. The real growth starts when you also learn how to handle issues, read errors carefully, and think through problems with patience.

For people preparing for the exam, I feel this is a reminder to learn more deeply. Do not stop with definitions and notes. Spend time understanding how deployments work, how troubleshooting happens, and why these areas matter in a real team environment.

Overall, this looks like a positive move. The guide feels more complete now, and in many ways, more aligned with what a data engineer actually does. Curious to know what others think.

Does this new 7-domain structure feel better to you as well?

Hi Databricks Community,

Sharing this in case it helps anyone preparing for the Databricks Data Engineer Professional EXAM.

BricksNotes just launched a free practice exam designed to help data engineers assess their readiness before taking the real exam.

What’s included:
• 60 scenario-based questions
• Coverage across all 10 exam domains
• Full 120-minute exam-style experience
• No signup required

The goal is simple: help learners understand where they stand, identify weak areas, and prepare with more confidence before spending money on the actual exam.

Hope it helps someone in their preparation journey.

You can try it here:
https://bricksnotes.com/practice/databricks-data-engineer-professional

https://preview.redd.it/kplnwuexjm0h1.png?width=1672&format=png&auto=webp&s=99d8f6628294a1795ff1c9838f117d4e7600b211

I have been following Genie since it first launched with AI/BI last year. Back then, I honestly thought it was mostly for business users. A chatbot on top of your data that could answer basic questions in plain English. Useful, but not something I thought data engineers really needed to care much about.

After seeing the new 2026 version, I completely changed my mind.

Genie is no longer just a business chatbot. The biggest change is Genie Code, which is basically an AI agent designed for data professionals. It can generate pipelines, debug failures, create dashboards, monitor systems, and work directly with Lakeflow and Unity Catalog. That part caught my attention immediately because it moves beyond simple Q&A and starts touching actual engineering workflows.

What surprised me most is how connected the whole system has become. It can pull context from dashboards, Genie Spaces, apps, metadata, documentation, and external systems like GitHub, Jira, and Confluence through MCP. Instead of only searching tables, it tries to understand relationships across the environment. That feels very different from the first version.

The operational side is also interesting. Genie Code can monitor pipelines, investigate failures, help with DBR upgrades, and respond to issues before teams even notice them. The more I read about it, the more it felt less like a chatbot and more like an assistant sitting beside the engineering team.

But honestly, the biggest takeaway for me is not the AI itself. It is what this means for data engineers.

A lot of people immediately jump to “AI will replace data engineers,” but I think the opposite is happening. These systems are only as good as the data foundation underneath them. If metadata is incomplete, if tables are messy, if naming conventions are inconsistent, or if documentation is missing, the AI layer will give poor answers confidently.

That means clean data modeling, governance, metadata, documentation, and data quality are becoming even more important than before. The engineers building those foundations become more valuable, not less.

I think the role is slowly shifting away from spending hours writing repetitive boilerplate transformations and more toward building trustworthy, AI-ready data systems.

One thing I keep noticing while learning Databricks through BricksNotes and the wider community is that the platform is moving very quickly toward AI-native data engineering. Features like Unity Catalog, Lakeflow, and now Genie all connect together. It feels like understanding metadata and governance is becoming just as important as understanding Spark itself.

Also interesting that Genie now has a full mobile experience on iOS and Android. Business users can access dashboards, apps, and chat directly from their phones, which means the underlying data quality matters even more because people are going to depend on these systems everywhere, not only during work hours.

Curious if anyone here is already using Genie or Genie Code in production. I would genuinely like to hear how the answer quality has been and whether your teams are changing how they approach metadata and documentation because of it.

The Databricks Data Engineer Associate exam changed on May 4, 2026.

The exam now has 7 domains instead of 5.

Two new domains were added.

The first new domain is CI/CD.

This includes:
• Databricks Repos
• Git integration
• Branching and commits
• Deploying Declarative Automation Bundles
• Using the Databricks CLI
• Moving code from dev to test to production

Databricks Asset Bundles is now called Declarative Automation Bundles, so learn the new name.

If you have never used Git or the Databricks CLI inside Databricks, spend some time practicing in the Free Edition. Connect a Git repo, make commits, and deploy bundles. Hands-on practice will help a lot.

The second new domain is Troubleshooting, Monitoring, and Optimization.

This includes:
• Reading the Spark UI
• Finding bottlenecks like data skew and excessive shuffling
• Understanding Liquid Clustering
• Predictive optimization
• Troubleshooting cluster and memory issues

Many courses do not teach Spark UI deeply, so try running queries yourself and checking the Spark UI. Compare good queries with inefficient ones to understand the difference.

Some existing domains also changed.

Ingestion now includes Lakeflow Connect along with Auto Loader and COPY INTO.

Governance now includes:
• Column-level masking
• Row-level security
• Attribute-based access control

You now need to understand security beyond basic GRANT permissions.

Lakeflow Jobs also tests three trigger types:
• Scheduled
• File arrival
• Table update

Know when to use each one.

Some product names also changed:
• Databricks Asset Bundles → Declarative Automation Bundles
• Delta Live Tables → Lakeflow Declarative Pipelines

The exam uses the new terminology, so update your study material if you are using older resources.

The exam format is still:
• 45 scored questions
• 90 minutes
• $200

There may also be extra unscored questions mixed into the exam.

For preparation, the original Academy courses still help for the old domains.
But for the two new domains, hands-on practice is very important.

Practice:
• Spark UI
• Git integration
• Databricks CLI
• Deployments using bundles

Also read the latest official exam guide PDF from the Databricks page.

Good luck to everyone preparing for the exam.

I want to share something that I wish someone told me when I started learning Databricks because it would have saved me months of confusion.

When I first opened Databricks, I did what most people do. I went straight to PySpark because every tutorial said that is what data engineers use. I spent weeks trying to understand RDDs,

DataFrames, transformations, actions, lazy evaluation, and the DAG all at once. I could follow along with the instructor but the moment I opened a blank notebook I had no idea where to start.

Then I took a step back and tried something different. I started with SQL.

Databricks runs SQL natively. I already knew SQL from a previous job. Within an hour I was querying tables, running aggregations, building views. I felt productive for the first time in weeks. That confidence changed everything.

Here is the order that worked for me and I genuinely believe it works for most people.

Start with SQL on existing tables. Databricks has sample datasets built in. Run SELECT statements. Do GROUP BY. Write JOINs. Get comfortable navigating data. If you already know SQL from any database this stage takes a few days not weeks.

Then learn Delta Lake through SQL. Create tables. Insert data. Update rows. Delete rows. Run DESCRIBE HISTORY and see the transaction log. Run SELECT VERSION AS OF and experience time travel.

This is where Databricks starts to feel different from other databases. Every table you create is automatically a Delta table so you get versioning, schema enforcement, and ACID transactions without configuring anything.

Then move to PySpark DataFrames. Now that you understand what the data looks like and how Delta tables work, PySpark makes way more sense. You understand what df.filter does because you already did WHERE in SQL. You understand what df.groupBy does because you already did GROUP BY. Lazy evaluation clicks faster because you have context for what the transformations are actually doing.

Then build pipelines. Take what you learned and chain it together. Read from a source. Transform. Write to a Delta table. Schedule it. Monitor it. This is where Lakeflow (the new name for Delta Live Tables) comes in. But it makes no sense if you skip the previous steps.

Then governance. Unity Catalog, permissions, data quality expectations. This feels like admin work when you learn it in isolation but once you have built a pipeline you understand exactly why it matters.

The mistake I made was trying to learn PySpark before I understood the data model. I was writing code without knowing what it produced. Once I started with SQL and built up from there everything fell into place faster.

One more thing. If you are on Free Edition you do not need to configure clusters. It is serverless. If a tutorial tells you to create a cluster and choose a runtime version that tutorial was written for Community Edition which no longer exists. Just open a notebook and start writing code.

Hope this helps someone who is feeling overwhelmed right now. Happy to answer any questions in the comments.

I took the Databricks Data Engineer Associate exam recently and wanted to share what actually came up because it was quite different from what I spent most of my time studying.

I went in thinking Delta Lake theory and platform architecture would be the big topics. They weren't. The exam is way more practical than I expected.

The first thing that caught me off guard was how heavily they test Auto Loader. Not just the basics but real scenarios. One question described a pipeline receiving 50,000 new files per day and asked which ingestion method to use and why. You need to understand when Auto Loader makes sense versus COPY INTO, how schema evolution works with mergeSchema, and the difference between directory listing and file notification mode. I probably got six or seven questions just on this one topic.

The second thing was lazy evaluation. I knew the concept but I wasn't prepared for how they test it. They give you a block of code with four or five DataFrame transformations and ask what happens when you run the cell. The answer is nothing happens because there is no action at the end. But the way they frame the questions makes you second guess yourself if you only memorized the definition without really understanding it.

Third was Lakeflow expectations. The old name was Delta Live Tables but they use Lakeflow in the exam now. You need to know the three expectation types and when to use each one. They gave me a scenario where the pipeline should log bad records but never drop them and I had to pick the right expectation decorator. Also know the difference between streaming tables and materialized views because that came up more than once.

Fourth was Unity Catalog permissions. Not just the three level naming pattern but actual grant scenarios. Something like a data analyst needs to read tables in the sales schema but should not be able to create new tables and you have to pick the correct grant statement. I got at least three or four questions like this.

Fifth was MERGE INTO. They really love this command. Upsert scenarios, deduplication, slowly changing dimensions. If you cannot write a MERGE statement from memory with the WHEN MATCHED and WHEN NOT MATCHED clauses you should spend an hour practicing just that before you sit for the exam.

What surprised me about what was not heavily tested. Cluster configuration was maybe one question. The architecture diagrams with control plane and data plane were one or two questions at most. Delta Sharing was one question. Spark internals like shuffle details were barely mentioned.

The biggest thing I wish I had done differently is spend less time reading documentation and more time actually running code. When you have actually executed a MERGE INTO on a real table and seen the results, the exam question feels like something you have done before instead of something you read about once. I used Databricks Free Edition for all my practice and it was more than enough.

Hope this helps someone who is preparing right now.

Feel free to ask anything about the exam in the comments and I will try to answer.