Parse_sql() function extracts table references, column names, functions, and parameters from SQL. It extracts without running the query. It also reports syntax errors.
more news https://databrickster.medium.com/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a
No table description? Not queried for 90 days?
See how to automatically tag the table and notify its owner in my new article.
https://medium.com/databrickscommunity/clean-up-the-mess-with-tag-automation-72ae119b6f35
https://www.sunnydata.ai/blog/databricks-unity-catalog-automated-tagging
It was a real pain, but now, with a few tricks, you can manage them better.
First, in Workspace files, next to the DABs folder or git repo, there is a small shortcut to show only tabs from that DABs folder or git repo. Alternatively, you can also use the switcher in Home next to Notebook.
If you need to organize your tabs differently, there is new functionality: spaces, which let you group them however you like.
We now have Genie Code Task in Lakeflow jobs. Can not yet send output to if/else, but more options for orchestration are planned.
In the Unity AI gateway, it is also possible to register an external model for which we pay the provider directly (OpenAI, Anthropic, etc.). In that case, Databricks now knows the prices for those models and can calculate, monitor usage, and alert or block based on budgets.
Skills are available in Unity Catalog. They use a similar concept to volumes and are integrated with the AI gateway. New REST endpoints for skills are coming, and a new tool to manage them, ucode, is already available.
I went to the Ingestion section and saw that Fivetran is no longer there. It was always there for many years. Also, at the same time, a few new Lakeflow connectors were added.
When working from the web experience in development mode with source-linked deployment set to true (the default for development mode), you can edit jobs and pipelines in the UI, and changes are automatically propagated to your YML files. So it is like the best of two worlds: ow-code and IaC. Please remember to review any changes in Git, especially non-UI elements such as variables or mutators.
Whole talk with audio https://www.databricks.com/dataaisummit/session/dabs-do-pro-all-best-tips-and-tricks
Genie ZeroOps is an autonomous agent that monitors, investigates, and proposes fixes for data pipelines, jobs, tables, and other assets, coming soon to your workspace. I tested it, checked how it works, and discussed the pipeline.
https://medium.com/databrickscommunity/genie-zeroops-a-hands-on-preview-b8fc46138087
https://www.sunnydata.ai/blog/genie-zeroops-hands-on-preview
Direct mode is GA and is now the default for bundles created from the workspace (since 13th August 2026); it will become the general CLI default on 26th August 2026. Let's see what direct mode brings and what will come next.
https://www.sunnydata.ai/blog/databricks-dabs-direct-mode-ga
It is great to see serverless in the compute section. Especially because I proposed it some time ago and lobbied for it during advisory meetings. Now we can set permissions for serverless compute. More options will come. I can say that it is my favorite recent improvement.
Now we can drop the .pbit file to Genie Code and ask for migrating to databricks dashboard. I tested in direct query mode, and everything was migrated; I just had to be patient, as the whole process took quite a long time. The process created metrics views for tables used in Power BI, and now I understand why the dashboard relationship option was added. I presented how easy it is to migrate in my last video.
New video just released. Databricks News: ZeroOps, DABs, Indexes, Genie, sandboxes, migration from PowerBI, secrets
video
https://www.youtube.com/watch?v=-e3tkcg21zw
article
Thanks to the new column_object_metadata we have access to storage-level information such as MIME type, ETag, and other metadata. That functionality is for external volumes, as in some use cases that data was needed for ingestion. Just remember that it is row-level data, so if you have a parquet file, it will be returned for every row.
Unity Catalog secrets operations Create, Update, List, Read, or Reference are now in audit tables as well. I think that, aside from a clear permission model, it is the biggest benefit of storing secrets in UC.
Now we can store secrets in Unity Catalog and use a 3-level namespace convention. Those secrets are no longer at the workspace level, as the old ones were, but at the catalog level. The biggest advantage is ease of managing permissions. There is READ SECRET, but also REFERENCE SECRET for scenarios when it can be used by other databricks services like Lakeflow Connections and not by programmatic access. Reading secrets in a notebook is still done by dbutils secrets get.
x10 performance,
x100 concurrency,
price the same,
impossible?
not in databricks
See benchmark https://www.sunnydata.ai/blog/databricks-rt-lakehouse-benchmark-results
Migration to direct mode is easier than ever; just add engine: direct to the old bundle, and it will be migrated automatically.
Don’t overwrite your code — make it immutable! With every deployment, thanks to immutable_folder, all files are copied to a new, read-only folder. They are not overwritten. The biggest benefit is that jobs already underway will not fail or produce unexpected results.
I benchmarked the new Lakehouse RT for billion-record tables, and it is crazy fast and crazy cheap, making it the first real-time database with open-format storage.
https://www.sunnydata.ai/blog/databricks-rt-lakehouse-benchmark-results