🛑 What's Lakehouse//RT?
Lakehouse Real-Time s a serverless compute built for low-latency, high-concurrency use cases. It offers sub-second latency on SQL read queries against your Unity Catalog tables that use Delta Lake or Apache Iceberg formats in cloud storage.

🛑 When to use it ?
Lakehouse RT is designed for operational analytics, BI and app serving and observability workloads.

🛑 How can I spin up a Lakehouse//RT compute ?
You create and manage Lakehouse//RT much like you do other SQL warehouses.

🛑 What's Reyden ?
It's name of the Engine powering Lakehouse//RT

Lakehouse Replay improves the quality and stability of future Databricks Runtime releases by automatically replaying a small subset of read-only workloads from your workspace against upcoming runtime versions before they reach production.

When a workload succeeds in production but fails on the upcoming runtime version Databricks identifies and fixes the regression before that version ships.

Lakehouse Replay samples workloads only from serverless compute but the regressions it catches improve every Databricks Runtime release, including both classic and serverless.

Running on serverless lets Databricks perform this testing on managed compute so the replay work is not billed to you.

Explore and learn the latest data and AI technologies

Experiment with the same unified data intelligence platform that’s used by millions of data and AI professionals

🛑 Genie Code
Ask Genie to analyze a dataset, clean a pipeline, or build a visualization, and it will write the code, execute it, interpret the results, and refine its approach based on what it finds. It's like having a data engineer and analyst working alongside you, available the moment you open Free Edition.

🛑 Serverless GPUs
Free Edition now includes access to GPUs subject to availability. Builders can take on advanced AI projects for free, while Databricks handles the compute behind the scenes.

🛑 Lakebase
Lakebase brings a fully managed Postgres-compatible database to Free Edition, purpose-built for data apps and AI agents.

🛑 Agent Bricks
Agent Bricks is our new framework for building production-ready AI agents on Databricks. It provides pre-built, composable components tools, memory, orchestration, and evaluation that let you move from idea to working agent in a fraction of the time.

🛑 Lakeflow Designer
Lakeflow Designer makes it easier to build data pipelines visually. You can design data flows, connect steps, and see how data moves through your pipeline without starting from a blank page. For learners, this makes data engineering easier to understand by letting you build real pipelines directly in Databricks.

https://preview.redd.it/uugeabbi936h1.png?width=1442&format=png&auto=webp&s=de84ddb9ea9cbe7733986d8c12c0ed4df8557fa2

Do you know that Workspace admins can configure the timeout by going to Settings > Compute and, under Serverless interactive, update the Serverless interactive execution timeout setting.

The default timeout is 2.5 hours and users can still override the timeout for an individual notebook by using spark.databricks.execution.timeout

https://preview.redd.it/1ormcrezkh5h1.png?width=2048&format=png&auto=webp&s=c4f48d9d9e8e0c12a50861cbb8092ea6fa8274a1

Create and schedule from anywhere => you can create, schedule, edit, and debug Databricks Jobs. No more clicking through settings pages or manual investigations from scratch. 

**Diagnose and fix failed runs =>**Say "Diagnose error" (or /fix) Genie Code can then triage, pull logs, explain root cause, and suggest a fix.

Edit by conversation =>"Change the schedule." "Add a notification." "Bump the cluster." The diff shows up inline, right in the chat, using an optimized API that feels snappy. 

**Smart change approval =>**Every edit passes layered safety checks

https://preview.redd.it/1c7js4rcsm4h1.png?width=1920&format=png&auto=webp&s=25de3882ee1dda6559be9976ac07e5f579f73a99

Managed Iceberg: You can now create, read, write, optimize, govern and share Iceberg tables directly in UC. Bonus Predictive Optimization remove the burden of maintenance which makes Databricks very easy to use, optimized and highly performant.

Iceberg V3: Adding Native support to row Tracking, Variant Data Type and Deletion Vectors across Managed, Foreign and Uniform enabled tables.

Foreign Iceberg and Credential Vending for Foreign Iceberg: you can now register, govern and securely query Iceberg tables managed in external catalogs

Iceberg compatible Materialized Views: You can now create high performance MVs in Databricks and expose them downstream as a native Iceberg Tables

Cross Engine Attribute-based access control: You can now enforce fine grained governance policies even for external engines through Iceberg REST Catalog SCAN APIs.

Catalog Federation Connectors: You can expand Unity Catalog's federation support Beyond AWS Glue, Snowflake Horizon ... and include Google Cloude Lakehouse and Palantir making unity Catalog your single Pane of Glass

Databricks is more open than ever

https://preview.redd.it/lfappxockm4h1.png?width=1920&format=png&auto=webp&s=1cdee88ba90d9d06d5b07c2096953f9cdbcd8029

1) Open APIs and credential vending

You can create, read and write to Iceberg Tables in Unity Catalog from any engines using UC's Iceberg REST Catalog APIs. You are able to use the engine that best fits your workloads whether it's Spark Trino, Flink, DuckDB or Snowflake without copying data or giving every engine broad storage permissions.

In addition, Unity Catalog vends credentials for federated Iceberg Tables providing a secure access via Open APIs even to the tables managed in external catalogs.

2)Catalog Federation

If you have many catalogs, we got you covered, Unity Catalog can govern Iceberg tables managed in other Catalogs whether it's AWS Glue, Snowflake Horizon, Palantir, Salesforce any many more.

3)Cross-Engine ABAC

Unity Catalog extends Attribute-based access control to Iceberg Clients using REST Catalog Scan APIs. What does it mean ? once an Admin create a policy when an External Iceberg engine requests access, Unity Catalog evaluates the applicable policies during server side scan planning. UC then return a filtered scan plan so the engine only read authorized data. Learn More

4)Zero copy secure Sharing for External and Cross-domain collaborations

You no longer require every recipient to use the same vendor ecosystem or copy data into another platform. With Sharing to Iceberg Clients, you can share live data externally with any recipient that supports the Iceberg REST Catalog API. Recipients can query shared data from Iceberg compatible clients such as Snowflake, Flink Trino... without manual ingestion.

You can also share Iceberg tables that are managed or cataloged outside Databricks as long as they are registered in Unity Catalog.

5) Performance and Format Innovation

UC uses AI to optimize your tables to make them faster and lower the operational overhead. Predictive Optimization finds the tables that require maintenance and which operation to run ( Vacuum, Optimize, Analyze...) and run it on your behalf. Every Operation is logged on a System table to everything is transparent.

With Iceberg V3, you get support for Deletion Vectors, Variant and Row Tracking across managed iceberg tables, Foreign iceberg tables and Uniform Enabled Managed tables.

One objective/One Goal : Open Tables that do not force you to choose between ecosystem interoperability and the performance capabilities required for production workloads

Learn more: https://www.databricks.com/blog/unity-catalog-and-next-era-apache-icebergtm

80% of enterprise data is locked inside PDFs, scans, emails and contracts and most teams still treat it as someone else's problem.

Document Intelligence on Databricks changes that. One SQL function (ai_parse_document), governed by Unity Catalog, integrated with Lakeflow for ingestion, Agent Bricks for structured extraction and Vector Search for RAG.

No stitched-together OCR vendors, no brittle Python glue, no separate platform to govern.

I put together with Archika Dogra a walkthrough showing how it actually works end-to-end from a folder of raw PDFs to queryable Delta tables and downstream agents.

▶️ https://youtu.be/sdG73gI143c

Curious to hear what use cases you're tackling invoices, contracts, claims, technical docs? Drop them in the comments.

80% of enterprise data is locked inside PDFs, scans, emails and contracts and most teams still treat it as someone else's problem.

Document Intelligence on Databricks changes that. One SQL function (ai_parse_document), governed by Unity Catalog, integrated with Lakeflow for ingestion, Agent Bricks for structured extraction, and Vector Search for RAG.

No stitched-together OCR vendors, no brittle Python glue, no separate platform to govern.

I put together with Archika Dogra a walkthrough showing how it actually works end-to-end from a folder of raw PDFs to queryable Delta tables and downstream agents.

▶️ https://youtu.be/sdG73gI143c

Curious to hear what use cases you're tackling invoices, contracts, claims, technical docs? Drop them in the comments.

1,800 : Spark Declarative Pipelines running on Databricks Lakeflow
86% to 99.6% : Improvement in job success rate
80% less transformation complexity

This is insane. If you have any questions about SDP let me know.

Databricks Student Fellows is an exclusive opportunity for university and college students who are passionate about computer science, AI, and data engineering to become leaders on their campus. Whether you are just beginning your coding journey or you are already diving deep into AI models, this program is designed to turn your data expertise into a career in data and AI. Student Fellows are standout contributors to their campuses, their projects, and beyond

ABAC policies, governed tags, and automated data classification are now generally available in Unity Catalog.

Governance teams define access rules and they apply automatically across the entire data estate. Sensitive data is discovered, tagged, and protected as it's created, with no manual configuration per table.

Together, they deliver consistent, scalable protection with less operational overhead and stronger compliance posture as data grows. https://databricks.com/blog/abac-row-filtering-and-column-masking-policies-governed-tags-and-data-classification-are-now