u/mwc360

Hi r/MicrosoftFabric community!

I'm Miles Cole from the Fabric Customer Advisory Team (CAT), joined by u/thanasaur (Jacob Knightly - PM Leader) and u/gobuddylee (Chris Finlan - PM Leader) from the Fabric Spark Team. We're excited to do a Fabric Spark edition of r/MicrosoftFabric's Experts + Engines AMA series.

The Fabric Spark team has been aggressively shipping to deliver better perf, capability, etc. - Runtime 2.0 is now GA with Apache Spark 4.1 and Delta Lake 4.2 (we are one of the first cloud platforms to GA these components). The Native Execution Engine supports a much broader set of query workloads, Incremental Liquid Clustering now makes Liquid Clustering the defacto data layout optimization. New monitoring, release channels, failure diagnostics, and developer experiences - but production data engineering is not just new features and benchmarks. A faster query is only useful if the workload is also reliable, understandable, and reasonably easy to operate.

With that in mind - bring us the real scenarios and the uncomfortable questions. What works today, and where do we still have work to do? Where would you like us to invest and innovate? Those are the conversations we want to have.

We're here to answer your questions about:

• What actually changes when you move to Fabric Runtime 2.0, including compatibility considerations (Spark 4.1, Delta Lake 4.2, Python 3.13, Java 21, and Scala 2.13) and new default features / configurations?
• Where the Native Execution Engine improves Spark SQL, DataFrame, UDF, complex-type, join, window, and CTE-heavy workloads, and how to verify that your work stays in the native path
• How Incremental Liquid Clustering changes data layout maintenance, write amplification, and the way you should think about data layout decisions
• Why Spark jobs queue or fail, how Efficient Scaledown works, and how to reason about reliability, concurrency, and capacity instead of guessing
• Where Lakehouse Query Explorer and Fabric Runtime release channels fit into a practical development and release workflow
• How we are approaching streaming and real-time Spark workloads, the tradeoffs involved, and how we can empower customer success in this space
• Designing data applications for performance and scalability

Tutorials, links and resources before the event:

• Fabric Runtime 2.0 (Generally Available)
• Inside Fabric Runtime 2.0: Spark 4 and Delta 4 in Action
• Faster Spark queries with the Native Execution Engine: New performance optimizations
• Improve performance for Python UDFs and complex data types with the Native Execution Engine
• Incremental Liquid Clustering in Microsoft Fabric
• More resilient Spark jobs with Efficient Scaledown (Preview)
• Job concurrency and queue monitoring for Spark with a capacity-level view (Generally Available)
• Simplify Spark failure diagnosis in Fabric (Preview)
• Lakehouse Query Explorer in Microsoft Fabric (Generally Available)
• Fabric Runtime Release Channels
• Spark Real-Time Mode

AMA Schedule:

• Start taking questions 24 hours before the event begins
• Start answering your questions at: August 25, 2026 8:00 AM PDT / August 25, 2026 3:00 PM UTC
• End the event after 1 hour

Ever wish you could guarantee non-duplicating runs of any arbitrary process? Well, if you are working Python anything is possible...

LMK if this is helpful for learning what various features do, how they work, etc. Feel free to comment with any special requests. I'm planning for this to be an expanded catalog but am starting with the following:

- Incremental liquid clustering

- Auto compaction

- Deletion vectors

Cheers!

If you've been challenged with how to optimize your Spark jobs, consider running through this performance engineering tutorial.

Install it into your workspace in minutes, then run the datagen SJD which will generate a full schema of Delta tables w/ ~ 600 commits giving you a real-world type schema that dozens of performance tuning exercises are all built around.

It's all set up to run no more than 8cores at once, so easy to get started and learn.

It currently covers code, table design, and execution knobs. Please share any feedback and we'll consider adding or adjusting scope!

Based on community feedback and frequent questions around what Delta Lake features are supported by Spark compared to Polars and DuckDB that come pre-installed in the Python runtime, we've updated our docs page around selecting the right Notebook kernel/engine.

Things change fast, but as of today the 30-point comparison is accurate. We will update over time as needed.

Additionally, check out the guidance for compute configuration with each kernel. We've seen frequent misconceptions around the minimum compute size / CUs that can be used for Spark. Did you know that you can run just 2 CUs (4 cores) as a single node and even still use a starter pool but with only 4 CUs (8 cores).

Give it a read, we hope this brings more clarity

Ok r/MicrosoftFabric - finally, here it is. It's either the worst or best kept secret since we did a soft launch @ FabCon: Fabric Jumpstart

Read the blog: Fabric Jumpstart – Discover what’s possible with M... - Microsoft Fabric Community

Please share any feedback, raise an issue on the GitHub page for any issues, and for all of you out there that have cool accelerators, demos, etc., would love to see you contribute and help the entire Fabric community!

Want to quickly experience, learn, or demo Microsoft Fabric... solved.