r/MicrosoftFabric

We’ve built a Microsoft Fabric development process using Deployment Pipelines for shared semantic models and Git branch-out to give each developer their own workspace.

For Dev/Test/Prod, we’d like to use shareable cloud connections authenticated with Workspace Identity. The problem is that when a developer branches out to their own workspace, the Workspace Identity shared cloud connection doesn’t seem to work when editing the model. perhaps I'm missing a configuration or setting?

it seems like the shared cloud connection works fine for refresh, however when you edit the model. I get a warning

To avoid semantic model refresh errors while developing, we end up changing the connection to a per user OAuth/user credentials in the developer workspace.

Our current thinking is:

• Developer branch-out workspace → OAuth connection
• Shared Dev → Workspace Identity connection
• Test/Prod → Workspace Identity connections
• Rebind the semantic model to the appropriate managed connection during integration/deployment

It works, but having developers switch authentication feels like extra complexity and potentially something we shouldn’t need to manage manually.

For teams using Fabric + Git + branch-out + shared semantic models, how are you handling connections and authentication across developer workspaces?

I'm also curious about what happens or needs to change when development is done outside of the workspace in power bi desktop.

Hi r/MicrosoftFabric community!

I'm Miles Cole from the Fabric Customer Advisory Team (CAT), joined by u/thanasaur (Jacob Knightly - PM Leader) and u/gobuddylee (Chris Finlan - PM Leader) from the Fabric Spark Team. We're excited to do a Fabric Spark edition of r/MicrosoftFabric's Experts + Engines AMA series.

The Fabric Spark team has been aggressively shipping to deliver better perf, capability, etc. - Runtime 2.0 is now GA with Apache Spark 4.1 and Delta Lake 4.2 (we are one of the first cloud platforms to GA these components). The Native Execution Engine supports a much broader set of query workloads, Incremental Liquid Clustering now makes Liquid Clustering the defacto data layout optimization. New monitoring, release channels, failure diagnostics, and developer experiences - but production data engineering is not just new features and benchmarks. A faster query is only useful if the workload is also reliable, understandable, and reasonably easy to operate.

With that in mind - bring us the real scenarios and the uncomfortable questions. What works today, and where do we still have work to do? Where would you like us to invest and innovate? Those are the conversations we want to have.

We're here to answer your questions about:

• What actually changes when you move to Fabric Runtime 2.0, including compatibility considerations (Spark 4.1, Delta Lake 4.2, Python 3.13, Java 21, and Scala 2.13) and new default features / configurations?
• Where the Native Execution Engine improves Spark SQL, DataFrame, UDF, complex-type, join, window, and CTE-heavy workloads, and how to verify that your work stays in the native path
• How Incremental Liquid Clustering changes data layout maintenance, write amplification, and the way you should think about data layout decisions
• Why Spark jobs queue or fail, how Efficient Scaledown works, and how to reason about reliability, concurrency, and capacity instead of guessing
• Where Lakehouse Query Explorer and Fabric Runtime release channels fit into a practical development and release workflow
• How we are approaching streaming and real-time Spark workloads, the tradeoffs involved, and how we can empower customer success in this space
• Designing data applications for performance and scalability

Tutorials, links and resources before the event:

• Fabric Runtime 2.0 (Generally Available)
• Inside Fabric Runtime 2.0: Spark 4 and Delta 4 in Action
• Faster Spark queries with the Native Execution Engine: New performance optimizations
• Improve performance for Python UDFs and complex data types with the Native Execution Engine
• Incremental Liquid Clustering in Microsoft Fabric
• More resilient Spark jobs with Efficient Scaledown (Preview)
• Job concurrency and queue monitoring for Spark with a capacity-level view (Generally Available)
• Simplify Spark failure diagnosis in Fabric (Preview)
• Lakehouse Query Explorer in Microsoft Fabric (Generally Available)
• Fabric Runtime Release Channels
• Spark Real-Time Mode

AMA Schedule:

• Start taking questions 24 hours before the event begins
• Start answering your questions at: August 25, 2026 8:00 AM PDT / August 25, 2026 3:00 PM UTC
• End the event after 1 hour

If a developer is comfortable with Spark on their desktops, on k8s, and on databricks, then should these Spark skills be easily transferred to Fabric?

I'm struggling with Spark in Fabric. People love Spark because it is a technology which is developed out in the open, and is well documented, and there are massive online communities who will assist with questions when a user runs into problems.

Yet this Spark in Fabric breaks those patterns and can be pretty disorienting. I am not talking about NEE or anything super-duper-top-secret or proprietary. (I normally disable that stuff, in any case). I'm simply talking about running a spark job, and interpreting the logs when something is misbehaving. This should be well within the reach of a conventional Spark developer. Right?

Lets take a simple example The pyspark notebook cell has hung, and when I look in the logs and it says this....

2026-08-19 14:39:16,055 INFO RpcAppSender [shutdown-hook-0]: RPC app sender closed
2026-08-19 14:39:16,055 INFO RpcAppSparkContextServer [shutdown-hook-0]: Closing remote SparkContext service at 10.0.yyy.xxx:18083, remoteSparkContext/remoteSparkContextEndpoint

What is this stuff? What is RpcAppSender? What is RpcAppSparkContextServer? If this stuff is printed all over my logs, shouldn't I be able to find a single search result on google? If something called a Spark Context is being "closed", then isn't that a message that a normal spark developer should be able to interpret/investigate by googling the log message?

I'm not objecting to Microsoft's prerogative to extend Apache Spark and innovate, and such. But the bothersome thing is that these extensions are undocumented and secretive and developers are at a loss to self-support. Any Spark developers coming from another ecosystem are not properly equipped to troubleshoot Fabric-specific behaviors. It is disorienting and unfamiliar. Even the log messages themselves are confusing; the purpose of logs should NOT be to create even more confusion. I'm almost of the opinion that Microsoft should stop calling their product Spark if the product is twisted to the point where even the log messages are unrecognizable and not meaningful for troubleshooting.

Sorry for the long rant. In the very least, there will be a search result the next time someone searches for "RpcAppSparkContextServer". As of today, that term was never heard, across the entire internet! One might think I made it up myself. lol.

Status: Degraded | Reported: Aug 19, 2026 at 7:40 AM UTC

---

Power BI customers with Tenants located in the North Europe J cluster may experience issues with publish and refresh operations. Engineers are investigating the issue and an update will be provided soon.

---

🤖 This post was sent from an automated and unattended service and cannot respond to questions or requests. For official updates, visit the Microsoft Fabric Service Status page.

This post contains content not supported on old Reddit. Click here to view the full post

It came to me in an epiphany today.
Fabric is upside down to my mental model.
In general, when I deploy or create something, I want THAT code first.
If I want to check or monitor something, I want that GUI first.
My expectations of a mature platform:

Authoring? Code.
Configuration? Code. Deployment? Git.
Policy? Declarative. Automation? API.
Monitoring? GUI.
Lineage? GUI

#Is my platform on fire?#

#👆 I want to click that#

And Fabric does almost everything upside down to the way most engineers I know develop:

Dataflows? GUI. Configuration? GUI. Deployment? GUI plus Workaround. Policy? N/A.
Automation? Not sure actually
Lineage? API

Is my platform on fire?
API

Monitoring?
Install a Workaround and it's code first.

No wonder I'm tired. 😂

-Your perpetually exhausted Data Engineer

--MonkeyDData

PS this is half in jest and half I finally put my finger on it

Is anyone facing issues with deployment pipelines. We have been using deployment pipelines regularly and all of a sudden deployment pipeline sync has stopped working correctly and is always showing out of sync for a specific semantic model.

We have a DEV -> UAT -> PROD setup for deployment pipelines.

There are no deployment rules changes. The only thing recently changed is that we added a new source table.

In fact the table is actually visible in the PROD workspace as well, changes are there and I can query the table in PBI service as well.

Despite multiple deployments from dev to uat and uat to prod, successful deployment doesn't result in code sync.

Thanks in advance.

https://preview.redd.it/q95fbly3pakh1.png?width=625&format=png&auto=webp&s=6d5882884ee0f541954b77cc869dfc62faaa2e46

This is now appearing above individual schedules for me. If I pass a variable from a local variable library to it and it's boolean = True it enables the schedule.

I have created 3x variables called dev_schedule, test_schedule and prod_schedule, then in the DEV pipeline I create 3x schedules and point to each individually.

It means when I promote to TEST and PROD only 1/3 of the schedules are activated meaning I can finally deploy pipelines without messing about with schedules once they hit the intended stage as they just work.

Hi all,

I'm considering a pattern for error handling in Fabric pipelines and I'm wondering if there are any documented limits I should be aware of.

The idea is that individual notebooks return a small JSON object via "notebookutils.notebook.exit()", for example:

{
"status":"failure",
"activity":"LoadSales",
"message":"Something went wrong"
}

The pipeline appends these to an Array variable:

[
  {"status":"success","activity":"LoadCustomer","message":"OK"},
  {"status":"failure","activity":"LoadSales","message":"Something went wrong"},
  ...
]

A final notebook receives the array, checks for failures, and raises a hard failure if any are found.

My questions are:

1. Is there a documented maximum size/number of elements for a Pipeline Array variable?

2. Is there a documented maximum size for a parameter passed from a Pipeline to a Fabric Notebook?

3. Are there any other limitations or gotchas with using this pattern?

I can see the documented 896 KB activity payload limit, but I haven't found a specific limit for Array variables or Notebook parameters.

I'm mainly interested in whether this would be a reasonable pattern for a reusable pipeline framework, assuming the messages themselves are kept small.

Thanks in advance!

Hey everyone, I just passed the DP600 with a score of 850 and with no experience in Power BI and Spark, and a modest experience with SQL, as I just finished my second semester in data science, coming from a completely different field (marketing).
It took my a lot of sleepless nights, some moments of doubt but I did it 😭
Outcomes:
- If you have the choice to take the exam in person, DO IT!
- 2 youtube channels: Will and Priyanka, I'm so thankful 🥺
- Keep taking the practice assessment till you feel confident
- Exam day : I had 1 case study at the end and isolated questions, a lot of KQL, T-SQL and DAX questions.
- Focus on understanding the "why" of everything and the optimization part.
Good luck!

Hey all,

Booking my DP-700 (Fabric Data Engineer Associate) soon and honestly getting a bit in my head about it, so figured I'd ask people who've already sat it.

Quick background: I'm pivoting into data engineering. Passed DP-900 earlier this year, then ran myself through a structured ~9-day DP-700 sprint covering the official MS Learn paths — Lakehouse/OneLake, ingestion (Pipelines/Dataflow Gen2/Copy Activity), Warehouse + T-SQL + star schema, Real-Time Intelligence (Eventstream/Eventhouse/KQL), and Fabric management/security/governance. For each topic I wrote my own notes and did active-recall questions instead of just watching videos. Plan is to finish with the official MS practice assessment, log every wrong answer by category (concept gap / tool-selection error / detail gap / misread / guess), and go fix the weakest areas before booking.

Not gonna lie, part of the nerves is from lurking here, pretty much every thread I find on this exam has people saying it was way harder than they expected, so that's been sitting in the back of my head through this whole prep.

So the actual questions for anyone who's taken it recently:

• Which questions or topic areas caught you most off guard? Was it a specific domain (ingestion/orchestration, monitoring/optimization, security, real-time/KQL), or more just tricky wording?
• Was there one area in particular that seems to trip up most people, in your experience or from what you've seen others say?
• How would you prepare, knowing what you know about the examn after taking it?
• How exactly did you use the "learn" page? Is there a strategy to this? It seems really hard to look things up in in general.
• The case study section is the part I feel least prepared for - it doesn't seem to map cleanly onto any single MS Learn module, more like it's testing whether you can pull the whole architecture together. What's the best way to prepare for that specifically, and is there any way to practice it beforehand, or do you just walk in and figure it out?
• Is CertLabs a reasonable stand-in for the real exam in terms of difficulty and question style, or does it skew easier/harder/differently than what you actually got on exam day?

Any tips on what to prioritize in these last stretches would be hugely appreciated.

This might be a little divisive, but what about bringing C# and .NET Core into Fabric somehow? I remember when you could use C# + Spark in Synapse, and honestly I didn't use it nor do I think it was a successful feature.

However, in Fabric, why not adopt .NET Core + C# in some way for a new notebook or workload type? It doesn't need to be Spark, but it would be a great selling feature and allow onboarding Microsoft based developers pretty easily who've been developing ETL and have been doing data engineering with .NET and SQL Server for ages.

It could be a new workload type altogether (again, not wanting it on top of Spark, but a new workload entirely and/or non-Spark notebook).

This is not a request, but merely a discussion and to feel out where the Microsoft product team could be going in regards to .NET Core + Fabric.

Thoughts?

Which of these is decimal and which is double?

Decimal or Double

This is a SQL EP for a lakehouse.

You might think you know the answer but you would probably be wrong. Here are the types from the corresponding types from the dataframe in spark (printSchema):

|-- Number_of_Packages: double (nullable = false)

|-- Pieces: decimal(22,8) (nullable = true)

Was this intentional? It seems very confusing. Almost like when PBI desktop's PQ environment shows "$" on top of all my numbers, regardless if they are currency-related or not.

When Fabric first announced dbt support, the feedback from this community was pretty consistent:

• "When will Lakehouse/Spark be supported?"
• "It's not useful without pipeline activities."
• "We need better Git integration."
• "We want to bring existing dbt projects into Fabric."
• "Don't make me rebuild everything from scratch."

Over the last few months, several of those asks have landed.

We've been working through that feedback, and we're excited to share the latest updates.

What you can do today

• Fabric Lakehouse Adapter — Run dbt projects directly against Fabric Lakehouses, bringing transformation logic closer to where data is prepared and stored. Build transformation layers on top of Lakehouse data, apply dbt models and tests, and align analytics engineering practices with Lakehouse-first architectures.

Blog link: Introducing Fabric Lakehouse support in dbt job fo... - Microsoft Fabric Community

https://preview.redd.it/x0oz46vjv2kh1.png?width=903&format=png&auto=webp&s=f596629240eef368c1e563991fcc9f14de61a1ef

• dbt Activity in Pipelines — Trigger dbt jobs as a first-class Pipeline activity and incorporate transformations into end-to-end orchestration workflows. It supports parameterization and allows selective model refresh in a metadata driven manner.

Interesting Fact: 33% of dbt job are run through the dbt activity in pipelines.

Doc link: Run dbt Job Activity in Fabric Pipeline - Microsoft Fabric | Microsoft Learn

https://preview.redd.it/kg9c7tenv2kh1.png?width=903&format=png&auto=webp&s=e888f3605bbdda8f78b1b5e853ba08e6d2dbb90d

• Import Schema — Bootstrap projects from existing Warehouse and Lakehouse assets by generating sources and metadata automatically.

https://preview.redd.it/5x07jo6tv2kh1.png?width=889&format=png&auto=webp&s=e47b4d7e8aa3d6f8529506ce0973a5a4ef5b6f24

• · Runtime v1.11 — Support for newer dbt Core versions through the latest runtime

https://preview.redd.it/149bx0awv2kh1.png?width=903&format=png&auto=webp&s=2f7e4e54f42e741f0cc6f3509f023dee8dfd69a9

• Import Existing Projects from GitHub — Bring existing dbt projects into Fabric and get started quickly without recreating project structures from scratch.

Docs link: Run dbt Projects from a GitHub Repository in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

https://preview.redd.it/jtqe0ln4w2kh1.png?width=774&format=png&auto=webp&s=5847d071b2259e1d9b72ea791cabcd5213447d11

• Public Package Support — Leverage community-maintained dbt packages, macros, and utilities directly within Fabric dbt projects.

https://preview.redd.it/fqnmtct6w2kh1.png?width=903&format=png&auto=webp&s=ccb63bb780a3a792ecb7119edde63b728782428d

What's on the horizon

• Monitoring Enhancements - Fine-grained step-by-step visibility into background execution, see whether dbt is installing dependencies, compiling, or executing models in real time.
• CI/CD Improvements — Multi-profile target support across environments and Variable Library integration.
• Docs Generate & Source Freshness — Native support for dbt documentation generation and source freshness monitoring.
• dbt Fusion + Versioning — Support for dbt Fusion and project versioning to simplify upgrades and reduce migration effort.
• Workspace Identity / SPN Support — Service Principal and Managed Identity support for enterprise authentication and automation scenarios.

Looking back at the original feedback

A lot of the top asks from the original dbt Job in Fabric announcement included:

1. Lakehouse support

2. Pipeline integration

3. Git-based onboarding

4. Public package support

It's great to see many of these capabilities now available, with additional investments already on the roadmap.

We'd love to hear what you're building with dbt on Fabric and what you'd like us to focus on next.

#MicrosoftFabric #dbt #dbtJobs #DataEngineering #AnalyticsEngineering #DataFactory #Lakehouse #OneLake #GitHub #CICD #FabricCommunity #pipelines #monitoring #UX #UI #dbtdocs #versionupgrade #Fabric #GA #dbtGA #updates #features

This post is a space to highlight a Fabric Idea that you believe deserves more visibility and votes. If there’s an improvement you’re particularly interested in, feel free to share:

• [Required] A link to the Idea
• [Optional] A brief explanation of why it would be valuable
• [Optional] Any context about the scenario or need it supports

If you come across an idea that you agree with, give it a vote on the Fabric Ideas site.