u/SmallAd3697

If a developer is comfortable with Spark on their desktops, on k8s, and on databricks, then should these Spark skills be easily transferred to Fabric?

I'm struggling with Spark in Fabric. People love Spark because it is a technology which is developed out in the open, and is well documented, and there are massive online communities who will assist with questions when a user runs into problems.

Yet this Spark in Fabric breaks those patterns and can be pretty disorienting. I am not talking about NEE or anything super-duper-top-secret or proprietary. (I normally disable that stuff, in any case). I'm simply talking about running a spark job, and interpreting the logs when something is misbehaving. This should be well within the reach of a conventional Spark developer. Right?

Lets take a simple example The pyspark notebook cell has hung, and when I look in the logs and it says this....

2026-08-19 14:39:16,055 INFO RpcAppSender [shutdown-hook-0]: RPC app sender closed
2026-08-19 14:39:16,055 INFO RpcAppSparkContextServer [shutdown-hook-0]: Closing remote SparkContext service at 10.0.yyy.xxx:18083, remoteSparkContext/remoteSparkContextEndpoint

What is this stuff? What is RpcAppSender? What is RpcAppSparkContextServer? If this stuff is printed all over my logs, shouldn't I be able to find a single search result on google? If something called a Spark Context is being "closed", then isn't that a message that a normal spark developer should be able to interpret/investigate by googling the log message?

I'm not objecting to Microsoft's prerogative to extend Apache Spark and innovate, and such. But the bothersome thing is that these extensions are undocumented and secretive and developers are at a loss to self-support. Any Spark developers coming from another ecosystem are not properly equipped to troubleshoot Fabric-specific behaviors. It is disorienting and unfamiliar. Even the log messages themselves are confusing; the purpose of logs should NOT be to create even more confusion. I'm almost of the opinion that Microsoft should stop calling their product Spark if the product is twisted to the point where even the log messages are unrecognizable and not meaningful for troubleshooting.

Sorry for the long rant. In the very least, there will be a search result the next time someone searches for "RpcAppSparkContextServer". As of today, that term was never heard, across the entire internet! One might think I made it up myself. lol.

Which of these is decimal and which is double?

Decimal or Double

This is a SQL EP for a lakehouse.

You might think you know the answer but you would probably be wrong. Here are the types from the corresponding types from the dataframe in spark (printSchema):

|-- Number_of_Packages: double (nullable = false)

|-- Pieces: decimal(22,8) (nullable = true)

Was this intentional? It seems very confusing. Almost like when PBI desktop's PQ environment shows "$" on top of all my numbers, regardless if they are currency-related or not.

I'm trying to use pyspark in fabric.

I keep having issues with my notebook, where the cells won't complete unless I cancel them. The wait cursor will continue to run until I click the little cancel button on the side.

I never see this behavior in the notebook cells on the databricks platform.
When I look in the "logs" shown under the cell, Spark seems to be displaying this over and over, and wont stop:

>

>2026-08-17 21:33:00,714 INFO YarnAllocator [Reporter]: Updating resource requests for ResourceProfile id: 0, target: 4, pending: 0, running: 4, executorsStarting: 0

... and occasionally this as well:

>2026-08-17 21:40:49,987 INFO KustoLogger [external-catalog-metrics-1]: type=METER, name=HiveExternalCatalog::dropTable::failure, count=0, m1_rate=0.0, m5_rate=0.0, m15_rate=0.0, mean_rate=0.0, rate_unit=events/second

If anyone has any ideas about what is going wrong, please let me know. The notebook is behaving like there are no executors, even though there are. Is there a way to recreate all my executors mid-notebook?

My son had an internship this summer in Alaska (Wasilla). He is 20 years old. He felt alarmed when he got sick with chills and a fever and went to urgent care or ER at a regional medical center.

They proceeded to give him every blood test known to medicine (eg. TROPONIN, QUANT?) And billed $14k to my insurance. They gave him fluids and sent him home.

He felt normal again, and worked the next day. There is no way he would have know that this trip to a doctor in Alaska would cost $14K. I have often vented my frustration about the insane nature of our healthcare system in the past, but he has little first-hand experience and was making rash decisions when he was feeling poorly.

After the dust settled, my plan discounted 7K, paid about 3K and wants us to pay 4K.

Now that this appears on my insurance, is there a way I can avoid paying the 4K and make them collect from him instead of me (and/or foul his credit not mine)? Seems like a reasonable course of action if he is no longer a minor. His credit is pristine, and maybe his future voting on health care law will allow this to be removed from his credit one day.

Is it dishonest to allow the plan to pay 3K and not pay my side? Is it dishonest for me to allow them to allocate this unpaid bill to my deductible?

As a side, I'm guessing the main problem is that he went to an ER instead of urgent care? Not sure, but I think an urgent care would have billed him a number I can stomach.

In the past our Excel Pivot Tables were the foremost tool for retrieving data from semantic models. Nowadays they seem to be the red-headed stepchild.

I see examples of regressions on a regular basis.

Today I noticed that Excel will no longer refresh the field list to detect new metadata changes. This should be happening when using the "refresh" command. It is a pretty fundamental requirement, since schema can change regularly (new fields added to existing tables).

Is this issue something that the Microsoft ASWL team is likely to care about? I can open a support case but it is likely to take a tremendous amount of effort, and would not want to invest in it if Microsoft is already aware (and doesn't care). There are other scenarios that demonstrate a diminished level of support for Excel nowadays.

Microsoft 365 MSO (Version 2607 Build 16.0.20228.20188) 64-bit

UPDATE: the field wasn't in the "perspective" used by this workbook, even though it was in the model itself. After adding to the perspective, the field came thru as expected.
I still stand by my original sentiment that Excel isn't getting prioritized as highly as it should nowadays.

Does anyone use the power bi connector that allows a model to be pointed at a dbt semantic layer? I never see this discussed so I'm assuming it isn't very popular.

I do see people discussing the use of dbt with Fabric DW and Fabric Spark. I'm guessing these are used by up to 5 pct of the Fabric community. (ie. relatively popular)

My impression is that the people who are using dbt in their data pipelines are probably landing data in a lakehouse before pulling it into a fabric semantic model. Im guessing it will be a long road before semantic models and dbt semantic layers are going to play nice. Also it seems like the power bi connector requires a dbt account and is probably not open source (let me know if this assumption is wrong).

Im not actively using dbt today, just curious.

Does everyone love the "multi-tasking" user interface?

I find myself fighting it constantly, and not using any of its benefits (if there are any). I've never liked using browser-hosted IDEs. This one is stretching the concept a LOT further than I'd like. All of my "multi-tasking" can be handled by the top-level tabs in the browser.

The Fabric multi-tasking is just weird. If I try to close my stuff, the things I worked on two hours ago keep following me around and I'm forced to dispatch them in a manual way. I spend lots of time micromanaging this U-I.

One of the silliest problems I have is that DF GEN2 will only allow three open dataflows at any time, and they are often buried in different windows and "fabric tasks", so I have to go on an easter egg hunt to find and close them before I'm allowed to get back to work.

Does anyone know how to disable this multi-tasking altogether? It is just maddening. I'm a simple guy when it comes to browser-hosted developer experiences. I only want one thing happening in each of the browser tabs. If Microsoft wants to create a high-quality VS-Code extension for Fabric, then I will have an appetite for any advanced U-I they may offer. But this effort to create a complex IDE inside of a web page is just nuts.

Does anyone know what logic is used to show that status marker in a DF GEN2? There is often a nagging red icon saying a validation error is in effect - even though it is not necessarily true. See below.

https://preview.redd.it/fqozq27blejh1.png?width=553&format=png&auto=webp&s=d734a548e68515b7fed9569d57621f488f08f7df

When I use the "check validation", the dialog says there is no validation problem. You can see the timestamps say that a validation error happened at 8/13 8:44 PM and another successful validation happened at 8/13 10:20 PM.

WHY am I being nagged about an irrelevant validation failure that happened at some arbitrary time in the distant past? These nagging indicators need to be removed, if they are raising alarms about a problem that doesn't exist anymore. There are real problems to worry about, and these bogus errors aren't one of them.

This should be an easy issue for Microsoft to investigate. Microsoft can search for whatever data is presented in these two places and find out why they aren't consistent. What I typically find in this situation is that there are two different PG teams involved in some way, and they just can't seem to get on the same page. (in this case, maybe one team manages the green icon - DF Team? - and a totally different team manages the red icon - Scheduling Team? and they just don't seem to have a coherent strategy to determine if validation is succeeding or failing.) Does that sound about right?

This shouldn't be so hard. What am I missing? What am I doing differently than the other DF GEN2 users?

For many, many years the DF GEN1 stuff would only support the output/storage of CSV files. That happened under the hood and there was nothing we could do to control it. It was very curious to me how the CSV format was put on a pedestal, when there are so many better options these days.

Finally we started noticing the use of parquet/deltalake under the hood, even without a custom destination.

And nowadays we can use a "custom destination" save to tables in a lakehouse, of course (deltalake table).

But lets assume that for a certain scenario we determine deltalake table is overkill; and we just want to drop my GEN2 output into parquet, under the "Lakehouse Files" destination. How can I accomplish that? When I click the "Lakehouse Files" option, it only supports CSV. Once again. Can folks tell me what is so wonderful about CSV files that this is always the default behavior for dataflows? Isn't it time to start defaulting to parquet or perhaps even json or avro?

If someone knows how to drop parquet from a DF GEN2 into Lakehouse Files, please let me know. Maybe there is a back-door edit to the code or something...

The docs say that ingestion gateways need to run continuously, when using "lakeflow managed connectors" for CDC:

https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/cdc-overview

https://preview.redd.it/4tm5vk5cumih1.png?width=904&format=png&auto=webp&s=c5470b4edf1b0bdd7d5729f90ad91d7dfab652c7

I've heard anecdotes from folks who say they can also launch this on an interval rather than continuously. And some time ago I also heard a Databricks rep saying that the product was evolving and that continuous ingestion would not always be a requirement.

Is there anything official on the roadmap to allow the ingestion gateway to run on an interval, and so we can decrease our compute costs? I realize there are risks involved when it is NOT running continuously (eg. transactions log files will grow, and the source database can't truncate its inactive transactions from the log)

However I think the responsibilities for managing these risks, and balancing the costs, should be left with the customer. If there is a source database that is being updated infrequently, then it is possible that the compute on the databricks side would cost even MORE money than the source database itself. That seems silly and unnecessary.

I'm surprised to see ADF so frequently in the reference architectures for Databricks.

First of all, ADF is not everyone's most popular tool. Secondly, even Microsoft is trying to evict customers from ADF, and is sending them over to Fabric. Thirdly, it seems long overdue for Databricks to have their own tool for the initial landing of data in the cloud, from an on-prem data source.

Whenever I see reference architectures like this in 2026, it is a bit shocking to me. I would have thought ADF would have a replacement by now, for customers that are databricks-centered:

https://preview.redd.it/eueufhnkblih1.png?width=617&format=png&auto=webp&s=b2c0b0df2856319f0878e763e528b2bc8868f5a1

I keep getting spam called every day by some sort of digital assistant from AT&T. When I pick up, there is no option when they call for taking my number off the list. It is hell.

I originally initiated by calling to check on the availability of fiber. (I learned they don't have it on my street.) Now I'm being continually tortured by their digital assistant.

This stuff needs to be illegal. If a company is going to initiate a call with me, there should be a person on the other end. Nobody wants to be getting calls from a digital assistant, without an option to be removed from the list.

What is up with the "mirroredwarehouses" in Fabric URL's, eg:

groups/123-whatever/access/mirroredwarehouses/123-whatever/directAccess

.. I would have thought that SQL EP was independent of Fabric mirroring.

Does anyone understand the history of why that "mirroredwarehouses" is in there? That part of the URL path seems really obscure and out of place. I suppose I shouldn't worry, but what I'm doing has nothing to do with mirroring. I'm just using a SQL endpoint on a vanilla lakehouse.

These SaaS developer tools can be so painful. I'd bet internal Microsoft teams aren't using them regularly enough to work out the kinks.

... otherwise I'd guess that an error like this would have been fixed long ago.

https://preview.redd.it/3uifumsdh8gh1.png?width=825&format=png&auto=webp&s=2f316450475ec000ffe4ebae6ee16c38a31fefda

Receiving an error like this one after waiting an hour is a real kick in the pants. And who gives me a refund for the CU's that have been wasted? Are those refunded to us at the moment of the DF failure??? Lol.

This PQ stuff would be infinitely better for developers if there was a button to retrieve the logs.

... wait a sec... what the ... they finally added a button. It says "Download detailed logs". Although I clicked it and it doesn't work by default: "Detailed logs download is not available because enhanced diagnostics is disabled for this on-premises data gateway. " Doh.

Developing software inside a web page has never been a high quality experience. It certainly isn't much fun for the developer, and now it seems that it is starting to become a problem for Microsoft as well.

Today in NCUS I started seeing a new error for the first time:

Can't open the Dataflow Gen2 (CI/CD) item

"You can only have 3 Dataflow Gen2 (CI/CD) items open at a time and currently there are 3 open already. Close one Dataflow Gen2 (CI/CD) item, and then try to open this one again."

Anyone see this before? As near as I can tell, the error is lying to me. I only have one DF GEN2 open in the browser window. How do I get past this? They need to add a button that says to open anyway. (I'm not sure WHY they think I have other stuff open. But even if I did, it probably isn't important and if it was important it would be captured by a commit in a git repo.)

I'm a newbie on .Net core (but have a lot of experience on Framework). I have a web/http client application that I've been using for years. It connects to a remove http service and transforms the response into a usable format.

I call it in a loop (many thousands of times per minute).

It always worked great in .Net Framework and wouldn't ever blow up. I never once saw it exhaust the outbound ephemeral ports.

Fast forward to now, and the code stops working at scale. After just a few minutes all my ephemeral ports are exhausted. Every round trip to HTTP seems to consume a new port. As I'm digging into the differences between .Net 10 and .Net framework, I discover that the class System.Net.ServicePoint is a faint shadow of what it used to be. There is no connection pooling, and ports are not reused.

How can I bring back the old behavior of ServicePoint? Will I need to refactor the entire application to support connection pooling? Is there any guidance from the Microsoft side to help us migrate apps that used to rely heavily on ServicePoint?

I just found a blog from a year ago where the Microsoft FTE claimed that the onelake shortcuts were able to use private network, and proceeded to step thru the entire demo.

See this blog in the fabric community:
Private ADLS Gen2 access made easy with OneLake Shortcuts: a step-by-step guide

At the end of the journey he clearly ENABLES public networking. You can find that screenshot in the section "Verify The Firewall Exception Has Been Applied" ...

https://preview.redd.it/53bcllpzz2dh1.png?width=802&format=png&auto=webp&s=ea5517d20771082b4c2675cf62aeaeda0c8aa653

In the final conclusion the blogger claims that he showed us how to use a shortcut to "access ADLS Gen2 storage behind a private endpoint". But that is clearly NOT what he showed us, since he ended up enabling public neworking (albeit with firewall rules and trusted workspace access).

The Microsoft proprietary networking tech is challenging enough, without these blogs that are clearly misleading. Why isn't he just acknowledging that the onelake shortcuts are reliant on public internet traffic?

Can anyone share the conceptual purpose behind this xmla.json that appeared in my git repo after syncing a DW in a workspace?

The DW in my workspace does NOT have a default semantic model. So I don't understand why some random TMSL is being produced when working with this DW artifact. Is this intentional? Is it supposed to be used as a sort of template for creating new models in the future?

I think it is a weird hack. I don't mind the fact that Microsoft tools will give us a way to autogenerate this TMSL. But the tooling for this should be found in another place. I don't think there is a way to influence the code of this file, so it really doesn't need to be part of my source control. They might as well generate/commit some random SELECT statements, and some random python scripts for retrieving data out of every table in the DW.

Does anyone know how to minimize or eliminate this weird code file? Oddly I find no authoritative docs about it whatsoever.