u/Personal-Quote5226

https://preview.redd.it/0cv0syf3t6kh1.png?width=630&format=png&auto=webp&s=c1ec0c97350cf0f408d7e1ae3edcd38eb2f5a60f

Looking to update the workspace for a semantic model, and I found this method which should work for Direct Lake lakehouses.  

import sempy_labs as labs
labs.directlake.update_direct_lake_model_lakehouse_connection(
    dataset="My Semantic Model",
    workspace="Super Workspace",
    lakehouse="Reporting",
    lakehouse_workspace="Super Duper Workspace"
)

It runs successfully, and it tells me that 

🟢 The expression in the 'My Semantic Model' semantic model within the 'Super Workspace' workspace has been updated to point to the 'Reporting' lakehouse in the 'Super Duper Workspace' workspace.

Perfect.

However, the actual binding result is weird.  

(See screenshot)

This is what it was before:

Source name: tablename
Source schema: dbo
Storage mode: Direct Lake
Data source type: Lakehouse
Server name: Lakehouse Name
Workspace: Super Workspace

I get that it's sempy-labs, but this functionality is documented as working.  Has anyone tried anything similar?

This might be a little divisive, but what about bringing C# and .NET Core into Fabric somehow? I remember when you could use C# + Spark in Synapse, and honestly I didn't use it nor do I think it was a successful feature.

However, in Fabric, why not adopt .NET Core + C# in some way for a new notebook or workload type? It doesn't need to be Spark, but it would be a great selling feature and allow onboarding Microsoft based developers pretty easily who've been developing ETL and have been doing data engineering with .NET and SQL Server for ages.

It could be a new workload type altogether (again, not wanting it on top of Spark, but a new workload entirely and/or non-Spark notebook).

This is not a request, but merely a discussion and to feel out where the Microsoft product team could be going in regards to .NET Core + Fabric.

Thoughts?

It used to be easy to open a file (video, image) in Files app and save it to my camera roll.

Suddenly, it's now very difficult to do that.

Could we just keep backwards compatibility with new features to not break people's workflows unexpectedly?

I know the supports not there now, but it would be ideal if we could spin up a spark session super fast.

Our workspaces have external integrations that necessitate the use of MPEs which lead to session start up times of 3 to 5 mins.

However, there are a significant number of use cases that don’t rely on any of the MPEs.

We find ourselves spinning up spark sessions, and waiting 5 minutes, and executing a quick job.

Wasted CUs; wasted lead time.

Interactive user frustration.

I know we can use custom pools, HC to mitigate this to a degree but that’s not what I want or prefer.

According to this thread, OptimizeWrite is enabled by default.

https://www.reddit.com/r/MicrosoftFabric/comments/1etyt62/new\_blog\_optimizing\_spark\_a\_deep\_dive\_into/. (u/mwc360 , u/frithjof_v)

It's not enabled from the default WriteHeavy resource profile, however. In fact, it's specifically set to null by the resource profile which would inevitably mean it uses the system default.

A notebook will return that OptimizeWrite is 'unset' by default.

However, that tells us if the spark config was set or not, but doesn't tell us if it's still 'default' set to true underneath the hood.

Given the reddit thread above where Microsoft is acknowledging it's enabled by default, can this be confirmed? Is it documented anywhere?

We don't want to use the WriteHeavy vs ReadHeavy(PBI/Spark) profiles. The reason is that we want most of these to always be set table by table as a table property. Otherwise, every single script has to either set the spark config, or unset the spark configs that are carried over from the Resource Profile (that we don't even want).

Example with WriteHeavy; it's always going to have OptimizeWrite binSize set to 128 and optimizeWrite not set. The other profiles always set vOrder and have optimizeWrite set to true.

We don't want to leave it to the engineers to remember to include overriding these values in each notebook because inevitably it will grow to be inconsistent and difficult to audit/verify over time.

Table properties take the guess work out of it and ensure that any scripts that do set or unset the config are intentional rather than just carried over from a Resource Profile that we didn't even want.

For others, like BinSize, we can override it in the spark session settings in the environment, but our tests show that even after deploying those settings, waiting a while, the config remains set to 128 and not our spark setting we provided in the environment.

When could there be a solution to this problem that doesn't involve custom code in each notebook to work around it?

Ideally:
--> Get custom resource profiles actually implemented, so that we can use them

--> Make resource profiles optional (so they don't override our custom spark settings in the environment) and can more easily allow us to use stable level properties without unsetting spark configuration in notebooks.

If we set or unset a spark configuration in a Notebook that is part of a HC session will this propagate to all notebooks in the same HC session?

I understand each notebook runs in an isolated REPL core, but I didn't see any reference regarding session level configuration ex:spark.conf.set("spark.microsoft.delta.optimizeWrite.enabled", "true").

As a follow up to my question here, https://www.reddit.com/r/MicrosoftFabric/comments/1oqzs8l/as_a_fabric_administrator_why_cant_i_see/ has anyone made any recent progress of getting access to a full inventory of connections in Fabric tenant (full inventory, not just ones associated to a Gateway).

A user leaves an org, we can't see their connections. They are dead, stuck dead in Fabric forever.
A user sets up a data exfiltration scheme and we have no way to audit or know the connection they used is even there; everything looks legit because you can't see what the connection.
Failed audits or upset auditors & compliance teams.
CISO getting upset.

..etc

Even for connections we can see, a Fabric administrator won[t even be able to take ownership unless explicitly granted to them; the owner leaves the org, that connection can be seen, but not even a Fabric admin can do anything with it. Connection becomes dead, maybe even used, but we can't administrate it.

Are there back door channels we can use to address this by getting access to a full list of connections and allowing administrators to take ownership of specified connections through some backdoor/service request or something else?

Is there any meaning to Workspace Identity Authorized Users? As far as I can tell it just lists the users that have access to the workspace. If you can contribute to the workspace, you are an authorized user ("can use identity"). If you are an admin, you can "edit all members".

Will there be some future meaning here? It gives the appearance of being able to restrict who can use an identity, but there doesn't seem to be any actual function behind it.

Will there be (or is there already some way) to prevent users from being able to use a WI other than making them a 'Viewer' which inherently doesn't allow doing much of anything anyway?

We're noticing the unexpected behaviour of a Fabric pipeline running on schedule but executing (submitted by) under the user who created or updated the schedule and not the last modified user of the pipeline.

If a user updates the schedule, it sets last modified by of the pipeline to that user.

We then update the pipeline to set the description by the SPN. Last modified by is set as the SPN.

When the schedule is triggered, the pipeline is 'submitted by', and therefore executed by the user who created the schedule and not the user who last modified the pipeline.

Therefore, the documentation from Microsoft regarding setting the description by an SPN in order to run a pipeline as that SPN appears to not actually work when scheduling the pipeline.

Can this be fixed? Is this expected?

I'm able to easily run a pipeline under an SPN security context when that pipeline last modified by is set to the SPN.

It also runs no problem under the SPN security context as a Viewer and without any workspace access at all to the workspace the pipeline is running in.

I understood that in order to run a pipeline the user context of that pipeline requires contributor or higher. For interactive pipeline runs from the UI that holds true.

However, it doesn't hold true for all scenarios and specifically when a pipeline is scheduled and executing under a user context that does not have Contributor access or higher to the workspace or does not have any access to the workspace at all.

Is this going to be the expected behaviour going forward?

If you have a small fact table that was about 1GB, would you use targetFileSize of 128m, 400m, or 1024m on the table if the goal was fast performance for PowerBI.

I'm looking forward to hearing about your experience here...

Perhaps my statement (title) is wrong, however, we are experiencing the same behaviour experienced here https://community.fabric.microsoft.com/t5/Data-Engineering/Change-Data-Feed-bug-Unable-to-reconstruct-state-despite-recent/td-p/4663530

As described here, https://www.mssqltips.com/sqlservertip/7962/microsoft-fabric-automatic-table-maintenance-checkpoint-statistics/ checkpointing is an optimization.

Official documentation across the board mentions the need for logRetentionDuration and file retention:
https://learn.microsoft.com/en-us/fabric/data-engineering/delta-lake-time-travel?tabs=sparksql

No indication of needing checkpointRetention or its impact.

Checkpointing is described as an optimization across the board in official channels, however those channels don't indicate that CDF/TT will fail if the optimization files get removed.

However, CDF fails if checkpoint files have been removed within the log retention window. Take the following scenario:

We have both the delta logs (with a long retention) and the unreferenced parquet files (not vacuumed and a long retention) for the CDF version. However, previous checkpoints have been automatically removed due to the default checkpointRetentionDuration being 2 days.

Therefore, retrieving CDF versions of records are always working if done within 2 days of the transaction, but always fail if done after 2 days even though we still maintain the delta log and the unreferenced files.

Many user forums indicate users having this same experience where the solution is to set checkpointRetentionDuration to the same duration as your logRetention.

Is this how it's supposed to work? Is this a confirmed bug? If it's by design, can Microsoft document it somewhere?

I imagine that the failure is due to reading the delta log, seeing the checkpoint, and no longer being able to reference the checkpoint file and instead of traversing the log for the cumulative changes (which it would do normally), it just fails. Considering the optimization isn't actually needed, it shouldn't fail there. It should revert to traversing the logs/parquet files for TT and CDF.

I’ve been involved in a few different discussions lately where setting Last Modified By on a pipeline came up and it has been noted that it works for SPN but not WI.

In case anyone is curious, it does actually work fine to set the last modified by of a pipeline to the WI.

When triggered via schedule, it runs as the WI security context.

I don’t suggest you do it necessarily, but if you want to run something that’s completely secretless, it should work fine.

The other options:
User accounts tokens will fail eventually, so that’s no good.

SPN token expiry mechanism isn’t documented, so last modified by SPN works but you have to periodically reset the SPN token as per many users experiences.

WI doesn’t have passwords or secrets. Thus, we can assume it won’t just stop working due to not being able to get a token.

Drawback: WI scoped to the workspace

Update from u/banner650 :‘Unfortunately, at this point in time, WI is treated like an SPN if it ends up as the Last Modified By for a Pipeline (or other item) and uses a captured refresh token that can expire. As a result, it is no more durable than an SPN. We have some work in flight right now to improve this situation, but I don't have a timeline that I know that I can share yet for it.’

Giving Workspace Identity access to anything allows anyone in the workspace with, or who gains, Contributor access in the workspace access to anything that the WI can access.

Authorization to anything outside the workspace should be strictly guarded.

Workspace Identity make it easy for anyone with contributor rights to access secrets, data sources, resources, etc that the WI can access.

I like WI. Sometimes we should still consider SPNs.

Do you agree? (I appreciate everyone’s response. This is mostly an exercise to help clarify thinking about WI vs SPN.)

Workspace Identify no longer defaults to providing 'Contributor' access to its workspace although you can grant 'Contributor' via 'Manage Users/Roles'.

Are there use cases that actually work with Workspace Identity that do not require Contributor access? Without contributor access, you can't execute things. If you can't execute things, WI security context is moot.

Am I missing something?

A client had a notebook that was a pretty simple MLV refresh. It finishes in less than 30 seconds.

It ended up running for 16 hours before session timeout.

Workspace session timeout is set for 6 hours (long I know, but that's not 16 hours).

It was not an HC session.

Job confirmed in spark to take less than a minute. The rest of the time was idle running up compute during that whole time.

Has anyone observed this behaviour? Any recommendations?

As per the MLV Refresh Activity documentation: "The Refresh Materialized Lake View activity refreshes a materialized lake view as part of a Fabric pipeline."

There is nothing in the documentation that shows you where you select the mlv that you would like to refresh.

In the actual implementation, you can select a Workspace and a Lakehouse.

There is no option in the connection or activity to select the actual materialized lake view that you wish to refresh.

I get that it's a preview feature, but I assumed that base functionality would be working? The fact that the base functionality isn't even documented in the Microsoft reference article indicated to me that the feature wasn't even complete or validated when the documentation was written?

https://learn.microsoft.com/en-us/fabric/data-factory/refresh-materialized-lake-view-activity

https://preview.redd.it/baypv2b0iedh1.png?width=1478&format=png&auto=webp&s=3e7d930ae09eb6a0224ed3280b236eb14cfc03b6