u/RazzmatazzLiving1323

Unity AI Gateway - expected public preview timeline?

Hi All,

Unity AI Gateway has many features I want to introduce to my company. The challenge is that it is still in "Beta" which means that even if I test it and it looks promising, introducing it even in dev will be an issue bc of pushback from Security teams.

Is there an expected timeline for Unity AI Gateway to come out of Public Preview? Not sure if I missed this announcement at DAIS 2026. Hoping that this offering is available in public preview soon.

EDIT: I'd also be curious to hear from those who have Unity AI Gateway enabled whether this is comparatively costlier compared to Microsoft AI Foundry / other Agent Registry Frameworks / AI Governance tools.

reddit.com
u/RazzmatazzLiving1323 — 3 days ago

Workload Identity Federation on Databricks (CI/CD)

For those who use Workload Identity Federation for their Databricks CI/CD workloads, do y'all use a Service Principal or a User Assigned Managed Identity underneath it? And why?

Databricks documentation states creating a federation policy for Service Principals. But wanted to clear this up since Managed Identities on Azure are treated as "Service Principals" in the Databricks Account.

I'd lean towards User Assigned Managed Identities since it removes the need to manage secrets (rotation, storage of secrets securely) and issues short lived tokens.

reddit.com
u/RazzmatazzLiving1323 — 5 days ago
▲ 2 r/gitlab

Syncing nested groups from Microsoft Entra ID

Hi there,

I am in the midst of rolling out a project where I need to sync nested groups in Microsoft Entra ID. I only need to sync one level deep i.e. sync the parent group and all the children groups nested within the parent group (the children group themselves do NOT contain groups).

I read some Gitlab documentation on "SAML Linked Groups" and was wondering whether Gitlab is able to support the above requested functionality.

Thank you!

reddit.com
u/RazzmatazzLiving1323 — 11 days ago

Databricks and Gitlab - 403 Git Folder Creation

I am unable to create Git folders in my Databricks workspace.

I have generated a PAT on Gitlab, given it the required scopes, verified I can clone a repo using the PAT on my VM.

But when I test on Databricks, I get a 403 error. My Gitlab admin has allow listed the Databricks control plane IP Addresses.

We use Gitlab with internet accessibility (secured with an allow list, no firewall).

Has anyone faced this and managed to resolve it? I also listed my Gitlab Enterprise base URL in the Databricks workspace settings to "only allow push/pull ..."

reddit.com
u/RazzmatazzLiving1323 — 12 days ago

Recommended pattern for fetching Managed Delta Tables from Databricks Unity Catalog for Python Typescript and React Web App hosted outside Databricks

I am currently building a Web App that is powered by data stored in Databricks, specifically Managed Delta Tables. What is the recommended pattern for fetching this data?

- A direct link to the ADLS Gen2 Path where the table is stored (read-only access to UC-managed storage)

- Using the UC RESTful API - concern here is that this adds an additional hop to the request and adds latency - the web app will contain dashboards/data that when filtered should quickly show updated data

- Caching layer - Redis to Cache Delta Table data with TTL of one day bc data is also refreshed nightly

Would you design the system similarly? What would you do differently?

reddit.com
u/RazzmatazzLiving1323 — 29 days ago

MSSQL Server Lakeflow Connect

Read up on auth options to MS SQL Server as a source DB and see that only basic auth is supported. How about integration with Entra ID as a IDP and using U2M or M2M?

I had assumed that since this is GA, there'd be more auth options. I'm hoping I'm missing something

reddit.com
u/RazzmatazzLiving1323 — 1 month ago

Unable to see other workspaces in current workspace to bind catalog to

I am working out of workspace A where I have created Catalog A. I want to bind Catalog A to workspace B. However, when editing the Catalog Workspace Bindings of Catalog A in Workspace A, I am unable to see Workspace B as a potential workspace for the binding. What am I missing here?

Could it be that workspace B was created with a separate metastore? What is the Databricks configuration/object that enables the ability to view other workspaces to bind a catalog to?

reddit.com
u/RazzmatazzLiving1323 — 1 month ago

Lakeflow Connect from MSSQL Server vs Apply Changes Into (Spark Declarative Pipelines)

If I want to ingest CDC data from a database such as MSSQL Server, which what's the difference between using Lakeflow Connect and a Spark Declarative Pipeline? What are the advantages of using one approach over another?

reddit.com
u/RazzmatazzLiving1323 — 1 month ago

DABs - state resolution nuances

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/direct#resource-state-diff-calculation

I was reading the above documentation to deepen my understanding of how DABs in direct mode manage state (recall that direct mode no longer uses Terraform under the hood and instead uses the Databricks Go SDK under the hood).

This is a snippet from the documentation that's pertinent to this discussion:

"The resources state diff calculation is done in two steps:

The local bundle configuration is compared to the snapshot configuration used for the most recent deployment. The remote state plays no role.

The remote state is compared to the snapshot configuration used for the most recent deployment.

The result is that:

databricks.yml resource changes are never ignored and will always trigger an update.

Resource fields not handled by the implementation do not trigger an inconsistent result error. These resources are deployed successfully by the direct engine, but this can result in a drift. The deployed resources are updated during the next plan or deploy."

The qn I have is, I want to manage grants on a UC Securable object, say a catalog by using two bundles. One bundle has Business Unit (BU) specific grants i.e. only Entra Groups belonging to the BU get granted permissions on the BU catalog.

The other bundle contains cross-BU grants i.e. grants on that catalog belonging to a certain BU to Entra Groups for another BU.

The reason for the above separation is because of a difference in Governance process where cross BU grants must be approved by a central governance committee whereas BU specific grants only need to be approved by BU specific data stewards.

The qn is, does the above model of using two bundles (local BU grants vs cross-BU grants) work? Or does it not bc of the possibility of the bundles being deployed at different times and reconciling against the remote state of the catalog grants?

TLDR: Are mutually exclusive BU grants applied independently in a non-destructive manner through reconciliation of bundle state only with its previous bundle state (as opposed to reconciling with the state of the remote object)?

u/RazzmatazzLiving1323 — 2 months ago

Tagging Catalogs and Schemas using DABs

Do DABs support Catalog/Schema tagging? I need this functionality to trigger GitHub CODEOWNER reviews on the basis of the tags... (InfoSec is looped in for restricted data classification grants)

reddit.com
u/RazzmatazzLiving1323 — 2 months ago

Knowledge bases in medallion architecture

Would you put knowledge bases in the bronze/silver/gold layer? The raw documents definitely reside in the bronze layer. But if I create AI Agents atop a volume storing the raw documents, then the knowledge base remains in bronze.

However, if I create vector embeddings/do chunking/create a vector search index, then these tables should be in the silver layer.

Am I on the right track?

reddit.com
u/RazzmatazzLiving1323 — 2 months ago

Catalog Workspace Bindings GA timeline in DABs?

As per title. Would appreciate any insights on this from the Databricks Team. I need this capability for cross BU Data sharing.

reddit.com
u/RazzmatazzLiving1323 — 2 months ago

I'm focused on static data masking as opposed to dynamic data masking (where the latter is done through RLS and CLS). I'm also not talking about encryption such as AES 256 where compromising the key compromises the Data.

If not supported natively by Databricks Unity Catalog, what are some partner companies that support tokenization and integrate well with UC?

reddit.com
u/RazzmatazzLiving1323 — 2 months ago