Read the announcement regarding Databricks Unity Catalog and how secrets can now be stored under a schema as opposed to using secret scopes backed by a secrets manager e.g. Azure Key Vault.
Does anyone know if the secrets are stored in plaintext in bundle state if using the direct deployment engine mode for DABs?
I need to maintain Terraform state in Gitlab and am glad that Gitlab supports both Terraform State and Modules. The challenge here is that Terraform state can contain secrets and in a highly regulated industry nobody, even maintainers, should be able to view Terraform state if it contains secrets.
Is there a dedicated/custom role in Gitlab that can be used to gate access to Terraform state?
Some may say that the Terraform state should not contain secrets, and understandably so...But if one were to consider it how would you gate access to it in Gitlab?
Business users (no technical background e.g. no Python or SQL) want to participate in the Silver to Gold Transformations.
First of all, I don't think business users shd manage schemas. But, if we entertain them being able to participate in the silver to Gold transformations, how would you do that taking into account their no code background?
I was thinking Lakeflow Builder and checking in code to Got for version Control. But that may be too technical for them too...
What's a workaround for the above? Can Databricks accommodate separate Databricks Accounts per Business Unit (BU) in a Microsoft Tenant?
Also, is there documentation on Azure for what data is exactly stored in the metastore?
Why? Regulatory constraint
I want ppl to be able to create bundles through Git Folders but not deploy them. How do I achieve this in Databricks?
This is necessary because of the regulatory environment my team operates out of where PRs need to be approved before DABs can be deployed via SPs in a GitHub pipeline.
Hi All,
Unity AI Gateway has many features I want to introduce to my company. The challenge is that it is still in "Beta" which means that even if I test it and it looks promising, introducing it even in dev will be an issue bc of pushback from Security teams.
Is there an expected timeline for Unity AI Gateway to come out of Public Preview? Not sure if I missed this announcement at DAIS 2026. Hoping that this offering is available in public preview soon.
EDIT: I'd also be curious to hear from those who have Unity AI Gateway enabled whether this is comparatively costlier compared to Microsoft AI Foundry / other Agent Registry Frameworks / AI Governance tools.
For those who use Workload Identity Federation for their Databricks CI/CD workloads, do y'all use a Service Principal or a User Assigned Managed Identity underneath it? And why?
Databricks documentation states creating a federation policy for Service Principals. But wanted to clear this up since Managed Identities on Azure are treated as "Service Principals" in the Databricks Account.
I'd lean towards User Assigned Managed Identities since it removes the need to manage secrets (rotation, storage of secrets securely) and issues short lived tokens.
Hi there,
I am in the midst of rolling out a project where I need to sync nested groups in Microsoft Entra ID. I only need to sync one level deep i.e. sync the parent group and all the children groups nested within the parent group (the children group themselves do NOT contain groups).
I read some Gitlab documentation on "SAML Linked Groups" and was wondering whether Gitlab is able to support the above requested functionality.
Thank you!
I am unable to create Git folders in my Databricks workspace.
I have generated a PAT on Gitlab, given it the required scopes, verified I can clone a repo using the PAT on my VM.
But when I test on Databricks, I get a 403 error. My Gitlab admin has allow listed the Databricks control plane IP Addresses.
We use Gitlab with internet accessibility (secured with an allow list, no firewall).
Has anyone faced this and managed to resolve it? I also listed my Gitlab Enterprise base URL in the Databricks workspace settings to "only allow push/pull ..."
I am currently building a Web App that is powered by data stored in Databricks, specifically Managed Delta Tables. What is the recommended pattern for fetching this data?
- A direct link to the ADLS Gen2 Path where the table is stored (read-only access to UC-managed storage)
- Using the UC RESTful API - concern here is that this adds an additional hop to the request and adds latency - the web app will contain dashboards/data that when filtered should quickly show updated data
- Caching layer - Redis to Cache Delta Table data with TTL of one day bc data is also refreshed nightly
Would you design the system similarly? What would you do differently?
Read up on auth options to MS SQL Server as a source DB and see that only basic auth is supported. How about integration with Entra ID as a IDP and using U2M or M2M?
I had assumed that since this is GA, there'd be more auth options. I'm hoping I'm missing something
I am working out of workspace A where I have created Catalog A. I want to bind Catalog A to workspace B. However, when editing the Catalog Workspace Bindings of Catalog A in Workspace A, I am unable to see Workspace B as a potential workspace for the binding. What am I missing here?
Could it be that workspace B was created with a separate metastore? What is the Databricks configuration/object that enables the ability to view other workspaces to bind a catalog to?
If I want to ingest CDC data from a database such as MSSQL Server, which what's the difference between using Lakeflow Connect and a Spark Declarative Pipeline? What are the advantages of using one approach over another?
I was reading the above documentation to deepen my understanding of how DABs in direct mode manage state (recall that direct mode no longer uses Terraform under the hood and instead uses the Databricks Go SDK under the hood).
This is a snippet from the documentation that's pertinent to this discussion:
"The resources state diff calculation is done in two steps:
The local bundle configuration is compared to the snapshot configuration used for the most recent deployment. The remote state plays no role.
The remote state is compared to the snapshot configuration used for the most recent deployment.
The result is that:
databricks.yml resource changes are never ignored and will always trigger an update.
Resource fields not handled by the implementation do not trigger an inconsistent result error. These resources are deployed successfully by the direct engine, but this can result in a drift. The deployed resources are updated during the next plan or deploy."
The qn I have is, I want to manage grants on a UC Securable object, say a catalog by using two bundles. One bundle has Business Unit (BU) specific grants i.e. only Entra Groups belonging to the BU get granted permissions on the BU catalog.
The other bundle contains cross-BU grants i.e. grants on that catalog belonging to a certain BU to Entra Groups for another BU.
The reason for the above separation is because of a difference in Governance process where cross BU grants must be approved by a central governance committee whereas BU specific grants only need to be approved by BU specific data stewards.
The qn is, does the above model of using two bundles (local BU grants vs cross-BU grants) work? Or does it not bc of the possibility of the bundles being deployed at different times and reconciling against the remote state of the catalog grants?
TLDR: Are mutually exclusive BU grants applied independently in a non-destructive manner through reconciliation of bundle state only with its previous bundle state (as opposed to reconciling with the state of the remote object)?
Do DABs support Catalog/Schema tagging? I need this functionality to trigger GitHub CODEOWNER reviews on the basis of the tags... (InfoSec is looped in for restricted data classification grants)
Would you put knowledge bases in the bronze/silver/gold layer? The raw documents definitely reside in the bronze layer. But if I create AI Agents atop a volume storing the raw documents, then the knowledge base remains in bronze.
However, if I create vector embeddings/do chunking/create a vector search index, then these tables should be in the silver layer.
Am I on the right track?
As per title. Would appreciate any insights on this from the Databricks Team. I need this capability for cross BU Data sharing.
Lakehouse federation typically uses OAuth, this the above qn.
I'm focused on static data masking as opposed to dynamic data masking (where the latter is done through RLS and CLS). I'm also not talking about encryption such as AES 256 where compromising the key compromises the Data.
If not supported natively by Databricks Unity Catalog, what are some partner companies that support tokenization and integrate well with UC?