r/databricks

Read the announcement regarding Databricks Unity Catalog and how secrets can now be stored under a schema as opposed to using secret scopes backed by a secrets manager e.g. Azure Key Vault.

Does anyone know if the secrets are stored in plaintext in bundle state if using the direct deployment engine mode for DABs?

Anyone used docling Parsing tool on databricks.

Me and my team started using this, though its a great tool. It has its own limitations. Example GILBERt issues happening here and there.

Any suggestions on to use databricks agent bricks ke free docling?

I’m currently new to Databricks, and I’m writing unit tests for a pipeline that is entirely in SQL, with Bronze, Silver, and Gold tables.

I asked how I should approach unit testing, and I was told that since unit tests are typically written in Python, I could wrap the pipeline logic in Python files and then use pytest to test those files.

But I’m confused about how this would work in a real production environment.

For example, if we push everything to GitHub and later make a change to the SQL pipeline, we would also have to make the same change in the corresponding Python files so that the unit tests continue to work. That means maintaining the same logic in two places, which doesn’t seem like a good production practice.

So what is the recommended approach for unit testing a SQL-based Databricks pipeline? Is there a way to test the actual SQL transformations directly without duplicating the pipeline logic in Python?

I’m still learning Databricks, so I’d really appreciate some guidance on the proper production approach.

Hi,

In the past when a new metastore was created, it automatically also did create an external location and storage credentials in the Unity Catalog metastore. Since a couple of months this does not seems to be true anymore. You now have to create these two artifacts upfront.

I was wondering how you can do this when there is no workspace that belongs to the metastore yet? Seems like impossible?

Thanks,

Joining Databricks soon—what do employees typically use for mobile? Personal phone with MobSec/MDM, or a separate device for privacy?

I have a fairly small bronze dataset with around 8 million rows in the bronze layer, however when I move to silver layer I have to perform an EXPLODE operation to an array in this table that makes it become a 5+ billion row dataset, plus I have to apply a deduplication/CDC into it. I haven’t been successful to do it via SDP without paying over a 100 USD and over 3 hours job run, whilst in a standard Lakeflow Job I can do that with less than 20 USD and within one hour. I’d really like to have it in SDP as it simplifies a lot the code, add features and so on, but is it possible to do it while keeping the costs low?

I’ve tried mainly streaming table as a sink for this data, I’m afraid to try materialized views as it looks to require serverless to do incremental refreshes plus it may still do full recomputing if it thinks it needs to do.

In my project we are storing json as raw response in string in bronze layer,now we have to parse and flatten json in multiple child table and clean in silver layer,we want this using spark declarative pipeline also wanted to make the framework reusable,any one worked on such usecase, welcome to discuss and collaborate

I feel really positive about all the recent progress of the Genie One which is backed up by all the business contexts, data, metrics views, pages and driven by Genie Ontology. It can get us answers cost effectively compared to external models connected to Databricks. Earlier it didn't have much unstructured data support, but it is now allowing connection to UC Volume as well as attaching the file option to a chat. There'll soon be image attachment options too where we can add screenshots as an additional context and ask questions like we do generally with ChatGPT or Gemini in our personal life.

Great days for businesses ahead 😀

Hey folks, I work on Databricks implementations at Abilytics and have been looking into Genie Code cost controls recently. I’m running into a frustrating blind spot with the compute side. The budgets work well for tracking Genie usage, but when Genie Code writes and runs code or triggers warehouse queries, that compute is billed separately and doesn’t seem to have a clear link back to the user or Genie request that caused it. The token usage is easy to track, while the warehouse or cluster cost just appears as normal compute. I’m also seeing some differences between the budget page and "system.billing.usage", which I understand can happen because of reporting delays, but it makes month end reconciliation harder. Has anyone found a clean way to attribute Genie triggered compute back to a specific user or team without manually piecing together AI Gateway logs and compute activity? I’d be interested to hear how others are handling this.

Im fairly new to databricks and trying to learn more about it. I’m trying to see examples online of databricks connecting to a work email, reading the contents and running a sql or python query which outputs an excel file and then sending it back go the requestor through email. I was reading AI agents can go about doing this but I would like more information on how to set up this process. Does anyone know of any examples or resources?

I wonder sometimes that our laptop hardware is getting so much better that sometimes it is easier to open the laptop, crunch the numbers, and save them if required. This workflow is much more hassle-free, cost effective and super fast compared to spinning up a Databricks notebook with a Spark cluster to explore a subset of my data.

I do understand that for daily jobs and large-scale processing, where a “heavy” amount of data needs to be processed, something like Databricks is super helpful.

Nothing is faster than local processing for small exploratory analysis, avoiding the whole network latency, cluster spin-up times, etc. Mind you small here can be upwards of 1 million records.

I wonder if the industry will start moving in this direction from everything cloud to a hybrid setup.

What’s your opinion?

How Databricks Lakehouse Replay reruns your read-only serverless queries on unreleased runtimes to catch regressions before they ship.

Lakehouse Replay, now in Public Preview, moves part of that burden to Databricks. Instead of you testing the new runtime, Databricks tests it against your queries, in your workspace, before the version is released to anyone.

Hi community.

I just wanted to ask about Databricks integrations with. For instance, is it possible build an #agent with my own NLP but connect it to WhatsApp?

I want to start a project like this, but I don't want any surprises or integration restrictions in the future.

Having read through and seeing some demos I still don’t understand if Genie Ontology is a real thing or some marketing fluff , we have been asked to compare against Palantir foundry’s ontology and I find very few comparisons apart from the data model and relationships , for example how do I show that Genie Ontology Knowledge Graph ?

Databricks just closed a $5B round at a $190B valuation — up from the $188B term sheet from July, after crossing $7B in revenue run-rate and growing 80%+ YoY.

We have S/4HANA on prem as a source and we need the data in Snowflake with low latency, including deletes.

What we have ruled out so far: ODP and RFC based extraction, because of SAP Note 3255746. OData on top of CDS views works but it is pull based, and it gives us no reliable way to capture deletes without reconciling full snapshots.

That leaves SAP's own event mechanisms, BOR, RAP and BTE, which push a notification whenever a business object changes. On paper that solves both the latency and the delete problem.

Has anyone actually landed this in Snowflake? Curious specifically about the landing path, whether you go Kafka connector, Snowpipe Streaming, or Openflow, and how you handle the initial full load and stitch it to the event stream without gaps or duplicates. Also interested in how people are doing the MERGE side, since the events arrive at table level and header and item rows do not always show up in order.