So very excited to share this demo + presentation with the one and only, Scott Haines, Staff Developer Advocate @ Databricks. The topic? Zerobus, which is a great option for easily ingesting event data at scale into Unity Catalog.
We do a demo and overview of the technology, talk about how it is similar & different to Kafka, when to use Real-Time Mode vs Zerobus, and much more!
Hope you enjoy this very technical overview!
First time building a Databricks’ Genie space using Genie Code. Surprisingly, you can get 80% of what you'd need with one prompt, with the other 20% being tailoring things even more with prompts. The key to making it happen? Spending time upfront on governance inside the Unity Catalog, especially leveraging its' documentation capabilities.
👉 Quick walkthrough of what I did here:
-Started off from the home screen on my Databricks workspace.
-Wrote a single prompt into Genie Code to create a Genie space, pointing at the schema containing a handful of dimensions & two fact tables.
-The tables and respective fields already had "Comments" in the Unity Catalog to document what they represent.
-Genie Code handled the Genie space creation, table relationships, created reusable measures, and created a handful of starter questions that would be appropriate for business users.
-I picked one of the suggested questions which leveraged "Agent Mode", a mode for complex questions.
-I asked a follow up question to have it give me some actionable recommendations.
👉 General recommendations:
-Proper governance is more important than ever. Spend time making the most out of Unity Catalog first to make the most out of the platform!
-Always review the configurations, logic, and code generated by coding agents, specially when money is involved!
-Become familiar with the different capabilities Databricks offers, and then use Genie Code to help you get started using the ones that make business sense to you, fast.
Hope you enjoyed this post!
In this demo for Lakeflow Designer you will see me:
-Pulling sales data from Unity Catalog AND from local Excel workbooks
-Building a one big table report and more specialized reports
-Exporting reports to Excel
-Storing outputs back into Unity Catalog tables for use by analysts, BI developers, and even business users using the Databricks' Excel connector
-Setting up a recurring pipeline so that the data is kept fresh automatically
-All of the above without writing any code
I hope you enjoy the video, but more importantly, that you try it out yourself AND give feedback to the folks at Databricks on how to make the product even better. There are still many out-of-the-box pieces not yet available, and I know that the amount of feedback Databricks gets from customers will affect direction and priorities!
SOC 2, HITRUST, and so many other security things! Security requires a multi-pronged approach. Databricks' Arun Pamulapati did a deep-dive into the Security Analysis Tool, a tool created by field engineers at Databricks to help you improve your organization's Databricks deployments security posture against threats.
This is a very technical deep-dive and I hope you enjoy it!
Link to repo: https://github.com/databricks-industry-solutions/security-analysis-tool
Link to Databricks' security best practices: https://www.databricks.com/trust/security-features/best-practices