New to Databricks, question about unit testing SQL pipelines
I’m currently new to Databricks, and I’m writing unit tests for a pipeline that is entirely in SQL, with Bronze, Silver, and Gold tables.
I asked how I should approach unit testing, and I was told that since unit tests are typically written in Python, I could wrap the pipeline logic in Python files and then use pytest to test those files.
But I’m confused about how this would work in a real production environment.
For example, if we push everything to GitHub and later make a change to the SQL pipeline, we would also have to make the same change in the corresponding Python files so that the unit tests continue to work. That means maintaining the same logic in two places, which doesn’t seem like a good production practice.
So what is the recommended approach for unit testing a SQL-based Databricks pipeline? Is there a way to test the actual SQL transformations directly without duplicating the pipeline logic in Python?
I’m still learning Databricks, so I’d really appreciate some guidance on the proper production approach.