I have an issue with some models running long, seemingly at random, and I want to monitor this by running EXPLAIN on the compiled sql in a pre hook so if something runs log I can look at what it planned to do and compare it to previous runs.
But I cannot get the compiled sql into the pre hook. Doesn't anyone have any ideas or suggestions?
Hello. I am new to dbt and data modeling. I am wondering what is better, select star or select specific columns at the top of the model, where you get data from ref in intermediate models. DBT courses don't really talk about it, but I have seen some blog posts where people said it is better to be specific. Now I wonder why one or another would be better. What are pros and cons. All I come up with is that if you suddenly need more columns, you have to add it in the source cte as well, so one more place to maintain.
But again, I'm quite new and haven't been exposed to many problems yet haha. What do you people do?
I just read an interesting article about using Apache Spark not only to transform data else also to enforce data contracts within pipelines.
The key idea: the problem isn't that jobs fail, but that they don't fail when they should. The pipelines keep running, but the data might be corrupted → silent errors.
The proposal:
This transforms pipelines into systems that guarantee quality, not just move data.
If you don't validate your data within the pipeline, you're relying on assumptions.