Dynamic Tables vs. Complex Pipelines - How Much of an Advantage Does Snowflake Really Have?
I’m trying to understand the practical advantage of Snowflake Dynamic Tables, especially for complex data pipelines.
Dynamic Tables seem very useful for simplifying incremental processing and reducing the need to manage orchestration. But does that advantage hold when a pipeline involves multiple platforms, multiple pipelines, external dependencies, and complex transformations?
At what level of pipeline complexity do Dynamic Tables start becoming less useful?
If a pipeline spans multiple tools/platforms, do you still need an external orchestrator like Airflow/Dagster?
How does this compare with the equivalent approach on Google Cloud (BigQuery + Dataform/Composer/Dataflow, etc.)?
Is the main advantage of Dynamic Tables actually simplifying pipeline management, or are there meaningful performance/cost benefits as well?
For large, production-grade pipelines, would you choose Dynamic Tables over a more traditional orchestration + SQL/ETL approach? Why?