We open sourced a YAML based tool for managing Snowflake warehouses, roles, and grants
▲ 10 r/analyticsengineering+1 crossposts

We open sourced a YAML based tool for managing Snowflake warehouses, roles, and grants

Disclosure up front: I work on this, so factor that in.

A problem we kept hitting with Snowflake: warehouses, roles, and grants end up spread across ad hoc SQL scripts, the console UI, and whatever the last person who touched them remembers. There wasn't a lightweight way to treat that layer as code.

So we built Snowcap and open sourced it. You describe warehouses, roles, and grants in YAML. snowcap plan shows exactly what will change before anything runs. snowcap apply makes the change. No state file to manage, no Terraform provider, no clicking through the console.

We timed the full loop once: install to a live warehouse running in Snowflake, about five minutes.

It's grown through PRs and issues with zero ad spend or outbound, which says more about the gap than anything we could write in a blog post.

Blog Link - https://datacoves.com/post/snowcap-getting-started

Happy to answer anything about the design or trade offs in the comments.

u/Data-Queen-Mayra — 13 days ago
▲ 5 r/DataBuildTool+1 crossposts

The best order to learn dbt

People ask where to start with dbt. Most answers say start with dbt Labs’ great tutorials, but miss other things learners should understand.

What actually helps is understanding why dbt even exists. Why not just use tool X or just use stored procedures? Once you get this, other things makes sense.

The order I suggest people learn dbt is to start with Git and getting comfortable with the terminal. dbt is just code, if you dont know what git commit, cd, and ls do, you will be lost. Then understand why data layers exist. Followed by data modeling concepts and star schema. Finally, you can learn dbt.

You don't need to master it all before you start. You just need enough to not be lost when you encounter them.

Happy to answer questions if you're early in your dbt journey.

Full learners’ guide with resources from people you should follow Bruno Lima and Zach Wilson on LinkedIn: https://datacoves.com/post/dbt-getting-started

u/Data-Queen-Mayra — 1 month ago
▲ 13 r/datawarehouse+3 crossposts

We built an open-source IaC tool for Snowflake, here's how it works

Most Snowflake setups end up as a mix of tools, scripts, and manual clicks. We built Snowcap to handle it all in one place: warehouses, roles, grants, masking policies, dynamic tables, etc.

No state file. It queries Snowflake directly on every run and generates the SQL to match your config. If someone makes a change outside the tool, it catches it next run.

We wrote up the full overview here: https://datacoves.com/post/snowcap-snowflake-infrastructure-as-code

Happy to answer questions if anyone's dealing with Snowflake RBAC or provisioning headaches.

u/Data-Queen-Mayra — 2 months ago

We put together a guide for setting up dbt with Snowflake from scratch and figured it might be useful here.

What it covers:

  • Python, venv, and dbt-snowflake install
  • Setting up the Snowflake user, role, warehouse, and database with the actual SQL
  • Key pair authentication end-to-end
  • profiles.yml and dbt_project.yml settings worth knowing about (transient tables, query tags, copy_grants, warehouse overrides)
  • Official Snowflake Labs packages worth adding: dbt_constraints and dbt_semantic_view
  • VS Code extensions the official Snowflake Extension, Power User for dbt, and SQLFluff
  • How Snowflake Cortex CLI and other AI tools fit into the workflow
  • Managing Snowflake infrastructure (roles, grants, masking, RBAC) alongside dbt

Anything we missed that you would add?

https://datacoves.com/post/dbt-snowflake

u/Data-Queen-Mayra — 2 months ago