9 out of 12 public dbt repos we audited have "phantom columns" in their docs
TL;DR: Most CI linters (dbt-checkpoint, dbt-project-evaluator) check if a column has a description, but almost nothing checks if the column in your YAML actually exists in your data warehouse. We scanned public production dbt projects and found that 9 out of 12 had significant doc-to-warehouse drift.
The Bug That Got Me Thinking
Found this in a public dbt repo recently:
A model called fact_customer_survey had a UNION.
- Branch 1 (line 30):
NULL as dissatisfacation_category(notice the extra 'a') - Branch 2 (line 51):
dissatisfaction_category(spelled correctly)
Because SQL takes union output column names from the first branch, the warehouse materialized DISSATISFACATION_CATEGORY.
Here's the kicker: The project’s YAML docs declared dissatisfaction_category (spelled correctly).
- The SQL ran fine daily.
- The documentation was "correct."
- The code was wrong.
- Any dashboard or analyst trusting the docs was querying a column that didn't exist.
How Bad Is This in the Wild?
Every time you run dbt docs generate, you produce two files:
manifest.json(what you claim exists in YAML)catalog.json(what the warehouse actually returns)
We compared declared columns vs. cataloged columns across verified production repos. Out of 12 active organizational projects:
- 9 out of 12 had phantom columns (documented in YAML, completely missing in the warehouse).
- Cal-ITP (BigQuery): 106 phantom columns
- Allvue Systems (Snowflake): 112 phantom columns + 323 data type mismatches (mostly declared
stringsitting on warehouseNUMBER) - Cook County Assessor (Athena): 11 phantom columns
- The "Ghost Repo" problem: 23 docs sites published a complete, polished YAML docs UI where
catalog.jsonwas an empty stub—meaning the warehouse was literally never introspected. One documented 1,280 models this way.
Where Does the Drift Come From?
When we audited the findings against actual model SQL:
- Renames / Deletions: Column was renamed or dropped in SQL, but the YAML entry was never cleaned up.
- Commented-out code: One project had a 20 KB cleaning projection inside a
/* ... */block. The live warehouse table had 173 raw Airbyte column names (WEEK STARTING 01/19/2025 - RESOURCES...), while the YAML proudly documented the clean columns someone intended to build. - Syntax accidents: Trailing commas in YAML names (e.g.,
- name: feed_type,).
Why This Is Becoming a Bigger Problem
When human analysts read dbt docs, they can spot a typo or realize a column was renamed.
But with dbt MCP servers and Text-to-SQL AI agents using dbt docs and manifests as ground truth context, a phantom column is an immediate failure. The agent attempts to query columns that aren't there or hallucinates transformations based on dead YAML.
A Few Questions for the Sub:
- Does anyone here actively diff
manifest.jsonagainstcatalog.jsonin their CI/CD pipelines? - How do you prevent documentation rot when engineers refactor/comment out SQL transformations?
- If you are already feeding dbt metadata to LLMs/AI agents, how are you validating that the schema you hand the agent actually matches production?
