Hey all, we have several columns in tables that are tagged for certain criteria like PII, Confidential etc. We are trying to build some cicd / devops pipelines to handle deployments to snowflake, and part of that is using create or alter on tables to avoid create or replace.

The problem we have is create or alter on a table with tags on columns gives you:

>CREATE OR ALTER does not support setting policies or tags

The tags are already set on the table, we arent trying to add or set new ones, just running the DDL from GET_DDL with a new column on the end. But the GET_DDL includes the tags, and we dont want the table to be emptied of data so need to use create or alter.

Anyone have any ideas how to work around this because its completely blocked any development of trying to have automated snowflake deployments of objects

I’m facing a strange issue in Snowflake/dbt today.

We have Viewpoint source tables ingested through Fivetran into Snowflake. Several staging models have a column named PURGE. These models have been running successfully every day for a long time.

Today, all models referencing PURGE suddenly started failing with:

SQL compilation error:
000904 (42000): invalid identifier 'PURGE'

The interesting part:

• The PURGE column still exists in the Snowflake landing/Fivetran table
• No source column was removed
• The dbt code hasn't intentionally changed
• Models that don't reference PURGE are working
• This works:

​

SELECT "PURGE"
FROM <table>
LIMIT 1;

• But this fails:

​

SELECT PURGE
FROM <table>
LIMIT 1;

with:

invalid identifier 'PURGE'

So it appears that unquoted PURGE is no longer being resolved as a column identifier, while quoted "PURGE" still works.

Does anyone know if there was a recent Snowflake behavior change / parser change / release that could cause this?

I'm especially interested in whether PURGE has recently started being treated as a keyword in some SQL context, because this broke many existing dbt models simultaneously without any apparent schema change.

Any insight into the exact Snowflake change would be appreciated.

Cloud data warehouses like Snowflake scale compute effortlessly and I find a lot of use in it, at the same time, coupling ad-hoc user dashboard interactions and autonomous agent polling directly to warehouse compute creates linear cost growth. Every time a user changes a date filter or an LLM queries a metric like 30-day active users, Snowflake runs full table scans and resource-heavy COUNT(DISTINCT) calculations over millions of cold records, preventing compute clusters from suspending

In my opinion relying solely on Snowflake's native result cache fails here sometimes, because result caching requires byte-for-byte identical SQL syntax; any slight filter tweak or timezone adjustment misses the cache and forces a full cluster spin-up. The remediation may be in multiple ways, for example, i think of introducing a universal semantic layer with declarative pre-aggregations into more of my projects. Know any good semantic layers? Heard thet some like cubedev can define pre-aggregation roll-up tables that are built asynchronously. Ideally, would like for my data to be stored in a kind of distributed, Rust-based columnar cache. Incoming queries hit these warmed roll-up partitions in milliseconds, so that also lets Snowflake compute warehouses to stay suspended between scheduled refresh intervals. but what solutions can you think of here, and what experience do you have?

A few things I noticed from my exam experience that might help others preparing:

The questions were not that lengthy. For me, it was mostly an elimination game between 2 likely options.
Quite a few questions around Telemetry, DMFs, Information Schema, and Account Usage/Account Schema functions — mainly understanding what does what.
Alerts and how they can send notifications using resources.

Iceberg Tables — got 1–2 questions.
Cortex AI functions, including functions like .SENTIMENT().
Result Cache vs Warehouse Cache.
Understanding spilling — where it happens and what it means.
Query Profile questions, including identifying when results came from cache.

I prepared using Coco Test and SkillCertPro. They were useful for practice, but don’t keep false hope that the exact same questions will appear in the exam. The questions can be different.

My biggest advice: keep your concepts clear about what each Snowflake functionality actually does.

For example:
Streams → capture CDC/change tracking
Tasks → schedule/automate SQL execution
DMF → data quality metrics
Resource Monitor → control/monitor warehouse credit usage

Once you understand the purpose of a feature, you can usually eliminate the wrong options even if you haven’t seen the exact question before.

Overall, it wasn’t as tough as I expected. I had zero prior experience with Snowflake and was genuinely keen to learn, so if you’re starting from scratch, don’t get intimidated.
Good luck to everyone preparing! 🚀❄️

We have S/4HANA on prem as a source and we need the data in Snowflake with low latency, including deletes.

What we have ruled out so far: ODP and RFC based extraction, because of SAP Note 3255746. OData on top of CDS views works but it is pull based, and it gives us no reliable way to capture deletes without reconciling full snapshots.

That leaves SAP's own event mechanisms, BOR, RAP and BTE, which push a notification whenever a business object changes. On paper that solves both the latency and the delete problem.

Has anyone actually landed this in Snowflake? Curious specifically about the landing path, whether you go Kafka connector, Snowpipe Streaming, or Openflow, and how you handle the initial full load and stitch it to the event stream without gaps or duplicates. Also interested in how people are doing the MERGE side, since the events arrive at table level and header and item rows do not always show up in order.

I’m trying to understand the practical advantage of Snowflake Dynamic Tables, especially for complex data pipelines.

Dynamic Tables seem very useful for simplifying incremental processing and reducing the need to manage orchestration. But does that advantage hold when a pipeline involves multiple platforms, multiple pipelines, external dependencies, and complex transformations?

At what level of pipeline complexity do Dynamic Tables start becoming less useful?

If a pipeline spans multiple tools/platforms, do you still need an external orchestrator like Airflow/Dagster?

How does this compare with the equivalent approach on Google Cloud (BigQuery + Dataform/Composer/Dataflow, etc.)?

Is the main advantage of Dynamic Tables actually simplifying pipeline management, or are there meaningful performance/cost benefits as well?

For large, production-grade pipelines, would you choose Dynamic Tables over a more traditional orchestration + SQL/ETL approach? Why?

I’ve been using Snowflake zero-copy cloning for a while, but I'm curious how other teams view its real-world impact

Is it actually a gamechanger for your day-to-day operations? I know storage is essentially free at creation, but once modifications happen, storage costs can increase

What does the actual storage cost and scalability impact look like while using this? Also, Databricks & GCP also provides the similar capability, is Snowflake's implementation still a key differentiator?

I attend the Snowflake Sydney World Tour , some amazing sessions , but I would like this to renamed to the Sydney CoCo - Summit . Some very good content unfortunately lost in the marketing hype of CoCo. The after parties were fun for networking . The bad , it was crowded and if you could walk across the hall without the hall without atleast 3 people bumping into you . The ugly was , the crows brought in less opportunity to mingle and have a chat that the Snowflake Sydney World Tour used to be . I have been going Snowflake Sydney World Tour for the past 5 years so its time to say . Thanks I rather go for the one in US or in NZ . What are your thoughts Sydneysiders ?

Hi all. I've configured Claude desktop in my org (as a POC) to leverage a Snowflake MCP object to get NL to SQL / data back. Just wondering if others have done this and it better to simply force users to use Snowflake Cowork instead when they don't need to build complex workflows that utilize other tools?

My understanding (which may be wrong) is that currently when users as questions from Claude Desktop, the SF role used must be the their default role. That seems to be limiting.

Any thoughts?

Hi I am closing in on 4year experience with Snowflake. My main skills are Snowflake and SQL. Can anyone guide/mentor me on how to improve quickly(what all topics to learn other than Snowflake and SQL which I know sufficienly well).I am currently working for a big 4 company. I would like to move ASAP.Also SnowPro Core certified.Feel free to DM tol if necessary

I am trying to pass Multiple files as a visual context to Claude-sonnet 4.6 using Prompt and TO_File() function

but getting error even though I'm not hitting context limit and also each pdf is 3mb so the cap of 22mb is also well within the limit.

Verified the file location too.

weirdly it was working yesterday and today it is just not able to take multiple files

heres the code

SELECT AI_COMPLETE(

'claude-sonnet-4-6',

PROMPT(

'Summarize both documents and highlight any differences. DOCUMENT 1: {0} DOCUMENT 2: {1}',

TO_FILE('@"DB"."SPC"."LAMS"/folder OS Car _26.pdf'),

TO_FILE('@"DB"."SPC"."LAMS"/folder OS Bike _26.pdf'),

'how many documents are there, and mention the months'

)

) AS two_file_test;

Snowflake Cortex practice is not getting free looks like .Any thoughts where to get free ?

Hi everyone,
I work mainly with SQL, reporting, and dashboards, but I’ve recently been given the opportunity to take on a larger data project at my company.

We have data across several systems and databases, including SQL Server, PostgreSQL, and a CRM. We’re currently bringing these sources into Snowflake, and that part is nearly complete. The next step is to clean and organize the data, create reliable semantic data models, and make it easier for people to use the data for reporting and analysis.

This is my first time working on a proper data warehouse project, so I’m trying to understand what I should learn before going too far in the wrong direction.
From what I’ve researched so far, I probably need to focus on dimensional modelling, fact and dimension tables, star schemas, dbt, data quality testing, and Snowflake best practices. But I’m not sure what the most practical learning path would be, or whether I’m missing anything important.

If you’ve worked on a similar project, what topics would you recommend focusing on first? Are there any courses, books, YouTube channels, or hands-on projects that helped you understand how to design good data models in Snowflake?

I’d really appreciate any advice or learning recommendations. Thanks!

Hi! I'm wondering if there are other business analysts in this subreddit, and what tools you use with Snowflake for reporting. I've used Alteryx in the past to prep the data for Power BI, and occasionally did basic SQL queries in Snowflake for ad hoc data pulls. What are you guys using to get from Snowflake data to reports?

Grateful to everyone in this community who has shared their SnowPro Certification journey, resources, and advice. I cleared my SnowPro Certification today! 🎉

I started with no prior Snowflake knowledge and prepared for about a month before taking the exam today. This community has been incredibly helpful, and I genuinely appreciate everyone who took the time to share their experiences and help others.

I’m going to take a little break now 😄. I’ll be back soon to share my exam experience, preparation strategy, resources, and a clear study path for anyone planning to take the certification.

Thank you all! 🙌

I had mostly worked on batch-based projects before and had played around with Kafka, Flink, and ClickHouse, but never really connected everything into one proper end-to-end flow. I am trying to assess how far this is from production use in terms of the techstack.

So I decided to build a real-time anomaly detection system for simulated sensor data from an oil and gas plant. With AI based RCA to re-confirm true positives. This is purely for learning purpose.

The basic flow was:

Sensor data -> Kafka -> Flink/Python consumers -> ClickHouse -> Grafana -> Alerts -> RCA analysis

https://preview.redd.it/kq7lsasdn7hh1.png?width=3196&format=png&auto=webp&s=d71d1d17171e8fb3a8b334064457b522ccb80e21

https://preview.redd.it/ycfvehgnp7hh1.png?width=3196&format=png&auto=webp&s=7bba1f0f1a67f65cb6242e5542c885f4746716fd

https://preview.redd.it/quxou46tm7hh1.png?width=3196&format=png&auto=webp&s=794dc1027646c3fbc5c8dea7dc2d3a0174a419c0

I had around $200 in cloud credits(vultr) from a hackathon, so I rented a fairly powerful VM and self-hosted everything on it. I specifically avoided managed services because I wanted to understand how the individual open-source tools worked.

The main stack was:

• Kafka for streaming
• PyFlink for windowing and aggregations
• Python and FastAPI consumers
• ClickHouse for analytical storage
• PostgreSQL for metadata
• Grafana and Streamlit for dashboards (UIs generated using codex/claude)
• Airflow for some batch experiments
• Claude for alert validation and root cause analysis
• Resend API for email alerts

Since I did not have real plant data, I built a Python generator that simulated readings for different assets and randomly injected faults/failures.

The data was sent to Kafka topics with multiple partitions. I also added retries using exponential backoff. After five failed retries, the event would be moved to a dead letter topic.

For anomaly detection, I kept it simple. Each sensor had an acceptable operating range, and Flink used one-minute windows to calculate averages and check whether the asset was operating within those limits.

If an anomaly was detected, an alert was pushed into another Kafka topic and then processed by a diff consumer.

On a single VM, I tested around 10,000 events per second, or roughly 600,000 events per minute. This generated around 1.5 GB of data per hour. I ran it for a few hours and most of the events were processed successfully.

The biggest issue was not processing. It was storage.

I used ClickHouse with a three-day TTL, but at that throughput the data obviously builds up quickly. In a real setup, I imagine ClickHouse would only store recent hot data, while compressed raw data would be stored in object storage as Parquet.

I also thought about how I would build a simpler managed version on GCP.

The flow could be something like: Sensors -> Pub/Sub -> Dataflow/Cloud Run -> BigQuery/Bigtable -> Cloud Storage

Pub/Sub would replace Kafka, while Dataflow could handle the stream processing and windowing. Cloud Run could be used for simpler event consumers and alert-processing services.

This would be much easier to operate, but I wanted to build the self-hosted version first so I could properly understand what the managed services were doing underneath.

I also added a separate consumer for alert analysis.

Whenever an alert was generated, a Claude-based agent would:

• Check whether it was likely a true positive
• Query recent and historical sensor data from ClickHouse
• Check connected equipment
• Review similar past alerts
• Compare the issue against the asset's RCM maintenance plan
• Suggest a possible failure mode, cause, and recommended action
• Check if there were similar alerts raised earlier and how these were rectified(using vector embeddings - using Nomic Text embeddings served using ollama to find similar alerts)
• Find related recommended actions from RCM for the particular equipment. Alerts are also embedded to help find these and improve the search results/agentic analysis.
• Send the analysis by email using resend

For example, if a pump had high vibration, the agent could also inspect connected equipment instead of assuming the pump itself was definitely the cause.

The RCM data includes failure modes, possible causes, effects, and recommended maintenance actions. The agent used this only to recommend next steps and prep the email alert.

A possible next step would be creating a reviewed work order in SAP or Maximo after an alert is validated.

Obviously, this is all running on one VM with no replication or high availability. I built it mainly to understand streaming systems, Kafka partitions, windowing, retries, dead letter topics, consumer lag, ClickHouse, event-driven workflows, and how the same architecture could later be moved to managed cloud services.

I would appreciate feedback on a few things:

1. Does the overall architecture make sense?
2. Is ClickHouse for hot data and object storage for historical data the right approach?
3. What could be improved before presenting this as a proper portfolio project?

Thanks for reading through till the end.