u/roksolana_shendiukh

Snowflake's free trial (30 days/$400 credits) is too short for a university thesis running several months – need something that doesn't expire mid-project. What are people using instead? Also curious if anyone knows of a longer free tier/subscription (student program, extended trial, etc.) for Snowflake specifically before I give up on it.

Task: We operate a toll road network with 200 sensor-equipped lanes across 15 locations. Each sensor captures license plate reads, timestamps, and lane metadata. We need to compute real-time traffic volume, average transit times between checkpoints, and flag anomalies for toll evasion detection. Design the data model.

I modeled it as a single accumulating snapshot fact table, grain = one row per vehicle crossing (entry -> exit). The row is inserted at entry with status = OPEN and all exit columns NULL, then updated in place when the exit read comes in – no join needed for evasion detection, just WHERE status = 'OPEN' AND entry_time < now() - window.

I have a confusion: does update-in-place on an accumulating snapshot actually hold up under real-time write load (concurrent OPEN -> COMPLETED updates), or does this call for an append-only/streaming pattern instead with the snapshot table only as a serving layer?

passed schema review for grain/fan traps – now want feedback on the real-time side

Hi everyone!
I'm preparing for a Data Engineering career and I'm trying to prioritize what to learn.
I know Python and SQL are essential, but I'm wondering how much emphasis I should put on algorithms and data structures.
For example, how often do you deal with things like trees, DFS/BFS, graphs, or other algorithmic problems in your day-to-day work?
Are these concepts mainly useful for coding interviews, or do they come up in real Data Engineering tasks?
What programming topics would you recommend focusing on the most?
Thanks!

I have a real-time counter (Redis, per-link clicks) and a nightly batch aggregate (Redshift, deduplicated, source of truth). They're expected to differ slightly, but I want to catch real drift (bug, broken pipeline) rather than let it go unnoticed.

My rough idea: a scheduled job that, once both are available for a day, compares the two totals per link and alerts if the gap exceeds some threshold – doesn't auto-correct either side.

What do people actually use for this in practice? A few options I'm considering:

\- simple scheduled diff job + alerting (Airflow -> CloudWatch/Slack)

\- storing reconciliation results as their own table for trend tracking

\- some kind of streaming-side checkpoint/watermark that flags "final" state instead of comparing after the fact

Curious what mechanisms are actually used, and what I might be missing.

Hi everyone!
I want to learn Apache Spark in depth, not just the DataFrame API.
I started with the official documentation, but I find it difficult to learn from because it isn't very interactive.
What resources would you recommend for someone who wants to really understand Spark?
I'm looking for recommendations on books, courses, YouTube channels, blogs, or hands-on projects. I'd especially like to understand Spark internals, architecture, optimization, and best practices.
If you were starting over today, how would you learn Spark?
Thanks!

If a consumer is down (or lagging) longer than delete.retention.ms, it can come back online and miss a tombstone entirely – meaning it never learns a key was deleted, and just keeps the stale last-known value forever. That's not an edge case, that's baked into how compaction works.
So is "compacted topic = changelog of truth" (as Kafka Streams/KTables imply) actually a safe abstraction, or does it just quietly break under any non-trivial consumer downtime – and if so, why does the ecosystem lean on it so heavily?

Building a click pipeline (~200M events/day, viral links causing 10-20x spikes). Events keyed by short_code, so a viral link's traffic all lands on one Kafka partition + one Redis shard – classic hot-key problem.
My fix: salt every key with a small fixed number (e.g. short_code:hash(x)%10), instead of trying to detect "hot" links first. Reasoning – detection adds its own failure point, and fixed salting costs almost nothing for normal links while auto-spreading load for viral ones. Aggregate on read (sum across the N sub-keys).
Trade-offs I know about: lose strict ordering per key, N reads instead of 1.
Is this actually how people handle this in production, or is there a better pattern I'm missing?