ingestr is quite fast, here's the benchmark
u/uncertainschrodinger — 14 hours ago
u/uncertainschrodinger
r/SQL r/istanbul r/snowflake r/Database r/BusinessIntelligence r/bigquery r/CLI r/AI_Agents r/analytics r/datasets r/learnSQL r/Polymarket r/DuckDB r/dataanalysis r/foss r/Clickhouse r/dataisbeautiful r/ETL
▲ 18 r/SQL
How do you decide which incremental strategy to use in your SQL models
Choosing the right incremental strategy matters: in bigquery it reduces the amount of data scanned, while in clickhouse it means less CPU, I/O, and background merging.
but tbh, data/analytics engineers sometimes optimize this too early, if a table is small or rarely changes, a view or `create+replace` is easier than a convoluted incremental model. Sometimes full refreshes are just good enough
Before choosing anything, I normally ask: do old rows change? is `updated_at` reliable? can rows disappear? is the data truly append only? do you have proper primary keys & partitions? and how late can updates arrive?
here's an overview:
* `append` -> the source is genuinely append only
* `merge` -> existing rows change and there's a stable primary key
* `delete+insert` or `time_interval` -> you can safely rebuild a complete group or time window
* SCD2 -> you need the full history, not just the latest version
with `merge`, NULL keys can be inserted again on every run because `NULL = NULL` isn't true, I have personally lost sleep over this... (in bigquery)
with `delete+insert` or `time_interval`, only delete what you can fully rebuild - incomplete partitions, bad boundaries, or late arrivals can create gaps or duplicates.
That's why I always add quality checks that catch issues the strategy can cause: `not_null` & `unique` for merge keys, duplicate checks for rebuilt windows, and row-count or freshness checks where they make sense.
In SQL pipelines the most frustrating (and silent killer) is that a pipeline status can be successful, but that only means the SQL ran, but doesn't guarantee that the result is correct. That's why quality checks are there to catch the other issues.
and if timestamps and lookback windows still can't capture changes reliably, CDC is an option... just with more state, replay logic, and operational headaches.
the image below is an example of a `time_interval` strategy and its rendered clickhouse query (don't mind the wrapped date overlappign with INSERT)
I'm curious what other criteria do people have in mind when they try to evaluate which strategy to go with? I think this is more relevant to analytics engineers, not so much the operational/transactional side.
u/uncertainschrodinger — 3 days ago
▲ 3 r/ETL
I helped a client migrate from Fivetran to Ingestr, cutting their cost by 10x
I wrote a whole article about it, link in comments, but I just want share a quick summary here.
This client had over 30 data sources they were ingesting from, it looked something like this:
- posgresql -> snowflake ~100m/rows/mo (some spikes to 1b/rows/mo)
- all other data sources -> snowflake ~100m/rows/mo (some spikes to 1b/rows/mo)
This was costing them almost $100k per year (~$8.5k/month) and it was especially hard to justify because the spikes would normally happen during busy seasons and they already had tight margins.
They had already started using ingestr (the free open source version) to offload some of their smaller jobs and running the jobs on an EC2 instance.
Once they migrated over their production database to snowflake job, they cut the cost significantly - the total server cost came to less than $500/month.
If this resonates to anyone, and you want to put on your resume "helped reduce ingestion cost by x amount", then check out ingestr repo and docs.
u/uncertainschrodinger — 6 days ago
how I learned why you shouldn't name an alias the same as the original column name
I wrote a query last week that ran fine on Postgres and DuckDB, and hard-errored on ClickHouse and BigQuery - this sent me down a rabbit hole for most of the day.
Here's what I had:
```
SELECT term, MAX(ranking_page_count) AS ranking_page_count
FROM ranked
GROUP BY term
HAVING MAX(ranking_page_count) >= 2
```
The CTE already had a column called ranking_page_count. I aliased MAX() of it to the same name, because why not, and then used that name again in HAVING.
So which one does HAVING actually filter by? Turns out that's a matter of opinion.
In Postgres, HAVING can’t see SELECT aliases at all. So it reads the column directly and lands on the same max anyway - no error, right answer.
DuckDB does let you use aliases in HAVING, but only as a fallback, and it won't put one inside an aggregate, so this also runs. This is the one that got me, since DuckDB is where I test locally.
BigQuery gives the alias priority over the column. So it read my query as MAX(MAX(...)) and gave the error "aggregations of aggregations are not allowed"
ClickHouse just swaps aliases in everywhere, so it gave code 184 illegal aggregation. it even fails when the alias isn't shadowing anything.
The thing that finally made it click for me was processing order. FROM, WHERE, GROUP BY, HAVING, then SELECT, then ORDER BY. Aliases get created in SELECT, so when HAVING runs the alias doesn't exist yet. That's why Postgres says no, and why everything else here is a vendor extension rather than four equally valid readings.
ORDER BY is the only clause that runs after SELECT, which is why it's the only clause where nobody argues.
What actually worries me is that it can go completely silent. Drop the aggregate from the alias and the loud error disappears:
```
SELECT term, ranking_page_count * 10 AS ranking_page_count
FROM ranked
GROUP BY term, ranking_page_count
HAVING MAX(ranking_page_count) > 4
```
Postgres and DuckDB filter on `ranking_page_count`
BigQuery and ClickHouse filter on `ranking_page_count * 10`
I get 1 row from the first two and 4 rows from the other two, and not one of them raises an error about it.
That's the version that ends up on a dashboard.
ok fine, I learned my lesson and won't name an aggregate after the column it aggregates...
If you work across different engines, this is your reminder to go check 🥲
u/uncertainschrodinger — 6 days ago
[OC] The infamous Peter Thiel's chess game history analysis
Sources: chess dot com API
Tools: BigQuery, Bruin (CLI for processing the data, data apps for visualization)
u/uncertainschrodinger — 7 days ago
▲ 6 r/ETL
Self healing pipeline agents - free webinar
I'm hosting a free webinar to go over how you can build self healing pipeline agents.
It will be a practical and technical webinar meant for mid/senior level data/analytics engineers.
Key topics:
- overview of the context that already exists in your pipelines
- methods for enriching the context and data governance
- tools, resources, and skills for agents to diagnose, investigate, analyze, test, and run things
The main idea is that your pipeline already contain some context, you can improve that context, then set up rules, procedures, and workflows for agents to help with the diagnosis and even pushing fixes.
Disclaimer; I'm a developer advocate at Bruin - in this webinar we will use Bruin to demonstrate things, but the concepts and strategies are applicable to any tools (i.e. dbt, airflow, etc.)
u/uncertainschrodinger — 14 days ago
▲ 0 r/dataanalysis
What's the difference between data apps and BI tools?
In my opinion, data apps are different in the sense that you can take actions and interact beyond just filters and drill downs.
for example, changing the projected growth percent or cogs and seeing how it impacts the bottom line, or even updating or inserting data directly from the data app (in a way, reverse ETL)
I was talking to a colleague and he said it's the same as regular dashboards but just my dynamic - but in my opinion the key difference is the ability to interact with the data itself and not just the charts.
I'm curious what others think
u/uncertainschrodinger — 22 days ago
▲ 6 r/ETL
Just some thoughts on incremental strategies
Choosing the right incremental strategy matters: in bigquery it reduces the amount of data scanned, while in clickhouse it means less CPU, I/O, and background merging.
but tbh, data engineers sometimes optimize this too early, if a table is small or rarely changes, a view or `create+replace` is easier than a convoluted incremental model - full refreshes aren't sexy, but sometimes they're good enough.
Before choosing anything, ask: do old rows change? is `updated_at` reliable? can rows disappear? is the data truly append only? do you have proper primary keys & partitions? and how late can updates arrive?
here's a simple overview:
* `append` -> the source is genuinely append only
* `merge` -> existing rows change and there's a stable primary key
* `delete+insert` or `time_interval` -> you can safely rebuild a complete group or time window
* SCD2 -> you need the full history, not just the latest version
with `merge`, NULL keys can be inserted again on every run because `NULL = NULL` isn't true, I have personally lost sleep over this...
with `delete+insert` or `time_interval`, only delete what you can fully rebuild - incomplete partitions, bad boundaries, or late arrivals can create gaps or duplicates.
That's why checks should match the strategy: `not_null` & `unique` for merge keys, duplicate checks for rebuilt windows, and row-count or freshness checks where they make sense. A successful pipeline only means the SQL ran, so quality checks are there to catch the other issues.
and if timestamps and lookback windows still can't capture changes reliably, CDC is an option... just with more state, replay logic, and operational headaches.
the image is an example of a bruin`time_interval` asset and its rendered clickhouse query
u/uncertainschrodinger — 23 days ago
▲ 11 r/dataisbeautiful
[OC] FIFA World Cup final teams' cumulative xG differential throughout the tournament
Sources: FIFA Training Centre public post-match reports
Tools: Python/pdfplumber, Bruin cli, BigQuery, and SVG
Limitations: The teams faced different opponents, so this describes their tournament paths rather than an opponent-adjusted rating
u/uncertainschrodinger — 29 days ago
▲ 2 r/analytics
I've consolidated some resources to help learn AI skills for data analytics and engineering
| Career path | Best starting resources | What you should build |
|---|---|---|
| Student or beginner | Kaggle Learn, CS50 AI, Google ML Crash Course, Microsoft AI Agents for Beginners | A Python notebook that loads data, trains a simple model, asks an LLM to explain results, and checks the explanation against the data |
| Software engineer | Anthropic Academy, Claude API development, OpenAI Agents SDK, Hugging Face Agents Course, LangChain Academy | A tool-using agent with structured outputs, tests, traces, and a human approval step |
| Data engineer | Data Engineering Zoomcamp, dbt Learn, Bruin Academy, Dagster University, Airbyte Academy | A pipeline that ingests data, transforms it, validates it, exposes metadata, and lets an agent query it safely |
| Analytics engineer | dbt Learn, Bruin Academy, Build an AI Data Analyst, LlamaIndex documentation | A semantic layer or context layer that defines metrics, entities, joins, examples, and freshness checks |
| Data analyst | Kaggle Learn, Codecademy AI for Data Analysis, Bruin Academy AI data analyst, dbt Learn | A repeatable analysis workflow where the AI writes SQL, explains assumptions, and you verify the result |
| Team lead or manager | Anthropic AI Fluency resources, OpenAI practical guide to building agents, Bruin Cloud AI agents, scheduled agents | A governance checklist: what agents can access, what they can change, who approves, and how answers are audited |
u/uncertainschrodinger — 2 months ago
I helped build an open source semantic layer tool
I feel like this year I've heard a lot of talk about semantic layer, every time the topic of "AI data this and that" comes up, inevitably people talk about semantic layer.
That's one of the reasons why we wanted to add a semantic layer to Bruin to allow users to define their semantic layer in the same repo that their pipelines are so that agents can get the full picture - ingestion, transformation, governance, and now the semantic layer.
It is still the early days for the semantic layer, but it works across all the platforms we support. I've tested it with my own data analyst agent and I've seen improvements in terms of how accurately it answers questions, but I'm curious what others think.
Has anyone tried using agents to analyze data with and without semantic layer? Did you see any improvements?
u/uncertainschrodinger — 2 months ago
I helped build a simple and fast data ingestion tool
There is an open-source data ingestion CLI tool called ingestr and it is one of the easiest way to move and copy data between databases and warehouses.
One way I've used it is to ingest data from a database/warehouse or third party sources like hubspot, google ads, etc. into a local duckdb to quickly analyze data - this is especially cool when given to an AI agent to quickly ingest data and analyze it for you locally.
GitHub: https://github.com/bruin-data/ingestr
Docs: https://getbruin.com/docs/ingestr/
u/uncertainschrodinger — 3 months ago
[OC] Wikipedia AI referenced articles growth since
Sources: Wikipedia MediaWiki Action API, Wikipedia Vital Articles / Level 4
Tools: Bruin cli, BigQuery, Bruin dac
Methodology
Universe. Two tiers (14,004 articles total, 11 top-level subjects, 110 sub-subjects). Tier 1: Wikipedia Vital Articles / Level 4 - 9,907 curated articles across all 11 subjects. Tier 2: 4,097 WikiProject Top/High-importance articles from Companies, Brands, Computing, Internet culture, and Business - added only to Society and social sciences (+2,735) and Technology (+1,362) to compensate for those areas being under-represented in Vital L4. Vital takes priority on collision.
AI seed list. 48 curated AI-topic articles spanning foundations (Artificial intelligence, Machine learning, Neural network, Deep learning, Supervised/Unsupervised/Self-supervised learning), architectures (Transformer, CNN, RNN, GAN, Diffusion model, Attention, LSTM), modern systems (LLM, GPT-3, GPT-4, ChatGPT, Claude, Gemini, LLaMA, BERT, Stable Diffusion, DALL-E, Midjourney, Generative AI, Foundation model), companies (OpenAI, Anthropic, DeepMind, Hugging Face), sub-fields (NLP, Computer vision, RL, Speech recognition, Symbolic AI, Machine translation, Robotics, Expert system), and cultural/policy (AI alignment, safety, ethics, AGI, existential risk, technological singularity, regulation, AI winter). Each canonical title is expanded with its current redirect aliases.
Snapshots. 14 semiannual snapshots at fixed dates (December 1 and May 1, Dec-2019 through May-2026). For each (article × date), the MediaWiki Action API returns the closest revision at or before the target date; body wikilinks (regex-extracted from wikitext, excluding namespace, self, and anchor links) are intersected with the AI alias list to count "AI references".
Pipeline. Raw scrapes -> staging joins -> subject/sub-subject/article aggregates. This dashboard queries staging.wat_ai_reference_counts directly. All assets run via Bruin cli on BigQuery; the dashboard renders via Bruin dac.
Limitations & caveats
Slicing & filtering. Gainer charts rank by absolute percentage-point gain since Dec 2019, not relative growth; the sub-subject chart shows the top 8 only. Both gainer charts and every small-multiples panel apply the same eligibility filter: n>=20 articles AND >=1 AI-referencing article at the latest snapshot. The 20-article floor avoids small-denominator noise (e.g. a 2-article sub-subject swinging to 50% on a single edit). Small-multiples panels show up to 7 sub-subjects (top by article count); panels with sparse AI uptake show fewer (History 2; Everyday life and Geography 3; Mathematics 4; Arts and Physical sciences 5; Biology & health 6).
Comparability. In the small-multiples grid, per-panel y-ranges are independent - compare shapes, not heights. The universe is not uniform across subjects: only Society and social sciences and Technology receive the WikiProject Top/High extension; the other 9 subjects are Vital L4 only. Cross-subject magnitudes therefore reflect both AI uptake AND uneven corpus composition.
What "AI reference" means. A structural body wikilink to one of 48 curated AI articles (plus current redirect aliases), not a semantic measure of AI content. Template-generated and navbox links are excluded; only editor-chosen body links count.
Scope. Universe is curated (Vital L4 + WikiProject Top/High in 5 categories = 14,004 articles), not a random or exhaustive sample of Wikipedia. English Wikipedia only. Results generalise to "important, well-edited articles", not to long-tail content.
Time. Some AI seed pages did not exist in 2019 (e.g. ChatGPT, GPT-4, Claude, Gemini, LLaMA, Stable Diffusion, Midjourney), so apparent growth partly reflects new AI vocabulary entering Wikipedia rather than only existing articles adopting new links. Snapshots are semiannual (Dec 1 / May 1), so spikes shorter than ~6 months and revisions reverted between snapshots are invisible. The MediaWiki API returns the closest revision at or before each snapshot date, so an article's state can be up to ~6 months stale relative to the next snapshot.
u/uncertainschrodinger — 3 months ago
Do you prefer building dashboards using a UI based BI tool or code?
On a scale of 1 to 3, what is your preference for building dashboards and visualizations?
1 -> fully no-code (drag-and-drop only) don't write queries or code, and often can't fully access or customize the underlying code behind charts or dashboards (e.g. Power BI, Data Studio)
2 -> hybrid (mostly drag-and-drop) drag-and-drop dashboard building, but you can also write/edit queries and view the underlying code and customize things (Grafana, Superset, Metabase)
3 -> fully code-driven (code-only) queries, layout, styling, interactions, and chart behaviour all defined in code (e.g. Plotly Dash, D3js, Streamlit)
u/uncertainschrodinger — 3 months ago
[OC] How Claude is used across occupations: directive, iteration, feedback, validation, learning (AEI v3)
Sources: Anthropic Economic Index v3 - collaboration patterns (CC BY 4.0; per-task directive / task_iteration / feedback_loop / validation / learning percentages and per-task usage_count_global weights), O*NET-SOC 28.3 (CC BY 4.0; 8-digit O*NET-SOC codes used to roll AEI tasks up to BLS SOC major groups via LEFT(onet_soc_code, 2)).
Tools: Bruin CLI (pipeline), BigQuery (warehouse), Bruin DAC (visualization).
Limitations: AEI also publishes none and not_classified collaboration buckets - both are dropped before renormalizing so the five shown patterns sum to 100 % within a group, which inflates each pattern's share by however much fell into those two buckets. Within-group percentages are weighted by AEI conversation count, so heavy tasks dominate the group composition; an unweighted view would be flatter. Groups with fewer than 1,000 weighted conversations are dropped. Construction & Extraction and Building & Grounds Cleaning have all-NULL collaboration fields in release_2026_01_15 and are also dropped, leaving 20 of the 22 BLS SOC major groups.
u/uncertainschrodinger — 3 months ago
▲ 3 r/istanbul
Co-Working offices in Kadikoy
Anyone have any experience with co-working offices in Kadikoy or surrounding area?
Most of the team lives around marmaray so we're looking for something preferably accessible by marmaray or M4.
We've checked out a few places but I'm curious to hear what place there are and what your experience has been.
u/uncertainschrodinger — 3 months ago
▲ 0 r/bigquery
Is BigQuery late to the AI game?
I've used BigQuery for a few years now and this past year I've seen so many different AI tools that help with everything from text-to-SQL to actually building reports and other features.
On one hand I understand they make their bread and butter from the actual warehouse and processing but as a user I would've liked to see more AI features integrated into the product. The new Gemini features work alright but it seems like an afterthought, like there's no way to build reports or visualizations, integrate into messaging apps, or connecting your context and semantics layers.
That was one of the reasons why I joined Bruin as a Developer Advocate recently because I wanted to be involved in building tools that address the stuff I wished I had as a data engineer. We just made our AI data analyst generally available. It connects to any warehouse like BigQuery, it imports the metadata of your datasets and creates a mental map of your data. You can also connect your dbt, airflow, dagster, or bruin pipeline repos to add additional context about your models.
The whole point is to have an agent that lives right inside your team and acts like a team member - from answering quick questions to preparing reports and even troubleshooting data & pipeline issues.
I was quite skeptical at first but we have dozens of clients using it and the more they use it the better the agent gets because it is self-correcting - every conversation and every correction further improves the context.
While I'm speaking about Bruin here, this is the general blueprint and framework for any organization to build themselves an AI data agent that does more than just text-to-sql.
u/uncertainschrodinger — 3 months ago
▲ 1 r/datasets
Tool for data ingestion, transformation, orchestrations, and analysis [self-promotion]
Disclaimer, I’m a developer advocate at Bruin. I previously worked in data analyst and then data engineering roles for almost 10 years, and now at this job I finally have the freedom to play around with data just for fun. This community has always been my go to place to find cool datasets.
That’s why I’m excited to share this announcement with you but I promise to keep the promotional talk very minimal.
I’m sure many of you use AI agents to analyze data, build dashboards, and share them with friends and others. Bruin has a lot of open-source tools for data ingestion, transformation, orchestration, and visualization. Today we are announcing the general availability of Bruin Cloud which is the managed service of those free open-source tools.
I’m personally excited because as a dev advocate I’ve focused mainly on our open-source tools but managing and deploying them locally is sometimes an obstacle for someone that just wants to play around with data - so the free tier (no payment required) version of Bruin Cloud will give you enough credits to get started to run your pipelines but more importantly analyze your data using the AI data analyst and dashboard builder.
Check out the open-source tools: https://github.com/bruin-data
If interested, feel free to check Bruin Cloud too: https://cloud.getbruin.com/register
u/uncertainschrodinger — 3 months ago
▲ 1 r/Polymarket
Sources: Meteostat, Open-Meteo, Polymarket CLOB.
Tools: Bruin CLI (pipeline), BigQuery (warehouse), Bruin DAC (visualization).
Limitations: Meteostat returns the METAR nearest the top of each UTC hour, so the alleged sub-hour spike at CDG on 2026-04-15 between 19:00 and 20:00 shows up as a recovery leg rather than a spike. The dashed price line is the last CLOB tick within each hour; intra-hour movement is not visible. Trader identity and on-chain wallet attribution are out of scope.
u/uncertainschrodinger — 4 months ago
▲ 223 r/dataisbeautiful
Sources: Meteostat, Open-Meteo, Polymarket CLOB.
Tools: Bruin CLI (pipeline), BigQuery (warehouse), Bruin DAC (visualization).
Limitations: Meteostat returns the METAR nearest the top of each UTC hour, so the alleged sub-hour spike at CDG on 2026-04-15 between 19:00 and 20:00 shows up as a recovery leg rather than a spike. The dashed price line is the last CLOB tick within each hour; intra-hour movement is not visible. Trader identity and on-chain wallet attribution are out of scope.
u/uncertainschrodinger — 4 months ago