Login

u/uncertainschrodinger

r/startups_promotion r/ETL r/snowflake r/StartupFuture r/startup r/DuckDB r/Polymarket r/BusinessIntelligence r/CLI r/bigquery r/AI_Agents r/Supabase r/dataisbeautiful r/analytics r/Database r/foss r/datasets r/Clickhouse r/istanbul r/SQL r/learnSQL r/dataanalysis
$10k in cloud credits for startups to run data pipelines
▲ 8 r/ETL

$10k in cloud credits for startups to run data pipelines

We are launching our startup program - providing $10k in cloud credits to eligible startups to run their data pipelines and use the AI agents for analytics and automating tasks. This is usually enough credits to last up to 2 years for a startup.

Check the website for eligibility criteria and full list of perks and benefits.

https://getbruin.com/startups/

If your startup or project is not eligible, you can still use Bruin completely free & open source and self-host it or use the free tier of Bruin Cloud. There are free templates to get started, and a slack community to get quick support from us.

Disclaimer: I'm a developer advocate at Bruin

u/uncertainschrodinger — 14 days ago
▲ 1 r/startups_promotion

Get $10k in Bruin Cloud credits

Bruin is a unified data platform (ingestion, transformation, governance, AI data agents) built on open-source tools.

50+ startups and larger companies rely on Bruin for their data pipelines, analytical workflows, and AI automations. Startups usually use Bruin for internal analytics to help manage their startup and grow the product; additionally, they use Bruin to manage the data that fuels their AI product.

We have launched our startup program, providing $10k of cloud credits that can be used to run data pipelines and use AI agents - this is enough credit for most startups to last up to 2 years.

Learn more about the program eligibility criteria and benefits: https://getbruin.com/startups/

https://preview.redd.it/sdhji1cilush1.png?width=2044&format=png&auto=webp&s=43edf9399214069de629dbc07469c994b1d53ac9

reddit.com
u/uncertainschrodinger — 14 days ago
▲ 5 r/learnSQL

Learn SQL using your coding agent

I've put together a free course that teaches you SQL using your coding agent.

This way you learn SQL but also how to use a coding agent in a data role.

After some feedback from this community, the course has been updated - please check it out and share your feedback: https://getbruin.com/learn/sql-ai

reddit.com
u/uncertainschrodinger — 17 days ago
▲ 0 r/dataisbeautiful

[OC] Cost of Living and Local Affordability by Country (September 2026)

Sources: Numbeo current country rankings and city rankings (crowdsourced; all indices relative to New York City = 100) - snapshot data taken on September 21, 2026.

Tools: Bruin cli (pipeline), BigQuery (warehouse), Bruin Cloud (visualization). Basemap geometry: Natural Earth via the ECharts world map.

Limitations: Numbeo is crowdsourced, so coverage and recency vary by place; indices are relative benchmarks, not official inflation or household-budget measures. The affordability gap is a derived index difference, not a literal share of income.

u/uncertainschrodinger — 17 days ago
▲ 3 r/ETL

Webinar today about building context layer

Hi everyone, I'm a developer advocate at Bruin and we're hosting a free webinar livestreamed to LinkedIn and Youtube. It is today at 2PM UTC.

The topic is about how to build a context layer for AI data & analytics engineers/analysts agents.

We will go over the different layers of context, what each one does, how to set it up, and how to maintain it.

The webinar is for all data/analytics engineers using any data stack - the content is not specific to Bruin and all concepts and techniques work with other tools (airflow, dbt, etc.)

You can register for the event in Luma or just check our LinkedIn page for the livestream.

https://luma.com/fykcqnbv

u/uncertainschrodinger — 1 month ago
▲ 6 r/StartupFuture+3 crossposts

Free data pipeline development support for startups

I will help build out your data pipeline using free open source tools, self-host or deploy to a free/low-cost infra. Bruin has a built-in connector for Supabase/Postgres for data ingestion, transformation, etc.

disclaimer: I'm a developer advocate at Bruin leading this program, and currently we are accepting a limited number of startups into this program so you might get waitlisted for now.

What you get:

- 30min initial meeting to scope things out and plan

- 1-2h of hands on 1:1 workshop to build out your data pipeline and deploy it

- dedicated Slack channel to answer questions

- follow up touchpoint meetings and community office hours

For who:

- best suited for SaaS startups

- you want to analyze their data beyond the basic reports

- you want to combine data from your app db (supabase) with other services (e.g. stripe, posthog, hubspot, GA4, GSC, etc.) for internal analytics and reporting

- you want to process data that goes back into your application

Why:
- free open source tools to get you the data and analytics without investing a lot of time and money

- get started on building your data stack before it's too late

more info: https://getbruin.com/startups/

u/uncertainschrodinger — 2 months ago
▲ 18 r/SQL

How do you decide which incremental strategy to use in your SQL models

Choosing the right incremental strategy matters: in bigquery it reduces the amount of data scanned, while in clickhouse it means less CPU, I/O, and background merging.

but tbh, data/analytics engineers sometimes optimize this too early, if a table is small or rarely changes, a view or `create+replace` is easier than a convoluted incremental model. Sometimes full refreshes are just good enough

Before choosing anything, I normally ask: do old rows change? is `updated_at` reliable? can rows disappear? is the data truly append only? do you have proper primary keys & partitions? and how late can updates arrive?

here's an overview:

* `append` -> the source is genuinely append only

* `merge` -> existing rows change and there's a stable primary key

* `delete+insert` or `time_interval` -> you can safely rebuild a complete group or time window

* SCD2 -> you need the full history, not just the latest version

with `merge`, NULL keys can be inserted again on every run because `NULL = NULL` isn't true, I have personally lost sleep over this... (in bigquery)

with `delete+insert` or `time_interval`, only delete what you can fully rebuild - incomplete partitions, bad boundaries, or late arrivals can create gaps or duplicates.

That's why I always add quality checks that catch issues the strategy can cause: `not_null` & `unique` for merge keys, duplicate checks for rebuilt windows, and row-count or freshness checks where they make sense.

In SQL pipelines the most frustrating (and silent killer) is that a pipeline status can be successful, but that only means the SQL ran, but doesn't guarantee that the result is correct. That's why quality checks are there to catch the other issues.

and if timestamps and lookback windows still can't capture changes reliably, CDC is an option... just with more state, replay logic, and operational headaches.

the image below is an example of a `time_interval` strategy and its rendered clickhouse query (don't mind the wrapped date overlappign with INSERT)

I'm curious what other criteria do people have in mind when they try to evaluate which strategy to go with? I think this is more relevant to analytics engineers, not so much the operational/transactional side.

https://preview.redd.it/6l30zdb6yvjh1.png?width=1080&format=png&auto=webp&s=b26f3060edf5417b6d9f40d4401d4c8ceb2a9997

reddit.com
u/uncertainschrodinger — 2 months ago
▲ 3 r/ETL

I helped a client migrate from Fivetran to Ingestr, cutting their cost by 10x

I wrote a whole article about it, link in comments, but I just want share a quick summary here.

This client had over 30 data sources they were ingesting from, it looked something like this:

  • posgresql -> snowflake ~100m/rows/mo (some spikes to 1b/rows/mo)
  • all other data sources -> snowflake ~100m/rows/mo (some spikes to 1b/rows/mo)

This was costing them almost $100k per year (~$8.5k/month) and it was especially hard to justify because the spikes would normally happen during busy seasons and they already had tight margins.

They had already started using ingestr (the free open source version) to offload some of their smaller jobs and running the jobs on an EC2 instance.

Once they migrated over their production database to snowflake job, they cut the cost significantly - the total server cost came to less than $500/month.

If this resonates to anyone, and you want to put on your resume "helped reduce ingestion cost by x amount", then check out ingestr repo and docs.

reddit.com
u/uncertainschrodinger — 2 months ago
▲ 56 r/DuckDB+4 crossposts

how I learned why you shouldn't name an alias the same as the original column name

I wrote a query last week that ran fine on Postgres and DuckDB, and hard-errored on ClickHouse and BigQuery - this sent me down a rabbit hole for most of the day.

Here's what I had:
```
SELECT term, MAX(ranking_page_count) AS ranking_page_count
FROM ranked
GROUP BY term
HAVING MAX(ranking_page_count) >= 2
```

The CTE already had a column called ranking_page_count. I aliased MAX() of it to the same name, because why not, and then used that name again in HAVING.

So which one does HAVING actually filter by? Turns out that's a matter of opinion.

In Postgres, HAVING can’t see SELECT aliases at all. So it reads the column directly and lands on the same max anyway - no error, right answer.

DuckDB does let you use aliases in HAVING, but only as a fallback, and it won't put one inside an aggregate, so this also runs. This is the one that got me, since DuckDB is where I test locally.

BigQuery gives the alias priority over the column. So it read my query as MAX(MAX(...)) and gave the error "aggregations of aggregations are not allowed"

ClickHouse just swaps aliases in everywhere, so it gave code 184 illegal aggregation. it even fails when the alias isn't shadowing anything.

The thing that finally made it click for me was processing order. FROM, WHERE, GROUP BY, HAVING, then SELECT, then ORDER BY. Aliases get created in SELECT, so when HAVING runs the alias doesn't exist yet. That's why Postgres says no, and why everything else here is a vendor extension rather than four equally valid readings.

ORDER BY is the only clause that runs after SELECT, which is why it's the only clause where nobody argues.

What actually worries me is that it can go completely silent. Drop the aggregate from the alias and the loud error disappears:
```
SELECT term, ranking_page_count * 10 AS ranking_page_count
FROM ranked
GROUP BY term, ranking_page_count
HAVING MAX(ranking_page_count) > 4
```

Postgres and DuckDB filter on `ranking_page_count`
BigQuery and ClickHouse filter on `ranking_page_count * 10`
I get 1 row from the first two and 4 rows from the other two, and not one of them raises an error about it.

That's the version that ends up on a dashboard.

ok fine, I learned my lesson and won't name an aggregate after the column it aggregates...

If you work across different engines, this is your reminder to go check 🥲

reddit.com
u/uncertainschrodinger — 2 months ago
▲ 0 r/dataisbeautiful

[OC] The infamous Peter Thiel's chess game history analysis

Sources: chess dot com API

Tools: BigQuery, Bruin (CLI for processing the data, data apps for visualization)

u/uncertainschrodinger — 2 months ago
▲ 6 r/ETL

Self healing pipeline agents - free webinar

I'm hosting a free webinar to go over how you can build self healing pipeline agents.

It will be a practical and technical webinar meant for mid/senior level data/analytics engineers.

Key topics:

- overview of the context that already exists in your pipelines

- methods for enriching the context and data governance

- tools, resources, and skills for agents to diagnose, investigate, analyze, test, and run things

The main idea is that your pipeline already contain some context, you can improve that context, then set up rules, procedures, and workflows for agents to help with the diagnosis and even pushing fixes.

Disclaimer; I'm a developer advocate at Bruin - in this webinar we will use Bruin to demonstrate things, but the concepts and strategies are applicable to any tools (i.e. dbt, airflow, etc.)

https://luma.com/t4a5jrin

u/uncertainschrodinger — 2 months ago
▲ 0 r/dataanalysis

What's the difference between data apps and BI tools?

In my opinion, data apps are different in the sense that you can take actions and interact beyond just filters and drill downs.

for example, changing the projected growth percent or cogs and seeing how it impacts the bottom line, or even updating or inserting data directly from the data app (in a way, reverse ETL)

I was talking to a colleague and he said it's the same as regular dashboards but just my dynamic - but in my opinion the key difference is the ability to interact with the data itself and not just the charts.

I'm curious what others think

reddit.com
u/uncertainschrodinger — 3 months ago
▲ 6 r/ETL

Just some thoughts on incremental strategies

Choosing the right incremental strategy matters: in bigquery it reduces the amount of data scanned, while in clickhouse it means less CPU, I/O, and background merging.

but tbh, data engineers sometimes optimize this too early, if a table is small or rarely changes, a view or `create+replace` is easier than a convoluted incremental model - full refreshes aren't sexy, but sometimes they're good enough.

Before choosing anything, ask: do old rows change? is `updated_at` reliable? can rows disappear? is the data truly append only? do you have proper primary keys & partitions? and how late can updates arrive?

here's a simple overview:

* `append` -> the source is genuinely append only

* `merge` -> existing rows change and there's a stable primary key

* `delete+insert` or `time_interval` -> you can safely rebuild a complete group or time window

* SCD2 -> you need the full history, not just the latest version

with `merge`, NULL keys can be inserted again on every run because `NULL = NULL` isn't true, I have personally lost sleep over this...

with `delete+insert` or `time_interval`, only delete what you can fully rebuild - incomplete partitions, bad boundaries, or late arrivals can create gaps or duplicates.

That's why checks should match the strategy: `not_null` & `unique` for merge keys, duplicate checks for rebuilt windows, and row-count or freshness checks where they make sense. A successful pipeline only means the SQL ran, so quality checks are there to catch the other issues.

and if timestamps and lookback windows still can't capture changes reliably, CDC is an option... just with more state, replay logic, and operational headaches.

the image is an example of a bruin`time_interval` asset and its rendered clickhouse query

https://preview.redd.it/iuyeqg05yyfh1.png?width=1716&format=png&auto=webp&s=8856da232b3f45e45b7226525b4ee2d933f3763b

u/uncertainschrodinger — 3 months ago
▲ 11 r/dataisbeautiful

[OC] FIFA World Cup final teams' cumulative xG differential throughout the tournament

Sources: FIFA Training Centre public post-match reports

Tools: Python/pdfplumber, Bruin cli, BigQuery, and SVG

Limitations: The teams faced different opponents, so this describes their tournament paths rather than an opponent-adjusted rating

u/uncertainschrodinger — 3 months ago
▲ 2 r/analytics

I've consolidated some resources to help learn AI skills for data analytics and engineering

Career path Best starting resources What you should build
Student or beginner Kaggle Learn, CS50 AI, Google ML Crash Course, Microsoft AI Agents for Beginners A Python notebook that loads data, trains a simple model, asks an LLM to explain results, and checks the explanation against the data
Software engineer Anthropic Academy, Claude API development, OpenAI Agents SDK, Hugging Face Agents Course, LangChain Academy A tool-using agent with structured outputs, tests, traces, and a human approval step
Data engineer Data Engineering Zoomcamp, dbt Learn, Bruin Academy, Dagster University, Airbyte Academy A pipeline that ingests data, transforms it, validates it, exposes metadata, and lets an agent query it safely
Analytics engineer dbt Learn, Bruin Academy, Build an AI Data Analyst, LlamaIndex documentation A semantic layer or context layer that defines metrics, entities, joins, examples, and freshness checks
Data analyst Kaggle Learn, Codecademy AI for Data Analysis, Bruin Academy AI data analyst, dbt Learn A repeatable analysis workflow where the AI writes SQL, explains assumptions, and you verify the result
Team lead or manager Anthropic AI Fluency resources, OpenAI practical guide to building agents, Bruin Cloud AI agents, scheduled agents A governance checklist: what agents can access, what they can change, who approves, and how answers are audited
u/uncertainschrodinger — 4 months ago
▲ 8 r/BusinessIntelligence

I helped build an open source semantic layer tool

I feel like this year I've heard a lot of talk about semantic layer, every time the topic of "AI data this and that" comes up, inevitably people talk about semantic layer.

That's one of the reasons why we wanted to add a semantic layer to Bruin to allow users to define their semantic layer in the same repo that their pipelines are so that agents can get the full picture - ingestion, transformation, governance, and now the semantic layer.

It is still the early days for the semantic layer, but it works across all the platforms we support. I've tested it with my own data analyst agent and I've seen improvements in terms of how accurately it answers questions, but I'm curious what others think.

Has anyone tried using agents to analyze data with and without semantic layer? Did you see any improvements?

reddit.com
u/uncertainschrodinger — 4 months ago
▲ 15 r/DuckDB+1 crossposts

I helped build a simple and fast data ingestion tool

There is an open-source data ingestion CLI tool called ingestr and it is one of the easiest way to move and copy data between databases and warehouses.

One way I've used it is to ingest data from a database/warehouse or third party sources like hubspot, google ads, etc. into a local duckdb to quickly analyze data - this is especially cool when given to an AI agent to quickly ingest data and analyze it for you locally.

GitHub: https://github.com/bruin-data/ingestr
Docs: https://getbruin.com/docs/ingestr/

u/uncertainschrodinger — 5 months ago
▲ 0 r/dataisbeautiful

[OC] Wikipedia AI referenced articles growth since

Sources: Wikipedia MediaWiki Action API, Wikipedia Vital Articles / Level 4

Tools: Bruin cli, BigQuery, Bruin dac

Methodology

Universe. Two tiers (14,004 articles total, 11 top-level subjects, 110 sub-subjects). Tier 1: Wikipedia Vital Articles / Level 4 - 9,907 curated articles across all 11 subjects. Tier 2: 4,097 WikiProject Top/High-importance articles from Companies, Brands, Computing, Internet culture, and Business - added only to Society and social sciences (+2,735) and Technology (+1,362) to compensate for those areas being under-represented in Vital L4. Vital takes priority on collision.

AI seed list. 48 curated AI-topic articles spanning foundations (Artificial intelligence, Machine learning, Neural network, Deep learning, Supervised/Unsupervised/Self-supervised learning), architectures (Transformer, CNN, RNN, GAN, Diffusion model, Attention, LSTM), modern systems (LLM, GPT-3, GPT-4, ChatGPT, Claude, Gemini, LLaMA, BERT, Stable Diffusion, DALL-E, Midjourney, Generative AI, Foundation model), companies (OpenAI, Anthropic, DeepMind, Hugging Face), sub-fields (NLP, Computer vision, RL, Speech recognition, Symbolic AI, Machine translation, Robotics, Expert system), and cultural/policy (AI alignment, safety, ethics, AGI, existential risk, technological singularity, regulation, AI winter). Each canonical title is expanded with its current redirect aliases.

Snapshots. 14 semiannual snapshots at fixed dates (December 1 and May 1, Dec-2019 through May-2026). For each (article × date), the MediaWiki Action API returns the closest revision at or before the target date; body wikilinks (regex-extracted from wikitext, excluding namespace, self, and anchor links) are intersected with the AI alias list to count "AI references".

Pipeline. Raw scrapes -> staging joins -> subject/sub-subject/article aggregates. This dashboard queries staging.wat_ai_reference_counts directly. All assets run via Bruin cli on BigQuery; the dashboard renders via Bruin dac.

Limitations & caveats

Slicing & filtering. Gainer charts rank by absolute percentage-point gain since Dec 2019, not relative growth; the sub-subject chart shows the top 8 only. Both gainer charts and every small-multiples panel apply the same eligibility filter: n>=20 articles AND >=1 AI-referencing article at the latest snapshot. The 20-article floor avoids small-denominator noise (e.g. a 2-article sub-subject swinging to 50% on a single edit). Small-multiples panels show up to 7 sub-subjects (top by article count); panels with sparse AI uptake show fewer (History 2; Everyday life and Geography 3; Mathematics 4; Arts and Physical sciences 5; Biology & health 6).

Comparability. In the small-multiples grid, per-panel y-ranges are independent - compare shapes, not heights. The universe is not uniform across subjects: only Society and social sciences and Technology receive the WikiProject Top/High extension; the other 9 subjects are Vital L4 only. Cross-subject magnitudes therefore reflect both AI uptake AND uneven corpus composition.

What "AI reference" means. A structural body wikilink to one of 48 curated AI articles (plus current redirect aliases), not a semantic measure of AI content. Template-generated and navbox links are excluded; only editor-chosen body links count.

Scope. Universe is curated (Vital L4 + WikiProject Top/High in 5 categories = 14,004 articles), not a random or exhaustive sample of Wikipedia. English Wikipedia only. Results generalise to "important, well-edited articles", not to long-tail content.

Time. Some AI seed pages did not exist in 2019 (e.g. ChatGPT, GPT-4, Claude, Gemini, LLaMA, Stable Diffusion, Midjourney), so apparent growth partly reflects new AI vocabulary entering Wikipedia rather than only existing articles adopting new links. Snapshots are semiannual (Dec 1 / May 1), so spikes shorter than ~6 months and revisions reverted between snapshots are invisible. The MediaWiki API returns the closest revision at or before each snapshot date, so an article's state can be up to ~6 months stale relative to the next snapshot.

u/uncertainschrodinger — 5 months ago