r/learndatascience

Ask questions across your Markdown notes using a fully local Graph RAG engine. Built for Obsidian vaults, works with any folder of Markdown files. Extracts entity-relation triples from wikilinks & YAML frontmatter, retrieves answers via hybrid search (vector + BM25 + temporal). Multilingual. No cloud. Runs on Ollama.

https://github.com/benmaster82/Kwipu

I’m working on a small project to adapt a statistical analysis skill for use inside ChatGPT in Excel.

The original skill came from Claude and already had a solid statistical foundation. It covered descriptive statistics, trend analysis, outlier detection, and hypothesis testing. However, when I started testing it in a spreadsheet environment, I noticed a gap.

The answers were often technically reasonable, but not always structured in a way that was useful for a business analyst, financial analyst, or FP&A user working inside Excel.

The goal is not to turn Excel into an academic statistics lab. The goal is to make statistical reasoning more usable for real business cases, such as:

• A/B testing campaign results
• Comparing sales performance between two segments
• Testing before/after changes after a training, promotion, or process improvement
• Comparing conversion rates
• Checking whether two categorical variables are related
• Identifying outliers or unusual business behavior
• Explaining whether a difference is likely real or just normal business noise

The biggest area I started refining is hypothesis testing.

I expanded the workflow so the skill does not immediately jump into a formula or test. Instead, it should first interpret the business question, identify the correct type of comparison, define the null and alternative hypotheses in plain language, check assumptions, select the right test, and then produce a structured business-readable conclusion.

The desired output is something like:

1. Business question being tested
2. Statistical test selected and why
3. Null and alternative hypotheses in plain English
4. Assumptions and caveats
5. Result and p-value
6. Effect size or business impact
7. Decision: reject or fail to reject the null hypothesis
8. Practical interpretation
9. Recommended business action

To test the skill, I created an Excel workbook with different business scenarios, and test it on ChatGPT for Excel using the /statistical-analysis prompt

The main question I wanted to answer is: “Can AI help a business analyst choose the right statistical method, explain it clearly, and turn the result into a better business decision?

This project is still an early iteration. My current focus is making the hypothesis testing section more reliable and useful inside Excel. Future improvements may include correlation and regression workflows, more finance-oriented examples, and better output formatting for spreadsheet-based reporting.

If interested, please give me your feedback. As a Excel user for more than 15 years I am very interested in your opinion on this. If you want to inspect the project repo, you can find it here: https://github.com/Ogzapatah1/statistical-analysis-skill-for-excel

I've spent the last few months building something and I'm finally at the point where I want to share it properly rather than just quietly hoping people find it.

The idea came from a frustration I kept seeing (and feeling myself): SQL tutorials teach the syntax fine but there's never a reason to care about the answer. You filter a table called employees, get a result, and nothing happens. Your brain doesn't bother keeping it.

I wanted to try a different approach. QueryCase teaches SQL through detective investigations. You get a briefing from Chief Fox (our mascot), a real database to query, and a mystery to crack. The JOIN matters when a suspect has an alibi. The WHERE clause matters when you're trying to find who entered the building at 22:13. The SQL is the tool for solving something, not the point in itself.

Here's what's actually in it:

• A structured learning path across 54 cases, going from Recruit through Rookie, Detective, Senior Detective, and Chief Detective. Each rank has drills and a level exam to pass before you progress.
• Sandbox mode where you can explore real datasets (IMDB movies, Spotify, sports stats, Steam games) and run whatever you want with no pressure and no mystery attached. Just free exploration against actual data.
• Everything runs in the browser using DuckDB WASM so there's nothing to install.

I'm a solo developer and this is genuinely early days. I'm sharing here because this community is exactly the kind of people I built it for, and I'd rather get honest feedback now than find out later I've built the wrong thing.

What's missing? What would make you actually stick with something like this versus what you've used before?

querycase.com if you want to take a look.

Any feedback appreciated!

Weekly analysis of AI & Data Science job postings from Indian job boards.

Sample size: 12,614 listings (till May 16, 2026).

---

**Top Skills — Full Breakdown:**

| Skill | Mentions |

|--------------------|----------|

| Python | ~2,600 |

| SQL | ~2,400 |

| Machine Learning | ~1,500 |

| Artificial Intelligence | ~1,050 |

| Azure | ~1,000 |

| Java | ~1,000 |

| AWS | ~800 |

| GCP | ~600 |

| Spark | ~600 |

| Data Analysis | ~550 |

---

**Key observations:**

**SQL is basically tied with Python now**

Gap is only 200 jobs. Everyone learns Python first but companies

still need SQL everywhere — pipelines, reporting, analytics layers.

If you skipped SQL thinking it's "old", reconsider.

**Azure quietly entered top 5**

~1,000 mentions. AWS was the default for years but Azure

is catching up fast in Indian enterprise hiring, especially in

BFSI and consulting. Both Azure + AWS together = ~1,800 jobs.

**Consulting firms are the real AI employers**

Top 10 companies hiring AI talent:

| Rank | Company | Jobs |

|------|------------|-------|

| 1 | TCS | ~360 |

| 2 | Accenture | ~340 |

| 3 | Leading Client | ~310 |

| 4 | Infosys | ~150 |

| 5 | EY | ~145 |

| 6 | Capgemini | ~130 |

| 7 | Amazon | ~110 |

| 8 | Databricks | ~105 |

| 9 | CGI | ~105 |

| 10 | IBM | ~100 |

EY and Capgemini in the top 6 is interesting —

Big 4 consulting is aggressively building AI/data practices.

Databricks at #8 means data engineering is very real demand.

"Leading Client" still at #3 = staffing firms hiding actual employers.

**City breakdown (expanded):**

| City | Jobs |

|------------|--------|

| Bengaluru | ~3,000 |

| Hyderabad | ~1,950 |

| Pune | ~1,200 |

| Chennai | ~850 |

| Mumbai | ~850 |

| Gurugram | ~550 |

| Remote | ~500 |

| Noida | ~480 |

Chennai entered the top 4 this time —

mostly TCS/Infosys/Accenture campuses expanding AI teams there.

---

**Takeaway from this week:**

The "learn GenAI or die" crowd is louder than the actual job market.

Real JDs: Python → SQL → cloud (Azure/AWS) → ML fundamentals.

That stack gets you through 80% of listings.

Tracking this weekly at getjobpulse.in if anyone wants the dashboard.

Anyone seeing Azure demand spike in their interviews too?

We are thrilled to invite you to try dashAI, an open-source that runs entirely on your own computer, without the need to write code. DashAI is designed to train and evaluate Machine Learning and generative AI models.

Some design decisions:

• No cloud dependency

• No external authentication or API keys

• Plugin architecture based on typed abstractions

• UI generated automatically from Pydantic schemas

• Support for predictive and generative models

• Explainability integrated into the workflow

• Extensions distributed through PyPI

We're trying to build something closer to an open-source alternative to cloud AutoML platforms while preserving transparency and local control.

Test the software: Download it and share your observations with us. Hearing your thoughts is our top priority during this early phase. Website: https://dash-ai.com/

Support open source: If you find the project valuable, we invite you to leave us a star on our GitHub: https://github.com/DashAISoftware/DashAI

Join the community: We are looking for users and contributors who want to get involved in refining this platform on Discord, Google Group or email.

We'd love feedback, bug reports and contributions.

Today I learned something that I think I rushed through when I first started learning ML.

The equation y = wx + b looks almost too simple, so I never paid much attention to it.

What finally clicked for me is that this isn’t just the equation of a line.

With one feature, you’re fitting a line.

With two features, you’re fitting a plane.

With n features, you’re fitting a hyperplane in n-dimensional space.

The equation barely changes: y = w₁x₁ + w₂x₂ + … + wₙxₙ + b

Another thing I didn’t know until today:

“Linear” doesn’t necessarily mean the relationship between x and y is a straight line. It means the model is linear in its parameters (the weights). So you can use features like x² or log(x) and it’s still linear regression.

That also helped me understand why linear models are still widely used in production—they’re simple, interpretable, and every weight has a meaning.

Kind of funny that I spent more time trying to understand transformers than the equation almost every supervised ML model builds on.

For people who’ve been doing ML for a while: I’m working through ML from first principles. What topic should I dive into next?

Still building on linear regression, equation is just the half part of thestory and the other half is how the model learns.

this is all what i already knew ok :)

I knew gradient descent minimized the loss function.

I knew linear algebra helped scale the equation.

I never connected the two in my head!!

What finally clicked today after reading more is that they’re solving two completely different problems.

Linear algebra lets the equation scale from one feature to thousands.

Calculus lets the model figure out which parameters minimize the error

Then my brain did a complete 180 on linear regression lol,

The loss function for ordinary least squares is convex.

which means gradient descent isn’t just “trying its best” to find a good solution. Right?

it can actually converge to the global minimum because the loss surface is convex, so there aren’t any local minima to get trapped in.

Then I found out something even cooler.

Gradient descent isn’t even necessary.

There’s an exact analytical solution:

θ = (XᵀX)⁻¹Xᵀy

No learning rate.

No iterations.

No waiting for convergence.

Just the optimal parameters.

The reason we still use gradient descent is because inverting huge matrices gets ridiculously expensive as the number of features grows.

So we intentionally trade an exact solution for a computationally cheaper approximation.

That was such an “ohhhh” moment for me.

This is basically me learning ML in public at this point :p

If I’ve messed something up or oversimplified anything, please roast the explanation, not me. I’d genuinely appreciate corrections from people who’ve been in the field longer :)

Dear Folks, I have created multiple content on Machine Learning(work in progress), and they are free. I am a data scientist and a post grad degree holder in AI/ML from IIT. To help the machine learning community with important Machine Learning Concepts, I have created multiple long form videos, and structured topicwise digestible contents structured as playlists for learning.

If you go through the first two playlists:

Introductory Machine Learning Concepts
Probability Foundations: Univariate Models

You might find helpful content, I have tried explaining with intuitions, derivations, and this is work in progress. For code implementations, scikit learn website has great content on them as well. In total they have 60+ topicwise videos so far, and I think they have the potential to help folks a lot in starting with concepts, or getting with mathematical concepts, or whether you are preparing for an AI/ML/Data job interviews etc.

When I sat for my interviews, I was grilled on my project, but majority of questions from my project tested more on foundational concepts and there know how’s.

These are FREE content on youtube. This is for the benefit of the learning community.

Link: https://youtube.com/@aayushsugandh4036?si=w5MKORU2fWzLRrAJ

Hi everyone,

For the last few weeks, I've been building a project called Confluence.

I originally started it because I struggled to build intuition while learning machine learning. I found myself constantly switching between notebooks, documentation, videos, and different visualization tools, and none of them really brought everything together.

The goal wasn't to replace scikit-learn or Jupyter notebooks. Instead, I wanted a place where I could experiment and immediately see what changing a hyperparameter actually does.

At the moment, the project includes:

• 38 machine learning algorithms
• 25 datasets (real-world + synthetic)
• Interactive decision boundary visualizations
• Training animations
• Prediction explanations
• Side-by-side algorithm comparison
• Algorithm encyclopedia
• Python code generation for experiments

Everything runs on a FastAPI backend using real scikit-learn models rather than browser-only simulations.

I'd really appreciate feedback from people who work with ML regularly.

What features would make a tool like this genuinely useful for learning or teaching machine learning?

Website:
https://confluence.website

GitHub:
https://github.com/mahirmlk/Confluence

Hello Folks,

Have you ever wondered why we use sigmoid function so often in Machine Learning? Although it gives us a probability, it comes from Exponential families, and this exponential family, subsumes many of the distributions, that we study in Machine Learning.

In this lecture, we understand exponential families, Directional derivatives(Gradients and Hessians), study mixture Models, and understand how domain knowledge in Probabilistic Graphical Models makes our life simpler to model joint probability densities.

Timeline breakup(in hours and minutes):
0:00-0:17 - Understanding exponential families.
0:17-0:27 - Deriving Sigmoid Function for Bernoulli.
0:27-0:48 - Understanding log partition function, convex functions and proving why positive definite of hessians imply convexity, and why convex needed?
0:48-1:04 - Directional derivates(deriving gradients and hessians)
1:04-1:26 - Maximum entropy derivation of the exponential family.
1:26-1:56 - Mixture Models(Gaussians and Bernoulli Mixture Models)
1:56-2:16 - Probabilistic Graphical Models
2:16-2:34 - Markov Chains
2:34-End - Inference and Learning, Plate Notation diagram of Gaussian Mixture Models.

If you have watched earlier of my lectures from the playlist, they will help. I try explaining as if I am a learner, to simplify complex concepts. Everything I write in whiteboard, and these are completely FREE lectures to mention.

Link: https://youtu.be/T1uTBtJ7aHU?si=rozXSTjtSqPaaYb5