I thought the community might find this interesting - I used entity resolution software (disclosure: from my company) to deduplicate the missing persons data from Venezuela and compare it to the list of patients in hospitals.

https://medium.com/tilo-tech/i-deduplicated-53-000-missing-persons-reports-from-venezuelas-earthquake-74f05c37521b

I recently pulled all my personal finance data into one place: monthly spending by category, savings rate, investment returns, debt paydown progress. Started in Excel but I'm thinking about moving to Python or a simple dashboard eventually.

The problem is I keep secondguessing which metrics actually tell a useful story versus which ones just look interesting but don't drive any real decisions. I track net worth monthly, for example, but I'm not sure that granularity adds value or just creates noise when markets swing around.

For those of you who've done personal finance analysis projects, which metrics did you actually check regularly and act on? And what did you expect to be useful but turned out to be kind of pointless in practice?

Also curious whether you built anything visual or just worked off raw tables. My instinct is that a simple savings rate trend line is genuinely more useful than a fancy dashboard, but maybe I'm wrong.

Would love to hear how others have approached this, especially around choosing the right level of detail without overcomplicating things. There seems to be a real gap between data that's interesting and data that actually changes behavior.

Iam looking for ideas from people working in data analytics. Was there a personal, work, or portfolio project that significantly improved your analytical thinking or technical skills? What made that project so valuable compared to learning from courses alone?

Curious how far people are taking this. Beyond using AI to write queries or clean data, is anyone building actual tools with it? Things like:

• interactive dashboards or KPI trackers
• report generators
• small internal apps for the team to use

Or does most of it stay inside Power BI / Tableau / a notebook and never really become a standalone thing?

And if you have built something standalone - what happened next? Did it get shared with the team, or did it just stay on your machine as a one-off?

Genuinely interested in where the line is these days between "AI helped me analyze" and "AI helped me build a thing other people use."

I'm preparing for a data analyst role and have been learning SQL, Power BI, and Python. One area I'm struggling with is domain knowledge and business thinking.

How did you learn about different business domains (e-commerce, banking, healthcare, etc.)? What resources or approach helped you the most?

Also, when you're given a business problem, how do you approach it? How do you break it down, decide which metrics to analyze, and identify the root cause?

I'd really appreciate any advice or resources that helped you when you were starting out.

I've noticed that many online courses focus heavily on tools like Excel, SQL, Python, and Power BI, but real-world work often requires skills that aren't emphasized enough. Looking back, what's one data analysis skill, mindset, or habit that made the biggest difference in your career? I'm especially interested in lessons that beginners usually overlook.

I'm currently learning python along with that have created study group for like like minded people let me know if you want to join

Hi all,

What’s the best way to match account names across two datasets when there are no common IDs and the naming conventions differ?

I was planning to use Python with fuzzy matching (using a similarity threshold), but are there any better tools or AI-based solutions you’d recommend for this kind of entity matching/data reconciliation?

Thanks!

Hello guys, for my research I need to analyze and organize large amounts of data from research papers. I am looking for an AI model that is best fit for this job like putting stuff into excel/spreadsheet nicely and organized in the way we want it.

I tried chatgpt premium and it seems very good, but I'm just wondering if there are any other models that are better for this.

Thank you

Hey everyone,

I work as a data analyst for a client with incredibly locked-down security. If you’ve ever worked in this kind of corporate environment, you know the drill: no access to cloud data warehouses, no advanced developer tools, nothing. My entire world is basically restricted to standard Excel and Power BI.

Recently, I hit a massive wall. I had to clean and analyze flat CSV files ranging anywhere from 4GB to 8GB. Trying to open these in Excel is a joke, and waiting for Power Query to crunch through the transformations was taking forever and completely freezing my machine.

Now, I’m not a professional developer by any means, but I was so frustrated with the tool limitations that I decided to see if I could build a lightweight, custom Enterprise SQL Workbench to handle the heavy lifting while keeping everything completely local to respect data integrity and security rules.

The backend is entirely Python-based, but I set it up so that my non-technical colleagues can use it without writing a single line of code. It pairs Streamlit for a clean browser interface with DuckDB for crazy fast, in-memory processing, and the Calamine engine to handle heavy Excel parsing.

>

What it actually does:

• Zero cloud or database setup: Everything runs locally inside an isolated memory sandbox. No servers to configure, and zero data leaves your machine.
• Handles massive files instantly: Because DuckDB processes data in columns (vectorized), it slashes through 4–8GB datasets and runs complex analytical queries in less than a second.
• Flexible Multi-File Loading: It lets you mount multiple datasets sequentially into your active session. You can either use Direct File Paths (great for instantly mounting huge files without making copies) or just drag and drop via standard Browser Uploads.
• Clean Query Editor: It integrates streamlit-ace so you get a proper dark-mode SQL editor right in your browser with syntax highlighting, line numbers, and a sidebar to explore your active table schemas.
• Direct-to-Disk Exporting: If a query pulls a massive result set that would crash a browser tab, it uses DuckDB streams to dump the entire output straight back onto your local hard drive as a .csv or .parquet file.
• Multi-Sheet Excel Support: It automatically splits and maps multi-sheet workbooks into individual, clean database tables.

The "One-Click" Magic for Colleagues

Since my teammates aren't developers either and don't use GitHub, I bundled the entire setup into a single .bat script launcher.

Now, all they have to do is double-click a desktop icon. The batch script quietly spins up an isolated virtual environment in the background, pulls the latest UI code directly from my GitHub, checks the dependencies, and launches the interface right in their default web browser. The coolest part? If I optimize the code on GitHub, their desktop launcher automatically grabs the update the next time they open it.

Give it a spin and let me know what you think!

I’ve made the repo public so anyone dealing with corporate data constraints can use it. Please feel free to grab the batch file, throw some of your heaviest datasets at it, and test it out for yourself!

Since I'm still learning the development side of things, I would love to hear your thoughts and suggestions:

• How does the processing speed feel compared to your usual Excel/Power Query workflows?
• Are there any specific SQL features or shortcuts you think I should add next?
• Any tips for further optimizing local memory when pushing past 8GB?

Check out the code or grab the script template here: 👉 GitHub Repository:https://github.com/Nikhil-Maske/sql-workbench

Let me know your feedback or if you run into any quirks while testing it!

Hello i wish some one can help me…
I have sheet with more than 200 product SKUs with names.. I work in a warehouse and it needs to check every product.
There is any way to make an app or other way to only write the product name then it give me product SKU to record it in the warehouse system.

I need it to be in my phone.

Was it SQL, Excel, statistics, data visualization, business understanding, or communication skills? Curious to hear what made the biggest difference in real-world work.

Have some queries on how to enhance domain knowledge. any materials, books, courses that I could use ?

I come from engineering background, the credit and banking knowledge hinders my ability to come up with better insights.

There are hundreds of teachers, coaches and mentors across YouTube, LinkedIn etc., but it feels like their real income comes from content creation or course sales, not from any real data work. I am genuinely curious: has anyone actually landed a data role in the last 5 years by following one of these roadmaps, especially without a tech degree and coming from a completely unrelated field?

Right now the whole thing looks like a machine designed to keep people learning forever. It seems like a large share of learners worldwide are essentially the target audience for these online advisors. Would genuinely love to be proven wrong. If you have seen real examples or experienced this yourself, I’d be interested to hear.

Iam curious which skills turned out to matter the most in real world projects compared to what is typically taught in courses or bootcamps.

I have a csv file which will be turned to an SQLite database (480k rows). Content: 5 years of real estate transaction statistics. I'll update the database twice a year with fresh data overwrite (I keep it 5 years).

I'll build a one page dashboard that prettyfies all that data with various graphs.

This is a "freemium" feature for very niche users so READ ops count will be limited.

With that context in mind, which simple, easy to use cloud database solution would you recommend? I'm a no coder, and have learned over the past 6 years how databases, backends, frontends work, i just can't write pure code. That's why simple / easy is important.

Thanks for reading.