r/data

With the growing demand for tech skills worldwide, where do you think the best opportunities exist for professionals in Data Analysis, Data Science, and Artificial Intelligence — both in the job market and freelance industry?

Which field currently offers:

More job openings?

Better freelance opportunities?

Higher income potential?

Easier entry for beginners?

I’d love to hear your thoughts and experiences from different industries and countries.

Hi

I just done my final uni project on analytics

I used python for cleaning

There were multiple data sets were involved (some are 1.8+million rows)

I have done my analysis and reviews and recommendations

The only thing I regretted is that i haven't cleaned data properly because the entire data is too messy and given in "raw txt" format by professor

Whatever i do with cleaning still some mistakes were

So i all want to ask you is

Suggest some youtube tutorials and books for me to improve data cleaning

And also which other software should i learn other than python for cleaning data

I'll just say it: we had really good dashboards. Like, genuinely good. Multiple tools, clean data pipelines, weekly reviews. I was proud of the setup.

And then a store underperformed for basically two quarters before it got escalated. Not because the data wasn't there. It was there the whole time. Just... nobody put it together until it was already a problem.

That one stung.

I spent a while trying to figure out what actually went wrong. The honest answer is embarrassing: we were measuring everything and investigating nothing. The dashboards showed the numbers. They didn't ask questions about the numbers. And we didn't have enough people with enough time to ask those questions across every location, every week.

Turns out we were relying on our best people to notice things. Which works fine until your best people are stretched thin, which is always.

What I actually changed after that:

Stopped thinking about analytics as "how do we surface information" and started thinking about it as "how do we make sure the right questions get asked, even when nobody has time to ask them."

Practically, that meant:

• Building explicit screening logic that flags locations before they need a full investigation. Just: does this location need attention right now, yes or no.
• Separating that from the investigation itself. Why is it flagged? What's driving it? What do we do? Different question, different process.
• Accepting that the analyst who's already worried about a store will dig. You need coverage for the stores nobody's worried about yet.

We eventually tried a few different approaches. One of them was a tool called Scoop that basically encodes how your best operator investigates and then runs that same logic across your whole portfolio automatically. Honestly I was skeptical of the AI framing at first but it's less "AI magic" and more "your best person's checklist, applied everywhere, every week." That reframe made it click for me.

Not saying it's the right answer for everyone. The underlying problem is the same regardless of what you use: investigation capacity doesn't scale the way data infrastructure does, and most BI stacks don't even try to solve that.

Anyway. Hard lesson. If anyone else has figured out how to solve the "we have the data but nobody's actually investigating it" problem I'd genuinely love to hear what worked.

"I got my first analyst role straight out of undergrad and started a part time masters at the same time. On paper I'm doing fine. Good performance reviews, my manager has me leading two projects now, decent grades in school.

But every single morning I open Slack and brace for the message that says ""we've reviewed your work and there's a problem."" When I get pulled into a meeting with no agenda I assume it's about me. When senior people on my team ask me a question I rehearse my answer 4 times in my head before speaking.

I don't think I'm bad at my job. I can defend my work and my logic when challenged. But there's this gap between what people see and what I feel and it's exhausting to maintain.

Talked to a friend who's been an analyst for 6 years and she said it doesn't really go away, you just get better at noticing when it's the anxiety talking vs. an actual signal. Is that the consensus or is she just being nice to me?

Posting this on a throwaway-feeling kind of morning. Coffee hasn't kicked in yet."

Our small business is getting overwhelmed with invoices lately, and manually entering everything into spreadsheets is starting to take way too much time. Looking for soft͏ware that can automatically capture invoice details (like vendor, date, totals, line items) from PDFs or email attachments so we don’t have to keep typing everything in or fixing errors after every upload.