r/DataScientist

So yeah. AI tool to talk to Epstein and his files

Hi everyone so I am woman getting a master degree in Data Science. I don’t want to directly work with data science but I plan to work as data analyst something related to this. The thing is I am really bad in math so I am struggling with the course. Of course there’s some self sabotage as I know that I need to study more, than I study the bare minimum and blank out in the days of assignments.

I really think it’s fascinating the statistic part and coding part, but I have always been the “humanities” person but seriously I don’t like to work with this! I even tried to work with content creation and I hated it. But I was really into the ananlytics/ADS part. Is that even possible?

Main question: can I be VERY good professional as data analyst being not a math person?

Thank you

I have 5 years of experience working in banking in Brazil, in analytics and data science. I'm currently a Senior Data Scientist, and now I'm planning to make a move to Europe.

My current English level is around B1/B1+, and I know that's probably not enough to work in my field because communication is very important (at least I think so haha)

I've saved a good amount of money over the past few years, and I'm thinking about doing an 8-month exchange program in Ireland, focusing heavily on English so I can reach fluency and get to a C1 level, which I consider enough to work in English. I'm not planning to hang out with Brazilians or spend my time travelling around Europe.

After the exchange, once I reach C1 English, I plan to apply for mid-level data science roles across Europe, with the goal of getting a work visa and eventually obtaining European citizenship.

I saw that in Spain it's possible to apply for citizenship after 2 years of residence for Brazilians, so I'm focused on starting there. I don't speak Spanish yet, but I believe that with English it should be possible to work anywhere while I study spanish.

Do you think this is a good plan? Am I being unrealistic about anything?

P.S. I forgot to mention that I have already received an opportunity from company in Spain offering visa sponsorship for English-speaking roles. I had to turn them down at the time because I didn't feel confident enough with my English, but that experience made me think this path might be realistic if I improve my language skills first.

Hey everyone,

Over the past few months, I’ve been building a spoken language identification (LID) model focused specifically on Indic languages and real-world conversational speech.

The model can automatically detect the spoken language directly from audio input, even in noisy telephony-style conversations.

Supported Languages

Hindi

English

Bengali

Marathi

Tamil

Telugu

Kannada

Malayalam

Gujarati

Punjabi

What the Model Handles

Short utterances

Call-center / telephony audio

Conversational speech

Background noise

Indian accents & regional variations

Some level of code-mixed speech

Tech Stack

PyTorch

Deep learning–based audio classification

Custom preprocessing pipeline

Audio embeddings + transformer/CNN experiments

Automated evaluation & benchmarking workflows

Biggest Challenges

One thing I underestimated was how difficult Indic spoken LID becomes in real-world data.

Some major issues:

Similar phonetics across languages

Hindi mixed with regional languages

Accent & dialect diversity

Imbalanced datasets

Extremely short voice samples

Noisy customer-support recordings

A lot of effort went into preprocessing, balancing, and improving robustness.

Potential Use Cases

IVR language routing

Multilingual voice assistants

ASR model selection

Customer support automation

Speech analytics

Voice AI systems for India

Current Focus

Right now I’m experimenting with:

Better short-utterance detection

Robustness on noisy audio

Improving confusion between related languages

Faster inference for production deployment

Looking for Feedback

Would especially appreciate:

Good Indic LID benchmarks/datasets

Ideas for handling heavy code-mixing

Production deployment suggestions

Interest in an open-source release

Happy to discuss architecture choices, datasets, or experiments if people are interested.

Early on everything looks fine:

good benchmark numbers,

clean demos,

decent validation results.

Then production starts and suddenly you’re chasing weird edge cases for weeks.

We had one vision pipeline where the actual model wasn’t even the main issue. The bigger headache turned out to be the data itself:

same images coming from different sources,

slightly different labels across batches,

missing metadata,

random scraped assets mixed with curated ones,

etc.

What made it worse is that most of this wasn’t obvious during training. It only started surfacing once we tried scaling the system and auditing failures properly.

At some point we stopped obsessing over architectures and spent more time cleaning ingestion and sourcing workflows instead.

What’s been the biggest hidden dataset issue in your projects?

5 days into a data analytics role at a 130-person manufacturer. Is this architecture overkill or am I thinking about this right.

The reporting situation I walked into is what you’d expect but worse than I initially described. We have zero data infrastructure. No data warehouse, no pipelines, no automation, nothing at all. Every single report across the entire company is built the same way - someone manually exports from SAP, pastes it into Excel, applies transformations that only they understand, and sends it out. That is the entire stack.

Just in the finance department alone I’ve identified at least 15 recurring reports done this way on the conservative side. Actuals vs plan, cost center reporting, sales register, board packages, forecasts, aging reports - all of them manual, all of them owned by specific individuals, none of them pulling from a shared data source, none of them guaranteed to reconcile with each other. When two reports reference the same underlying SAP data and show different numbers nobody can immediately explain why because the logic is undocumented and lives inside workbooks that have been manually maintained for years.

And that’s just finance. Other departments are running their own reports the same way and I genuinely don’t know how many exist company wide. Could easily be double. Could be more. Nobody has a full inventory of what reports exist, who owns them, or how they’re built.

Estimated manual reporting hours in finance alone is 60 to 100 hours a month conservatively. That’s not people being inefficient. That’s what happens when Excel is your entire data infrastructure and every reporting cycle depends on tribal knowledge and individual availability. When the person who owns a report is out the report either doesn’t go out or goes out wrong.

The other critical piece of context - we have an ERP migration coming mid to late 2027. Moving off SAP 2017 which has no API to something new TBD. Every manually built report in existence today gets rebuilt from scratch when that happens unless there’s an abstraction layer sitting between the source system and the reports. That’s not a small problem. That’s a massive organizational risk on top of an already complex migration.

I’m basically the only person here thinking seriously about any of this. My instinct is that cleaner spreadsheets aren’t the answer. The problem isn’t the Excel files, it’s that there’s no real infrastructure underneath them.

The architecture I keep coming back to is SAP exports feeding into a Microsoft Fabric lakehouse then into a Power BI semantic model. The appeal is that when the company eventually migrates off SAP the reporting layer survives. You swap the data source not rebuild everything from scratch. Every report that was built before the migration works the day after the migration.

What I’m genuinely unsure about is whether Fabric is overkill for a company this size, whether starting with a small F2 proof of concept makes sense before pushing for anything bigger, or whether I should be more patient and work within what exists until I actually understand the business better.

What y’all think. I’m new to data engineering and would appreciate the help!

EDIT: Added more context.

I am a student who has completed Pandas ,Numpy ,EDA ,ML (I can make models and deploy it on streamlit and a little bit using flask) and now I am moving for Tensor flow.

So as of now I am so much confused and want to do an internship but for what role I should apply.

I applied so many times but 90% of them are paid internships. Can anyone help me to get an internship trust me I want to do it and I will give my 100% to the role I get.

Most of what I’ve come across is either scattered across different platforms or clearly more oriented toward institutional use, so it’s hard to figure out what people actually rely on for basic research. Looking for things like yields, ratings, maturities, and a simple way to compare different issues. What do you personally use for this? 👍

I am confused and need some guidance.

I am working as a data analyst in a healthcare firm for past 2 years now.

I wanted to transition to data scientist but my current company or team has no such opportunity.

I prepared for the transition made Resume.....been applying for past 2 months. But getting rejected from everywhere.

I went 3 rounds interview in another healthcare consulting firm for the position of data scientist but they have rejected me.

Went 2 rounds in another company for the role of ML Engineer ( AI interview + Assessment) .... Another online assessment for DS role.....but those rounds were default means prolly they were sent to everyone who applied.

The other assessment I have given so far for 5 companies are for Business Analyst role. One more interview for business analyst role.

Got rejected or ghosted from them as well.

I don't have any masters degree on data science since lot of companies ask for it. I was considering to do a online MTech on DS after I made the DS switch. But without switch, I am not very sure to invest money in a Masters.

Reached out to some people for how did they transitioned... but no reply.

My performance hasn't been good in my current job. I will probably get laid off within 2 months. I am burnt out and don't want to actually pursue a career in consulting and that's why I started studying 9-10 months ago for DS.

Be brutally honest and tell me what I should do