r/DataEngineeringPH

hi! im a comsci freshgrad planning to take the datasci route.

for context, i took up datasci, machine learning, and deep learning electives during undergrad (and my undergrad thesis will be published next month, which involves both ml/dl.)
moreover, since im a comsci graduate, so python and sql are something im alr familiar with.

however, upon job hunting for a 3-4 weeks, i realized that most data science roles are senior level, and only very few companies offer junior data scientist roles huhu. the anxiety of not getting any job related to datasci has been eating me up lowk😭

honestly idk why im writing this haha maybe to just rant or to ask tips or maybe to seek some reassurance that the job market is gon get better for me, the next few months😭😭

I’m trying to understand how people decide whether messy data can be safely repaired or whether it should be left alone/escalated.

For example, suppose you discover:
missing values
duplicates
invalid-looking values
inconsistent formats
unusual/outlier values

Some of these seem straightforward to fix, while others may require knowing the underlying business context.

How do you decide whether a problem is safe to automatically repair?

And are there situations where the data technically violates an expected rule but you would not change it because you don’t know what the correct value should be?
I’d especially appreciate examples from real projects.

Hi! I graduated with a BS in Mathematics and I'm currently waiting for my deployment with the DepEd for my return of service for my scholarship. I really want to work with data and turn actionable insights from it. Right now, I'm just stuck with my ROS.

As of now, my only plan is to study Python and some libraries. Also, to create some projects that I can relate to teaching so I can at least enjoy my ROS. Can you guys suggest any ideas that I can do? So that once I finish my ROS, it will be easier for me to apply in the field of data science, and I'm hoping it won't be an entry-level position. Thanks in advance!

Throughout my career, I was trained to ingest broad, transform narrow.

Meron akong ginawang pipeline na ini-ingest yung snapshot ng lahat ng data from a source via API once on a daily basis. Around 6k rows daily

Ngayon yung data na yun, ginagamit sa dashboard. Some transformations done before it reaches the presentation layer. Sa meeting namin kanina, may biglang pinapaexclude na subset ng data given certain filter/criteria. Pinapakita yung data na yun before. Bigla lang pinaaalis ng stakeholder

So our "lead" engineer insists na wag nang iingest yung data na yun given gusto namang i-exclude sa dashboard. Even claimed na it is the best practice. I argued na why not sa dashboard layer na lang i-exclude since hindi naman dapat mahirap isama sa where clause yun.

Best practice ba talaga na iexclude from the ingestion step pa lang? Kahit mga non-data stakeholders sa meeting kanina off dun sa "best practice" na sinabi niya eh

Hi, I came across this term while checking out airflow and it seems like a good way of keeping track of mutations happening in your data. How did you implement this in your job? I read there are quite a few ways to do this but I'm curious ano ginamit nyo professionally. 😁