Mali ba ako dito? Di ko sure eh.
Throughout my career, I was trained to ingest broad, transform narrow.
Meron akong ginawang pipeline na ini-ingest yung snapshot ng lahat ng data from a source via API once on a daily basis. Around 6k rows daily
Ngayon yung data na yun, ginagamit sa dashboard. Some transformations done before it reaches the presentation layer. Sa meeting namin kanina, may biglang pinapaexclude na subset ng data given certain filter/criteria. Pinapakita yung data na yun before. Bigla lang pinaaalis ng stakeholder
So our "lead" engineer insists na wag nang iingest yung data na yun given gusto namang i-exclude sa dashboard. Even claimed na it is the best practice. I argued na why not sa dashboard layer na lang i-exclude since hindi naman dapat mahirap isama sa where clause yun.
Best practice ba talaga na iexclude from the ingestion step pa lang? Kahit mga non-data stakeholders sa meeting kanina off dun sa "best practice" na sinabi niya eh