Built a localized tax data pipeline processing 150K fixed assets using Polars and DuckDB (feedback on business logic vectorization?)
Hey everyone,
I’m a recent B.Com Computer Applications graduate with a CS background, working on an end-to-end Tax Data Engineering vault (Project TTA).
I wanted to move away from standard Pandas iterative workflows and test how DuckDB + Polars handles complex, state-dependent corporate tax logic across 150,000 records.
What the pipeline does:
• Ingests 150k asset records directly from a local DuckDB vault emulating an SAP S/4HANA sub-ledger via zero-copy `.pl()` exports.
• Vectorizes non-linear statutory rules (e.g., Section 179 dollar-for-dollar phase-out caps, conditional ADS vs. GDS routing).
• Aggregates entity-level dates to evaluate whether Q4 additions breach a 40% threshold, dynamically flipping depreciation lookup matrices from Half-Year to Mid-Quarter conventions.
• Writes finalized arrays back to DuckDB with strict DECIMAL(18,4) casting for idempotency.
I recorded a raw technical walkthrough of the code and output tables: https://youtu.be/yvQlwaBQOvY
Repo: https://github.com/ibrahimworksfr-code
Would appreciate any feedback on how I can further optimize the Polars query plan or handle out-of-core streaming as I scale this to 5M ledger rows in Phase 2.