Research infra: Does a table format really add any significant value if you can just sync a predictable Parquet layout to local NVMe?
Hey everyone,
I’m looking at data -> research workflows for Mid-Frequency Trading (MFT) ML pipelines using Python (Ray, Polars).
A common industry trend is using table formats like Apache Iceberg or Delta Lake on object storage (S3). However, if you already enforce a highly static, predictable directory layout (e.g., equities/exch=.../year=.../ with <50 optimized Parquet files per leaf), I'm struggling to see the value.
In a high-performance research environment, it seems far more practical to treat object storage strictly as a cold source of truth, sync the required historical partitions directly onto the compute nodes' local NVMe scratch disks, and run active Python training loops entirely on local NVMe.
If you are caching a predictable folder structure down to local NVMe anyway, does an object-store table format buy us anything substantial, or is it just added complexity?
For those working on Quant Platform or QR Infrastructure teams: Do you actually query Iceberg/Delta tables directly from cloud storage during active research, or do you use the "Cloud Archive -> Local NVMe Hot Compute" pattern?
Thanks!