PySpark on Hugging Face lowers storage costs and save I/O, since HF storage buckets do intra and inter-files deduplication

"Once uploaded, never duplicated"

Examples of dedupe-compatible formats:
- Parquet, Arrow, Lance, WebDataset
- JSON Lines, JSON, CSV, text
- Media folders (images, audio, videos, pdf, etc.)

I feel like this kind of storage should be the default everywhere, since the benefits are great and transparent for users. What do you think ? Why no other cloud provider is doing it ?

I worked on how HF deduplicates any Parquet data by chunk at intra and inter-files levels.
Lower storage cost and faster I/O for everyone. This is super useful so here is how it works:

How it works

1. data frameworks encode data in formats like Parquet in memory or on disk (row groups / columns / pages)
2. the resulting file is passed to the File system interface for upload, and points hf:// paths to the correct client: Xet
3. the Xet client uses a rolling hash algorithm to chunk the Parquet file in a smart way, aims for robust chunks boundaries every ~64KB, and asks the server which chunk already exist online
4. the HF Storage Buckets server says with the chunk that already exist so the client only uploads new data. In practice, only the parts of the Parquet pages containing new data are uploaded. Even better if the Parquet file is written with Parquet Content Defined Chunking which makes pages boundaries robust to insertions/deletions
5. the Xet backend stores the physical chunks as blocks of 64MB for spatial locality into the CAS (Content Addressable Storage)

(I did a talk on this recently so I can elaborate if needed / share the slides btw)

Open Source tools

Data frameworks like pandas, datafusion and pyspark (via a custom data source) support both file system interfaces and Parquet Content Defined Chunking. Pandas and pyspark have Parquet CDC via pyarrow (docs), while datafusion has CDC via arrow-rs and the parquet crate (docs).

The file system implementations are open source as well at huggingface-hub (docs) (python) for fsspec/pandas and at OpenDAL (docs) (rust) for arrow-rs / datafusion.

The Xet client is open source as well, at huggingface/xet-core

Results

- Data is never downloaded twice: reduced I/O operations and outbound traffic

- Data is never stored twice: reduced storage costs

- All this process is transparent and you only see the files as usual

Here is how HF deduplicates any kind of files by chunk at intra and inter-files levels.
Lower storage cost and faster I/O for everyone.

How it works

1. data frameworks encode data in formats like Parquet in memory or on disk

2. the resulting file is passed to the File system interface (fsspec/opendal) for upload, and points hf:// paths to the correct client: Xet

3. the Xet client uses a rolling hash algorithm to chunk the Parquet file in a smart way, aims for robust chunks boundaries every ~64KB, and asks the server which chunk already exist online

4. the HF Storage Buckets server says with the chunk that already exist so the client only uploads new data. In practice, only the parts of the file containing new data are uploaded

5. the Xet backend stores the physical chunks as blocks of 64MB for spatial locality into the CAS (Content Addressable Storage)

Let me know if you have questions about any of these points, I'm happy to elaborate !

Results

- Data is never downloaded twice: reduced I/O operations and outbound traffic

- Data is never stored twice: reduced storage costs

- All this process is transparent and you only see the files as usual