r/apachespark

PySpark on Hugging Face lowers storage costs and save I/O, since HF storage buckets do intra and inter-files deduplication

"Once uploaded, never duplicated"

Examples of dedupe-compatible formats:
- Parquet, Arrow, Lance, WebDataset
- JSON Lines, JSON, CSV, text
- Media folders (images, audio, videos, pdf, etc.)

I feel like this kind of storage should be the default everywhere, since the benefits are great and transparent for users. What do you think ? Why no other cloud provider is doing it ?

Trying to implement a Realtime-valid streaming job, but got an error STREAMING_REAL_TIME_MODE.BATCH_BATCH_JOIN_NOT_SUPPORTED.

Pretty sure that this is because I'm using a view (that joins 2 tables) as a static part of the stream-static join, but I can't find any references to this error OR that there's a limitation (either for structured Streaming or the realtime mode in particular) that says "you can't join a stream to a pre-joined dataframe".

Does anyone know where can I read more about this?

And on the other side of the question, I get the restriction is probably due to sheer complexity of having to manage (potentially) multiple incoming records to the joined batch, but I still didn't see anything mentioned about this anywhere.

Edit: it's definitely a limitation of the Realtime mode specifically

I have ~4 years in Data Engineering (Kafka, Elasticsearch, K8s, Redis, Ops, SRE, DevOps and many things etc.), and I'm now going deep on Spark — joins, shuffles, skew, the Spark UI. Not looking for someone to spoon-feed me, just someone I can rubber-duck with or who'll tell me when my mental model is wrong. Seriously, solo learning this Apache Spark is becoming very difficult for me as I have started to make my hands dirty. Anyone up for occasional async questions on Discord/Reddit? Pleaseeeee...

Whether you’re just starting with Apache Spark or already building production-grade pipelines, here’s a curated collection of must-read resources:

Learn & Explore Spark

• Getting Started with Apache Spark
• Understanding Spark Architecture

Performance & Tuning

• Optimizing Spark Performance
• Partitioning & Caching Strategies

Real-Time & Advanced Topics

• Structured Streaming Tutorial
• Data Lakehouses & Spark’s Role

🧠 Bonus: How ChatGPT Empowers Apache Spark Developers

👉 Which of these areas do you find the hardest to optimize — Spark SQL queries, data partitioning, or real-time streaming?