u/jseeker528963

If you work with GCP Dataproc, you've probably hit this wall: Google doesn't provide official local container images that match Dataproc cluster components.

Testing spark apps or running CI integration tests usually meant either spinning up a live Dataproc cluster on GCP (costs $, takes time, needs lot of setup) or using vanilla Spark images that lack Dataproc-specific component versions.

We built multi-arch (linux/amd64 and native Apple Silicon linux/arm64) compatibility images pinned to official Dataproc versions: ( on dockerhub )

• Dataproc 3.0.0 (Spark 4.1.2, Hadoop 3.5.0, Hive 4.2.0, Java 21, Python 3.12)
• Dataproc 2.3.34 (Spark 3.5.3, Hadoop 3.3.6, Hive 3.1.3, Java 11, Python 3.11)
• Dataproc 2.2.85, 2.1.117, and 2.0.161

Quick Start:

bashdocker run -d --name dataproc \
  -p 9870:9870 -p 8088:8088 -p 18080:18080 -p 4040:4040 \
  agentcloud/dataproc:3.0.0-debian13

Run PySpark:
docker exec -it dataproc pyspark

What's included out of the box:

• Single-node YARN ResourceManager UI (:8088) & HDFS NameNode (:9870)
• Spark History Server (:18080)
• Hive Metastore & HiveServer2 (:10000, :9083)
• Client-mode submit and shared catalog preconfigured
• Runs as unprivileged user (localcloud)

Primary use cases:

1. Local dev: Fast feedback loops on your laptop without paying GCP bills or dealing with cloud network latency.
2. CI/CD: Run full pytest + Spark integration tests in GitHub Actions as a service container without cloud credentials.
3. AI Coding Agents: Local sandbox for Claude Code, Antigravity, or Cursor agents to test generated PySpark code safely.

*this is an Independent project, not affiliated with Google.*