r/googlecloud

If you work with GCP Dataproc, you've probably hit this wall: Google doesn't provide official local container images that match Dataproc cluster components.

Testing spark apps or running CI integration tests usually meant either spinning up a live Dataproc cluster on GCP (costs $, takes time, needs lot of setup) or using vanilla Spark images that lack Dataproc-specific component versions.

We built multi-arch (linux/amd64 and native Apple Silicon linux/arm64) compatibility images pinned to official Dataproc versions: ( on dockerhub )

• Dataproc 3.0.0 (Spark 4.1.2, Hadoop 3.5.0, Hive 4.2.0, Java 21, Python 3.12)
• Dataproc 2.3.34 (Spark 3.5.3, Hadoop 3.3.6, Hive 3.1.3, Java 11, Python 3.11)
• Dataproc 2.2.85, 2.1.117, and 2.0.161

Quick Start:

bashdocker run -d --name dataproc \
  -p 9870:9870 -p 8088:8088 -p 18080:18080 -p 4040:4040 \
  agentcloud/dataproc:3.0.0-debian13

Run PySpark:
docker exec -it dataproc pyspark

What's included out of the box:

• Single-node YARN ResourceManager UI (:8088) & HDFS NameNode (:9870)
• Spark History Server (:18080)
• Hive Metastore & HiveServer2 (:10000, :9083)
• Client-mode submit and shared catalog preconfigured
• Runs as unprivileged user (localcloud)

Primary use cases:

1. Local dev: Fast feedback loops on your laptop without paying GCP bills or dealing with cloud network latency.
2. CI/CD: Run full pytest + Spark integration tests in GitHub Actions as a service container without cloud credentials.
3. AI Coding Agents: Local sandbox for Claude Code, Antigravity, or Cursor agents to test generated PySpark code safely.

*this is an Independent project, not affiliated with Google.*

I’m the CTO of MobiusEngine.ai . we are a fast growing startup and we want to make the process of job applications free and easy and we want to help people land jobs faster.

Today, October 7, our Google Cloud promotional credit balance dropped by approx $35,000!!!!

Our investigation found four old, previously near-idle Gemini API keys suddenly generating massive amounts of traffic. We deleted all four, and the traffic dropped to near zero within minutes.

I’m posting this to document what happened and track the resolution publicly, in real time, as we learn more. Hopefully, the details help other teams protect themselves and give Google useful feedback on making GCP’s billing visibility and abuse controls better.

We are finding HUGE gaps in the billing transparency and just even opening a support case already!

First, thank you to everyone at GCP helping us investigate and work toward a resolution. I appreciate the time and effort. This is a stressful situation for a small company, and that help matters. This is literally an existentially crisis

What we know so far

• All four keys went from roughly one request per day to heavy usage at the same minute, around 7 AM Pacific.
• They generated approximately 1.17 million requests combined through 4:06 PM.
• Requests were split almost equally across the four keys, consistent with automated key rotation.
• Our separate production key stayed within its normal traffic range.
• Usage included substantial image generation with Gemini 3 Pro Image, which our product does not use at anything approaching this scale.
• We deleted the four keys at 4:18 PM. By 4:22 PM, traffic attributed to them was near zero.

How much was lost?

Approximately $35,000 in promotional credits, based on the balance change. This was credit consumption, not a $35,000 charge to our card. Those credits still represent a substantial part of our startup’s operating runway.

The billing export initially showed approximately $12,900 in Gemini usage during the first few incident hours, with later hours still missing. One hour alone showed approximately $5,466.

The final unauthorized amount is not yet confirmed. We need the remaining billing data and must subtract legitimate usage.

Why it was difficult to understand what was happening

The credit balance was falling, but the billing views we initially checked did not clearly explain why. Promotional credits offset the charges, so net cost appeared as $0. The billing export was also incomplete when we checked.

During an active incident, we needed to see which credentials and models were consuming credits, how quickly, and how to stop it. Piecing that together took time.

How did the keys leak?

We don’t know yet.

The affected keys were managed through Google Secret Manager and referenced by our services, while some older Cloud Run services also had the keys available as environment variables. We also found development keys in local development logs, but those were not the four affected production keys.

Our coding/AI agents did not have access to the affected production keys or our production Secret Manager secrets. We have also found no matches for the affected keys in the repository histories we checked.

At this point, we have not found evidence that the affected keys were exposed through GitHub, our coding agents, or a network breach. We’re continuing to investigate the actual compromise path and don’t want to speculate before we have evidence.

Our responsibility

The keys were restricted to the Gemini API, but had no additional application restrictions. We also had no budget alert or configured spend limit in place. Those were gaps on our side.

We’re reviewing the remaining credentials, how secrets are stored and accessed, and our monitoring and usage controls.

What we’re asking Google for

An investigation, reconciliation of the unauthorized usage, and restoration of the promotional credits consumed by it.

I’ll update this post with:

• The final confirmed amount.
• What we learn about the compromise.
• Google’s response and the credit-restoration decision.
• The protections we implement and any limitations we encounter.

I’ll distinguish confirmed findings from estimates and correct anything that changes. I’ll also give credit to the people helping us along the way.

If you’ve dealt with compromised Gemini keys—particularly where promotional credits were consumed—what helped with the investigation and resolution? What controls have actually worked for you?

My hope is that sharing this openly helps us recover, helps other startups avoid the same situation, and helps all of us improve GCP.

Update 1 — October 7, approximately 5:15 PM Pacific: All four affected keys are deleted. Traffic attributed to them has dropped to near zero. Final billing reconciliation and the credit-restoration outcome remain pending.

Update 2 - October 7, 5.49pm: Had a call with our representative and another contact person. We appreciate their willingness to help but unfortunately neither of us were able to get to a proper conclusion on

• Has the unauthorized usage actually stopped?
• How much more spending hasn’t shown up yet?
• Were only these keys affected, or was more of our GCP environment exposed?
• Is a spending limit now in place, and will it stop further charges?
• Who at Google owns this, when will we get answers, and will our credits be restored?
• Can your abuse team determine whether these requests came through a known resale proxy or matched other compromised-key incidents?

Unfortunately there is a 7+ hour lag on usage & when it reflects on Billing!! So we very well could be in the negative balance by now. This is unsettling

https://preview.redd.it/gdh53fcl95uh1.png?width=1958&format=png&auto=webp&s=c54efb9321576735562559c71d5de7bd2632f5a2

About 6 months ago, an AI agent went rogue during our site testing and silently spammed our Maps API. We take full responsibility for the lack of strict monitoring, but the timeline was brutal: the 50% and 100% budget alerts hit our inbox exactly 45 minutes apart (1:17 PM and 2:02 PM).

A lightning-fast budget drain in under 45 minutes, followed by a 2-month email battle with Google Support.

By the time we reacted, our flat €0 usage spiked to €182.45 plus VAT, totaling €220.76.

The cursed April candle. A flat €0 usage suddenly spiking to €182.

The €220 aftermath. We appreciate Google’s generosity, but it has its limits. They waived the main debt, but still asked us to cover their tax and operational costs — €19.44. We had no choice but to pay it.

Legally, Google is covered. Practically, their UI is a trap. Like many, we assumed the API would hard-stop after the free tier limit. Google engineered a system where configuring a strict $0 hard cap is incredibly non-obvious, leaving developers to fight the UI individually to avoid unexpected bills.

To Google’s credit, after a 2-month dispute involving 5 different employees, they acknowledged it was an unintentional technical spike on our end (not a malicious exploit). They granted us a "one-time courtesy credit" to waive the primary debt. We still had to pay €19 out of pocket for non-refundable taxes, which we accepted as our "stupidity tax."

The Fix: Because of this nightmare, I built a tool to fix Google's native interface so our team wouldn't have to deal with this again. It’s a browser extension that acts as a visual overlay directly on console.cloud.google.com.

• It auto-calculates the exact safe daily/minute limits for your active APIs.
• It visually highlights exactly where to paste those limits to guarantee a $0 bill.
• It runs 100% locally. Zero telemetry, no external network requests. It just reads the DOM.

If anyone else wants to secure their GCP billing, I put the Chrome extension here:https://chromewebstore.google.com/detail/free-maps-calculator-for/bempibaijbfokllpdhkjlokmpeokkpkh

We’ve seen dozens of threads here with people raging about this exact trap. It usually ends in just venting or someone sliding into the comments to sell a paid setup service. I know it’s a simple tool, and hardcore geeks might say "just read the docs," but the UI is genuinely broken.

I got tired of seeing the same complaints for a year without anyone actually building a free fix. So I decided to just build it. Take it and use it.

I am trying to enable billing and get a working Places API key for my project, but every time I try with my Debit card or credit card and try to verify it here, it deducts the small amount but then does not verify it.

I am getting this error:

https://preview.redd.it/dumnqlgu1vth1.png?width=726&format=png&auto=webp&s=ad921e36f37421e5c0ee66ad1fab3ab2448ff7ad

I have tried with different credit cards, different debit cards, waited 24 hours and tried, tried with a different gmail account, also checked my payments profile and the address matches with the address on mt cards, but nothing worked.

Is there anyone who has faced the same error? If yes, please do help me out.

Ok so multi cloud management at our place has quietly turned into aws rules, gcp rules and azure vibes. Dev teams own more infra now, and every day we get a pile of PRs with new queues, random IAM in prod, an extra db in gcp and an EKS tweak in aws... all technically fine, zero architectural context.

The bigger problem is we now have three different ideas of what governed infra is supposed to look like. how are you deciding what gets standardized across clouds and what stays provider specific?