$35,000 of Gemini API lost in 3 hours!!! Please help.
I’m the CTO of MobiusEngine.ai . we are a fast growing startup and we want to make the process of job applications free and easy and we want to help people land jobs faster.
Today, October 7, our Google Cloud promotional credit balance dropped by approx $35,000!!!!
Our investigation found four old, previously near-idle Gemini API keys suddenly generating massive amounts of traffic. We deleted all four, and the traffic dropped to near zero within minutes.
I’m posting this to document what happened and track the resolution publicly, in real time, as we learn more. Hopefully, the details help other teams protect themselves and give Google useful feedback on making GCP’s billing visibility and abuse controls better.
We are finding HUGE gaps in the billing transparency and just even opening a support case already!
First, thank you to everyone at GCP helping us investigate and work toward a resolution. I appreciate the time and effort. This is a stressful situation for a small company, and that help matters. This is literally an existentially crisis
What we know so far
- All four keys went from roughly one request per day to heavy usage at the same minute, around 7 AM Pacific.
- They generated approximately 1.17 million requests combined through 4:06 PM.
- Requests were split almost equally across the four keys, consistent with automated key rotation.
- Our separate production key stayed within its normal traffic range.
- Usage included substantial image generation with Gemini 3 Pro Image, which our product does not use at anything approaching this scale.
- We deleted the four keys at 4:18 PM. By 4:22 PM, traffic attributed to them was near zero.
How much was lost?
Approximately $35,000 in promotional credits, based on the balance change. This was credit consumption, not a $35,000 charge to our card. Those credits still represent a substantial part of our startup’s operating runway.
The billing export initially showed approximately $12,900 in Gemini usage during the first few incident hours, with later hours still missing. One hour alone showed approximately $5,466.
The final unauthorized amount is not yet confirmed. We need the remaining billing data and must subtract legitimate usage.
Why it was difficult to understand what was happening
The credit balance was falling, but the billing views we initially checked did not clearly explain why. Promotional credits offset the charges, so net cost appeared as $0. The billing export was also incomplete when we checked.
During an active incident, we needed to see which credentials and models were consuming credits, how quickly, and how to stop it. Piecing that together took time.
How did the keys leak?
We don’t know yet.
The affected keys were managed through Google Secret Manager and referenced by our services, while some older Cloud Run services also had the keys available as environment variables. We also found development keys in local development logs, but those were not the four affected production keys.
Our coding/AI agents did not have access to the affected production keys or our production Secret Manager secrets. We have also found no matches for the affected keys in the repository histories we checked.
At this point, we have not found evidence that the affected keys were exposed through GitHub, our coding agents, or a network breach. We’re continuing to investigate the actual compromise path and don’t want to speculate before we have evidence.
Our responsibility
The keys were restricted to the Gemini API, but had no additional application restrictions. We also had no budget alert or configured spend limit in place. Those were gaps on our side.
We’re reviewing the remaining credentials, how secrets are stored and accessed, and our monitoring and usage controls.
What we’re asking Google for
An investigation, reconciliation of the unauthorized usage, and restoration of the promotional credits consumed by it.
I’ll update this post with:
- The final confirmed amount.
- What we learn about the compromise.
- Google’s response and the credit-restoration decision.
- The protections we implement and any limitations we encounter.
I’ll distinguish confirmed findings from estimates and correct anything that changes. I’ll also give credit to the people helping us along the way.
If you’ve dealt with compromised Gemini keys—particularly where promotional credits were consumed—what helped with the investigation and resolution? What controls have actually worked for you?
My hope is that sharing this openly helps us recover, helps other startups avoid the same situation, and helps all of us improve GCP.
Update 1 — October 7, approximately 5:15 PM Pacific: All four affected keys are deleted. Traffic attributed to them has dropped to near zero. Final billing reconciliation and the credit-restoration outcome remain pending.
Update 2 - October 7, 5.49pm: Had a call with our representative and another contact person. We appreciate their willingness to help but unfortunately neither of us were able to get to a proper conclusion on
- Has the unauthorized usage actually stopped?
- How much more spending hasn’t shown up yet?
- Were only these keys affected, or was more of our GCP environment exposed?
- Is a spending limit now in place, and will it stop further charges?
- Who at Google owns this, when will we get answers, and will our credits be restored?
- Can your abuse team determine whether these requests came through a known resale proxy or matched other compromised-key incidents?
Unfortunately there is a 7+ hour lag on usage & when it reflects on Billing!! So we very well could be in the negative balance by now. This is unsettling
