Does anyone else track their caffeine levels?
u/pythononrailz — 5 days ago
r/GoogleGeminiAI
$35,000 of Gemini API lost in 3 hours!!! Please help.
I’m the CTO of MobiusEngine.ai . we are a fast growing startup and we want to make the process of job applications free and easy and we want to help people land jobs faster.
Today, October 7, our Google Cloud promotional credit balance dropped by approx $35,000!!!!
Our investigation found four old, previously near-idle Gemini API keys suddenly generating massive amounts of traffic. We deleted all four, and the traffic dropped to near zero within minutes.
I’m posting this to document what happened and track the resolution publicly, in real time, as we learn more. Hopefully, the details help other teams protect themselves and give Google useful feedback on making GCP’s billing visibility and abuse controls better.
We are finding HUGE gaps in the billing transparency and just even opening a support case already!
First, thank you to everyone at GCP helping us investigate and work toward a resolution. I appreciate the time and effort. This is a stressful situation for a small company, and that help matters. This is literally an existentially crisis
What we know so far
- All four keys went from roughly one request per day to heavy usage at the same minute, around 7 AM Pacific.
- They generated approximately 1.17 million requests combined through 4:06 PM.
- Requests were split almost equally across the four keys, consistent with automated key rotation.
- Our separate production key stayed within its normal traffic range.
- Usage included substantial image generation with Gemini 3 Pro Image, which our product does not use at anything approaching this scale.
- We deleted the four keys at 4:18 PM. By 4:22 PM, traffic attributed to them was near zero.
How much was lost?
Approximately $35,000 in promotional credits, based on the balance change. This was credit consumption, not a $35,000 charge to our card. Those credits still represent a substantial part of our startup’s operating runway.
The billing export initially showed approximately $12,900 in Gemini usage during the first few incident hours, with later hours still missing. One hour alone showed approximately $5,466.
The final unauthorized amount is not yet confirmed. We need the remaining billing data and must subtract legitimate usage.
Why it was difficult to understand what was happening
The credit balance was falling, but the billing views we initially checked did not clearly explain why. Promotional credits offset the charges, so net cost appeared as $0. The billing export was also incomplete when we checked.
During an active incident, we needed to see which credentials and models were consuming credits, how quickly, and how to stop it. Piecing that together took time.
How did the keys leak?
We don’t know yet.
The affected keys were managed through Google Secret Manager and referenced by our services, while some older Cloud Run services also had the keys available as environment variables. We also found development keys in local development logs, but those were not the four affected production keys.
Our coding/AI agents did not have access to the affected production keys or our production Secret Manager secrets. We have also found no matches for the affected keys in the repository histories we checked.
At this point, we have not found evidence that the affected keys were exposed through GitHub, our coding agents, or a network breach. We’re continuing to investigate the actual compromise path and don’t want to speculate before we have evidence.
Our responsibility
The keys were restricted to the Gemini API, but had no additional application restrictions. We also had no budget alert or configured spend limit in place. Those were gaps on our side.
We’re reviewing the remaining credentials, how secrets are stored and accessed, and our monitoring and usage controls.
What we’re asking Google for
An investigation, reconciliation of the unauthorized usage, and restoration of the promotional credits consumed by it.
I’ll update this post with:
- The final confirmed amount.
- What we learn about the compromise.
- Google’s response and the credit-restoration decision.
- The protections we implement and any limitations we encounter.
I’ll distinguish confirmed findings from estimates and correct anything that changes. I’ll also give credit to the people helping us along the way.
If you’ve dealt with compromised Gemini keys—particularly where promotional credits were consumed—what helped with the investigation and resolution? What controls have actually worked for you?
My hope is that sharing this openly helps us recover, helps other startups avoid the same situation, and helps all of us improve GCP.
Update 1 — October 7, approximately 5:15 PM Pacific: All four affected keys are deleted. Traffic attributed to them has dropped to near zero. Final billing reconciliation and the credit-restoration outcome remain pending.
Update 2 - October 7, 5.49pm: Had a call with our representative and another contact person. We appreciate their willingness to help but unfortunately neither of us were able to get to a proper conclusion on
- Has the unauthorized usage actually stopped?
- How much more spending hasn’t shown up yet?
- Were only these keys affected, or was more of our GCP environment exposed?
- Is a spending limit now in place, and will it stop further charges?
- Who at Google owns this, when will we get answers, and will our credits be restored?
- Can your abuse team determine whether these requests came through a known resale proxy or matched other compromised-key incidents?
Unfortunately there is a 7+ hour lag on usage & when it reflects on Billing!! So we very well could be in the negative balance by now. This is unsettling
u/aaghashm — 7 days ago
Google is making changes that I don't understand...
I understand that artificial intelligence is expensive, mainly because of the infrastructure and other factors. But what I don’t understand is why on earth they’re removing the Pro model from the Google One Plus plan—it was one of the reasons I subscribed to the plan, especially because it helped me when reviewing drafts of my stories and discussing complex topics. But I think I’ll switch to Claude. I’ll keep my Google One plan, but without the AI and Gemini features—I’ll just use it for occasional, less in-depth tasks.
u/Unable_Comparison190 — 7 days ago
▲ 11 r/GoogleGeminiAI
This is a damn declaration of war 😭
This is why I usually just press AI mode from Google instead of using the app...
u/mrman_4200 — 7 days ago
▲ 12 r/GoogleGeminiAI
Next gemini flash incoming?
got these choices twice in two weeks. will we get gemini 3.9 flash or gemini 4 flash? 👀
u/alovoids — 8 days ago
Why does Gemini pro sometimes answer my question from a totally different question asked?
Gemini Pro, Windows app. Occasionally when I ask Gemini Pro a specific technical question (where the answer should be in the notebook sources I'm asking about) it occasionally replies with an answer that I may have asked hours or days ago about a very different subject? Is this a Window GP app issue or is this a thing? Example: Does my router have an automatic backup setting to backup the configuration file regularly? It replies with some answer about a different technical subject I may have asked in the past.
u/marklyn59 — 7 days ago
Benchmark notes: Kimi K3 reaches 88/98; Qwen3.8-Max is stronger than its score looks; Gemini 3.6 Flash slips vs 3.5
I ran Kimi K3, Qwen3.8-Max and Gemini 3.6 Flash on the current 98-task MindTrial set with the same Python executor available to all models.
A few results stood out.
Kimi K3 was the biggest surprise: 88/98, up from 66/98 for Kimi K2.6, with hard errors dropping from 22 to 1. That ties Claude Opus 5 for the highest raw pass count among single models in this set. The downside is runtime: about 8h48m summed across the tasks.
Qwen3.8-Max finished at 74/98, but that number hides a much stronger solver. It went 39/39 on text and had 92.5% accuracy on completed tasks. Of its 18 hard errors, 15 were response-parsing failures; in a number of those cases the useful answer was there, but the outer response was malformed. So strict output discipline is currently a major weakness.
There was also an interesting API/agent interaction with both Kimi and Qwen. In the original runs using provider-enforced JSON object/schema output, neither model used the supplied Python tool across the benchmark. I reran using API response_format: text while keeping the same JSON schema in the prompt. Tool use came back immediately: 387 calls for Kimi and 423 for Qwen, with much better task performance. For Qwen, the constrained-output path also seemed to interfere with how reasoning translated into actions/final answers.
Gemini 3.6 Flash was less impressive: 74/98 versus 77/98 for Gemini 3.5 Flash. It was faster (~1h45m vs ~2h13m) and used fewer tokens overall, but made more Python calls (712 vs 597). In this workload the extra tool use often looked exploratory rather than efficient.
The strict scores are still the scores—I did not repair malformed model answers after the fact. I just found the differences in failure mode and tool behavior interesting.
Results/data: http://www.petmal.net/shared/mindtrial/results/2026-08-06/mindtrial-eval-all-models-03-2026_26.html
u/Correct_Tomato1871 — 11 days ago
3.8 Flash and DSH = 1700+ tok/s ???
Today, I added Gemini to Deepseek Harness, and it is showing, what I thought, was crazy tok/s ranges.
The usual range is between 600 - 800 tok/s, but I have seen it hit 1700 tok/s. Is this possible, or is this a flaw with DSH?
u/mtul — 7 days ago