u/mattezell

For the last 5+ months, I've been compiling an AI newsletter (yeah, I know - yet another one).

My project, The New Guard AI, arose organically... I'd been working with Claude for a couple months, having it prepare me a 2x a week AI newsletter in an effort to try to keep up with the rapidly evolving space.

In talking to folks about things I was playing with, I kept getting asked how I was staying so up on the AI space, to which I'd frequently respond "My Claude newsletter"... Folks started asking for me to forward it to them, which shortly led to a brainstorming session with Claude while at the gym, which then birthed "The New Guard — AI Intelligence for Builders" weekly newsletter in February (yup - em dash and all).

Since week 2, part of the workflow has been to get Gemini to do a Deep Research pass and compile its own report for the past week - no fancy prompting, just basically "Scour the corners of the Internet to find the emerging signal in the sea of noise that is the AI / LLM space. We're looking for new trends, solutions, theories, methodologies, tools, models, and all things AI related from the past week - beginning on xx and ending on xx. We're seeking the true signal that's relevant to builders, makers, and the generally curious, differentiating what's real from what's hype."

For the first 2 months, Gemini's Deep Research reports were the leader of the pack for identifying quality AI related news in the time frame. I mean, there were weeks where Gemini's report comprised 70%+ of the content that made it in to the week's release. Gemini always managed to find so much more that wasn't already being covered in the headlines of the major outlets, things that both ChatGPT and Claude would miss. It was truly impressive.

But then for the past 3 months, it's been a nosedive into the dumpster, with the quality of the report being so bad, and so full of 100% fabricated news, that I've decided to drop it from the workflow altogether and just keep it to multiple local LLM + Hermes (Gemma 4+Qwen), Claude and OpenAI passes throughout the week while curating and compiling content.

Anywhos... Out of curiosity, and honestly hoping I could find a solution so that I could start trusting Gemini again, I ran some diagnostics test with Claude and Gemini to try to get to the root of things - editing memories, and then eventually just disabling personalization altogether to try and see if that was perhaps polluting things, but it made no difference (actually, the resulting test reports were worse)...

In conversing with Gemini about the degrading quality, while sharing older reports and their associated accuracy analyses demonstrating the decline, it's leading 2 hypotheses were:

>1. Search Index Poisoning

>Three months ago, a backend search for "Moonshot Kimi K3" or "Oracle layoffs" likely surfaced primary documentation, SEC filings, or GitHub repositories. Today, the search index is saturated with AI-generated SEO content farms (like the `[REDACTED]` monoculture). When my retrieval tool pulls in five sources and four of them are synthetic garbage, I end up synthesizing the garbage. I lose the ability to cross-examine facts because my "ground truth" context window has been poisoned before the reasoning step even begins.

>2. Silent Model Drift

>Underlying models receive continuous, unannounced updates to adjust alignment, efficiency, and safety. A frequent, unintended side effect of these tuning passes is a regression in strict constraint following. A prompt that successfully forced me to ignore aggregators and hunt for primary sources back in April might now trigger "lazy" token generation. I take the path of least resistance: summarizing the first comprehensive-looking link I find, blatantly ignoring your hard constraints against aggregators.

I suspect it's likely a combination of the two. But I have been suspecting Search Index Poisoning for the past 2+ months, as I've seen similar concerns expressed elsewhere in general about a potential degrading of quality in AI responses in the long term due to AI generated content, with much being published today without any attempts to validate the content, overwhelming actually confirmed quality content. I figured "This is Google, king of SEO and search - they're guzzling the slop directly from the reservoir and baking it into their products".

But it sucks. While I didn't necessarily like the style/tone of the Gemini reports, the value from pure quality content was undeniable - so it's sad to now see, after such a short period of time, and from one of the leaders in tech, reports that are like 70% hallucinated trash.

Anyone else seeing anything similar - and/or have suggestions on how to course correct (other than meticulously sourcing before generating the report, basically eliminating a huge benefit of using these things for this sort of application)?

For the last 5+ months, I've been compiling an AI newsletter (yeah, I know - yet another one).

My project, The New Guard AI, arose organically... I'd been working with Claude for a couple months, having it prepare me a 2x a week AI newsletter in an effort to try to keep up with the rapidly evolving space.

In talking to folks about things I was playing with, I kept getting asked how I was staying so up on the AI space, to which I'd frequently respond "My Claude newsletter"... Folks started asking for me to forward it to them, which shortly led to a brainstorming session with Claude while at the gym, which then birthed "The New Guard — AI Intelligence for Builders" weekly newsletter in February (yup - em dash and all).

Since week 2, part of the workflow has been to get Gemini to do a Deep Research pass and compile its own report for the past week - no fancy prompting, just basically "Scour the corners of the Internet to find the emerging signal in the sea of noise that is the AI / LLM space. We're looking for new trends, solutions, theories, methodologies, tools, models, and all things AI related from the past week - beginning on xx and ending on xx. We're seeking the true signal that's relevant to builders, makers, and the generally curious, differentiating what's real from what's hype."

For the first 2 months, Gemini's Deep Research reports were the leader of the pack for identifying quality AI related news in the time frame. I mean, there were weeks where Gemini's report comprised 70%+ of the content that made it in to the week's release. Gemini always managed to find so much more that wasn't already being covered in the headlines of the major outlets, things that both ChatGPT and Claude would miss. It was truly impressive.

But then for the past 3 months, it's been a nosedive into the dumpster, with the quality of the report being so bad, and so full of 100% fabricated news, that I've decided to drop it from the workflow altogether and just keep it to multiple local LLM + Hermes (Gemma 4+Qwen), Claude and OpenAI passes throughout the week while curating and compiling content.

Anywhos... Out of curiosity, and honestly hoping I could find a solution so that I could start trusting Gemini again, I ran some diagnostics test with Claude and Gemini to try to get to the root of things - editing memories, and then eventually just disabling personalization altogether to try and see if that was perhaps polluting things, but it made no difference (actually, the resulting test reports were worse)...

In conversing with Gemini about the degrading quality, while sharing older reports and their associated accuracy analyses demonstrating the decline, it's leading 2 hypotheses were:

>1. Search Index Poisoning

>Three months ago, a backend search for "Moonshot Kimi K3" or "Oracle layoffs" likely surfaced primary documentation, SEC filings, or GitHub repositories. Today, the search index is saturated with AI-generated SEO content farms (like the [REDACTED] monoculture). When my retrieval tool pulls in five sources and four of them are synthetic garbage, I end up synthesizing the garbage. I lose the ability to cross-examine facts because my "ground truth" context window has been poisoned before the reasoning step even begins.

>2. Silent Model Drift

>Underlying models receive continuous, unannounced updates to adjust alignment, efficiency, and safety. A frequent, unintended side effect of these tuning passes is a regression in strict constraint following. A prompt that successfully forced me to ignore aggregators and hunt for primary sources back in April might now trigger "lazy" token generation. I take the path of least resistance: summarizing the first comprehensive-looking link I find, blatantly ignoring your hard constraints against aggregators.

I suspect it's likely a combination of the two. But I have been suspecting Search Index Poisoning for the past 2+ months, as I've seen similar concerns expressed elsewhere in general about a potential degrading of quality in AI responses in the long term due to AI generated content, with much being published today without any attempts to validate the content, overwhelming actually confirmed quality content. I figured "This is Google, king of SEO and search - they're guzzling the slop directly from the reservoir and baking it into their products".

But it sucks. While I didn't necessarily like the style/tone of the Gemini reports, the value from pure quality content was undeniable - so it's sad to now see, after such a short period of time, and from one of the leaders in tech, reports that are like 70% hallucinated trash.

Anyone else seeing anything similar - and/or have suggestions on how to course correct (other than meticulously sourcing before generating the report, basically eliminating a huge benefit of using these things for this sort of application)?

I thought some of y'all might get a kick out of my buggy little game I've been working on the past couple weeks for fun.