u/umur957

Went through Cerebras' writeup on their internal knowledge base — 15,000 questions a day from employees, automations and agents, three months after launch. The architecture is less interesting than what they had to do to Slack, so that's what I want to focus on.

Why plain vector search dies on chat data

Information density varies by orders of magnitude. "yeah sure Mike" sits in the same channel as a 40-line kernel explanation, and under cosine similarity the short one wins far more often than it should. A single message out of its thread is usually meaningless anyway.

Their fix: four fused signals per thread

• Full-text search — exact tokens. Error strings, flag names, hostnames. Embeddings reliably lose here and lexical match is unbeatable.
• Embedding search — paraphrase. Connects "restore is hanging" to "checkpoint stalled".
• IDF — boosts rare tokens (obscure config flags), suppresses filler ("sounds good", "thanks").
• Age decay — same answer from yesterday beats the one from 6 months ago referencing deprecated infra.

But the preprocessing does more work than the retrieval

Two steps before any of the above, and I think these matter more:

Thread distillation — an LLM normalizes each thread into a searchable one-line question + summary + resolution + systems and code referenced. That distilled doc gets embedded, not the raw transcript. Raw text is kept for FTS only.

Bursting — a "burst" is a run of consecutive messages from the same author. They prepend the thread topic for context and embed it separately, but only if it clears a gate: rare-token IDF >= 4.0, or >= 200 chars, or it got reactions. This is what rescues the one deeply technical tangent buried at message 47 that any thread-level summary would flatten away.

Fusion: RRF at k=60

Score contribution is weight / (60 + rank), summed across retrievers. The summation is the entire point:

• 3rd place in three retrievers → 3 × 1/63 = 0.048
• 1st place in exactly one → 1/61 = 0.016

Consensus beats peak. It isn't a ranker, it's a consensus builder.

Then ~20 candidates go to a small reranker scoring 0–10 against the query, top 10 survive. And the step people skip: re-attach surrounding context to the winners. If a wiki section wins, its neighbors come with it, so the model sees headers, preconditions and caveats instead of an orphaned chunk.

Code side

CocoIndex (open source, Rust core, Tree-sitter chunking) keeps 40GB+ repos synced by re-embedding only what a commit touched. Sync state and the embedding store live in the same database.

The design decision underneath all of it

Don't force people into a "single source of truth" platform — nobody wants to discuss a pull request inside a Google Doc. Pull from where the data already is. Every source, Slack thread to hardware netlist, lands as a row in the same embeddings table behind the same interface. Custom sources are just plugin scripts: a team opens a PR with a small Python module that reads their system and emits rows in that shape.

Also worth noting: the retrieval primitives are deliberately LLM-free. Model calls only happen at the edges — planning and synthesis. That's what makes it cheap enough for agents to hammer 15k times a day, and why the same pipeline serves a web UI and an MCP client identically.

---

Original Cerebras writeup (read this first if you only have time for one): https://www.cerebras.ai/blog/how-we-built-our-knowledge-base

Disclosure: I also did a ~10 min video walkthrough of the full pipeline, linked here — it's my channel, and the narration is AI-assisted. https://www.youtube.com/watch?v=FgKHjzoiMN4&t=4s

The bursting quality gates are the part I'd most want other people's numbers on. IDF >= 4.0 and 200 chars feel like they'd need retuning per org — has anyone tried burst-level embedding on their own chat data?

Google Search Central's documentation states plainly that "from Google Search's perspective, optimizing for generative AI search is still SEO," and explicitly discourages the GEO-specific tactics going around — artificial content chunking, llms.txt files, and so on. Their position is that foundational best practice is sufficient.

Meanwhile Chen et al. (Navigating the Shift, arXiv 2601.16858, University of Toronto) ran a large-scale comparison of Google Search results against leading generative AI services and found the two diverge significantly in consulted source domains, domain typology (earned vs. owned vs. social), query intent, and information freshness.

These read as contradictory. I don't think they are, and the reconciliation is the part worth internalizing:

Google is describing its own surface accurately. AI Overviews are retrieved from Google's index. If you're indexed and you rank, you're in the candidate pool. For AIO specifically, "it's still SEO" is defensible and the overlap data supports it — AIO overlap with organic is the highest of any engine.

Google is not describing the surfaces it doesn't own. ChatGPT, Perplexity, and Claude aren't retrieving from Google's index and aren't bound by Google's ranking logic. This is where the divergence Chen et al. measured actually lives, and Google has no incentive and no standing to give guidance about it.

So the practical split:

- If your traffic is dominated by AIO exposure: Google's advice is basically correct. Do the fundamentals, don't buy the chunking snake oil.

- If you care about ChatGPT and Perplexity: you're optimizing for retrieval systems with different corpus preferences, a heavy recency bias, and a very different domain mix. Reddit alone accounts for roughly 40% citation frequency across LLMs and 46.5% of Perplexity's citations. That is not a distribution any amount of on-page work reproduces.

The uncomfortable implication is that a meaningful share of "AI visibility" work isn't on-page optimization at all — it's presence in the specific third-party corpora these systems over-index on. Which is a very different discipline from what most of us have been selling.

Interested in whether anyone has data on how stable that domain mix is over time. If the corpus preferences shift quarterly, most GEO strategies have a very short shelf life.

Traditional search is undergoing its biggest paradigm shift in decades
. With Google's AI Overviews and AI Mode rolling out globally, up to 93% of searches in AI mode do not result in a single click
. In fact, overall zero-click searches have skyrocketed to 68% in the US—and when an AI answer appears, 83% of users find their answer without clicking through to any website
. Gartner even predicts traditional search volume will drop by 25% this year
.
This doesn't mean SEO is dead
. But the game has completely changed. The correlation between being ranked #1 on Google and being recommended by AI has plummeted from 75% to just 17%
.
To survive and thrive, brands must transition from traditional SEO to AEO (Answer Engine Optimization) and GEO (Generative Engine Optimization)
. In this video, we break down the exact strategies, research, and tools you need to optimize your brand for AI search engines like ChatGPT, Claude, Gemini, and Perplexity

Hey, here is my youtube video:

On May 12, SAP announced a strategic investment in n8n at a $5.2 billion valuation. And it’s not just an investment: n8n is being embedded natively inside Joule Studio, SAP’s agent-building environment. General availability is targeted for Q3 2026 — so, a few months out.
The numbers give some context: n8n has 1.7 million monthly active developers and 1,400+ enterprise customers. SAP has roughly 300,000 enterprise customers. So a bottom-up open-source tool is getting plugged into the distribution channel of the world’s largest ERP vendor.
Three things I read from this:
1. Enterprise automation and the “AI agent” world are meeting inside the same interface for the first time. Teams that couldn’t justify the cost of a full Integration Suite deployment were already using n8n unofficially — that’s now becoming official.
2. The value is shifting from tool knowledge to process knowledge. Learning n8n takes weeks; knowing where a procurement process actually breaks takes years. The second one is what will differentiate people.
3. SAP-specific nodes are still on the roadmap. Which means this is a genuinely early window for anyone building experience in this space.
What I’m curious about: anyone here working on the SAP side? Is n8n already being used unofficially in your organization, or is it hitting the security/governance wall?
I’m considering moving into this area and would appreciate input from anyone with hands-on experience.