u/kumard3

if your orm creates tables or columns in camelcase, and at some point you drop into raw sql to query one directly, you'll eventually write select * from subAccount and get "relation does not exist," which is confusing when the table is right there in the migration file.

postgres folds unquoted identifiers to lowercase by default. subAccount unquoted becomes subaccount, and if the actual table was created quoted, which is what most orms do to preserve the case you gave them, subaccount genuinely doesn't exist, only "subAccount" does. the fix is quoting it: select * from "subAccount". every camelcase identifier from your orm needs the same treatment the moment you touch it by hand.

burned an hour on this recently on a table i'd looked at a hundred times, because the error doesn't tell you it's a casing problem, it just tells you the table isn't there.

what's your team's convention, always quote camelcase identifiers by habit, or avoid camelcase in the schema entirely to sidestep it?

a knowledge base kept returning duplicated chunks, a 492-character source produced two chunks instead of one. the chunking file on disk looked fixed for weeks, so everyone assumed it had shipped.

it hadn't. the "fixed" file was uncommitted, origin/main, the thing that actually deploys, still imported the old splitter, deployed code never matched the working tree. separately, the splitter emitted the trailing ~200 chars of a chunk as its own standalone chunk even when the whole input fit in one chunk, so 492 chars became a full chunk plus a 200-char tail of itself.

worse, re-embedding never overwrote the old vector. ids were built with a random suffix, so a re-process appended a fresh set instead of replacing anything, and re-saving the same source spawned duplicate vector sets forever.

fix: a recursive splitter with real sliding overlap, plus a deterministic id used as both the vector store id and the row id, with a purge before every write so re-processing is idempotent. the same random-suffix, no-purge pattern turned up in 6 other upload paths, one helper had become 7 versions of one bug.

before trusting "the fix is already in the file," has anyone made a habit of diffing origin/main against their working tree first?

learned this the expensive way. our mcp server required a key on the very first initialize call. any agent, scanner, or curious dev probing it with no key yet got a 401 before it could even list tools. an independent agent-readiness scan read that as connection failed and zeroed out a chunk of the score, about 22 points, for a server that was actually fine.

the fix was separating discovery from execution. initialize and tools/list are open to anyone, no auth, so an agent can see the whole toolset before committing to anything. a key is only required on actual tool execution, with a clean 401 and a www-authenticate header pointing at the oauth flow underneath (dynamic client registration, pkce, refresh rotation).

gate the door, not the lobby. an agent that can't list your tools without a key just assumes you don't have any.

disclosure, this happened on our own server, that's where this comes from.

anyone else testing their mcp server from a genuinely unauthenticated caller before shipping it?

we had a webhook endpoint that would spike to 5x baseline traffic in under a minute with no warning, saturating the origin and dropping events during the spikes.

the fix was a small edge function in front of it doing three things in order: return an instant 200 to the caller so the upstream never sees our latency, forward the payload to origin asynchronously after responding, and if that forward fails, buffer the event to durable storage at the edge instead of losing it.

we're now absorbing about 195 million requests a month at the edge for roughly $92, origin only ever sees successful-forward traffic, and there have been zero dropped events since, verified up to 10mb payloads.

the trap that cost a day building it: you cannot read the request body after you've already returned the response, the runtime throws. tiny test payloads pass because they fit in the initial chunk, real payloads over a few hundred bytes fail, so it only breaks in prod.

anyone running a similar pattern in front of an origin on a major cloud, and where do you draw the line between edge-buffering it versus needing a real queue behind origin?

I finally sat down and queried my own database instead of looking at the signup counter, and the number that actually mattered was one I had never computed.

227 real orgs. 173 created an inbox. 102 created an API key. 93 ever received a single email. So 59% of everyone who signed up never saw the product do the one thing it exists to do.

Then I narrowed it. 46 of them created both an inbox and an API key and still received nothing. Of those, 20 made API calls. 2,253 of them. Zero emails received, zero sent, zero webhooks fired.

That is twenty people who read the docs, got a key, wrote code, and then sat in a polling loop against an empty inbox until they gave up. Every one of them looks identical to a tire kicker in my signup chart.

The part that stung is that I cannot tell whether nobody ever sent mail to those addresses, or whether mail arrived and was rejected before it reached my database. I never logged inbound rejections. So the highest intent group of users I have is also the group I have the least data about.

Two things I got wrong that I think generalise:

I was measuring the funnel in terms of things the user creates. Account, inbox, key. Those are cheap and all of them lie. The only honest activation event is the first time the system produces a result the user did not create themselves. For me that is one inbound email. Everything before it is furniture.

And I was only logging success. My tables record mail that arrived. Nothing records mail that was refused, or a poll that returned empty for the four hundredth time. A user hammering an endpoint and getting nothing back is the loudest signal in the system and it produced no rows anywhere.

The rest of the numbers are fine, which is what makes it annoying. Anyone who gets past five emails sticks around for a month or two. Retention after activation is genuinely good. I have been pouring traffic into a bucket with a hole where the handle should be.

Spending this week on those 46 instead of on anything that makes new signups.

building an auto-reply agent that answers inbound mail on its own, the failure mode that actually worried me wasn't a bad answer, it was two automated systems replying to each other in a loop. your agent auto-replies to a vacation responder, the responder acks, your agent reads the ack as a new message and replies again, forever.

the guardrails that stop it: check the auto-submitted and precedence headers (rfc 3834) before replying at all, detect no-reply senders, cap replies per thread, and stamp outbound with a marker header so the agent recognizes its own prior reply and doesn't answer itself. separately, draft-first mode queues the reply for a human to approve instead of auto-sending, and the agent can escalate_to_human when it's genuinely unsure, which marks the thread and fires a webhook.

disclosure, i build one of these, so i'm biased toward thinking about it this way.

has anyone here actually hit the ping-pong loop in production, or is it more of a designed-around-it-before-it-happened thing for most people?

a german-language bot sent "finansielle Freiheit," with the first syllables of "finanzielle" rendered in georgian script, the phonetic transliteration of "finans," then it switched back to latin mid-word.

first guess was pipeline corruption or a bad string replacement. the trace ruled that out: output length matched the sent message length exactly, replacement count was zero, no rag characters nearby, single iteration, no tool calls. the model just sampled cross-script phonetic tokens on its own.

worth knowing if you're chasing something similar: this generation of reasoning models appears to ignore temperature. it was set to 0 and the model still sampled at some default, presumably how a rare cross-script swap slips through.

the fix wasn't a prompt change, since this sits below the prompt layer. it's a latin-dominant output guard that detects script anomalies, re-rolls once, then strips on a second failure.

only 2 occurrences across all messages over a 4-day window, different assistants and accounts, both german. rare, but invisible until a customer pastes a screenshot at you.

has anyone else seen cross-script sampling glitches on newer reasoning models, and did setting temperature actually do anything for you?

if you're building anything multi-turn and not structuring prompts for caching, your bill is probably several times higher than it needs to be. this isn't a model choice or a retrieval trick, it's purely how you order the prompt.

the mechanic: put everything static (system instructions, tool definitions, few-shot examples, anything that doesn't change turn to turn) at the front, and put whatever actually changes (the latest user message, freshly retrieved context) at the end. caching works on a prefix match, so a cached prefix only helps if nothing above the dynamic part moved.

the number that matters: the break-even point on a cache write is roughly 3 reads, below that you're not saving anything. most agent loops do dozens of reads against the same system prompt in one session, so the break-even clears almost immediately. the mistake I see most is people tucking dynamic content near the top for convenience, a timestamp, a session id, which busts the cache every single turn without anyone noticing why costs didn't drop.

how are you structuring prompts to maximize cache hits, and has anyone measured the actual before/after on their bill?

some records that should get populated after an event completes were just missing, no rows at all, for a subset of our configs. no errors anywhere, nothing failed, the data just never showed up.

the cause was a config setting on a third-party service we integrate with. it has a webhook-style callback url set per config, and on a chunk of our configs that field was null. no callback url means no callback fires, means our side never learns the event happened, means the row never gets written. it looked completely inert from our side because there was nothing to alert on, an unset callback doesn't error, it just quietly does nothing.

some of these configs had apparently been sitting in that state for years, predating anyone currently on the team.

the fix was a one-time patch setting the callback url on every affected config, going forward only, it doesn't backfill anything that already happened while it was broken.

third-party config is code, and code you don't audit on a schedule drifts silently for years with zero signal. how do you track config drift on external services you don't own the source of truth for?

we run two separate codebases against the same database, which is already fragile, but what actually broke it was one cli flag.

our schema-push tool has an "accept data loss" flag you pass when it warns a push is about to drop columns or tables. someone passed it without reading the warning closely, on a push from one of the two repos. the tool did exactly what it said: dropped every table and column that existed in the live database but wasn't declared in that repo's own schema file, which included tables the other repo owned. that repo's schema was correct for itself, it just had no idea the other repo's tables existed.

the fix that holds is a ci check that diffs schema declarations across both repos and blocks a merge on drift, plus a doc comment marking which repo owns each mirrored table so it's not a guessing game in review.

if you don't control the flag, control the input to it: never accept a data-loss warning you haven't read line by line against a database more than one codebase writes to.

anyone running multiple services against one shared database, how do you keep schema declarations in sync?

the problem: an upstream we depend on rate-limits per tenant, but our workers pull from a single shared queue. one noisy tenant hitting its limit doesn't just slow itself down, it stalls every other tenant whose jobs happen to be queued behind it, because the whole worker backs off as if the limit were global.

the design we landed on instead of a worker fleet per tenant, which solves it but is heavier than the problem deserves: a redis token bucket keyed by tenant id, sitting in front of one shared queue. workers check the bucket before pulling a tenant's job, and skip to the next job if that tenant is out of tokens rather than blocking on it. one queue, fair scheduling across tenants, no per-tenant infrastructure.

the actual insight isn't the redis part, it's decoupling rate-limit state from job state. the queue doesn't need to know about limits at all, it just needs a cheap check before deciding whether to work a job or move on.

has anyone run token-bucket-per-tenant against a single shared queue at real scale? curious what breaks first, contention on the bucket keys or fairness under sustained pressure.

a chat widget kept saying "i don't have that on file" for answers clearly in its knowledge base. the same kb answered correctly in the playground and on a separate voice path. swapped models, no change. queried the index directly, it returned the right chunks. both dead ends.

the bug was upstream of retrieval entirely. the widget built its query from the visitor's bare last message. on a follow-up turn the subject is often missing, especially when the bot named it, not the visitor: "hours?", bot answers with a location, then "what's the cost breakdown?" embed that alone and you retrieve the wrong chunks, so the model truthfully says it doesn't know. it only ever sees what the retriever hands it. the playground ran multi-query, hyde, rrf, and a rerank pass, the widget was a plain single-query top-6, same index, opposite behavior.

fix was building the query from the last few turns instead of just the latest message, which cleared most of the "kb is broken" reports on its own.

before you blame your embedding model or re-ingest anything, are you actually logging the query your retriever sends, not just the answer it returns?

published our mcp server package this week and hit three things that would've quietly broken it for every user while looking fine locally.

npm's 2fa web-auth url prints redacted in a non-tty shell, auth/cli/***, so if you're publishing from an automated session the real link never exists anywhere to open. running the publish under a pseudo-tty gets npm to write the real url to a log you can pull from.

separately, a bin path with a ./ prefix gets silently stripped by npm's pack-time validation, warning only, and resolving env vars at module load instead of lazily inside the tool call crashes the server on startup for anyone who imports it before setting env. both pass every local test, because locally you already have env set and you're not running the published artifact.

the check that actually catches it: pipe an initialize handshake through npx on the published version and confirm you get json back.

disclosure, we ship an mcp server ourselves, that's where this came from.

anyone else got a pre-publish checklist for mcp packages specifically?

we had a column tracking which channel a user last messaged on, and it would get stuck on the first channel forever, even after they clearly switched.

two things were stacked. the read side pulled that column from a cached snapshot instead of re-querying, bug one on its own. the deeper one was in the write path: the upsert used on conflict do nothing, fine for columns you genuinely don't want touched, but it meant the channel column never updated on conflict either, since do nothing means nothing, not "nothing except this one column."

fixed it by re-fetching the value on read instead of trusting the snapshot, and changing the upsert to on conflict do update scoped to just the channel column, so untouched columns stay untouched and the one that should change gets explicit permission to.

such an easy thing to get backwards writing the conflict clause fast: do nothing is not a synonym for do nothing to this specific column i care about.

how do you scope conflict updates when only some columns on a row should actually change on conflict?

a voice assistant with a knowledge base attached behaved like one with no KB, answering from base model knowledge, even though it was wired up and multiple KBs were attached.

the retriever tool pointed at assistantKnowledgeBases[0]. that join had no orderBy, so index 0 was whatever the database returned first, in practice the oldest record. the platform allows only one KB tool per agent. when that oldest KB was archived or had zero completed sources, the retrieval handler returned an empty list and the model quietly fell back to its own knowledge, never touching a newer KB with real content sitting right next to it.

about 2,670 of 20,120 KB-enabled voice agents, roughly 13%, had their first-by-id KB pointing at zero completed sources, one in eight answering from the model the whole time. fix was loading the related rows, filtering out archived and deleted, picking the newest KB with a completed source, then re-syncing.

empty retrieval and no retrieval look identical from the outside. if your setup only supports one retriever per agent, has anyone built real selection logic for which one gets attached, or is index 0 more common than I'd like to think?

our cloudwatch bill sat around $321 a month, almost entirely log ingest, not storage, not alarms, about 23 gb a day of ingest. before digging in, the assumption was genuine traffic volume.

it wasn't. the single biggest source was one error line, repeated 848,000 times in 24 hours, coming from 7 sources with stale oauth tokens retrying against an upstream that kept rejecting them. seven misbehaving clients, not real load.

separately, and this cost real time before it clicked: our logs flow through docker to a log forwarder to cloudwatch, and the forwarder wraps the underlying json log inside another json envelope. a plain field:value log line actually lands in cloudwatch nested inside a "log" key as an escaped string. any parse rule written against the inner shape matches zero rows, silently, because you're one layer of escaping off from what you're looking at.

between fixing the retry storm and the parse rules, there was real savings on that ingest line without touching anything that mattered.

anyone else found their biggest log-cost driver was a handful of misbehaving clients rather than actual scale?

a chat agent's CRM writes landed intermittently, some fields saved, some stayed blank, no visible error. replaying the exact conversation extracted the value fine, so the logic wasn't the problem.

the agent calls its own backend over HTTP to run tools, and that request passes through a db-pressure circuit breaker with a bypass allowlist. the tool paths weren't on it. whenever the pool tripped pressure, latched for 30s, every tool call in that window got a 503. the retry wrapper only retried about 12 seconds, so the pressure window outlived the retries and the write was gone for good. each field was a separate call, so it looked like a random coin flip.

the pressure was self-inflicted: the per-process db pool was capped at 120 while the actual pooler allowed 10,000. at peak, processes queued locally at 120 and the breaker read that as database pressure, while the pooler had thousands of connections free. about 200 tool-call 503s a day, fleet-wide, before the fix.

anyone else had a load shedder mistake its own internal traffic for the external load it was built to protect against?

we had a bug where our own bot would go quiet right after sending a message, like it accidentally armed its own away mode on itself.

our dedup for "is this our own message echoing back in" was a single-use token, push on send, pop on receive, keyed by a content hash. worked fine until the third-party platform retried the outbound webhook delivery, which happens more than you'd think. the retry landed as a second event, the token was already gone, so it fell through the normal ingest path and got treated as a brand new inbound message. outbound detection never got a chance to catch it.

the real lesson is about dedup design generally: any scheme with exactly one token per event is one retry away from failing, because the upstream is allowed to deliver twice and your dedup can only survive that once.

do you assume every webhook can be delivered more than once by default, or does that assumption only show up after it bites you?

if you run a local RAG setup and your retrieval keeps surfacing near-duplicate chunks, check these two things before you blame the embedding model. both bit me and both are easy to miss.

first bug, the overlap tail. I was using a paragraph splitter configured with maxLength 1000 and overlap 200. when a source fits in a single chunk, that overlap setting still emits the trailing ~200 chars of the chunk as a second standalone chunk. so a 492-character document became two entries: chunk0 with the full 492 chars, and chunk1 with a 200-char tail that is just a substring of chunk0. nothing about it errors, your store just has a contained duplicate of part of every short doc. switching to a recursive character splitter with a sliding window that emits one chunk when the text is under the chunk size fixed it.

second bug, and the worse one, random vector ids. my vector ids were ${source}_${index}_${randomUuid}. that random suffix means re-embedding a source never overwrites the old vectors, it appends a brand new set. every time a doc got re-processed, I spawned another full duplicate set in the store, forever. the fix is a deterministic id, ${source}_${index}, used as both the vector store id and the row id in my metadata table, plus a purge-before-write so re-processing a source is idempotent and range-deletable.

I added 12 regression tests around the chunker after this. and the real kicker: the same random-suffix, no-purge pattern had been copy-pasted into 6 other upload paths, so one un-centralized helper was actually 7 latent versions of the same bug. centralize the id function and the purge, or you ship the duplication everywhere.

anyone else hit the overlap-tail-on-short-docs thing with off-the-shelf chunkers? curious which local chunking libs handle the under-chunk-size case cleanly.

if you run RAG over multi-turn chat, the single biggest accuracy lever is not your answer prompt and not your reranker. it is a small condensing prompt that turns the conversation so far into one standalone, self-contained question before you embed anything.

here is why it matters. on a follow-up turn the subject is usually missing from the user's literal message, and often the subject was named by the bot, not the user. think: user asks "hours?", bot answers about a specific location, user then asks "and the cost breakdown?". if you embed "and the cost breakdown?" on its own, you retrieve the wrong chunks and the model correctly says it does not have the info. the model only ever sees what the retriever hands it.

the rewrite prompt that fixes this, in plain terms:

• give it the last few user turns plus the most recent assistant turn as context, capped to something small like ~300 chars so you do not embed an entire transcript.
• instruct it: "rewrite the user's latest message as a single standalone question that makes sense with no prior context. resolve pronouns and references using the conversation. keep any entity the assistant introduced, like a place or product name. output only the rewritten question, do not answer it."
• a couple of guardrails that matter in practice: if the latest message is already self-contained, return it unchanged; if it is pure chit-chat with no question, pass it through rather than hallucinating a question; never let the rewrite add facts that were not in the conversation.

so "and the cost breakdown?" becomes "what is the cost breakdown for [the location the assistant just named]?", which actually retrieves the right chunks.

the cheap non-LLM version is to just concatenate the last 3 user turns plus the latest assistant turn into the query. it captures most of the win. the LLM rewrite is the real upgrade because it disambiguates instead of just stuffing context.

how are you all writing your condensing prompt? specifically, how do you stop it from "helpfully" answering the question or inventing a subject when the turn is genuinely ambiguous?