u/kumard3

I'm building an agent that replies to email, and any stranger can email it. So before the model reads a message I run a dumb regex scorer on it. It looks at the subject, the text and the raw HTML, since hidden stuff tends to be in the HTML.

Phrases like "ignore previous instructions", "you are now" or "forward all emails" add points. So do zero width characters, instructions in HTML comments, and CSS that hides text like display:none or font-size:0. Hidden CSS is 12 points alone and 25 if one of those phrases also matched. 60 and up out of 100 is high risk.

I don't block high risk mail. The agent still gets it, the system prompt just gets a line saying it was flagged and to only deal with the actual request or escalate to a human.

This misses anything reworded or not in English, I know. Has anyone moved to a small classifier for this, and was the latency per message worth it?

My API ran out of Neon compute quota at the end of August. Aug 24 to 29, anything that hit the database failed. /health returned {"status":"ok"} the entire time since it never queried anything, and I didn't hear about it from any alert.

Normally you'd just put a query in /health. Problem is Neon suspends compute after 5 idle minutes, I can't turn that off on my plan, and an uptime monitor pinging a DB route every minute would keep it awake 24/7. That burns exactly the quota I ran out of.

What I ended up doing: a Prisma client extension on $allOperations that saves the time of the last query that worked and the last one that threw. /health just reads those two timestamps. Failure is the newest thing in the last 5 min, 503 with the error. Success, ok. No queries at all, it says idle. A single failed query gets wiped by the next good one so it only stays red if it keeps failing.

Idle still bugs me though, at 3am with zero traffic it knows nothing. Anyone found a way around that without waking the db?

My email API has an endpoint that just holds the HTTP request open until a matching mail arrives, instead of a webhook or a polling loop on the caller side. GET /v1/inboxes/:id/wait, 30s default timeout, clamped to 120 max, checks Postgres every second, 200 with the email or a 408.

What caught me out was the since param, not the timeout. Defaulting it to now is right when you are waiting for the next mail to come in. For the OTP shortcut it was wrong, since the code is normally already in the table by the time an agent asks for it, so you would block the full 30s and then 408 on mail you already had. That one defaults to five minutes back.

Still on my list: the loop does not notice if the caller hung up, it keeps querying until the timeout runs out. The SSE endpoint right next to it checks stream.aborted on every pass and I never did the same here. That is basically why I capped the wait at 120s.

Anyone else doing this in Node, do you hold the request open or push over SSE?

if your orm creates tables or columns in camelcase, and at some point you drop into raw sql to query one directly, you'll eventually write select * from subAccount and get "relation does not exist," which is confusing when the table is right there in the migration file.

postgres folds unquoted identifiers to lowercase by default. subAccount unquoted becomes subaccount, and if the actual table was created quoted, which is what most orms do to preserve the case you gave them, subaccount genuinely doesn't exist, only "subAccount" does. the fix is quoting it: select * from "subAccount". every camelcase identifier from your orm needs the same treatment the moment you touch it by hand.

burned an hour on this recently on a table i'd looked at a hundred times, because the error doesn't tell you it's a casing problem, it just tells you the table isn't there.

what's your team's convention, always quote camelcase identifiers by habit, or avoid camelcase in the schema entirely to sidestep it?

a knowledge base kept returning duplicated chunks, a 492-character source produced two chunks instead of one. the chunking file on disk looked fixed for weeks, so everyone assumed it had shipped.

it hadn't. the "fixed" file was uncommitted, origin/main, the thing that actually deploys, still imported the old splitter, deployed code never matched the working tree. separately, the splitter emitted the trailing ~200 chars of a chunk as its own standalone chunk even when the whole input fit in one chunk, so 492 chars became a full chunk plus a 200-char tail of itself.

worse, re-embedding never overwrote the old vector. ids were built with a random suffix, so a re-process appended a fresh set instead of replacing anything, and re-saving the same source spawned duplicate vector sets forever.

fix: a recursive splitter with real sliding overlap, plus a deterministic id used as both the vector store id and the row id, with a purge before every write so re-processing is idempotent. the same random-suffix, no-purge pattern turned up in 6 other upload paths, one helper had become 7 versions of one bug.

before trusting "the fix is already in the file," has anyone made a habit of diffing origin/main against their working tree first?

learned this the expensive way. our mcp server required a key on the very first initialize call. any agent, scanner, or curious dev probing it with no key yet got a 401 before it could even list tools. an independent agent-readiness scan read that as connection failed and zeroed out a chunk of the score, about 22 points, for a server that was actually fine.

the fix was separating discovery from execution. initialize and tools/list are open to anyone, no auth, so an agent can see the whole toolset before committing to anything. a key is only required on actual tool execution, with a clean 401 and a www-authenticate header pointing at the oauth flow underneath (dynamic client registration, pkce, refresh rotation).

gate the door, not the lobby. an agent that can't list your tools without a key just assumes you don't have any.

disclosure, this happened on our own server, that's where this comes from.

anyone else testing their mcp server from a genuinely unauthenticated caller before shipping it?

we had a webhook endpoint that would spike to 5x baseline traffic in under a minute with no warning, saturating the origin and dropping events during the spikes.

the fix was a small edge function in front of it doing three things in order: return an instant 200 to the caller so the upstream never sees our latency, forward the payload to origin asynchronously after responding, and if that forward fails, buffer the event to durable storage at the edge instead of losing it.

we're now absorbing about 195 million requests a month at the edge for roughly $92, origin only ever sees successful-forward traffic, and there have been zero dropped events since, verified up to 10mb payloads.

the trap that cost a day building it: you cannot read the request body after you've already returned the response, the runtime throws. tiny test payloads pass because they fit in the initial chunk, real payloads over a few hundred bytes fail, so it only breaks in prod.

anyone running a similar pattern in front of an origin on a major cloud, and where do you draw the line between edge-buffering it versus needing a real queue behind origin?

I finally sat down and queried my own database instead of looking at the signup counter, and the number that actually mattered was one I had never computed.

227 real orgs. 173 created an inbox. 102 created an API key. 93 ever received a single email. So 59% of everyone who signed up never saw the product do the one thing it exists to do.

Then I narrowed it. 46 of them created both an inbox and an API key and still received nothing. Of those, 20 made API calls. 2,253 of them. Zero emails received, zero sent, zero webhooks fired.

That is twenty people who read the docs, got a key, wrote code, and then sat in a polling loop against an empty inbox until they gave up. Every one of them looks identical to a tire kicker in my signup chart.

The part that stung is that I cannot tell whether nobody ever sent mail to those addresses, or whether mail arrived and was rejected before it reached my database. I never logged inbound rejections. So the highest intent group of users I have is also the group I have the least data about.

Two things I got wrong that I think generalise:

I was measuring the funnel in terms of things the user creates. Account, inbox, key. Those are cheap and all of them lie. The only honest activation event is the first time the system produces a result the user did not create themselves. For me that is one inbound email. Everything before it is furniture.

And I was only logging success. My tables record mail that arrived. Nothing records mail that was refused, or a poll that returned empty for the four hundredth time. A user hammering an endpoint and getting nothing back is the loudest signal in the system and it produced no rows anywhere.

The rest of the numbers are fine, which is what makes it annoying. Anyone who gets past five emails sticks around for a month or two. Retention after activation is genuinely good. I have been pouring traffic into a bucket with a hole where the handle should be.

Spending this week on those 46 instead of on anything that makes new signups.

building an auto-reply agent that answers inbound mail on its own, the failure mode that actually worried me wasn't a bad answer, it was two automated systems replying to each other in a loop. your agent auto-replies to a vacation responder, the responder acks, your agent reads the ack as a new message and replies again, forever.

the guardrails that stop it: check the auto-submitted and precedence headers (rfc 3834) before replying at all, detect no-reply senders, cap replies per thread, and stamp outbound with a marker header so the agent recognizes its own prior reply and doesn't answer itself. separately, draft-first mode queues the reply for a human to approve instead of auto-sending, and the agent can escalate_to_human when it's genuinely unsure, which marks the thread and fires a webhook.

disclosure, i build one of these, so i'm biased toward thinking about it this way.

has anyone here actually hit the ping-pong loop in production, or is it more of a designed-around-it-before-it-happened thing for most people?

a german-language bot sent "finansielle Freiheit," with the first syllables of "finanzielle" rendered in georgian script, the phonetic transliteration of "finans," then it switched back to latin mid-word.

first guess was pipeline corruption or a bad string replacement. the trace ruled that out: output length matched the sent message length exactly, replacement count was zero, no rag characters nearby, single iteration, no tool calls. the model just sampled cross-script phonetic tokens on its own.

worth knowing if you're chasing something similar: this generation of reasoning models appears to ignore temperature. it was set to 0 and the model still sampled at some default, presumably how a rare cross-script swap slips through.

the fix wasn't a prompt change, since this sits below the prompt layer. it's a latin-dominant output guard that detects script anomalies, re-rolls once, then strips on a second failure.

only 2 occurrences across all messages over a 4-day window, different assistants and accounts, both german. rare, but invisible until a customer pastes a screenshot at you.

has anyone else seen cross-script sampling glitches on newer reasoning models, and did setting temperature actually do anything for you?

if you're building anything multi-turn and not structuring prompts for caching, your bill is probably several times higher than it needs to be. this isn't a model choice or a retrieval trick, it's purely how you order the prompt.

the mechanic: put everything static (system instructions, tool definitions, few-shot examples, anything that doesn't change turn to turn) at the front, and put whatever actually changes (the latest user message, freshly retrieved context) at the end. caching works on a prefix match, so a cached prefix only helps if nothing above the dynamic part moved.

the number that matters: the break-even point on a cache write is roughly 3 reads, below that you're not saving anything. most agent loops do dozens of reads against the same system prompt in one session, so the break-even clears almost immediately. the mistake I see most is people tucking dynamic content near the top for convenience, a timestamp, a session id, which busts the cache every single turn without anyone noticing why costs didn't drop.

how are you structuring prompts to maximize cache hits, and has anyone measured the actual before/after on their bill?

some records that should get populated after an event completes were just missing, no rows at all, for a subset of our configs. no errors anywhere, nothing failed, the data just never showed up.

the cause was a config setting on a third-party service we integrate with. it has a webhook-style callback url set per config, and on a chunk of our configs that field was null. no callback url means no callback fires, means our side never learns the event happened, means the row never gets written. it looked completely inert from our side because there was nothing to alert on, an unset callback doesn't error, it just quietly does nothing.

some of these configs had apparently been sitting in that state for years, predating anyone currently on the team.

the fix was a one-time patch setting the callback url on every affected config, going forward only, it doesn't backfill anything that already happened while it was broken.

third-party config is code, and code you don't audit on a schedule drifts silently for years with zero signal. how do you track config drift on external services you don't own the source of truth for?

we run two separate codebases against the same database, which is already fragile, but what actually broke it was one cli flag.

our schema-push tool has an "accept data loss" flag you pass when it warns a push is about to drop columns or tables. someone passed it without reading the warning closely, on a push from one of the two repos. the tool did exactly what it said: dropped every table and column that existed in the live database but wasn't declared in that repo's own schema file, which included tables the other repo owned. that repo's schema was correct for itself, it just had no idea the other repo's tables existed.

the fix that holds is a ci check that diffs schema declarations across both repos and blocks a merge on drift, plus a doc comment marking which repo owns each mirrored table so it's not a guessing game in review.

if you don't control the flag, control the input to it: never accept a data-loss warning you haven't read line by line against a database more than one codebase writes to.

anyone running multiple services against one shared database, how do you keep schema declarations in sync?

the problem: an upstream we depend on rate-limits per tenant, but our workers pull from a single shared queue. one noisy tenant hitting its limit doesn't just slow itself down, it stalls every other tenant whose jobs happen to be queued behind it, because the whole worker backs off as if the limit were global.

the design we landed on instead of a worker fleet per tenant, which solves it but is heavier than the problem deserves: a redis token bucket keyed by tenant id, sitting in front of one shared queue. workers check the bucket before pulling a tenant's job, and skip to the next job if that tenant is out of tokens rather than blocking on it. one queue, fair scheduling across tenants, no per-tenant infrastructure.

the actual insight isn't the redis part, it's decoupling rate-limit state from job state. the queue doesn't need to know about limits at all, it just needs a cheap check before deciding whether to work a job or move on.

has anyone run token-bucket-per-tenant against a single shared queue at real scale? curious what breaks first, contention on the bucket keys or fairness under sustained pressure.

a chat widget kept saying "i don't have that on file" for answers clearly in its knowledge base. the same kb answered correctly in the playground and on a separate voice path. swapped models, no change. queried the index directly, it returned the right chunks. both dead ends.

the bug was upstream of retrieval entirely. the widget built its query from the visitor's bare last message. on a follow-up turn the subject is often missing, especially when the bot named it, not the visitor: "hours?", bot answers with a location, then "what's the cost breakdown?" embed that alone and you retrieve the wrong chunks, so the model truthfully says it doesn't know. it only ever sees what the retriever hands it. the playground ran multi-query, hyde, rrf, and a rerank pass, the widget was a plain single-query top-6, same index, opposite behavior.

fix was building the query from the last few turns instead of just the latest message, which cleared most of the "kb is broken" reports on its own.

before you blame your embedding model or re-ingest anything, are you actually logging the query your retriever sends, not just the answer it returns?

published our mcp server package this week and hit three things that would've quietly broken it for every user while looking fine locally.

npm's 2fa web-auth url prints redacted in a non-tty shell, auth/cli/***, so if you're publishing from an automated session the real link never exists anywhere to open. running the publish under a pseudo-tty gets npm to write the real url to a log you can pull from.

separately, a bin path with a ./ prefix gets silently stripped by npm's pack-time validation, warning only, and resolving env vars at module load instead of lazily inside the tool call crashes the server on startup for anyone who imports it before setting env. both pass every local test, because locally you already have env set and you're not running the published artifact.

the check that actually catches it: pipe an initialize handshake through npx on the published version and confirm you get json back.

disclosure, we ship an mcp server ourselves, that's where this came from.

anyone else got a pre-publish checklist for mcp packages specifically?

we had a column tracking which channel a user last messaged on, and it would get stuck on the first channel forever, even after they clearly switched.

two things were stacked. the read side pulled that column from a cached snapshot instead of re-querying, bug one on its own. the deeper one was in the write path: the upsert used on conflict do nothing, fine for columns you genuinely don't want touched, but it meant the channel column never updated on conflict either, since do nothing means nothing, not "nothing except this one column."

fixed it by re-fetching the value on read instead of trusting the snapshot, and changing the upsert to on conflict do update scoped to just the channel column, so untouched columns stay untouched and the one that should change gets explicit permission to.

such an easy thing to get backwards writing the conflict clause fast: do nothing is not a synonym for do nothing to this specific column i care about.

how do you scope conflict updates when only some columns on a row should actually change on conflict?

a voice assistant with a knowledge base attached behaved like one with no KB, answering from base model knowledge, even though it was wired up and multiple KBs were attached.

the retriever tool pointed at assistantKnowledgeBases[0]. that join had no orderBy, so index 0 was whatever the database returned first, in practice the oldest record. the platform allows only one KB tool per agent. when that oldest KB was archived or had zero completed sources, the retrieval handler returned an empty list and the model quietly fell back to its own knowledge, never touching a newer KB with real content sitting right next to it.

about 2,670 of 20,120 KB-enabled voice agents, roughly 13%, had their first-by-id KB pointing at zero completed sources, one in eight answering from the model the whole time. fix was loading the related rows, filtering out archived and deleted, picking the newest KB with a completed source, then re-syncing.

empty retrieval and no retrieval look identical from the outside. if your setup only supports one retriever per agent, has anyone built real selection logic for which one gets attached, or is index 0 more common than I'd like to think?

our cloudwatch bill sat around $321 a month, almost entirely log ingest, not storage, not alarms, about 23 gb a day of ingest. before digging in, the assumption was genuine traffic volume.

it wasn't. the single biggest source was one error line, repeated 848,000 times in 24 hours, coming from 7 sources with stale oauth tokens retrying against an upstream that kept rejecting them. seven misbehaving clients, not real load.

separately, and this cost real time before it clicked: our logs flow through docker to a log forwarder to cloudwatch, and the forwarder wraps the underlying json log inside another json envelope. a plain field:value log line actually lands in cloudwatch nested inside a "log" key as an escaped string. any parse rule written against the inner shape matches zero rows, silently, because you're one layer of escaping off from what you're looking at.

between fixing the retry storm and the parse rules, there was real savings on that ingest line without touching anything that mattered.

anyone else found their biggest log-cost driver was a handful of misbehaving clients rather than actual scale?

a chat agent's CRM writes landed intermittently, some fields saved, some stayed blank, no visible error. replaying the exact conversation extracted the value fine, so the logic wasn't the problem.

the agent calls its own backend over HTTP to run tools, and that request passes through a db-pressure circuit breaker with a bypass allowlist. the tool paths weren't on it. whenever the pool tripped pressure, latched for 30s, every tool call in that window got a 503. the retry wrapper only retried about 12 seconds, so the pressure window outlived the retries and the write was gone for good. each field was a separate call, so it looked like a random coin flip.

the pressure was self-inflicted: the per-process db pool was capped at 120 while the actual pooler allowed 10,000. at peak, processes queued locally at 120 and the breaker read that as database pressure, while the pooler had thousands of connections free. about 200 tool-call 503s a day, fleet-wide, before the fix.

anyone else had a load shedder mistake its own internal traffic for the external load it was built to protect against?