u/donewitheverything26

I set up a Postgres MCP server on a read only role last week so the analysts could ask questions without pinging me every hour.

The first real question was how many orders did we lose last month. Claude looked at orders, saw status_cd with values 1 to 6, decided 4 was cancelled and gave a clean number with a nice little breakdown. 4 is refunded. Cancelled is 6. Nobody wrote that down anywhere. It lives in an enum in the app code and in my head.

The number was off by about a third and it read exactly like a right answer. No I'm assuming, no hedge. The analyst was about to put it in a deck.

So now I'm stuck on how to give an agent the meaning of fields without me turning into a full time documentation service:

1. Comments on the columns in Postgres and hope the server passes them through
2. A markdown data dictionary in the system prompt
3. A view layer with human names on everything
4. Make the agent ask before it interprets any code column

What are people actually doing here? and has anyone got it to say I don't know what 4 means instead of picking one?

Partner sends us a monthly export. Different customer ID scheme to ours, no mapping table, no docs, and whoever built it on their side left last year.

Right now I fuzzy match on lower(trim(email)) plus surname and eyeball a sample before it goes anywhere. It works well enough that nobody's complained, which is not the same as it being right. I have no idea what my false match rate is.

The part that bothers me is that a bad join doesn't announce itself. A wrong customer count looks identical to a right one.

So what do people actually do here. Do you block on something cheap first to cut the comparison space, compute a score and hold back anything under a threshold, or just accept a fuzzy match and monitor downstream? And if you threshold, where did the number come from?

We take CSVs from about thirty suppliers, all with their own header conventions, and there's a model step in the Airflow DAG that maps incoming headers to our canonical fields. Been in prod since March.

Offline it scores 94% on a labelled set an analyst put together over two afternoons, which I don't think is representative of much. In production there are no labels. Nobody goes back and confirms that shipped_dt was actually ship date.

We caught the first real failure because someone in finance said a monthly total looked low. A supplier renamed a column, it got mapped to a field it had no business in, and it had been doing that for five weeks. That was embarrassing.

Two things I've tried since.

Logging the model's own confidence. Useless in the way everyone says — the wrong mappings came back high. No threshold in there separates anything.

Second model on the same input, alarm on disagreement. Agreement sat around 97% and stayed there straight through the five weeks the failure was live. It's still in a notebook, never made it into CI. Great Expectations covers nulls and types and has nothing to say about whether a mapping is correct.

Where I keep landing is that I don't have an output I can validate, I have one I'd have to re-derive to validate, and at that point I'm not sure what the first one is for.

For people running a model step where prod has no labels — what do you actually alarm on? Output distribution, something downstream, or is the honest answer a sample and a human every week?

Inherited a schema where roughly half the columns are self-explanatory and the rest are things like flag_3 and val_b. Person who built it left. There's a Confluence page describing six columns, last edited before most of them existed.

I've been using COMMENT ON COLUMN because it lives with the database and can't drift into a stale wiki. Downside is nobody looks at it, it doesn't show up anywhere people work, and I've no way to know if a comment is still true after a migration.

Things I'm unsure about:

does anyone actually keep COMMENT ON up to date at scale, or does it rot the same as the wiki just less visibly

if a column's meaning changes but the name doesn't, is there anything that catches that, or is it purely a review discipline problem

and for the columns nobody can explain at all, do you leave them, drop them, or keep them with a comment saying unknown

I've been profiling the values to guess — cardinality, null rate, distributions — which narrows it but never gets me to what the thing means.

Been going back and forth on this and I don't think I've seen it written down anywhere.

Chunking a document is well covered. A row isn't a document. If I embed 4471 | 2024-03-02 | 88.50 | 1 | NULL I get something close to meaningless, and it sits near every other row of similar numbers regardless of what the table is about.

Options I've tried. Serialise the row as key-value text, which works and makes the column names do almost all the semantic lifting, which is fine until the columns are called val_b. Generate a description of each column once with a bigger model and prepend that to every row, which is better and multiplies your token cost. Or don't embed rows at all, embed a table-level summary and route the actual question to SQL.

The third one feels correct and then falls apart the moment a question spans two tables that don't share a key.

what are people doing in production here? and if you embed rows, do you put column stats in the chunk or just the values

Partner sends us a monthly export. Different customer ID scheme to ours, no mapping table, no docs, and whoever built it on their side left last year.

Right now I fuzzy match on lower(trim(email)) plus surname and eyeball a sample before it goes anywhere. It works well enough that nobody's complained, which is not the same as it being right. I have no idea what my false match rate is.

The part that bothers me is that a bad join doesn't announce itself. A wrong customer count looks identical to a right one.

So what do people actually do here. Do you block on something cheap first to cut the comparison space, compute a score and hold back anything under a threshold, or just accept a fuzzy match and monitor downstream? And if you threshold, where did the number come from?

Genuine question, I keep going back and forth on this.

Agent calls a SQL tool. Query comes back with 400 rows. Obviously you don't put 400 rows in context. So what do you put.

What I'm doing right now is dumb. Truncate to the first 20 rows and a row count. It works for "how many customers churned" and falls apart the second the question needs anything about the shape of the result, because the model has no idea whether those 20 rows are representative or whether the interesting stuff is in row 300.

Tried summarising the result with a second call. Better answers, but now every tool call is two model calls and the latency doubled on a step that used to be fast.

The other thing that bites is column names. If the query returns something like val_b or flag3, the model will confidently interpret it as whatever seems plausible from the question. It doesn't ask. It doesn't flag it. It just decides.

So:

Are you passing raw rows, a computed summary, or some schema-plus-sample hybrid? Does anyone compute stats server-side and return those instead of rows? And has anyone found a way to make the agent say "I don't know what this column is" rather than guessing, without stuffing a data dictionary into every prompt?

I do marketing at an early stage data infra startup. Ten days ago we did the whole launch properly, or what I thought was properly. LinkedIn, twitter thread, long form reddit posts. everything UTM tagged, reviewed twice, real numbers in the copy, no hype. I was proud of it.

signups: zero. not a few. zero.

Best reddit post got \~1000 views in twelve hours. 2 upvotes, 2 comments. Another one, 194 views and 1 upvote. then I crossposted the first to a related sub and got 0, which I deserved, thats the exact thing every guide says not to do and I did it anyway because I was tired and wanted reach.

two things I got wrong. views mean nothing, I felt great about that 1000 for most of a day before I did the division. and the ask was way too big, going from cold post straight to "sign up" for a technical product nobody's doing that from their phone at lunch.

the one thing that worked was a stranger leaving a long reply on a question I asked, with a number in it I didn't have. one comment, worth more than every impression combined.

so I'm asking things now instead of announcing things.

anyone else had a launch flop and figured out the real mistake after? wondering if "reach doesn't convert" is just how it goes early on

I do marketing at an early stage data infra startup. Ten days ago we did the whole launch properly, or what I thought was properly. LinkedIn, twitter thread, long form reddit posts. everything UTM tagged, reviewed twice, real numbers in the copy, no hype. I was proud of it.

signups: zero. not a few. zero.

Best reddit post got ~1000 views in twelve hours. 2 upvotes, 2 comments. Another one, 194 views and 1 upvote. then I crossposted the first to a related sub and got 0, which I deserved, thats the exact thing every guide says not to do and I did it anyway because I was tired and wanted reach.

two things I got wrong. views mean nothing, I felt great about that 1000 for most of a day before I did the division. and the ask was way too big, going from cold post straight to "sign up" for a technical product nobody's doing that from their phone at lunch.

the one thing that worked was a stranger leaving a long reply on a question I asked, with a number in it I didn't have. one comment, worth more than every impression combined.

so I'm asking things now instead of announcing things.

anyone else had a launch flop and figured out the real mistake after? wondering if "reach doesn't convert" is just how it goes early on

Partner sends over an export last month. 63 columns, maybe 20 named something a human could read. Rest are val_b, metric_14, one that's just "flag3." No ID matching anything in our warehouse, no data dictionary, guy who built it left in 2023.

Two days gone. Almost none of it pipeline work. Just me squinting at distributions going "this one's between 0 and 1 and 40% null so it's probably a rate of something," then hand-checking matches to see if I was making it up. Which I was, twice.

So what do you actually do here:

Profile everything up front, or start joining and work backwards from the wreckage? I go back and forth. Profiling is probably right but I get impatient.

At what point do you just email whoever sent the file? Held out way too long on this one, mostly pride.

Anything better than fuzzy matching on a few fields and hoping, when there's no shared key at all?

That last one is where I'm stuck. Not the matching, the knowing when to stop trusting it. A 60% confident match is worse than none because it doesn't fail loudly, it quietly wires two wrong things together and you find out four dashboards later when someone asks why the numbers look weird.

Curious if anyone has a real system or if it's vibes and coffee for everyone.