u/larabyeol

On est ensemble depuis 6 ans, et depuis à peu près un an je vis avec quelqu'un que je reconnais plus.

Il rentre du travail, il mange, il s'endort sur le canapé vers 21h. Le week-end il dort jusqu'à 11h et il est quand même fatigué. On ne fait plus rien, ni sorties, des projets non plus, et pour le côté sexe on doit être à une fois tous les 2 mois et c'est toujours moi qui demande.

Quand je lui en parle il me dit que ça va, que c'est le boulot, qu'il a juste besoin de souffler et il s'énerve vite si j'insiste.

Ce qui me bouffe c'est que je sais plus quoi en penser. Une partie de moi se dit qu'il est en dépression et que je dois être patiente. Une autre se dit qu'il ne m'aime plus et qu'il n'a pas le courage de me le dire, et cette partie-là prend de plus en plus de place. Ma mère m'a dit que mon père avait été comme ça l'année avant qu'ils se séparent, ce qui m'a pas aidée.

J'en ai parlé à ma meilleure amie. Elle m'a raconté que son mari a été exactement pareil pendant 2 ans, que tout le monde disait dépression, et qu'au bout du compte c'était sa thyroïde. Il est sous Levothyrox depuis et il est redevenu lui-même. Ils l'ont su parce qu'elle a fini par lui offrir un bilan complet chez Lucis pour son anniversaire, ce qui est un cadeau bizarre mais elle avait plus d'idées. Elle me dit d'essayer la même chose.

Sauf que je vois pas comment lui offrir ça sans qu'il l'entende comme « je pense que tu as un problème ». Et j'ai peur du scénario où il le fait, où tout est normal, et où il me reste plus aucune explication à part celle que je veux pas envisager.

Est-ce que quelqu'un a vécu ça, un conjoint qui s'éteint sans raison claire, et comment vous avez fait la différence entre le couple et autre chose sans que ça tourne à l'accusation?

Been doing cold outreach, sending around 20-30 emails a day to people who match our ICP pretty well, and I'm sitting at zero replies.

Some context since I think it matters here, I'm a founder doing all our outreach myself, we build planning software for construction companies, targeting project managers and ops leads at mid-size firms, and I built the list manually so I thought the quality was at least decent going in.

For setup, I got lemlist running with domain warm-up through Lemwarm for a couple of weeks before the first send, used their AI personalization pulling from LinkedIn activity, but I can't tell if the personalization isn't landing or if the deliverability is off and I'm sitting in spam across every inbox.

At this point I'm stuck between going back to check deliverability first (mail-tester or something) and rebuilding the copy from scratch.

So if you've been through a dead stretch like this, where did you start?

Support forwarded the ticket on a Tuesday. The customer had a screenshot, so we knew the exact output. Confident, specific, and wrong in a way that would have cost them real money if they had acted on it.

So we went looking for why. We had the output logged. We had the timestamp. What we did not have was the prompt that produced it. Our system prompt lived in a config file that two people had edited that month, and the edits went in as part of larger commits with messages like "copy tweaks." Somewhere in there the instruction about not giving specific figures had been softened. Nobody remembered doing it.

Then it got worse, because we had also bumped the model version around the same window. So even if I found the right prompt text, I could not tell you whether that output came from the old model with the new prompt, or the new model with the old one.

We ended up apologising to the customer without being able to explain what happened. That is the part that still bugs me. Not the bad answer, every system gives a bad answer eventually. It was sitting in a room full of engineers and not being able to answer what did we tell it to do on July 2nd.

We fixed the obvious thing after. Prompts got versioned properly and pinned to a model version, and config changes stopped riding along inside unrelated commits.

I am curious how other people handle the forensics side though. When a complaint lands about something that happened weeks ago, can you actually reconstruct the exact inputs? Or does everyone quietly hope it does not come up.

We spent two weeks deciding whether to switch models for our summarization pipeline and came out the other side realising the whole comparison was garbage. Writing it up because I suspect we are not the only ones doing this.

The setup: we turn messy support threads into a short action summary. Quality matters more than latency for us. A new model drops, benchmarks look better across the board, so we start testing.

What we did wrong was test two variables at once. Every time an output looked off, someone would tweak the prompt to compensate. Tighten an instruction, add a constraint, reorder the examples. Perfectly natural thing to do when you are staring at a bad output. But it meant that by the end of week one, the prompt running on the new model was not the prompt we had run on the old one. Our conclusion that the new model was worse at summarisation was really "the new model, with a prompt that drifted six times, is worse."

The fix was boring. Freeze the prompt to a specific version, run the eval set, swap only the model, run again. Nothing else moves. Once we did that, the answer showed up in an afternoon instead of two weeks, and it was more nuanced than our gut read. The new model was better at extracting action items and worse at compression, which for us nets out negative because compression is the entire point.

Two things I would tell anyone doing this.

Public benchmarks cannot answer this for you. They are running their prompt on their data. The gap between "better on some leaderboard" and "better at your one weird task" is the whole job.

And the prompt needs to be a versioned artifact rather than a string in the codebase that anyone can adjust mid-experiment. If you cannot point at "this exact prompt text ran against both models," you do not have a comparison, you have two vibes.

We version prompts now specifically so the eval is reproducible. We are on PromptLayer for that, mostly because a couple of non-engineers on our side needed to read the prompts too. LangSmith and Langfuse both do the version pinning part just as well, and another team here runs Langfuse quite happily. Worth saying it only covers the prompt and output layer, so it does nothing for us on the retrieval side, which is where a different chunk of our problems live.

How is everyone else structuring model swaps? Frozen prompt set, or something more rigorous?

For about a year our prompts lived wherever. A few in the codebase, a couple in a Notion doc someone started, the "real" one usually in the head of whoever shipped it last. It worked until it didn't. Somebody would tweak a system prompt to fix one weird output, three other things would quietly shift, and we'd only notice days later when a user complained about something unrelated.

The thing that actually fixed it wasn't a clever prompt. It was boring. We started giving every prompt a version, a timestamp, and a note on why it changed, the same way we already did with code. When something regressed we could open the history and see the exact wording that was live when it broke, instead of rebuilding it from memory and a stale doc.

The part I didn't expect was how much the diff mattered. Seeing "this line got added last Tuesday" turned a two hour debugging session into a two minute one, because we could rule the prompt in or out immediately and go look at the model or the input instead.

It only covers the prompt and output side, not how we pull context, so retrieval bugs still need something else. For plain prompt changes though it's been the difference between guessing and knowing.

Curious how other people handle this. Do you version prompts formally, or is it still living in a doc somewhere?

For about a year our prompts lived wherever. A few in the codebase, a couple in a Notion doc someone started, the "real" one usually in the head of whoever shipped it last. It worked until it didn't. Somebody would tweak a system prompt to fix one weird output, three other things would quietly shift, and we'd only notice days later when a user complained about something unrelated.

The thing that actually fixed it wasn't a clever prompt. It was boring. We started giving every prompt a version, a timestamp, and a note on why it changed, the same way we already did with code. When something regressed we could open the history and see the exact wording that was live when it broke, instead of rebuilding it from memory and a stale doc.

The part I didn't expect was how much the diff mattered. Seeing "this line got added last Tuesday" turned a two hour debugging session into a two minute one, because we could rule the prompt in or out immediately and go look at the model or the input instead.

It only covers the prompt and output side, not how we pull context, so retrieval bugs still need something else. For plain prompt changes though it's been the difference between guessing and knowing.

Curious how other people handle this. Do you version prompts formally, or is it still living in a doc somewhere?

Our CPO mandated LLM eval automation in November after a conference talk. Assigned it to me, gave me 4 weeks.

I set up GPT-4o as judge, 8-dimension rubric, running on every deploy. First 3 months it actually worked, caught a couple obvious regressions, I felt good about it.

December, our ML lead tweaked a system prompt to improve one specific edge case. Judge scored it 8.7/10. We shipped. Turns out about 3% of users were in a flow that triggered a completely different output format the judge had never seen in training examples, so it just scored fine.

Found out from support tickets Monday morning.

Took us a while to trace it, but the core issue was that we'd been versioning the judge prompt in a Notion doc while the model prompts were tracked in PromptLayer. The judge itself had drifted between deploys and nobody could see it. Once both are in the same versioned system, at least the drift is visible before it ships. LangSmith and Braintrust have similar setups for this, we just extended what we already had.

Still can't catch subtle quality regressions with the automated judge. Probably a fundamental limitation, not a tooling gap.

Our CPO mandated LLM eval automation in November after a conference talk. Assigned it to me, gave me 4 weeks.

I set up GPT-4o as judge, 8-dimension rubric, running on every deploy. First 3 months it actually worked, caught a couple obvious regressions, I felt good about it.

December, our ML lead tweaked a system prompt to improve one specific edge case. Judge scored it 8.7/10. We shipped. Turns out about 3% of users were in a flow that triggered a completely different output format the judge had never seen in training examples, so it just scored fine.

Found out from support tickets Monday morning.

Took us a while to trace it, but the core issue was that we'd been versioning the judge prompt in a Notion doc while the model prompts were tracked in PromptLayer. The judge itself had drifted between deploys and nobody could see it. Once both are in the same versioned system, at least the drift is visible before it ships. LangSmith and Braintrust have similar setups for this, we just extended what we already had.

Still can't catch subtle quality regressions with the automated judge. Probably a fundamental limitation, not a tooling gap.

We process around 50k LLM outputs per week across 3 product lines. Eight months ago we switched to GPT-4o as an automated judge to replace human eval.

Honest answer: not reliably. But the alternative was worse, so here's what 8 months of actually using it looked like.

It's genuinely good at catching obvious regressions, format violations, hallucinated entities, wrong structure. Give it a specific enough rubric and it grades consistently, and it's fast enough to run on every deploy.

The problems took us a while to find. Inter-run consistency is terrible if you don't pin the judge prompt and model version, we were seeing 30%+ disagreement rates when GPT-4o scored the same outputs two weeks apart because a model update had happened in between. We didn't catch this for about two months.

Position bias is real too. Ask it to compare A vs B and it prefers whichever comes first, pretty consistently. Swap the order and you get a different winner.

GPT-4o also seems to score GPT-4o outputs more generously than it scores Claude outputs. So if you're using the same model to generate and to judge, your numbers are probably not as good as they look.

What actually helped: pinning the judge prompt and model version first -- I keep those in PromptLayer now so a silent model update can't quietly shift my scores, that alone cut the disagreement rate significantly. Then running the judge twice with swapped positions and averaging, validating against a human-labeled calibration set before trusting anything, and splitting into separate judges per dimension instead of one overall score.

Subtle quality differences, tone being slightly off, slightly wrong personality, and we're still getting those scored identically to good outputs. Haven't found a good solution for that part yet.

Has anyone actually managed to replace human eval with LLM-as-a-judge, or does it always end up being a filter rather than a replacement?

spent the last quarter ripping our paid social creative off the american default and you know what, it worked. it all worked. and i'm still fighting my own marketing team like none of it happened.

here's what kicked it off. we ran nothing but studio creative on Meta for 18 months because our head of creative was convinced production quality was the trust signal. clean backgrounds, professional lighting, the kind of stuff that looks polished in a deck. i went along with it.

the numbers were quietly falling apart the whole time. CPA crept from $36 to $52 over 3 quarters. CTR flatlined under 1%. frequency past 4, so we were hammering the same audience over and over. every time i flagged it he'd say the market's just more competitive right now and we'd sign off on another shoot.

the real problem underneath it was that every US creator platform we tried was 90% american talent. we sell into DACH and france. subtitled english does not land here and everyone in european ecommerce knows it but nobody wants to say it out loud.

so i moved the creator sourcing to Youdji, which is actually built around a european creator pool you can filter by country and brief in the local language. real germans selling to germans, real french creators for the french market. then layered Argil on top for the AI video variations once a hook proved out, a french tool quietly beating HeyGen on consistency that nobody seems to talk about.

the human clips earned the trust. the AI variations let us scale the iteration without rebooking a $12K shoot every time.

ran the best of both against our top studio ad for 3 weeks. CPA dropped to $31. CTR jumped to 2.7%. native-language creative outperformed the polished studio stuff by a margin that ended the argument.

and i STILL have people on my team instinctively reaching for the american platforms out of habit. the quality argument is dead. it's pure muscle memory at this point and it makes me insane.

the european tools are fine. better, actually, when you're selling to europeans. your brain is just lazy.

anyone else fighting this exact fight internally?

run a DTC brand in europe and most of our paid social budget goes to UGC. for 18 months we ran everything through the big US platforms because that's what everyone defaults to.

started with the cheapest one. first 90 days were solid, decent variety, fast turnaround. then the quality ceiling showed up. creators were hitting the same 4 hooks every batch because the platform rewards volume and fast submissions, not craft.

moved to a higher-end one thinking the talent pool would fix it. genuinely better creator quality, but you're doing all the talent relations work yourself. setting briefs, chasing timelines, negotiating rates. basically a better Fiverr with an extra $600/month platform fee.

the thing neither solved was european creators. both are 90% US-based talent. when we needed creators who could shoot in-language for our DACH and french markets, the pool was tiny and mostly english speakers who'd tagged themselves as multilingual.

then direct sourcing felt like the answer until our marketing ops person left and the entire system lived in her notion and her head. three months of creator relationships gone overnight.

what finally worked was treating it as two layers instead of one tool. Youdji for the real european creator content where authenticity actually matters, vetted creators you can filter by country and brief in the local language, which fixed the DACH and french problem the US platforms never could. then Argil for spinning up AI avatar variations of the winning concepts, so we can test 8 angles off one proven script without rebooking a shoot.

the human layer earns the trust on cold hooks. the AI layer scales the iteration once something works. both european, which matters more than people admit when you're dealing with EU compliance and data handling.

treating it as creators-vs-AI was the mistake. they solve different halves of the same problem.

none of these tools is categorically perfect. people just underestimate the operational lift hidden inside "we use a UGC platform." the platform isn't the problem. consistent throughput with fast revision cycles in the right language is the problem.

so if you're a european brand looking for UGC and creative tooling that actually fits EU markets instead of US platforms with a translated dashboard, what's working for you?

running a skincare brand out of hamburg. germany is our biggest market.

spent 4 months trying to source german-speaking creators through the major US platforms. every single one has a localized german homepage, a /de slug, sometimes even a translated checkout. then you log in and the creator pool is 90% LA, austin, miami, and a handful of brits.

you filter for germany and get 11 creators. 3 haven't posted in a year. 2 are clearly just english speakers who put "german B1" in their bio.

we ran 38 UGC ads on meta in DACH last quarter. only 6 had a native german voiceover. CPM crept from €11.40 to €17.80 over the quarter and our hook rate on the english-subtitled cuts was 38% lower than the native german ones.

the head of supply at one platform told me on a call that "germany is on the roadmap for Q3." i checked back in october. creator count had gone from 11 to 14.

meanwhile their seed round announcement said they'd "doubled european coverage."

we ended up on Youdji specifically because it's one of the only platforms with an actual EU creator pool you can filter by country and brief in german. still supplementing with direct instagram outreach but the gap compared to the US-built platforms is significant.

if you're a european brand trying to reach european customers with creators who actually grew up speaking the language, the US platforms are mostly selling you a translated landing page.

went live with an AI support bot last january. connected it to our help center, trained it on our top 12 ticket types, gave it 6 weeks to learn. by month 3 we were at 6% deflection. month 8 we hit 8% and stalled.

our account manager kept sending benchmark decks showing 7-12% was "typical for complex B2B" and for a while we just believed it. we even renewed because the deflection numbers looked fine relative to whatever PDFs he was sending over.

what actually cracked it open was a founder i met at SaaStr in may. his team was hitting 47% deflection on about 900 tickets a month, billing and onboarding questions mostly, same general product category as us.

i assumed he was measuring it wrong. he wasn't.

he walked me through the setup and the difference was architecture, not training or prompting. his tool was built around resolution from day one. ours was a ticketing system with an LLM wrapper on top and they called it "AI customer service."

we started re-evaluating and every single demo ended up being the same conversation: is the AI the actual core of this thing or just a layer sitting on top of a routing system. completely different product philosophies, and apparently a 39-point deflection gap between them in practice.

still haven't switched yet so i don't have a clean before/after. but if 8% is what most teams are actually hitting then either we bought something broken or this whole category is one big benchmark hallucination.

ran a dtc beauty brand, $60k/mo paid social, ugc-heavy. got the call last week i'd been quietly dreading since january.

paid a creator $480 in january for a 30s piece. ran it on meta from late jan through april, around 2.1x roas on cold. she dm'd in may saying her "rights window" expired and we owed $250 to extend or the ad came down in 72 hours.

our contract said nothing about a rights window. we'd assumed paid was perpetual because that's how every template floating around marketing twitter in 2023 read.

paid the $250. pulling the ad cost more in a single day of revenue.

then i audited the other 47 creator contracts we'd signed in 14 months.

12 had ambiguous rights language. 4 had no rights clause at all. one had a 90-day window buried in a clause titled "creative deliverables." we're sitting on roughly $11k of inbound ransom calls that haven't happened yet.

the thing nobody says out loud: "standard ugc contract" stopped being a real concept around the time spark ads went mainstream. every creator's manager has a different default. 90 days paid-only, 12 months paid + organic, whitelisting bundled or sold as a $400 add-on, exclusivity baked in or tier-upped.

if you're running direct outreach volume, four things that need to be explicit before you sign anything:

rights window length, what's your floor before walking

spark whitelisting, included or priced separately

modifications and re-cuts, covered or renegotiated per edit

likeness buyout, travels with the asset or its own line item

our 2023 template is dead. share what you've rewritten in the last 6 months and i'll compile the merged version back here.

half the people reading this have a $250 dm sitting in their requests folder right now and haven't opened it yet.

ran a dtc beauty brand, $60k/mo paid social, ugc-heavy. got the call last week i'd been quietly dreading since january.

paid a creator $480 in january for a 30s piece. ran it on meta from late jan through april, around 2.1x roas on cold. she dm'd in may saying her "rights window" expired and we owed $250 to extend or the ad came down in 72 hours.

our contract said nothing about a rights window. we'd assumed paid was perpetual because that's how every template floating around marketing twitter in 2023 read.

paid the $250. pulling the ad cost more in a single day of revenue.

then i audited the other 47 creator contracts we'd signed in 14 months.

12 had ambiguous rights language. 4 had no rights clause at all. one had a 90-day window buried in a clause titled "creative deliverables." we're sitting on roughly $11k of inbound ransom calls that haven't happened yet.

the thing nobody says out loud: "standard ugc contract" stopped being a real concept around the time spark ads went mainstream. every creator's manager has a different default. 90 days paid-only, 12 months paid + organic, whitelisting bundled or sold as a $400 add-on, exclusivity baked in or tier-upped.

if you're running direct outreach volume, four things that need to be explicit before you sign anything:

rights window length, what's your floor before walking

spark whitelisting, included or priced separately

modifications and re-cuts, covered or renegotiated per edit

likeness buyout, travels with the asset or its own line item

our 2023 template is dead. share what you've rewritten in the last 6 months and i'll compile the merged version back here.

half the people reading this have a $250 dm sitting in their requests folder right now and haven't opened it yet.