We A/B tested ai content generator copy against a human rewrite for a fintech. The gap was ugly.
Fifteen years writing copy and brand strategy, mostly fintech and SaaS. I have made my peace with the tools, mostly. But I keep running the same test for clients and the result keeps embarrassing the machine, so here is one with numbers.
A fintech client wanted to cut their content budget. Fair. The pitch they got elsewhere was to run everything through an ai content generator and ship volume. Before they committed I asked to run a straight test on their highest-traffic comparison page, the one that actually drives signups.
Two versions. One was the generator output with light cleanup. The other I rewrote by hand, same offer, same structure, same length. Split the traffic evenly for three weeks.
The generated version: signups held about flat versus the old page, bounce a little higher. The hand-written version: signups up a bit over a third, and time on page noticeably longer.
The difference was not quality in some precious sense. It was specificity. The AI version explained what the product does. The human version named the exact moment the reader is stuck, the thing their finance team says no to, the fee they got burned by last time. The machine writes the category. A person writes the reader.
I am not here to tell you never to touch the tools. Volume has its place for the boring middle of a funnel. But on the pages where money actually changes hands, generic costs more than it saves, and most people never measure it because the cheap version feels productive.
Curious if anyone here has run the same head to head on a page that matters, and whether your numbers agree or I am just protecting my own job.