We stopped ranking email capture by opt-in rate and some of our winners stopped winning
Disclosure, since the sub asks for it. I work on the onsite widget side, so the tests below ran across client stores, and I am not able to publish per-client percentages without their sign-off. No product names either, and no links.
For a long stretch we ranked opt-in widgets by opt-in rate, wheel beat form, form beat banner, ship it, move on. Then we started reading revenue per session on the same tests. Some of the rankings flipped.
The split runs at session level, revenue per session is arm revenue over arm sessions, and opt-in rate comes out of that same denominator, which is what lets a widget collecting fewer addresses win the comparison. Then a second read at 30 and 90 days, because first-order revenue and third-month revenue kept disagreeing.
Gamification wins the opt-in comparison almost every run we have logged, spin-to-win and scratch cards both. People like the animation, they spin, the address goes in the box, and the list grows on a schedule you can promise a client.
Product-finder quizzes collected fewer addresses per session and in several niches came out ahead on revenue per session anyway, which took a few tests to believe. Two stores in the same niche, comparable traffic, the same two widget types, opposite winners. So the quiz result is a thing that happened in our data, and it predicted nothing about the store one niche over.
Push opt-in rate up while AOV slides and you have run a winning test that pays you less money. Same story with repeat purchase rate. That one shows up a quarter later, when nobody is checking the widget report anymore.
The mechanism is still a hypothesis for me. The wheel plausibly pulls in discount hunters who redeem once and go quiet, while four questions about hair type or a dog's weight filter for someone already mid-purchase and hand you zero-party data for the flows afterwards. I can see the pattern in aggregate and I cannot prove why.
So, the questions I came with. What do you optimize email capture against, opt-in rate, first-order revenue, or LTV? Has anyone watched gamified signups underperform on LTV over a full year? And how do you attribute revenue to a popup cohort without double counting people who were going to buy anyway?