How big does an eval dataset actually need to be?
We're an early-stage startup (3 engineers) and have been shipping AI features for about 6 months. Up to this point our testing has basically been me and one other engineer eyeballing outputs in staging before each release, plus whatever users report after.
I finally got time carved out this sprint to set up actual evals (been looking at Braintrust, Langfuse, Arize, etc.) and the tooling side seems pretty straightforward. What I'm stuck on is the dataset itself. So far I've hand-picked ~20 examples from our logs that cover our main use cases plus a few edge cases that have burned us before. And it honestly feels embarassingly small. Every guide I find is super vague on this. Some say start small and iterate, others are throwing around numbers in the hundreds or thousands.
Also unsure about sourcing. Pulling real inputs from production logs feels like the obvious move since it reflects what users actually do, but our logs are full of repetitive/low-effort prompts. I could write synthetic cases to fill the gaps, but then I feel like I'm just testing for stuff I already know to look for.
So for anyone who's set this up, how big was your dataset when you started with? Did you grow it over time or do a big upfront push? And what's your rough split between real production data vs synthetic?