Non-technical PM here, need to eval a "best photo" ranking feature in 3 weeks with zero eng support. How would you approach this?
Working on a "Summer Recap" style feature (think Google Photos Memories), a rule-based lexical scoring system picks the top 15 photos from a user's library based on tags and assigned weights/scores for each tag. My job is to figure out if this logic is actually good enough to ship or not.
Problem is I'm non-technical, I don't have engineering bandwidth to build a proper eval pipeline, and I've got about 3 weeks to get to a ship / no-ship decision.
What I've figured out so far:
- Need a rubric for what "good" means (no screenshots/blur/duplicates, diverse across days and events, feels like an actual summer highlight reel)
- Ideally want to score sets of 15 photos, not just single best-photo picks
- Was thinking of using Claude/GPT vision as an "AI judge" against the rubric instead of manual human rating, since I don't have a rater team
- Considered building an actual lightweight no-code tool for this (Claude artifacts can call the API directly) instead of manually uploading photos into chat one set at a time
Questions for anyone who's done something similar:
Has anyone actually shipped an AI-judge-based eval like this without an eng team? What broke, what worked?
Any no-code / low-code tools you'd recommend for batch-running images through a vision model and logging structured scores (thinking Zapier, Make, Airtable + API, Google Sheets + Apps Script, that kind of thing)?
For a rule-based (not ML) recommender/ranking system specifically, is there a simpler eval approach I'm missing, given I don't need to retrain anything, just score outputs?
How many test cases would you consider the minimum for a credible go/no-go call in this kind of timeline?
Not looking for a full research operation, just need something defensible enough to bring to my manager for a ship decision. Any playbooks, war stories, or "don't bother, just do X" advice welcome.