Before adding more training data, check whether the labeling rule is actually stable
I keep seeing CV projects where performance stalls and the first response is to add more images or try another model. Sometimes that helps. But sometimes the model is being asked to learn a rule that people haven’t agreed on.
A partially visible object, an uncertain boundary, or something cut off by the frame can all produce different “correct” annotations. More data just scales that inconsistency.
A simple check is to take 20–30 difficult images and have two people label them independently. Then review the disagreements, not just the agreement score. Each recurring disagreement becomes a written rule with one positive and one negative visual example. Run the same test again on a fresh sample before scaling.
I’d use a similar check for auto-labeling: measure missed objects and correction time per image, not only inference speed. Fast pre-labels aren’t useful if every image still needs a full review.
Disclosure: I work at Supervisely, a computer vision platform. This is a platform-independent observation.
What annotation edge case caused the most trouble in your dataset?