How we stopped QA scores depending on who marked the call
We had a problem a lot of teams probably have: the same call could score 91% from one marker and 73% from another. Agents stopped trusting feedback and started asking "who marked it?" instead of "what can I improve?"
What fixed it was a monthly 60-minute calibration session:
Everyone scores the same call solo, blind, no chat
Reveal all scores side by side
Only discuss the questions where scores differ, lowest scorer explains first
Agree the score AND write down why in the scorecard guidance
We track two numbers each session: % of questions where every marker agreed (aim 80%+) and the gap between highest and lowest overall score (aim under 10 points).
Biggest surprise: most disagreements were about N/A. One marker was using it to avoid giving zeros.
Happy to share the session template if anyone wants it. How do other teams handle this?