A good AI QA scorecard for customer service is built for the reason the customer got in touch. It keeps a small shared core (tone, clarity, resolution) and changes its heaviest metrics and critical errors by contact reason. Billing tickets are judged mainly on the right amount and eligibility rules. Technical tickets are judged on diagnosis and escalation. Cancellations are judged on correct eligibility and required disclosures. Revelir AI builds scorecards this way for RevelirQA, an AutoQA engine that performs QA to the standard of a human quality analyst. Revelir calibrates it with each customer until more than 95 out of 100 evaluations are judged correct when that customer's own QA analysts review and validate a sample. Scoring 100% of conversations matters, but most auto QA tools already claim it. The harder problem is grading each conversation against criteria that fit its contact reason.
TL;DR
- RevelirQA grades conversations to the standard of a human quality analyst. It is calibrated per customer to over 95% accuracy, as validated by the customer's own QA analysts on a sample of evaluations. 100% coverage is table stakes.
- One fixed scorecard for every ticket averages away the errors that matter most. Use a shared core plus contact-reason metrics.
- Weight metrics by where an error costs the business most: policy accuracy for money, escalation for technical issues, disclosures for cancellations.
- Every AI grade should cite the quote it judged and the SOP passage it checked against.
- Score AI chatbots and human agents on the same contact-reason scorecard to see which handles each reason better.
What is an AI QA scorecard, and why should it change by contact reason?
The case for contact-reason scorecards starts with the definition. An AI QA scorecard applies that framework through software, so each criterion must be written precisely enough for a model to grade it the same way every time.
The reason to split by contact reason is simple: the cost of an error depends on why the customer called. On a single scorecard with fixed weights, an agent who is polite and quick but quotes the wrong refund amount can still score well overall. The one error that matters gets averaged against several that do not.
A driving examiner works the same way. Mirror checks count on every test, but the manoeuvre decides which mistake fails you. A missed blind-spot check during a motorway merge is dangerous in a way a slightly wide parallel park is not, so the examiner weights errors by where the crash risk sits. A contact-reason scorecard weights errors by where the business risk sits.
What should every contact-reason scorecard share?
Before splitting by contact reason, keep a shared core so agents in different queues are still compared fairly. Traditional scorecards assess product knowledge, problem-solving and customer satisfaction [dialpad.com]. A practical core looks like this:
- Greeting and personalization: did the agent greet the customer professionally [turabit.com]?
- Tone and empathy: appropriate to the customer's state.
- Response clarity: could the customer act on the answer?
- Resolution quality: was the issue closed, or a clear next step given?
A good scorecard balances quantitative and qualitative criteria [turabit.com]. Keep the core small so contact-reason metrics carry real weight.
What should an AI QA scorecard measure for each contact reason?
With the core in place, the contact-reason layer is where most of a scorecard's value sits. Common contact-reason groups include billing, technical issues, account management, general inquiries, and returns or cancellations.
| Contact reason | Highest-weight metrics | Critical error (auto-fail) | Evidence the AI should cite |
|---|---|---|---|
| Billing and payments | Correct amount or fee explained; eligibility checked | Wrong refund amount quoted | Billing and refund policy |
| Technical issues | Troubleshooting steps followed; fix confirmed | Wrong escalation path | Troubleshooting SOP, escalation procedure |
| Account management | Verification completed before changes; correct process | Change made without required verification | Account security SOP |
| General inquiries | Answer accuracy; first-contact resolution | Incorrect product or policy information | Knowledge base article |
| Returns and cancellations | Eligibility applied correctly; retention offer within policy | Incorrect eligibility criteria; missed required regulatory disclosure | Returns policy, compliance documents |
Three patterns stand out:
- Policy accuracy dominates wherever money or eligibility is involved. A grader without access to the policy cannot know a quoted amount is wrong. Revelir reports that its policy-aware scoring flags wrong refund amounts, incorrect eligibility criteria, missed regulatory disclosures and wrong escalation paths, which a generic QA scorecard does not detect.
- Escalation compliance matters most where frontline agents cannot resolve the issue.
- Tone weight should rise on cancellations and complaints, where how the customer feels at the end shapes retention.
How do you build contact-reason scorecards step by step?
Knowing what to measure is half the job. The other half is writing each metric so an AI grades it the way your best analyst would.
- Classify contact reasons from real tickets, not only the helpdesk dropdown, which agents often fill in loosely.
- Map each reason to the SOP that governs it. If no document exists, the AI has nothing to check against.
- Pick the metric type: binary (disclosure given or not), multi-option (which escalation route), or scored (clarity on a scale).
- Name one or two critical errors per reason that fail the ticket regardless of other scores.
- Write each metric as a gradable question with a passing and a failing example.
- Calibrate against human analysts on a sample before go-live.
Frameworks such as the COPC Customer Experience Standard and ISO 18295 for customer contact centers are useful references for structuring quality criteria.
How do you check that an AI scorecard is grading correctly?
A contact-reason scorecard is only useful if its grades can be trusted, which makes calibration the real test of any AutoQA tool. Intryc's 2026 buyer's guide compares AI QA platforms on calibration, disputes and coaching, alongside coverage [intryc.com]. Three practices separate trustworthy grading from guesswork:
- Define accuracy before you measure it. Agree with QA leads what counts as a correct grade, then have them validate a sample.
- Require evidence on every score. RevelirQA records the model, prompt, documents retrieved and reasoning for each criterion on every evaluation.
- Keep humans in charge. QA leads can override any grade, agents can contest scores, and disagreements feed back into calibration. RevelirQA triggers no warnings, rankings or discipline.
Should AI chatbots and human agents share the same scorecard?
The same logic applies to chatbots: by default, yes, with extra criteria added for AI-handled conversations. A shared contact-reason scorecard shows which reasons the bot handles better or worse than people. It also shows where AI errors cluster and whether the fix is a knowledge base update, a prompt change or a new escalation rule.
Frequently Asked Questions
What is AutoQA in customer service?
AutoQA (or auto QA) is automated quality assurance that scores customer conversations against a QA scorecard without manual review. Manual sampling typically reviews only a small share of interactions.
Do I need a separate scorecard for every contact reason?
No. Use one shared core with contact-reason modules. RevelirQA supports custom metrics configurable per team and per contact reason.
What does RevelirQA's over 95% accuracy mean?
Revelir calibrates the AI with each customer until the customer's own QA analysts, reviewing a sample of evaluations, judge more than 95 out of 100 correct. It is reached per customer, not out of the box.
What should count as a critical error?
Any mistake that creates financial, compliance or security exposure, such as a wrong refund amount or a missed required disclosure.
Can AI QA grade conversations in languages other than English?
Yes. Xendit and Tiket.com run RevelirQA on conversations in English and Indonesian in production.
About Revelir AI
Revelir AI is a Singapore-headquartered company building RevelirQA, AI customer service QA software for enterprise teams worldwide. RevelirQA retrieves each customer's own SOPs before scoring, so contact-reason metrics are judged against the actual policy. Every score carries an auditable reasoning trace, and the same QA scorecard applies to human agents and AI chatbots. It connects to any helpdesk via API, including Zendesk and Salesforce.
Ready to see what contact-reason QA looks like on your own tickets? Learn more or book a conversation with the team at Revelir AI.
References
- Google (google.com)
- Customer Service QA Scorecard: How to Build One | Dialpad (dialpad.com)
- Best AI QA Software for Customer Service: Coaching, Multichannel QA & Disputes (2026) | Intryc (intryc.com)
- How to Boost Customer Experience with a QA Scorecard (turabit.com)