RevelirQA performs quality assurance to the standard of a human quality analyst, calibrated with each customer to over 95% accuracy as validated by the customer's own human QA analysts on a sample of evaluations. A quality assurance scorecard is a structured set of weighted criteria, tied to your actual policies and customer outcomes, used to score customer service conversations consistently across every agent and channel. Building one from nothing means working through five steps in order: define the outcome you're protecting, translate your SOPs into scorable criteria, decide how each criterion is weighted (including any auto-fail items), pilot the scorecard against a small batch of real conversations, then calibrate it against disagreements between reviewers. Teams that skip the pilot step almost always end up rewriting their scorecard within the first month, because criteria that look clear on paper turn out to be ambiguous the moment two different QA analysts score the same call differently.

TL;DR

• RevelirQA delivers quality assurance at the standard of a human quality analyst, calibrated per customer to over 95% accuracy as validated by your own QA analysts on a sample of evaluations. A good QA scorecard is not a checklist, it's a weighted QA scorecard that maps directly to policies, SOPs, and the customer outcomes your business actually cares about.
• Start with outcomes (first-contact resolution, compliance, retention risk), not with generic "soft skills" criteria copied from a template.
• Pilot the scorecard on real conversations before rolling it out. Disagreement between reviewers during the pilot is the signal that a criterion needs rewriting, not a sign the reviewers are wrong.
• Manual QA scorecards are traditionally applied to a small sample, usually 1-5% of tickets, which means most scorecards are validated on far less data than teams assume.
• AutoQA platforms like RevelirQA can apply the same scorecard to 100% of conversations once it's built, but the scorecard design work described here still has to happen first, whether scoring is done by a person or an AI.

About the Author: This article is written by the team at Revelir AI, which builds RevelirQA, an AutoQA platform that scores customer service conversations to the standard of a human quality analyst. Revelir has worked directly with CX and QA teams at Xendit and Tiket.com to design, digitize, and calibrate scorecards running in production across thousands of tickets a week.

What Is a QA Scorecard and Why Does Structure Matter More Than Content?

A QA scorecard is a documented QA scorecard that breaks a customer service interaction into specific, scorable criteria, each weighted according to how much it affects the outcome the business cares about. The structure matters more than any individual criterion because a badly structured scorecard produces inconsistent scores even when the underlying judgment is sound. If two QA analysts read the same ticket and reach different conclusions about whether an agent "showed empathy," the problem usually isn't the analysts, it's a criterion that was never defined precisely enough to score the same way twice [abstrakt.ai]. The fix is to write criteria as observable behaviors rather than traits: not "was polite" but "used the customer's name at least once and did not interrupt the stated issue before acknowledging it." Prominent industry frameworks such as the COPC CX Standard, ISO 18295, ITIL, and guidance from ICMI and HDI all converge on this same principle: specificity in criteria definition is what makes a scorecard repeatable across reviewers.

How Do You Decide What to Measure First?

Building on the structural point above, the first real decision isn't what criteria to include, it's what outcome the scorecard exists to protect. Teams with no existing system often start by brainstorming a long list of behaviors to check, which produces a scorecard that's exhaustive but unfocused. A better starting point is to pick one or two business outcomes, such as first-contact resolution, chargeback or complaint rate, or customer retention, and work backward to the agent behaviors that most plausibly move those numbers. A call center scorecard built around resolution and compliance will look very different from one built around upsell conversion, even if both are scoring the same conversations [sqmgroup.com][callforce.global]. For most customer service teams, three categories of criteria cover the majority of what matters:

• Policy and compliance criteria: Did the agent follow the documented SOP, disclose required information, and avoid saying anything that creates regulatory exposure? This is where standards like GDPR, HIPAA, PCI-DSS, TCPA, and CCPA often show up directly in scorecard line items, particularly for fintech, healthcare, and telecom teams.
• Resolution criteria: Did the agent correctly diagnose the issue, use the right internal process, and close the loop without requiring a follow-up contact?
• Communication criteria: Was the response accurate, clear, and appropriately paced for the channel (chat, email, voice)?

Auto-fail items deserve a separate category. These are criteria serious enough that a single miss, such as sharing a customer's personal data with the wrong party, zeroes the entire score regardless of how well everything else went [voxjar.com]. Mixing auto-fail items into the regular weighted total dilutes their severity and is one of the more common design mistakes in first-time scorecards.

How Should Criteria Be Weighted on a QA Scorecard?

Weighting is where most scorecards either earn or lose credibility with the agents being scored. A flat scorecard, where every criterion is worth the same number of points, implicitly tells agents that a missed greeting matters as much as a compliance violation, which is rarely true and agents notice the mismatch quickly. The more defensible approach is to weight criteria by their actual downstream cost: compliance and resolution-blocking errors should carry more weight than stylistic or tone-based criteria [abstrakt.ai][scorebuddycx.com]. A practical way to set weights without guessing is to look at a sample of past escalations, complaints, or churned accounts and ask which criteria, if they had been caught earlier, would have prevented the outcome. That's a small analytical exercise, but it roots the scorecard in evidence rather than intuition, and it gives QA leaders a defensible answer when an agent asks why one miss cost more points than another.

Criterion Type	Typical Weight Logic	Example
Auto-fail	Overrides total score; single miss fails the ticket	Disclosing customer data to the wrong party
Compliance / policy	Heavily weighted, directly tied to regulatory or SOP risk	Required disclosures, refund policy adherence
Resolution	Moderately to heavily weighted, tied to repeat-contact rate	Correct root-cause diagnosis, correct escalation path
Communication / tone	Lightly weighted, unless tied to a measurable CX outcome	Clarity, empathy, pacing

How Do You Pilot a New Scorecard Before Rolling It Out?

Once weighting is drafted, the next step is proving the scorecard actually produces consistent scores, and that only happens by testing it on real conversations before anyone depends on it. The standard method is to select a handful of recent tickets, ideally ones that span easy and difficult cases, and have two or more QA analysts score them independently using the draft scorecard [balto.ai]. Compare the results. Any criterion where reviewers disagree is not a reviewer problem, it's a definition problem, and it needs rewriting before the scorecard goes live. This step is frequently skipped by teams building their first scorecard because it feels like it slows down the rollout, but it's the single highest-leverage step in the whole process: a scorecard that passes a ten-ticket pilot with high reviewer agreement will hold up far better at scale than one that's never been tested against real disagreement. Think of it the way a new survey instrument gets piloted before a full rollout: the goal isn't to confirm the questions sound reasonable, it's to confirm that different people interpret them the same way when real, messy data is in front of them.

What Happens After the Scorecard Goes Live?

A scorecard is never finished once it's deployed, because the conversations it scores keep changing as products, policies, and customer behavior evolve. Most QA teams building their first system run into the same bottleneck a few months in: a scorecard designed for manual review only ever gets applied to a small sample, often cited around 1-5% of total tickets, because reading and scoring conversations by hand doesn't scale with ticket volume. That sampling limit isn't a flaw in the scorecard itself, it's a constraint of the review method. A scorecard built carefully but scored on 2% of conversations can still miss a policy-miss pattern concentrated in the other 98%, simply because no reviewer happened to pull those tickets. The market reflects this gap: contact center quality assurance software is projected by one 2026 industry report to grow at a 9.24% compound annual rate through 2035, driven largely by teams trying to close that coverage problem. This is where automated quality assurance comes in as a scaling option, not a replacement for the design work above. AutoQA, or auto QA, tools apply a scorecard's logic to every conversation rather than a sample, but they only work as well as the scorecard they're given. RevelirQA, for example, is built to apply a customer's own scorecard and SOPs, retrieved via RAG, to 100% of conversations, scoring to the standard of a human quality analyst and calibrated per customer until independent review by the customer's own QA analysts confirms over 95% of its evaluations are judged correct. That calibration process is itself a more rigorous version of the pilot step described above, just run continuously rather than once.

Frequently Asked Questions

How many criteria should a QA scorecard have?
Most functional scorecards land between 8 and 15 criteria. Fewer than that risks missing important behaviors; more than that tends to slow reviewers down and increase inconsistency between scorers [scorebuddycx.com].

Should every team use the same QA scorecard template?
No. Templates are a useful starting structure, but the specific criteria should reflect your own SOPs, compliance obligations, and the outcomes you're trying to protect, not a generic list [abstrakt.ai][voxjar.com].

What's the difference between a QA scorecard and a coaching form?
A scorecard scores a conversation against defined criteria; a coaching form uses those scores to identify specific, actionable feedback for an agent. The scorecard is the measurement, coaching is the follow-up action.

How often should a scorecard be reviewed and updated?
A quarterly review is a reasonable baseline, with ad hoc updates whenever a policy changes significantly or a new recurring issue type emerges that the current scorecard doesn't capture well.

Can a QA scorecard apply to both AI chatbots and human agents?
Yes, if the criteria are written around the resolution and compliance outcome rather than human-specific behaviors like tone of voice. Teams running both AI and human service increasingly need one scorecard both can be measured against, which is part of why platforms like RevelirQA evaluate AI agents and human agents on the same QA scorecard.

Do regulatory requirements need to be built into the scorecard itself?
Yes, for any team operating under standards such as GDPR, HIPAA, PCI-DSS, TCPA, or CCPA, compliance criteria should be explicit, auto-fail items on the scorecard rather than folded into general "professionalism" categories.

About Revelir AI

Revelir AI builds RevelirQA, an AI quality assurance platform that scores customer service conversations to the standard of a human quality analyst, calibrated per customer to over 95% accuracy as validated by independent review from the customer's own QA analysts. Founded in Singapore in 2025 by Rasmus Chow, Revelir works directly with each customer to calibrate the AI against their own scorecard and SOPs. The platform runs in production at companies including Xendit and Tiket.com, scoring thousands of tickets a week across English, Indonesian-language, Thai, and Tagalog service operations. Every score comes with a full reasoning trace, including the model, retrieved documents, and reasoning used, giving QA and compliance teams an auditable record behind every result.

If your team has a scorecard but no practical way to apply it beyond a small manual sample, visit Revelir AI to see how auto QA can apply your existing scorecard to every conversation, not just the ones a reviewer happened to pick.