What Is a QA Scorecard and Why Do Most of Them Fail?

A QA scorecard is a structured rubric used to evaluate customer service conversations against a defined set of criteria, typically covering compliance, tone, resolution quality, and policy adherence. In theory, it is the cornerstone of service quality management. In practice, most scorecards are undermined by three structural problems before a single ticket is reviewed.

The three core failure modes:

• Sampling bias. Most teams manually review 2-10% of conversations. The conversations that get reviewed are rarely random. According to Insight7, vague evaluation metrics and insufficient training for evaluators are among the most common reasons QA scorecards fail agents, but the sampling problem compounds both: you cannot fix what you cannot see.
• Evaluator inconsistency. When humans score conversations, scores shift based on who is reviewing, what mood they are in, and how they personally interpret a rubric criterion. Two evaluators reading the same transcript will score it differently. That variance is noise inside your quality data.
• Generic benchmarks. Most scorecards score against industry-standard criteria, not your policies. If your refund policy has a specific 48-hour commitment and your scorecard does not encode that, agents are being scored against a standard that does not reflect your actual customer promise.

What Does "Policy-Based Evaluation" Actually Mean?

Policy-based evaluation means the scoring engine retrieves your actual knowledge base, SOPs, and customer service policies before it scores each conversation, not after, and not based on a generic rubric someone built from a template.

The mechanism matters. Using retrieval-augmented generation (RAG), the AI ingests your documentation into a vector database. When evaluating a conversation, it retrieves the relevant policy context before producing a score. This means the score reflects whether the agent followed your rules, not a hypothetical industry average.

Why this changes the output:

This is the difference between a compliance check and a quality intelligence system.

Why Does 100% Coverage Matter So Much?

Sampling at 5-10% does not just mean you miss 90% of conversations. It means the conversations you miss are not randomly distributed. High-volume, routine tickets get reviewed. Edge cases, late-night shifts, and conversations handled by newer agents are systematically underrepresented.

According to Voiso, agent scorecards should be reviewed at least quarterly, with higher frequency during periods of scaling or goal changes. But frequency of review is only meaningful if the underlying data is complete. Reviewing a biased sample more often does not reduce the bias.

The business cost of sampling gaps:

• A policy violation pattern that appears in 3% of tickets is invisible at 5% sampling
• Agent coaching is based on the conversations selected for review, not the conversations that most need it
• Systemic failures (a broken escalation path, a miscommunicated policy) hide inside the unreviewed 90%

At Xendit and Tiket.com, Revelir AI's RevelirQA scoring engine processes every conversation, applying the same rubric consistently across thousands of weekly tickets. What changes is not just coverage; it is the quality of the coaching signal. Agents receive feedback grounded in actual performance, not a curated slice of it.

What Is Sentiment Arc and Why Does "Resolved" Not Mean "Satisfied"?

Resolved is a binary status. Satisfied is a human experience. The gap between them is where churn hides.

A ticket marked resolved tells you the conversation closed. It does not tell you the customer started frustrated and ended neutral, which is a retention risk disguised as a success metric. It does not tell you 15% of tickets this week started positive and ended negative, and that they all share a common contact reason.

Sentiment arc tracks two distinct data points per conversation:

• Customer Sentiment (Initial): How the customer felt at the start of the interaction
• Customer Sentiment (Ending): How they felt when the conversation closed

This is meaningfully different from a post-conversation CSAT survey, which captures a single retrospective data point from a fraction of customers who bother to respond. Sentiment arc is derived from the conversation itself, applied to every ticket, and comparable across time.

ThinkOwl's research on agent scorecards reinforces that measuring outcomes without measuring experience creates a blind spot in quality management. A resolution rate of 95% means nothing if sentiment is degrading across a category of contacts.

How Should You Evaluate AI Agents Alongside Human Agents?

This is the question most QA programs have not yet answered. As companies deploy AI chatbots and virtual agents alongside human representatives, they face a fragmented quality picture: human agents scored by one rubric, AI agents either not scored at all or scored by a different standard.

The correct approach is a unified rubric applied to both. If your policy says refund requests must be acknowledged within one message and resolved within 48 hours, that standard applies whether the conversation was handled by a human or an AI agent. Anything else creates a two-tier quality system that obscures where failures are actually occurring.

RevelirQA evaluates AI and human agents under the same policy-grounded rubric, giving CX leaders a single quality view across their entire operation. This matters especially for compliance-sensitive industries like fintech, where an audit trail covering AI-handled conversations is increasingly a regulatory expectation, not a nice-to-have.

Frequently Asked Questions

References