Sampling a fraction of support tickets and calling it quality assurance is not a programme, it is a placebo. When a team reviews 10% of conversations, the 90% that goes unseen still shapes customer outcomes, agent behaviour, and churn risk. The real cost is not in the tickets you catch, it is in the systemic patterns, policy violations, and sentiment collapses buried in the ones you never open. Genuine quality management requires full coverage, consistent scoring, and evidence that connects individual conversations to business outcomes.

About the Author: Revelir AI builds AI customer service software for enterprise teams processing thousands of tickets per week. With production deployments at Xendit and Tiket.com, Revelir has direct, operational experience with the limits of manual QA in high-volume, multilingual environments.

What Does "10% QA Sampling" Actually Mean in Practice?

QA sampling is the practice of selecting a subset of support conversations for human review against a scoring rubric. Industry data indicates contact centres typically review between 1% and 5% of customer interactions, meaning the vast majority of conversations are never evaluated. It is the default approach at most support organisations because it appears cost-effective: reviewers evaluate a manageable batch, flag trends, and pass results to team leads.

The problem is structural. A 10% sample of 10,000 weekly tickets means 9,000 conversations are never evaluated. If a policy mis-application is occurring in 8% of tickets, a 10% random sample may catch a handful of cases, but it will never surface the pattern with statistical confidence. The reviewer sees individual errors; the organisation never sees the systemic failure.

• Selection bias: Reviewers naturally gravitate toward escalations, flagged tickets, or recent interactions, skewing results toward already-visible problems.
• Inconsistency between reviewers: Two QA analysts scoring the same ticket often assign different ratings because human judgment drifts across time, mood, and fatigue.
• Coaching based on the wrong data: Agents receive feedback derived from a non-representative sample, which can reinforce or punish behaviour that does not reflect their actual patterns.

What Are the Real Financial Costs Hiding Inside a Manual QA Process?

Manual QA is rarely treated as a cost centre, but its financial drag is measurable. Research indicates that repetitive manual test cycles can account for up to 40% of total dev and QA costs, in addition to the downstream fixes they fail to prevent ^[1]. Industry analysis of manual testing and quality processes estimates hidden costs per team member running into tens of thousands of dollars annually in rework, delayed detection, and re-escalation ^[5].

Why Does Sampling Fail to Scale as Ticket Volume Grows?

Sampling does not scale, it compounds. As ticket volume grows, a fixed percentage sample represents an increasingly arbitrary slice of reality. A team handling 2,000 tickets per week sampling 10% reviews 200 conversations. At 20,000 tickets per week, the same percentage nominally reviews 2,000 tickets, but the ratio of unseen to seen conversations remains identical, and the cost of human review grows linearly with the team size needed to sustain it ^{[4] [6]}.

For high-growth operations in fintech or travel, where contact reasons shift week-to-week based on product changes, promotions, or regulatory updates, a static sample means quality data is always at least one cycle behind operational reality. By the time a trend surfaces in a sampled review, the customer impact has already accumulated.

What Does "Sentiment Arc" Reveal That Resolved Tickets Hide?

A resolved ticket looks like a success in any standard reporting system. The ticket is closed, the CSAT survey goes out, and the metric moves. But resolution and satisfaction are not the same thing. A customer who started a conversation frustrated and ended it neutral has not recovered loyalty. A customer who started positive and ended negative after a routine interaction is a measurable churn risk, regardless of whether the ticket was technically resolved.

This distinction is the sentiment arc: the trajectory from how a customer felt at the start of a conversation to how they felt at the end. Manual QA sampling does not surface this at scale. A reviewer reading one ticket can note tone, but cannot identify that 15% of conversations this week followed the same deteriorating arc and share a common contact reason.

Revelir Insights tracks both opening and closing sentiment across 100% of conversations. The result is not a single data point per ticket but a population-level view of how customer emotion is shifting, which contact reasons are generating tone deterioration, and which agents are consistently moving customers from negative to positive.

How Should Enterprises Think About Moving From Sampling to Full Coverage?

The shift from sampling to complete coverage is not simply about doing more of the same thing faster. It requires a different architecture.

1. Ingest your own policies, not generic benchmarks. A QA scoring engine that evaluates conversations against industry averages tells you how you compare to an abstraction. An engine that retrieves your actual SOPs before scoring every ticket tells you whether your team is following your rules. RevelirQA ingests knowledge bases and SOPs into a vector database, ensuring every score reflects your specific standards.
2. Separate scoring from reviewing. Human reviewers should handle calibration, appeals, and complex edge cases, not routine scoring. AI handles volume; humans handle judgment.
3. Build an audit trail into every evaluation. For compliance-sensitive industries, every score needs a reasoning trace: which documents were retrieved, which prompt was used, and why the score was assigned. This is not a nice-to-have; it is a regulatory requirement for fintech teams.
4. Evaluate AI agents under the same rubric as human agents. As organisations deploy AI for ticket deflection, quality oversight must cover the AI's conversations too. A unified rubric across human and AI agents gives CX leaders a complete quality picture.

Frequently Asked Questions

At very low ticket volumes (under a few hundred per week), a 10% sample may be sufficient to detect large effect sizes. At enterprise scale, it is not. The sample is too small to surface systematic patterns with confidence, and selection bias further degrades reliability.

A scoring engine evaluates conversations against a defined rubric and produces structured scores with reasoning. An agent takes action autonomously in conversations. RevelirQA is a scoring engine; it does not participate in conversations, it evaluates them after the fact.

Purpose-built AI customer service software can evaluate conversations in the language they were conducted, provided the underlying model has multilingual capability. Revelir has demonstrated this in Indonesian-language, high-volume environments at Xendit and Tiket.com.

Without a reasoning trace, an AI score is an assertion without evidence. If an agent disputes a low score, or a regulator requests documentation of compliance monitoring, a system that produces only a number with no supporting rationale is inadequate. Every RevelirQA score includes the prompt, retrieved documents, and reasoning behind the evaluation.

No, and it should not be positioned that way. AI handles consistent, high-volume scoring. Human analysts handle calibration, rubric design, edge case review, and coaching conversations. The goal is to redirect human judgment toward decisions that require it, not to eliminate it.

Coaching based on a complete dataset is categorically more actionable than coaching based on a sample. Managers can identify an agent's actual pattern across hundreds of conversations rather than inferring behaviour from a handful of reviewed tickets, reducing both over-correction and missed development opportunities ^[3].

Most enterprise AI customer service platforms integrate via API with major helpdesks including Zendesk and Salesforce. Revelir connects to any helpdesk via API, meaning QA coverage is not limited by which ticketing system the organisation uses.

References

1. The Hidden Cost of Manual Testing: Why Your IT Team is Burning Out (www.rimo3.com)
2. The Hidden Costs of Manual Ticket Resolution: How AI Automation Improves MSP Margins - AI powered automation for MSPs (zofiq.ai)
3. 5 Hidden Costs of Manual QA Programs - Insight7 - Call Analytics & AI Coaching for Customer Teams (insight7.io)
4. The Hidden Cost of Manual QA in Growing SaaS Teams | SystemClarity - Technical Leadership for Growing Systems (systemclarity.work)
5. The 2026 Quality Tax: Why AI-Assisted Development Didn't Actually Shrink Your QA Budget | Bug0 (bug0.com)
6. Manual vs Automated Testing: $1M Cost Gap (2026) | Autonoma (www.getautonoma.com)