A small QA team can control quality across tens of thousands of travel and e-commerce conversations if it stops grading tickets and starts setting the standard that an AutoQA engine applies. RevelirQA, Revelir AI's AI quality assurance platform, is calibrated with each customer until that customer's own human QA analysts review a sample of its evaluations and judge more than 95 out of 100 correct. It performs QA to the standard of a human quality analyst. Full conversation coverage is table stakes because most auto QA tools already claim it. Revelir's view comes from production work at Tiket.com, an Indonesian travel platform, and Xendit, an Indonesian fintech. Both run RevelirQA on thousands of tickets per week.

TL;DR

• RevelirQA performs QA to the standard of a human quality analyst, calibrated with each customer to over 95% accuracy as validated by the customer's own QA analysts on a sample of evaluations. Full coverage is table stakes; the real differentiator is human-level judgment.
• The real test is whether scoring checks agents against your actual refund, rebooking and returns policies.
• Automated quality assurance moves a small QA team out of grading and into calibrating, auditing and coaching.
• Weight QA scorecard metrics by contact reason. A flight cancellation and an order-status query carry very different risks.

About the Author: This article was written by the Revelir AI team. Revelir builds RevelirQA, an AutoQA scoring engine in production at Tiket.com and Xendit, where it grades blended human and chatbot customer service conversations in English and Indonesian.

Why does a small QA team struggle to keep up in travel and e-commerce?

A small team needs a different operating model because of a ratio: conversation volume grows with bookings and orders, and QA headcount rarely does. Zendesk reported in 2024, citing its Customer Service Quality Benchmark Report, that only 2 percent of customer service interactions are manually evaluated on average [zendesk.com]. Travel and e-commerce make that gap expensive for three reasons:

• Policy density. Fare rules, refund eligibility and return windows vary by product. That makes it easy for an agent to give a confident but wrong answer.
• Volume spikes. Holiday peaks, flight disruptions and sales events push up total volume and the number of exception cases at the same time.
• Channel spread. Customers expect the same tone and service quality on every channel [google.com], and better service builds loyalty and sales [zoko.io].

Revelir uses an illustration to show the maths. Thirty agents with 20 reviews each comes to 600 reviews out of 50,000 monthly conversations, or 1.2% coverage. An error that happens once in every 200 conversations produces 250 failures a month. The sample would be expected to catch about 3 of them.

What should a small QA team stop doing, and what should it start doing?

Since sampling cannot scale, the answer is to change how analysts spend their hours rather than to hire more of them.

Task	Manual sampling	AutoQA
Grading	Analysts score a small sample by hand	AI scores every conversation, and analysts validate a sample of the AI's grades
Scorecard	A document each reviewer applies using their own judgment	Criteria, weights and critical errors built as grading metrics
Finding problems	Depends on what the sample contains	Work from a ranked list of issues
Coaching	Feedback on single tickets	Coaching on each agent's recurring gaps
Disputes	Analyst and agent debate the grade	Overrides feed back into calibration

What makes AutoQA trustworthy enough to replace most manual review?

That role shift only works if the AI gives the grades your analysts would have given. This is why judgment matters more than coverage. Check for four things:

• Calibration against your analysts. Your own QA team should measure accuracy by validating a sample of AI grades. Your internal validation is the meaningful benchmark.
• Policy grounding. The AI should check answers against your SOPs. That is how it catches a wrong refund amount, not only poor tone.
• A reasoning trace. Each score should show the quoted exchange, the policy passage it was checked against and the reasoning behind the grade.
• Human control. QA leads should be able to override grades, and agents should be able to contest them.

Calibration works like onboarding a new QA analyst. The new hire grades the same tickets as your most experienced reviewer. The two discuss every disagreement until their scores consistently match. You end up trusting the new hire because their judgment was tested against someone you already trust. AI calibration uses the same mechanism, with your analysts as the reference point.

RevelirQA works this way. It cites the SOP passage behind each judgment and lets QA leads override grades. It triggers no automatic warnings, rankings or discipline. Rendy D. of tiket.com described the result: "We've manually reviewed tickets for years. Revelir is the first product that has made AI ticket review at scale actually usable."

Which QA metrics matter most for travel and e-commerce conversations?

Calibration tells you whether the AI grades well, and the QA scorecard decides what it grades. Prioritise QA metrics where a wrong answer costs money or trust:

Contact reason	Typical failure	QA metric
Flight or hotel cancellation	Wrong refund amount quoted	Binary: refund matches policy
Rebooking or schedule change	Wrong escalation path	Binary: escalation compliance
Order return	Incorrect eligibility criteria	Binary: eligibility stated correctly
Delayed delivery	Vague next steps	Scored: resolution quality and tone
Payment dispute	Missed required disclosure	Binary, marked as a critical error

Weight criteria by contact reason. Also compare customer sentiment at the start and end of each conversation. A ticket marked "resolved" can still end with a customer angrier than when they started. That is a real risk when satisfaction and loyalty drive e-commerce customer experience [shinedezigninfonet.com].

How do you roll out automated quality assurance with a small QA team?

Once the scorecard is set, the order of the rollout decides how much analyst time you actually save:

1. Convert your scorecard into grading metrics. Make each one binary, multi-option or scored, and mark the critical errors.
2. Collect current SOPs for each contact reason: refunds, rebooking, returns and escalation.
3. Agree on accuracy definitions and targets with your QA leads in week one.
4. Calibrate on a sample until your analysts sign off on the AI's grades.
5. Keep validating a sample after launch, especially after policy changes.
6. Send patterns in contact reasons and recurring issue types to your product and operations teams, as well as into coaching.

In Revelir deployments, Revelir's own team builds the scorecard, ingests the SOPs and connects the helpdesk. Nobody on the customer side has to write prompts.

Frequently Asked Questions

What is AutoQA?

AutoQA, or auto QA, is automated quality assurance for customer service. AI scores conversations against a QA scorecard, instead of analysts grading a small sample by hand.

Does AutoQA replace QA analysts?

No. It replaces most manual grading. Analysts still own the scorecard, validate AI grades, resolve disputes and lead coaching.

What does RevelirQA's over 95% accuracy mean?

Revelir calibrates RevelirQA with each customer. When that customer's human QA analysts review a sample of its evaluations, they judge more than 95 out of 100 correct. Each customer reaches this level through calibration. It is not an out-of-the-box figure.

Should chatbots and human agents share one scorecard?

Yes. A shared scorecard shows which contact reasons the chatbot handles worse than humans. It also helps you tell whether the fix is a knowledge base update, a prompt change or a new escalation rule.

How often should humans check AI grades?

Continuously, on a sample. Re-check after policy changes such as new fare rules or seasonal return windows.

How should you trial an AutoQA tool?

Run a time-boxed pilot on part of your scorecard, with accuracy targets agreed by your QA leads before grading starts. Revelir's six-week paid pilot grades an agreed subset of 10,000 conversations this way.

About Revelir AI

Revelir AI builds RevelirQA, an AI quality assurance platform that grades customer service conversations to the standard of a human quality analyst, calibrated with each customer to over 95% accuracy as validated by the customer's own QA analysts on a sample of evaluations. RevelirQA scores conversations against each customer's own SOPs and QA scorecard, applies one scorecard to human agents and AI chatbots, and attaches a full reasoning trace to every score. It connects to any helpdesk via API, including Zendesk and Salesforce, and is in production at Tiket.com and Xendit.

If your ticket volume has outgrown your QA team, see how calibrated AutoQA performs on your own conversations. Get in touch with Revelir AI at https://www.revelir.ai/.