RevelirQA, the auto QA engine from Revelir AI, performs QA to the standard of a human quality analyst. Revelir calibrates it with each customer to over 95% accuracy, meaning that when the customer's own QA analysts review and validate a sample of its evaluations, more than 95 out of 100 are judged correct. That standard is the real test in a build vs buy decision. Coverage is not. Scoring 100% of conversations is table stakes that most AutoQA tools already claim, and a prompted LLM can do it too. A prompt can ask the model to explain its reasoning, but on its own it cannot agree with your analysts or check each judgment against your current SOPs. Revelir AI does this calibration work in production at Xendit and Tiket.com. Our conclusion is that most CX teams should buy, and only teams prepared to own calibration permanently should build.

TL;DR

• RevelirQA performs QA to the standard of a human quality analyst. It is calibrated with each customer to over 95% accuracy, as validated by the customer's own QA analysts on a sample of evaluations. 100% coverage is table stakes that most AutoQA tools already claim.
• A prompted LLM is a scoring script. A QA program also needs SOP retrieval, a calibration loop and a reasoning trace on every score.
• The prototype is cheap. The ongoing engineering and calibration work is not [dev.to][esynergy.co.uk].
• Judge any option, built or bought, by analyst-validated accuracy on your own tickets.

About the Author: Revelir AI builds RevelirQA, AI customer service QA software that Xendit and Tiket.com run in production on thousands of tickets per week in English and Indonesian. This guide draws on our experience calibrating AI scores against enterprise QA teams.

What is the difference between prompting your own LLM and buying an AutoQA platform?

Both paths usually run on the same kind of large language model. The real decision is who owns the work around the model.

• Build: You send transcripts and your QA scorecard to a general-purpose model such as GPT, Gemini or Claude, then parse and store the scores. You rent the model and build everything else yourself [dev.to].
• Buy: You use an automated quality assurance platform. The vendor engineers and maintains ingestion, policy retrieval, calibration and audit logging.
• Train your own LLM: This is rarely justified for QA because of the cost, data and talent involved [esynergy.co.uk].

Why doesn't a well-written prompt produce analyst-level QA scores?

If the model is similar on both paths, score quality depends on what surrounds the prompt. A prompt alone is missing three things:

• Your policies. A model can judge tone in general terms. It cannot know whether a quoted refund amount, eligibility rule, required disclosure or escalation path was correct unless the right SOP is retrieved for that ticket.
• Your interpretations. Research on AI agents observes that these systems typically assume an expert user with well-formed preferences [alphaxiv.org]. A QA prompt makes the same assumption about your scorecard. Criteria like "took ownership" are rarely that well defined.
• A way to correct itself. The same errors keep appearing until someone traces them and fixes the cause.

Think about how a new QA hire learns. On day one they apply the scorecard literally. They become reliable only after calibration sessions, where they grade the same tickets as senior analysts and each disagreement becomes a shared rule. The prompt is the scorecard, and calibration is the training.

What does building auto QA in-house actually involve?

Closing that gap is the actual build. Here is what an in-house team owns beyond the prompt:

Component	What you maintain	Why it is hard
Helpdesk connection	API pulls, conversation filters, field mapping	Integration with existing systems is a core cost of in-house AI
SOP retrieval	Knowledge base in a vector database, retrieved by contact reason	If the wrong document is retrieved, the model gives a confident but wrong score. Production RAG is its own discipline
QA metrics	Each criterion set up as a binary, multi-option or scored metric, with weights and critical errors	Vague criteria produce inconsistent verdicts
Calibration loop	Analyst samples, disagreement tracking, fixes	It never finishes, because policies keep changing
Observability	Model, prompt, retrieved documents and reasoning logged for every score	LLM-as-judge setups need their own monitoring
Model changes	Re-validation after updates or switches	A new model version can change verdicts on unchanged tickets

How should you prove an AI QA score is accurate?

Every row in that table exists to serve one measure. Accuracy is the share of AI evaluations that your own QA analysts judge correct when they review a sample. Use the same process whether you build or buy:

1. Agree with your QA leads on what "correct" means for each metric before scoring starts.
2. Sample across contact reasons, languages, and both human and chatbot conversations.
3. Have analysts check each verdict and its cited evidence, one metric at a time.
4. Trace every disagreement to its cause: the wrong document was retrieved, the metric wording was ambiguous, or the model reasoned badly.
5. Fix, re-score and re-sample. Repeat after every policy or model change.

Revelir runs this process with each customer. In the paid pilot, RevelirQA grades an agreed subset of 10,000 conversations over six weeks. It is measured against accuracy definitions and targets set with the customer's QA leads in week one. Every score shows the model, prompt, retrieved documents and reasoning, and QA leads can override any grade.

When does building your own auto QA make sense, and what does each path cost?

With that accuracy bar defined, the decision comes down to who can meet it at a sustainable cost. Building can make sense when:

• you have engineers and analysts with time permanently reserved for calibration;
• your scorecard is short and mostly about tone, with little policy checking;
• scoring must sit inside in-house infrastructure that no vendor can reach.

Buying usually wins when:

• you operate in a regulated industry and need a defensible trail behind every score;
• you run AI chatbots alongside human agents and want one consistent QA scorecard;
• you score in more than one language or across several helpdesks.

On cost, the token bill is the visible part. Each evaluation sends a transcript plus the retrieved policy passages, so spend grows with both volume and context length. Teams renting general models are often surprised by it [dev.to]. The bigger cost is people: engineers to maintain the pipeline and analysts to run calibration for as long as the system exists. Buying turns that into a known structure. Revelir charges a one-time deployment fee covering the scorecard build, SOP ingestion, helpdesk integration and calibration until your QA leads sign off. After that, you pay a usage fee per conversation graded. The $10,000 pilot fee is credited in full against the deployment fee if you proceed.

Frequently Asked Questions

Can we just use Claude or GPT to score tickets?
Yes, for a prototype. Production QA also needs SOP retrieval, calibration against your analysts and logging for every score.

What accuracy should auto QA reach?
Set the target with your QA leads and measure it on samples your analysts validate. RevelirQA is calibrated with each customer until that customer's analysts judge more than 95 out of 100 sampled evaluations correct.

Doesn't every auto QA tool score 100% of conversations?
Most say they do. Zendesk QA describes analysis across all interactions without manual sampling. Loris states it analyzes 100% of conversation-based interactions. Since coverage is common, the quality of judgment is what sets tools apart.

Does auto QA replace QA analysts?
No. Analysts shift from sampling tickets to calibrating the AI, reviewing disputes and coaching agents. In RevelirQA, no action is triggered automatically. The QA lead decides what to do with the results.

Can automated quality assurance score AI chatbots?
Yes, as long as the chatbot produces a conversation log via API. RevelirQA applies the same scorecard to chatbot and human conversations, so their scores are directly comparable.

About Revelir AI

Revelir AI is headquartered in Singapore and builds RevelirQA, an AI quality assurance platform for enterprise customer service teams. RevelirQA performs QA to the standard of a human quality analyst, calibrated with each customer to over 95% accuracy as validated by that customer's QA analysts. It scores conversations against the customer's own SOPs, retrieved via RAG, and attaches a full reasoning trace to every score. Xendit and Tiket.com run it in production, with SaaS or dedicated-tenant deployment and connections to any helpdesk via API.

If you are weighing build against buy, hold both options to the same standard: your analysts' judgment on your own tickets. To see how RevelirQA is calibrated to your QA team, visit Revelir AI.