Indonesian digital banks can monitor quality in a blended queue by grading every chatbot and human conversation against one QA scorecard built from the bank's own SOPs, with a recorded reason behind each score. The real test is judgment. An AutoQA system has to spot a missed disclosure or a wrong escalation path the way an experienced QA analyst would. Revelir AI's RevelirQA performs QA to that human-analyst standard. Revelir calibrates it with each customer to over 95% accuracy: when the customer's own QA analysts review a sample of its evaluations, more than 95 out of 100 are judged correct. Indonesian fintech Xendit and travel platform Tiket.com run it in production on thousands of tickets a week, across blended chatbot and human queues in English and Indonesian. It also scores 100% of conversations, but most auto QA tools already claim that coverage.
TL;DR
- The bar is human-analyst judgment, not coverage. RevelirQA is calibrated per customer to over 95% accuracy, as validated by the customer's own QA analysts on a sample of evaluations. Scoring 100% of conversations is table stakes.
- One scorecard for bot and human. Add handoff criteria, because the handoff belongs to neither side by default.
- Compare by contact reason. Aggregate chatbot scores hide where the bot is confidently wrong.
- Keep the reasoning. In a regulated bank, why a score was given matters as much as the score.
About the Author: Written by the team at Revelir AI, whose AI quality assurance platform scores blended chatbot and human queues in production for Indonesian enterprises including Xendit and Tiket.com, in English and Indonesian.
Why does a shared chatbot and human queue break traditional QA?
A blended queue breaks traditional QA because one conversation now has two authors, while most quality programs were designed to grade one. Manual review was already thin. Chatbots add problems that sampling alone does not address:
- Errors repeat at scale. A human who misquotes a transfer limit misleads one customer. A chatbot reading an outdated policy gives the same wrong answer to everyone who asks, until someone fixes the source.
- Containment is not quality. A conversation the bot "resolved" may be a customer who gave up. Containment shows where a conversation ended, not whether it ended well.
- Meaning beats keywords. Indonesian customers often mix Bahasa Indonesia and English in a casual register. Keyword checks miss what was actually promised.
The handoff is the blind spot. Scoring the bot and the human separately is like timing each runner in a relay but never watching the baton exchange: a dropped baton happens in the transfer zone, which shows up in neither runner's split. If the bot fails to pass on that a customer reported an unauthorised transaction, the human starts blind, and neither half looks wrong on its own.
What do Indonesian regulators expect from customer service quality monitoring?
The practical consequence: a customer-facing chatbot is a model whose outputs need monitoring, and QA scores become part of that evidence. A score without a traceable reason is hard to defend in an audit. Regulations are updated, so check OJK's official publications for current requirements.
How should a digital bank score chatbot and human conversations on one QA scorecard?
Since regulators judge outcomes rather than channels, the cleanest design is a single QA scorecard for both, plus a few criteria for AI-handled conversations and handoffs. Shared criteria make scores directly comparable. Separate QA scorecards turn every comparison into an argument about definitions.
| QA metric | Chatbot failure | Human failure |
|---|---|---|
| Policy accuracy | Quotes a fee from a stale knowledge article | Misremembers eligibility criteria |
| Required disclosure | Skips a mandated warning in a scripted flow | Omits it under time pressure |
| Escalation compliance | Keeps answering a suspected fraud case | Routes to the wrong team |
| Response accuracy | Hallucinated answer delivered with confidence | Guesses instead of checking |
| Handoff quality | Passes no context to the human | Re-asks what the customer already said |
| Tone and empathy | Cheerful template on an angry complaint | Curt reply late in a long shift |
Weights should vary by contact reason. A disclosure miss on a loan enquiry deserves more weight than a tone slip on a card-delivery question.
Where should a bank look when chatbot and human scores diverge?
With both channels on one scorecard, the most useful view is quality by contact reason, because an overall chatbot score averages harmless intents with risky ones. The pattern points to the fix:
| Pattern | Likely cause | Fix |
|---|---|---|
| Bot fails policy accuracy on one contact reason | Outdated or missing source content | Update the knowledge base |
| Bot is accurate but skips steps or misjudges tone | Flow or prompt design | Change the prompt |
| Bot holds cases humans should take | Missing trigger | Add an escalation rule |
| Humans fail where the bot passes | Training gap | Targeted coaching |
How can a bank trust AutoQA scores enough to act on them?
None of this analysis is useful if QA leads doubt the scores, so trust in AutoQA and automated quality assurance has to be earned through calibration and transparency, not a vendor's headline number.
- Calibrate against your own analysts. QA leads should define what "correct" means, review a sample of AI evaluations, and sign off before go-live. Revelir calibrates RevelirQA this way until more than 95 out of 100 sampled evaluations are judged correct by the customer's analysts.
- Require a reasoning trace. Each RevelirQA score records the model, the exact prompt, the SOP passages retrieved and the reasoning per criterion, on every evaluation.
- Keep humans in charge. QA leads should be able to override grades and agents to contest them, with disagreements feeding back into calibration. Nothing should trigger discipline automatically.
Frequently Asked Questions
What is AutoQA in customer service?
AutoQA, or auto QA, is automated quality assurance that uses AI to grade conversations against a QA scorecard instead of relying on manual sampling.
Should chatbots and human agents share a QA scorecard?
Yes, with added criteria for AI conversations and handoffs, so results are directly comparable.
Is containment rate a good chatbot quality metric?
Not alone. It shows where a conversation ended, not whether the answer was correct and compliant.
Can AutoQA score Indonesian-language conversations?
RevelirQA scores English and Indonesian in production. Scoring meaning rather than keywords matters because code-mixing is common.
How accurate is AI QA scoring?
It depends on calibration. RevelirQA is calibrated per customer to over 95% accuracy, as validated by the customer's QA analysts on a sample of evaluations.
About Revelir AI
Revelir AI, headquartered in Singapore, builds RevelirQA, an AI quality assurance platform for enterprise teams that performs QA to the standard of a human quality analyst. Before each evaluation it retrieves the customer's own SOPs via RAG, applies the same QA scorecard to chatbot and human conversations, and records a full reasoning trace behind every score. It connects to any helpdesk via API, including Zendesk and Salesforce, and is available as SaaS or a dedicated tenant for teams with data residency requirements. Xendit and Tiket.com run it in production on thousands of tickets a week.
If your bank is putting chatbots and human agents in the same queue, see how RevelirQA scores both against your own policies at revelir.ai.
