How Indonesian Digital Banks Monitor Customer Service Quality When Chatbots Share the Queue

Published on:
November 13, 2026

How Indonesian Digital Banks Monitor Customer Service...

Indonesian digital banks can monitor quality in a blended queue by grading every chatbot and human conversation against one QA scorecard built from the bank's own SOPs, with a recorded reason behind each score. The real test is judgment. An AutoQA system has to spot a missed disclosure or a wrong escalation path the way an experienced QA analyst would. Revelir AI's RevelirQA performs QA to that human-analyst standard. Revelir calibrates it with each customer to over 95% accuracy: when the customer's own QA analysts review a sample of its evaluations, more than 95 out of 100 are judged correct. Indonesian fintech Xendit and travel platform Tiket.com run it in production on thousands of tickets a week, across blended chatbot and human queues in English and Indonesian. It also scores 100% of conversations, but most auto QA tools already claim that coverage.

TL;DR

  • The bar is human-analyst judgment, not coverage. RevelirQA is calibrated per customer to over 95% accuracy, as validated by the customer's own QA analysts on a sample of evaluations. Scoring 100% of conversations is table stakes.
  • One scorecard for bot and human. Add handoff criteria, because the handoff belongs to neither side by default.
  • Compare by contact reason. Aggregate chatbot scores hide where the bot is confidently wrong.
  • Keep the reasoning. In a regulated bank, why a score was given matters as much as the score.

About the Author: Written by the team at Revelir AI, whose AI quality assurance platform scores blended chatbot and human queues in production for Indonesian enterprises including Xendit and Tiket.com, in English and Indonesian.

Why does a shared chatbot and human queue break traditional QA?

A blended queue breaks traditional QA because one conversation now has two authors, while most quality programs were designed to grade one. Manual review was already thin. Chatbots add problems that sampling alone does not address:

  • Errors repeat at scale. A human who misquotes a transfer limit misleads one customer. A chatbot reading an outdated policy gives the same wrong answer to everyone who asks, until someone fixes the source.
  • Containment is not quality. A conversation the bot "resolved" may be a customer who gave up. Containment shows where a conversation ended, not whether it ended well.
  • Meaning beats keywords. Indonesian customers often mix Bahasa Indonesia and English in a casual register. Keyword checks miss what was actually promised.

The handoff is the blind spot. Scoring the bot and the human separately is like timing each runner in a relay but never watching the baton exchange: a dropped baton happens in the transfer zone, which shows up in neither runner's split. If the bot fails to pass on that a customer reported an unauthorised transaction, the human starts blind, and neither half looks wrong on its own.

What do Indonesian regulators expect from customer service quality monitoring?

The practical consequence: a customer-facing chatbot is a model whose outputs need monitoring, and QA scores become part of that evidence. A score without a traceable reason is hard to defend in an audit. Regulations are updated, so check OJK's official publications for current requirements.

How should a digital bank score chatbot and human conversations on one QA scorecard?

Since regulators judge outcomes rather than channels, the cleanest design is a single QA scorecard for both, plus a few criteria for AI-handled conversations and handoffs. Shared criteria make scores directly comparable. Separate QA scorecards turn every comparison into an argument about definitions.

QA metric Chatbot failure Human failure
Policy accuracyQuotes a fee from a stale knowledge articleMisremembers eligibility criteria
Required disclosureSkips a mandated warning in a scripted flowOmits it under time pressure
Escalation complianceKeeps answering a suspected fraud caseRoutes to the wrong team
Response accuracyHallucinated answer delivered with confidenceGuesses instead of checking
Handoff qualityPasses no context to the humanRe-asks what the customer already said
Tone and empathyCheerful template on an angry complaintCurt reply late in a long shift

Weights should vary by contact reason. A disclosure miss on a loan enquiry deserves more weight than a tone slip on a card-delivery question.

Where should a bank look when chatbot and human scores diverge?

With both channels on one scorecard, the most useful view is quality by contact reason, because an overall chatbot score averages harmless intents with risky ones. The pattern points to the fix:

Pattern Likely cause Fix
Bot fails policy accuracy on one contact reasonOutdated or missing source contentUpdate the knowledge base
Bot is accurate but skips steps or misjudges toneFlow or prompt designChange the prompt
Bot holds cases humans should takeMissing triggerAdd an escalation rule
Humans fail where the bot passesTraining gapTargeted coaching

How can a bank trust AutoQA scores enough to act on them?

None of this analysis is useful if QA leads doubt the scores, so trust in AutoQA and automated quality assurance has to be earned through calibration and transparency, not a vendor's headline number.

  • Calibrate against your own analysts. QA leads should define what "correct" means, review a sample of AI evaluations, and sign off before go-live. Revelir calibrates RevelirQA this way until more than 95 out of 100 sampled evaluations are judged correct by the customer's analysts.
  • Require a reasoning trace. Each RevelirQA score records the model, the exact prompt, the SOP passages retrieved and the reasoning per criterion, on every evaluation.
  • Keep humans in charge. QA leads should be able to override grades and agents to contest them, with disagreements feeding back into calibration. Nothing should trigger discipline automatically.

Frequently Asked Questions

What is AutoQA in customer service?
AutoQA, or auto QA, is automated quality assurance that uses AI to grade conversations against a QA scorecard instead of relying on manual sampling.

Should chatbots and human agents share a QA scorecard?
Yes, with added criteria for AI conversations and handoffs, so results are directly comparable.

Is containment rate a good chatbot quality metric?
Not alone. It shows where a conversation ended, not whether the answer was correct and compliant.

Can AutoQA score Indonesian-language conversations?
RevelirQA scores English and Indonesian in production. Scoring meaning rather than keywords matters because code-mixing is common.

How accurate is AI QA scoring?
It depends on calibration. RevelirQA is calibrated per customer to over 95% accuracy, as validated by the customer's QA analysts on a sample of evaluations.

About Revelir AI

Revelir AI, headquartered in Singapore, builds RevelirQA, an AI quality assurance platform for enterprise teams that performs QA to the standard of a human quality analyst. Before each evaluation it retrieves the customer's own SOPs via RAG, applies the same QA scorecard to chatbot and human conversations, and records a full reasoning trace behind every score. It connects to any helpdesk via API, including Zendesk and Salesforce, and is available as SaaS or a dedicated tenant for teams with data residency requirements. Xendit and Tiket.com run it in production on thousands of tickets a week.

If your bank is putting chatbots and human agents in the same queue, see how RevelirQA scores both against your own policies at revelir.ai.