Mixed-script conversations, where a single support ticket contains Bahasa Indonesia, Arabic, and Latin characters simultaneously, are not an edge case in global customer service. They are routine. A meaningful share of Southeast Asian, Middle Eastern, and diaspora customers naturally blend scripts within a single message. The core challenge for AI QA scoring is not detecting which language is present; it is maintaining policy-consistent evaluation when the linguistic ground shifts mid-conversation. Done correctly, a multilingual QA scoring engine evaluates policy compliance, tone, and resolution quality regardless of how many scripts appear in one ticket. Done poorly, it either skips non-Latin segments or applies the wrong policy context entirely.
- Mixed-script tickets are common in high-volume global support, not exceptions to plan around.
- Effective AI QA scoring requires per-turn language detection, not document-level language tagging.
- Scoring policy compliance across scripts demands that the QA engine retrieve the correct SOP version before evaluating, not after.
- Sentiment analysis breaks down in mixed-script threads unless the model evaluates the arc across the full conversation, not just individual turns.
- A unified QA scorecard applied to 100% of conversations removes the blind spots that manual sampling creates in multilingual environments.
What Makes Mixed-Script Tickets Different From Standard Multilingual Support?
Most multilingual QA frameworks are designed for one language per ticket, which is precisely why mixed-script conversations expose their limits. A standard multilingual setup assumes the system can route a ticket to the correct language model and evaluate it end-to-end. Mixed-script tickets break that assumption because the unit of language is no longer the ticket; it is the individual turn or even the individual sentence [1].
Consider a support conversation where a customer opens in Bahasa Indonesia, switches to Arabic script to clarify a proper noun, and closes with an English-language order number. A QA system that classifies the ticket as "Indonesian" and evaluates it as such will miss the semantic content of the Arabic segment and may misinterpret the response to it.
The three specific failure modes this creates:
- Policy retrieval mismatch: The wrong SOP version is retrieved because the system tagged the ticket under one language.
- Sentiment mis-scoring: Frustration expressed in Arabic within an otherwise calm Indonesian thread gets ignored, distorting the sentiment arc.
- Compliance gaps: Responses to the non-dominant-script segments are not evaluated at all [1].
How Does AI QA Scoring Detect and Handle Script Switching in Real Time?
Building on the failure modes above, the harder question is how a scoring engine should respond when it detects a mid-conversation script shift. Robust handling requires three components working in sequence.
| Component | What It Does | Why It Matters for QA |
|---|---|---|
| Per-turn language detection | Identifies the script and language at the message level, not the ticket level | Ensures each response is scored against the right policy context [1] |
| Context-aware retrieval (RAG) | Retrieves the relevant SOP or policy document for the detected language before scoring | Prevents a policy designed for one market from being applied to another [2] |
| Cross-turn conversation evaluation | Maintains a coherent quality assessment across the full thread, not just per-message. Captures resolution quality and sentiment arc across the whole interaction | Ensures each response is evaluated in its full context [4] |
Research on small and large language models evaluated against multi-turn customer service conversations confirms that context summarisation across turns is essential for accurate scoring; models that evaluate turns in isolation produce systematically lower accuracy on resolution and policy compliance metrics [2].
"A scoring engine that reads each turn in isolation is, by definition, unable to evaluate whether the issue was resolved. Resolution is always a property of the full thread."
Why Is Sentiment Analysis Particularly Unreliable in Mixed-Script Threads?
Sentiment analysis is the most fragile component in mixed-script QA, and it is worth examining why before looking at fixes. Most sentiment models are trained on monolingual corpora. When a model trained primarily on English or Indonesian encounters Arabic script mid-thread, it either skips the segment or forces it through an incorrect sentiment framework [1].
The cultural dimension compounds the technical one. Politeness markers, hedging language, and expressions of dissatisfaction vary significantly not just between languages but between scripts within languages. Romanised Arabic ("3arabi") carries different register cues than formal Arabic script [5]. A QA system that treats these as equivalent will consistently under-detect frustration in mixed-script threads.
Practical implications for QA teams:
- Track sentiment as an arc from conversation start to end, not as a single score per ticket.
- Flag tickets where the opening and closing sentiment diverge significantly, regardless of script.
- Do not rely on keyword-based sentiment detection for non-Latin scripts; it fails on colloquial and mixed-register text [1].
What Should a QA Scorecard Cover for Mixed-Script Conversations?
A related but distinct question is how to structure the QA scorecard itself when the conversations being evaluated are linguistically diverse. The answer is that the scorecard criteria should be language-agnostic, while the policy documents retrieved for each evaluation are language-specific [3].
A well-designed scorecard for mixed-script environments should include:
- Policy compliance per segment: Did the support team member follow the correct SOP for each language segment they responded to?
- Completeness of response: Did the support team member address all parts of the customer's message, including those written in a non-dominant script?
- Tone consistency: Was the support team member's register appropriate across the full thread, accounting for cultural norms in each language present?
- Resolution confirmation: Was the issue confirmed as resolved in a way the customer understood, in their preferred language? [3]
RevelirQA implements this through custom scoring metrics, binary, multi-option, or scored criteria, configured per team and retrieved via RAG against the team's own SOPs before every evaluation. Xendit and Tiket.com run this at scale across Indonesian-language, English, and mixed-script ticket volumes every week.
How Do You Maintain Scoring Consistency Across Human and AI Agents in Multilingual Environments?
Stepping back from the technical detail, a separate concern is consistency: how do you ensure that an AI chatbot handling Arabic-script queries and a human support team member handling Indonesian queries are held to the same standard? [3]
The answer is a single, unified QA scorecard applied to both, with the policy retrieval layer doing the work of finding the right standard for each conversation. This removes the common failure mode of having separate QA processes for bot and human traffic, which produces incomparable quality metrics and hides systemic issues [3].
Key principle: the scorecard is constant; the policy documents retrieved are context-dependent. This is the only architecture that produces auditable, comparable scores across a multilingual, mixed-agent support operation [7].
Frequently Asked Questions
Q: Can AI QA scoring handle code-switching, where customers mix two languages in a single sentence?
Yes, but only if the underlying model was trained or fine-tuned on code-switched data. Models trained purely on monolingual corpora will misparse code-switched text. Per-turn detection helps but does not fully resolve intra-sentence switching [2].
Q: Does a mixed-script conversation require separate QA scorecards for each language?
No. The scorecard criteria should be unified. The policy documents retrieved for scoring should be language-specific. Keeping them separate avoids inconsistent quality benchmarks across your support operation [3].
Q: How do you ensure the AI retrieves the right policy for a mixed-script ticket?
Through per-turn language detection combined with a RAG layer that matches detected language to the correct SOP version before scoring. Document-level tagging is insufficient for mixed-script tickets [1].
Q: What is the biggest risk of using manual QA sampling for multilingual support?
Reviewers tend to pull tickets in languages they can read, which systematically under-samples non-dominant-language tickets. This creates blind spots specifically in the languages most likely to contain compliance or tone issues [1].
Q: Can sentiment analysis work reliably in Arabic or Thai script?
It can, but it requires models fine-tuned on those scripts and cultural contexts. Generic sentiment models trained on English data perform poorly on Arabic or Thai text, especially colloquial or mixed-register content [5].
Q: Should AI chatbots and human support team members be scored using the same QA criteria?
Yes. Running separate QA processes for bots and humans produces incomparable metrics and hides quality gaps. A unified scorecard applied consistently to both gives CX leaders an accurate picture of their full support operation [3].
Q: How does a QA scoring engine handle Romanised scripts, such as Romanised Arabic or Bahasa Gaul?
Romanised scripts require the model to recognise informal transliterations, not just standard orthography. This is a training data problem: models need exposure to colloquial Romanised text from the target markets, not just formal written forms [6].
About Revelir AI
Revelir AI builds RevelirQA, an AI scoring engine that evaluates 100% of customer service conversations against a company's own policies and QA scorecard, eliminating the sampling bias of manual review. Every score carries a full reasoning trace covering the model used, documents retrieved, and the reasoning applied, making it suitable for compliance-critical environments. RevelirQA scores both human support team members and AI chatbots on a unified QA scorecard, giving CX and support operations teams a single, consistent view of quality across their entire operation. RevelirQA is in production at Xendit and Tiket.com, processing thousands of tickets per week across English, Indonesian, Thai, and Tagalog.
Want to see how RevelirQA handles mixed-script ticket volumes in your support environment?
References
- Multilingual QA Best Practices for Contact Centers… (www.oversai.com)
- Frontiers | Can small language models handle context-summarized multi-turn customer-service QA? A synthetic data-driven comparative evaluation (www.frontiersin.org)
- Run One QA System Across AI and Human Support Conversations (www.intercom.com)
- How to evaluate multi-turn conversations - Blog - Braintrust (www.braintrust.dev)
- Multilingual Conversational AI: Key Use Cases Explained | Rasa Blog (rasa.com)
- How to Build a Multilingual AI Agent (2026 Implementation Guide) - Invent (www.useinvent.com)
- Multilingual AI CX: How to Serve Global Customers at Scale (www.voiceflow.com)