When Intercom's Fin AI Agent resolves a ticket end to end, there is no human agent to review, which means traditional QA scorecards built around tone, greeting scripts, and hold-time etiquette simply don't apply. The scoring question shifts from "did the agent follow the script" to "did the agent, human or AI, retrieve the right policy, apply it correctly, and resolve the issue without creating downstream risk." RevelirQA performs QA to the standard of a human quality analyst, calibrated per customer to over 95% accuracy as validated by that customer's own QA analysts on a sample of evaluations. Coverage across every ticket is now close to standard practice among AutoQA (auto QA) vendors; what separates a usable automated quality assurance layer from a cosmetic one is whether its judgment actually holds up against a trained human reviewer, and whether it can score an AI agent's reasoning, not just a transcript.

TL;DR

• RevelirQA performs QA to the standard of a human quality analyst, calibrated per customer to over 95% accuracy as validated by the customer's own QA analysts on a sample of evaluations. Scoring 100% of conversations is table stakes most AutoQA tools now claim.
• Fin AI Agent's resolution logic (knowledge retrieval, multi-step workflows, handoff triggers) introduces new failure modes that legacy human-agent QA scorecards were never designed to catch.
• Scoring an AI agent requires visibility into what it retrieved and why, not just what it said; a transcript alone cannot tell you if the agent reasoned correctly.
• Manual sampling, the old way of doing QA, reviews roughly 5 percent of interactions industry-wide, which is structurally incapable of catching a systemic AI reasoning error before it repeats at scale.
• A QA scorecard built for the Fin era needs to evaluate policy grounding, workflow completion, and handoff judgment, across both AI and human agents, on one consistent QA scorecard.

About the Author: This guide is written by the team at Revelir AI, whose RevelirQA engine runs in production scoring thousands of support conversations a week for enterprise clients including Xendit and Tiket.com, across both human agents and AI chatbots on helpdesk platforms including Intercom.

What Does Fin AI Agent Actually Do Inside a Support Workflow?

Fin AI Agent is Intercom's native AI agent, launched in March 2023, designed to resolve customer service conversations autonomously by retrieving answers from a company's knowledge sources and executing multi-step workflows. It operates across channels and languages, and integrates with Intercom's own helpdesk as well as third-party systems like Zendesk and Salesforce, and custom tools through APIs, MCP, and Data Connectors. Intercom's 2026 product documentation states Fin achieves a 76 percent average resolution rate, meaning roughly three out of four conversations it touches are closed without human intervention. That resolution rate is the headline number most teams track. It is also the number that obscures the harder question: of the tickets Fin resolves, how many were resolved correctly, against the company's actual policy, versus resolved in a way that merely closed the conversation?

Why Doesn't a Traditional QA Scorecard Work on an AI Agent?

A traditional QA scorecard was built to catch human behavior: did the agent use the right greeting, show empathy, follow escalation steps. Fin doesn't get tired, doesn't go off-script out of frustration, and doesn't forget to log a case number. Its failure modes are different in kind, not just degree. An AI agent can retrieve a policy document that's outdated, misapply a correct policy to a case it doesn't actually fit, or complete a multi-step workflow in a technically valid sequence that still produces the wrong outcome for the customer. None of these show up as a tone violation or a compliance checklist miss. They show up only if someone checks whether the reasoning behind the resolution was sound, which a transcript alone can't reveal. This is the core reason AutoQA built for human-agent scorecards tends to under-perform when pointed at an AI agent's conversations: it's scoring the wrong layer of the interaction.

What New Failure Modes Does Fin AI Agent Introduce That QA Needs to Catch?

Building on the scorecard mismatch above, the practical question for support operations teams is which specific failure modes need a dedicated scoring criterion. Four come up repeatedly in high-volume environments:

• Policy drift. Fin retrieves from a knowledge base that changes over time. If the underlying documents are stale or contradictory, Fin can confidently apply an outdated refund window or fee schedule.
• Workflow abandonment. Multi-step workflows can terminate early if a required input is missing, producing a resolution that looks complete in the transcript but skipped a verification step.
• False resolution. A conversation can be marked resolved because the customer stopped responding, not because the issue was actually fixed. Resolution rate alone cannot distinguish the two.
• Handoff judgment. Knowing when to escalate to a human is itself a decision that can be scored right or wrong, independent of whether the automated steps up to that point were correct.

Each of these requires a QA scorecard criterion written specifically for AI-agent reasoning, not borrowed from a human-agent QA scorecard.

How Should Support Operations Teams Score Fin AI Agent Conversations Differently?

A related but distinct question is what a scorecard built for this reality actually looks like in practice. The scoring unit has to shift from "what did the agent say" to "what did the agent retrieve, and was the retrieval the right basis for the response." That means QA metrics need entries like policy-grounding accuracy, workflow-completion integrity, and escalation-trigger correctness, scored with the same rigor as empathy or script adherence used to receive. RevelirQA approaches this by retrieving the customer's own SOPs and policy documents via RAG before scoring each conversation, so the evaluation is measured against what the agent (AI or human) should have done in that specific company's policy environment, not a generic industry benchmark. That same scorecard then applies identically whether the conversation was handled by Fin, by a different AI agent layer, or by a human rep, which is the only way to get a single, comparable quality picture across a support operation running both.

Why Does Manual QA Sampling Fall Short Once Fin Is in the Mix?

Manual sampling was the default QA method before AutoQA existed, and it typically covers around 5 percent of interactions industry-wide. That ceiling was always a constraint on human-agent QA; it becomes a sharper problem once Fin is resolving the majority of tickets. If Fin applies an outdated policy to one ticket, there's a reasonable chance it applies the same outdated policy to the next thousand tickets that match that pattern, because it's executing the same retrieval logic each time. A QA process sampling 5 percent of tickets might catch that error on ticket 40 or miss it entirely until a customer complaint surfaces weeks later. Scoring 100% of conversations isn't a nice-to-have against this failure mode, it's the only way to catch a systemic AI error before it compounds. This is table stakes now, not a differentiator. Most AutoQA vendors already claim full coverage; the real question is whether the AI doing that scoring reasons at the level a trained human QA analyst would.

How Do You Know If Your AutoQA Tool Is Actually Accurate, Not Just Scalable?

Coverage alone doesn't prove quality. Building on the previous point, a tool that scores 100% of conversations is only as useful as the judgment behind each score, and that judgment needs to be checked against a human standard, not assumed. RevelirQA is calibrated with each customer specifically for this reason: the customer's own human QA analysts review a sample of RevelirQA's evaluations, and calibration continues until more than 95 out of 100 of those evaluations are judged correct. That figure is earned per customer through this review process, not claimed as a generic out-of-the-box benchmark. Every RevelirQA score also carries a full reasoning trace, including which model was used, which policy documents were retrieved, and the reasoning that produced the score, which matters for Fin conversations specifically because a QA team needs to see that the AI checked the right policy before resolving, not just that it produced a plausible-sounding answer.

Frequently Asked Questions

Does Fin AI Agent replace the need for a QA layer entirely?
No. Fin resolves conversations; it doesn't evaluate whether those resolutions were correct against company policy. That's a separate function, which is where AutoQA sits.

Can RevelirQA score Fin AI Agent conversations directly inside Intercom?
RevelirQA integrates with helpdesks via API, which covers Intercom alongside platforms like Zendesk and Salesforce, scoring conversations regardless of whether they were handled by an AI agent or a human rep.

Is a 76 percent resolution rate the same as a 76 percent accuracy rate?
No, and conflating the two is a common mistake. Resolution rate measures whether a conversation closed without human intervention. It says nothing about whether the resolution was correct against policy, which requires a separate QA evaluation.

How does RevelirQA's accuracy figure get validated?
Through calibration specific to each customer: that customer's human QA analysts review a sample of RevelirQA's evaluations and confirm more than 95 out of 100 are correct. It is not a fixed, universal benchmark.

What's the difference between AutoQA and auto QA?
They're the same category, written two ways. Both describe automated quality assurance software that scores customer service conversations at scale instead of relying on manual sampling.

Does RevelirQA evaluate AI agents and human agents on the same scorecard?
Yes. RevelirQA applies one consistent QA scorecard across both, which is necessary for teams running Fin or another AI agent alongside human reps who want a single, comparable view of quality.

What security standards does Fin AI Agent meet?
Intercom states Fin meets SOC 2 Type II, HIPAA, GDPR, and CCPA standards, along with ISO 27001 and ISO 42001 certifications, which matters for regulated industries evaluating AI agents for compliance-sensitive support volume.

About Revelir AI

Revelir AI builds RevelirQA, an AI AutoQA platform that performs quality assurance to the standard of a human quality analyst across 100% of customer service conversations, calibrated per customer to over 95% accuracy as validated by that customer's own QA analysts. Founded in 2025 by Rasmus Chow (YC W22), Revelir is headquartered in Singapore and runs in production at enterprise clients including Xendit and Tiket.com, scoring thousands of conversations a week across English, Indonesian, Thai, and Tagalog. The platform scores against each customer's own SOPs retrieved via RAG, evaluates AI agents and human agents on one consistent scorecard, and gives every score a full audit trail covering the model, retrieved documents, and reasoning behind it. Built for global enterprises running high-volume support operations, Revelir is architected for any enterprise support operation running on global helpdesks like Intercom, Zendesk, or Salesforce.

If your support operation is running Fin AI Agent alongside human reps and your QA process still can't tell you whether the AI's resolutions hold up against your own policy, it's worth a conversation. Learn more at Revelir AI.