RevelirQA, Revelir AI's AutoQA engine, performs QA to the standard of a human quality analyst. Revelir calibrates it with each customer until the customer's own QA analysts, reviewing a sample of its evaluations, judge more than 95 out of 100 correct. It also scores every conversation, which most AutoQA tools already claim. You prove coaching worked by showing three things: the QA scorecard criterion you coached moved for the agent you coached, the change held over time, and something else did not cause it. When the line moves on a scorer your own analysts have validated, the scorer is far less likely to be the cause, so you can focus on ruling out the other explanations. Revelir AI's view comes from running automated quality assurance in production at Xendit and Tiket.com, on thousands of tickets per week.

TL;DR

• RevelirQA performs auto QA to the standard of a human quality analyst. It is calibrated with each customer to over 95% accuracy, as judged by the customer's own QA analysts on a sample of evaluations. It scores every conversation, which most AutoQA tools already claim.
• Coaching proof means a change in one criterion, for one agent, measured against a baseline and sustained for weeks.
• A handful of manual reviews per agent is too noisy to show this. At five reviews a month, a single ticket swings a pass rate by 20 points.
• Track the coached criterion, repeat misses and critical errors. Treat CSAT as a lagging signal.
• Before you claim a win, rule out changes in contact-reason mix, SOP updates and scorecard edits.

About the Author: Revelir AI builds RevelirQA, an AI quality assurance platform for customer service. Founded in Singapore in 2025 by Rasmus Chow (YC W22), Revelir grades production volume at Xendit and Tiket.com against each company's own QA scorecard and SOPs.

What does "proving coaching worked" actually mean?

Proving coaching worked means showing a measurable, sustained change in the exact behaviour you coached, large enough to stand out from normal week-to-week variation. Most teams stop at "the agent's overall score went up." That number blends a dozen criteria, so it cannot tell you whether the coaching session caused anything.

• Specific: tie the claim to one scorecard criterion, such as "quoted the correct refund eligibility."
• Per-agent: team averages hide the agents who did not change.
• Baselined: compare against the same agent's performance before coaching.
• Sustained: a one-week bump is not a learned behaviour.

Proof is the step that closes the loop back to the scorecard.

Why can't a small manual sample show per-agent movement?

Building on that definition, the first obstacle is arithmetic, not effort. Human analysts make good judgments, but they can only review so many tickets. Revelir's own illustration shows the scale of the gap. Thirty agents with 20 reviews each produce 600 reviews out of 50,000 monthly conversations, which is 1.2% coverage.

Think of judging whether a basketball player's free throws improved after a clinic by watching five shots. One extra make moves the rate by 20 points, so "improvement" and luck look identical. Per-agent QA samples work the same way. When an agent has five reviewed tickets on a criterion, one ticket decides whether the coaching looks like it worked.

• Variance: small samples produce large swings with no behaviour change behind them.
• Selection: reviewers pull different ticket types each month, so the mix changes as well as the agent.

Why does scorer accuracy matter more than coverage for coaching proof?

Scoring every conversation solves the sample-size problem, but it raises a sharper one: a trend line is only as honest as the scorer behind it. Picture a bathroom scale that reads two kilos differently every morning. If you are trying to lose one kilo, the scale cannot tell you whether you did. The measurement error has to be smaller than the change you are trying to detect.

An AutoQA tool that scores 100% of tickets but often disagrees with your analysts creates the same blind spot. If its misreads cluster on certain contact reasons, an agent's score can move simply because their ticket mix changed. This is why RevelirQA's accuracy is defined per customer. Revelir calibrates the scoring engine until your QA analysts, reviewing a sample of evaluations, judge more than 95 out of 100 correct, and your team signs off before it goes live.

• Every score carries a trace with the model used, the prompt, the documents retrieved and the reasoning. A QA lead can inspect a suspicious jump instead of trusting it blindly.
• QA leads can override grades and agents can contest them. Disagreements feed back into calibration.

How do you set up per-agent scorecard tracking, step by step?

With a trustworthy scorer in place, the remaining work is experimental discipline. The steps below turn scorecard data into a coaching test.

Step	What to do	Why it matters
1. Make criteria observable	Write criteria specific enough that two analysts would agree on them [evaluagent.com]. Balance business requirements with customer needs [evaluagent.com].	Vague criteria produce scores that move for no reason.
2. Set a baseline	Record the agent's pass rate on the target criterion over several weeks before coaching.	Without a "before," there is no "after."
3. Coach one criterion	Log the date, the criterion and the example ticket used.	Coaching on five things at once makes attribution impossible.
4. Measure an equal window	Compare a post-coaching window of the same length as the baseline.	Unequal windows distort pass rates.
5. Use a comparison	Check the agent's uncoached criteria and peers on the same criterion.	If everyone improved, something other than coaching did it.
6. Recheck later	Revisit the criterion a month on.	This separates learned habits from short-term compliance.

Which QA metrics show coaching impact?

The steps above need the right numbers to work. The useful pattern is to separate leading signals from lagging ones.

Metric	What it tells you	Signal type
Coached-criterion pass rate	Whether the specific behaviour changed	Leading
Repeat-miss rate	Whether the same gap keeps recurring	Leading
Critical error count	Whether high-risk misses fell	Leading
Score by contact reason	Whether improvement holds across ticket types	Diagnostic
CSAT, resolution, sentiment arc	Whether customers felt the difference	Lagging

Outcome metrics remain the end goal. Use leading QA metrics to confirm the behaviour changed first, then look for the outcome.

How do you avoid crediting coaching for movement it didn't cause?

Even with clean metrics, a moving score is only evidence once you rule out the alternatives. Common confounders include:

• Check whether the SOP version changed during your window. RevelirQA shows which SOP version each grade was checked against.
• Contact-reason mix: an agent moved to easier queues will look improved.
• Revelir's AI Coach gives each agent a daily focus drawn from their own tickets and flags repeat gaps as persistent. It never triggers warnings or discipline. The QA lead decides what happens next.

Frequently Asked Questions

How long before coaching shows up in QA scores?
It depends on ticket volume per criterion. With every conversation scored, most agents generate enough tickets within a few weeks to show a readable trend.

What does "over 95% accuracy" mean for RevelirQA?
Revelir calibrates the AI with each customer until that customer's QA analysts, reviewing a sample of evaluations, judge more than 95 out of 100 correct. It is reached per customer, not out of the box.

Is AutoQA coverage enough to prove coaching?
No. Coverage reduces sampling noise, but without validated accuracy, score movement may reflect the scorer rather than the agent.

Can I track AI chatbots the same way?
Yes. RevelirQA applies the same QA scorecard to human and AI agents, provided the chatbot produces a conversation log via API.

Should overall QA score be the main coaching metric?
No. Use criterion-level pass rates. Overall scores blend too many behaviours to support attribution.

Do agents see their own trends?
In Revelir, agents can see what a score was based on, contest it, and ask the AI Coach why they were marked down.

About Revelir AI

Revelir AI builds RevelirQA, an AI quality assurance platform for customer service that grades conversations at the standard of a human quality analyst. It is calibrated with each customer to over 95% accuracy: when that customer's QA analysts review a sample of its evaluations, more than 95 out of 100 are judged correct. It retrieves each customer's own SOPs via RAG before scoring, attaches a full reasoning trace to every evaluation, and shows per-agent and per-team scores over time so leaders can see whether coaching worked. RevelirQA runs in production at Xendit and Tiket.com, integrates with any helpdesk via API, including Zendesk and Salesforce, and serves enterprise teams globally.

If you want coaching results you can defend in front of your leadership team, see how Revelir AI calibrates AutoQA to your own analysts at https://www.revelir.ai/.