RevelirQA grades customer service conversations at the standard of a human quality analyst. It is calibrated with each customer until that customer's own QA analysts validate more than 95 out of 100 sampled evaluations as correct. RevelirQA also scores 100% of conversations, which most auto QA tools already claim as table stakes. What makes the data trustworthy is accurate judgment within each contact reason. Before scoring a ticket, it retrieves the SOP documents for that ticket's contact reason, so each check is measured against the policy that actually applies. With this foundation, you can find which SOPs are skipped by contact reason without reviewing every ticket.
TL;DR
- RevelirQA grades customer service conversations at the standard of a human quality analyst, calibrated per customer until that customer's QA analysts validate over 95% of a sample of its evaluations as correct. Full coverage is table stakes on top of that.
- A skip only counts where the step was required: skip rate = misses divided by tickets where that step applied to the contact reason.
- Manual sampling suits deep coaching reviews. It is too thin to rank skips across many contact reasons.
- Where a skip appears tells you why it happens. Skips by one team member across many reasons point to coaching. Skips by most team members within one reason point to the SOP, the tooling or the training.
- The same scorecard should grade chatbots and human team members, so you can compare them on the same steps.
About the Author: This guide is written by the team at Revelir AI, the Singapore-based company behind RevelirQA. RevelirQA is an automated quality assurance engine that grades customer service conversations against each customer's own SOPs, and it runs in production at enterprises including Xendit and Tiket.com.
What counts as an "SOP skip" in customer service QA?
Before you can count skips by contact reason, you need a definition precise enough for a machine and a human analyst to agree on. An SOP skip is a required step for a given contact reason that the team member omitted or got materially wrong. Common forms include:
- An omitted step, such as no identity check before an account change
- A wrong refund amount quoted
- Incorrect eligibility criteria given
- A missed required regulatory disclosure
- The wrong escalation path
The words "for a given contact reason" do most of the work here. An identity check may be mandatory for account changes and irrelevant to a delivery-status question. If your QA scorecard marks the check as failed on every ticket where it did not happen, your skip data fills up with tickets where the step never applied.
Why can't manual sampling show skips by contact reason?
That conditional logic is exactly where sampled review breaks down. Revelir puts manual QA coverage in customer service at only 1% to 5% of tickets. Revelir illustrates the effect with an example: 30 team members with 20 reviews each is 600 reviews out of 50,000 conversations, or 1.2% coverage. At that rate, an error occurring once in 200 conversations produces 250 failures a month, and only about 3 would be caught.
Now split that sample across a dozen contact reasons. Low-volume reasons, which are often the high-risk ones like disputes or account closures, may get a handful of reviews or none. Sampling still has real value. A human reading one ticket closely catches nuance, and that analyst is the reference point for calibrating any automated system. Sampling is simply the wrong tool for measuring how often something happens across many segments.
How do you set up SOP skip tracking by contact reason?
Building on that gap, the fix is to grade everything with logic that knows which steps apply to which reason. A practical sequence:
- Settle your contact reason list. Each reason should map to a distinct set of SOP steps. If two reasons share identical steps, merge them for QA purposes.
- Break each SOP into checkable steps. Write each step as a metric, for example "Verified account ownership before discussing balance: Pass / Fail / Not applicable." The "Not applicable" option keeps the denominator honest.
- Flag critical steps. Track disclosures and escalation paths separately from tone items, because one miss can carry compliance exposure.
- Tag contact reason from the conversation itself. Team member-selected dropdowns can drift from what the customer actually asked. RevelirQA's ticket enrichment assigns a contact reason, sentiment and recurring issue type to every ticket.
- Grade only the applicable steps. RevelirQA retrieves the SOPs for each ticket's contact reason before scoring. Custom metrics and QA scorecard weights can be configured per contact reason.
- Calibrate before trusting the numbers. QA analysts review a sample, dispute wrong grades, and the grading is adjusted until it meets your target.
- Compute skip rate correctly: failed instances of a step divided by tickets where that step applied.
How do you read the contact reason by SOP step grid?
Once the grid exists, the skill shifts from finding skips to diagnosing them. Think of how a mechanic reads tyre wear. Uneven wear on one tyre points to that wheel. The same wear pattern on all four tyres points to alignment, which is a problem with the whole car. Skips work the same way. Where they cluster tells you whether the cause sits with one person or with the system.
| Pattern in the grid | What it usually means | Where the fix lives |
|---|---|---|
| One team member misses several steps across many reasons | An individual habit or knowledge gap | Coaching for that team member |
| Most team members miss the same step in one reason | The SOP is unclear, the step is awkward in the tools, or training skipped it | An SOP rewrite, a tooling change or team training |
| A step's skip rate jumps after a policy change | The change reached the document but not the floor | Rollout communication |
| One team skips more than peers on the same reason | Local guidance has diverged from the SOP | Team lead calibration |
| The chatbot misses a step your team follows | A knowledge base gap, prompt issue or missing escalation rule | Bot configuration |
Revelir also connects to Claude via MCP. A Head of CX can ask questions like "Which contact reason is growing fastest?" and get an answer backed by ticket data.
How do you know the automated grades are right?
None of this diagnosis matters if the underlying grades are wrong, so trust has to be earned and checkable. With RevelirQA, it works like this:
- Per-customer calibration. Over 95% accuracy means the customer's own QA analysts validated more than 95 out of 100 sampled evaluations as correct. It is reached through calibration with each customer, not delivered out of the box.
- Evidence on every failure. Each failed metric shows the quoted team member text and the name and version of the SOP it was checked against.
- A full trace. Every score records the model used, the exact prompt, the documents retrieved and the reasoning behind each criterion.
- Human control. QA leads can override grades, and team members can contest them. Disagreements feed back into calibration. Nothing happens automatically: RevelirQA triggers no warnings, rankings or discipline.
Retrieval-based scoring evaluates the meaning of what the team member said against the substance of the policy. This is why it can catch a wrong refund amount that keyword matching would pass.
Does this work for AI chatbots as well as human team members?
A related question is whether bots skip SOPs too. They do, and often consistently. In customer service, chatbot misses show up as the wrong policy applied at scale, missed escalations and hallucinated answers. RevelirQA applies the same rubric to any chatbot platform that produces a conversation log via API. This shows which contact reasons the AI handles better or worse than your team, and whether the fix is a knowledge base update, a prompt change or a new escalation rule.
Frequently Asked Questions
How many contact reasons should we track?
Track enough reasons to separate the ones with different SOPs. Fewer, cleaner reasons give you more reliable skip rates per cell.
Can we rely on helpdesk tags for contact reason?
Partly. Deriving the reason from the conversation itself reduces drift from rushed or mistaken tagging.
How is skip rate different from a QA score?
A QA score blends many criteria into one number. Skip rate isolates one step within one contact reason, which tells you exactly what to fix.
Does AutoQA replace QA analysts?
No. Analysts calibrate the system, override grades and decide what action to take.
Which teams is this suited to?
Revelir states its fit as teams with 30 or more customer service professionals and 5,000 or more conversations a month.
How do we test it?
Revelir's paid pilot runs for six weeks on an agreed subset of 10,000 conversations, with accuracy targets set with your QA leads in week one.
About Revelir AI
Revelir AI builds RevelirQA, an AutoQA engine that grades customer service conversations at the standard of a human quality analyst. It is calibrated with each customer to over 95% accuracy, as validated by that customer's QA analysts. RevelirQA retrieves your SOPs for each contact reason before scoring, enriches every ticket with contact reason and sentiment, and records a full audit trail on every evaluation. It runs in production at Xendit and Tiket.com on thousands of tickets per week and integrates with any helpdesk via API.
If you want to see which SOPs your team skips, broken down by contact reason, talk to the team at Revelir AI.
References
- Google (google.com)