RevelirQA, Revelir AI's AutoQA engine, performs quality assurance to the standard of a human quality analyst. Revelir calibrates it with each customer until the customer's own QA analysts review a sample of its evaluations and judge more than 95 out of 100 correct. Fair scoring on any QA scorecard comes down to one question: does this criterion apply to this conversation at all? The fair answer has three parts:

• Treat "not applicable" as its own outcome and remove it from both the numerator and the denominator.
• Decide applicability with an explicit trigger rule based on the customer's situation, not on what the agent chose to do.
• Audit N/A calls as closely as Fails.

Scoring N/A as a Pass inflates agents with easy ticket mixes. Scoring it as a Fail punishes agents for issues the customer never raised. Most AutoQA tools already grade 100% of conversations, so coverage is table stakes. The harder problem is that at full coverage, a bad N/A rule distorts every score rather than a few.

Why does "N/A" distort agent QA scores in the first place?

The N/A problem is a denominator problem. Every QA scorecard assumes each criterion can be judged on each conversation, and most conversations break that assumption. A password reset has no refund amount to quote correctly. A one-line thank-you has no escalation path to follow. Something still has to go in that cell, and whatever goes there moves the agent's score.

Under manual sampling, the damage stayed small. Manual QA reviews only a small sample of customer service conversations. At that volume, an inconsistent N/A call touched few scores, and reviewers often corrected it by instinct. Auto QA, or automated quality assurance, reverses this. One N/A rule now runs thousands of times a week, so any bias in it becomes systematic.

Treatment	What it does	Effect on scores	When acceptable
N/A counted as Pass	Adds a free point	Inflates agents who handle simple contacts	Almost never
N/A counted as Fail	Adds a free zero	Deflates agents who handle simple contacts	Never, for agent scoring
N/A excluded	Removes the criterion from numerator and denominator	Reflects only what the agent was asked to do	The default, with reporting safeguards

Which N/A scoring method keeps agent scores fair?

Building on that comparison, exclusion is the right default. A worked example shows why the other two methods flip rankings rather than just nudging them. Take a 10-criterion QA scorecard and two agents (illustrative numbers):

• Agent A handles password resets. Four criteria apply, and A passes three of them.
• Agent B handles refund disputes. All ten criteria apply, and B passes eight.

Method	Agent A (password resets)	Agent B (refund disputes)	Who looks better
N/A as Pass	9/10 = 90%	8/10 = 80%	A
N/A as Fail	3/10 = 30%	8/10 = 80%	B, by a wide margin
N/A excluded	3/4 = 75%	8/10 = 80%	B, narrowly

Universities faced the same problem with pass/fail courses. They leave those courses out of the GPA instead of counting them as an A or an F. A GPA averages graded performance, and an ungraded course contains no performance to average. N/A criteria work the same way. The ticket gave the agent no chance to perform, so it should not move the average.

Exclusion has a cost of its own. Agent A's score rests on four data points, so one miss costs 25 points, while one miss costs Agent B only 10. The fix for that sits in the reporting layer, covered below.

How do you decide whether a QA criterion actually applies?

A related but harder question is who decides that a criterion is N/A. Exclusion is only fair if the applicability call is right. The most damaging error is a false N/A: for example, the escalation criterion is marked N/A because the agent never escalated, even though the customer's issue required escalation. That turns the most serious failure on the scorecard into an invisible one.

The fix is to write every criterion as a pair:

• Trigger: the customer situation that makes the criterion apply. Base it on what the customer said or needed, never on what the agent did. "Applies if the agent offered a refund" is a bad trigger. "Applies if the customer requests money back on an eligible order" is a good one.
• Standard: what the agent must do once the trigger fires. If the trigger fires and the standard is not met, the result is a Fail, not an N/A.
• Scope: the contact reasons the criterion can ever apply to. Don't run it on contact reasons where it is structurally impossible.

Psychometric scales spell out, item by item, which responses earn a point [psyteachr.github.io]. Quality assessment tools in clinical research are expected to be tailored to each review rather than applied generically [ncbi.nlm.nih.gov]. A QA scorecard deserves the same precision: an explicit rule per criterion, written for your operation.

How should N/A-heavy results be reported to agents and managers?

Once applicability is settled, the remaining risk sits in reporting, where small denominators make honest scores look volatile. Health measurement researchers have published comparisons of practical methods for handling "not applicable" item responses [pmc.ncbi.nlm.nih.gov]. It is a known measurement problem with deliberate solutions, not a quirk of customer service.

• Report pass rates per criterion, each with its applicable count shown next to it.
• Set a minimum applicable count with your QA leads before ranking agents on any single criterion.
• Compare agents within the same contact reason, not across mixes that differ structurally.
• Keep critical errors outside the average. A missed regulatory disclosure should fail the evaluation regardless of how many other criteria were excluded.

How do you check that your N/A rules are still working?

Even well-written triggers drift as products and policies change, so N/A decisions need the same audit as Passes and Fails.

• Treat N/A rate as a QA metric. Track it per criterion and per agent. A sudden rise points to a policy change, a trigger that no longer matches reality, or agents steering conversations away from obligations.
• Include N/A calls in calibration samples. Human analysts should validate them alongside graded criteria.

RevelirQA makes its scoring reviewable:

• Custom metrics are configurable per team and per contact reason.
• The engine retrieves the SOPs relevant to each ticket's contact reason before scoring.
• Every evaluation carries a reasoning trace that cites the SOP passage it checked against.
• QA leads can override any grade, agents can contest scores, and those disagreements feed back into calibration.

Frequently Asked Questions

Should N/A ever count as a Pass?

Almost never. Counting it as a Pass rewards agents for conversations that tested nothing, which inflates scores for anyone with a simple ticket mix.

What is a false N/A?

A false N/A is a criterion marked not applicable when it should have applied. It usually happens because the trigger was based on the agent's action rather than the customer's need.

Does excluding N/A make some scores unstable?

Yes. Fewer applicable criteria means each miss weighs more. Show applicable counts and set minimums before ranking agents on a criterion.

How is AutoQA different from manual QA for N/A handling?

Manual reviewers make N/A calls case by case. AutoQA applies one rule to every conversation, so the rule must be explicit and audited.

Should chatbot conversations use the same N/A rules?

Generally yes. A consistent QA scorecard keeps human and AI agent scores comparable, and you can add criteria specific to AI-handled conversations where needed.

How often should N/A rules be reviewed?

Review them whenever a policy or product changes, and whenever N/A rates shift sharply for a criterion or an agent.

About Revelir AI

Revelir AI, headquartered in Singapore, builds RevelirQA, AI quality assurance software that scores conversations to the standard of a human quality analyst. Revelir calibrates it with each customer to over 95% accuracy, as validated by that customer's QA analysts on a sample of evaluations. RevelirQA grades human and AI agents against each team's own SOPs using retrieval-augmented generation, with criteria configurable per team and contact reason and a full reasoning trace on every score. It runs in production at enterprise teams, including Xendit and Tiket.com, on thousands of tickets per week in English and Indonesian.

If N/A handling is quietly bending your agent scores, see how calibrated, auditable scoring works in practice at Revelir AI.

References

1. Quality Assessment of Diagnostic Accuracy Studies - NCBI - NIH (ncbi.nlm.nih.gov)
2. Practical methods for dealing with 'not applicable' item responses in the AMC Linear Disability Score project - PMC (pmc.ncbi.nlm.nih.gov)
3. 9 AQ data and recap | Level 1 Data Skills (psyteachr.github.io)