How to QA Shopify AI Tickets in 45 Minutes a Week

If you do not read the tickets, you are flying on resolution rate alone. Resolution rate lies. QA is where you catch the polite wrong refund.

Who this is for

Support leads who just turned an agent live and need a ritual, not a 20-page QA program.

The actual problem

Teams either sample nothing or try to score every ticket. Both fail. You need a small, nasty sample and a place to put the fix.

Every miss should become a rule, an article, or a permission change. If the miss only becomes a Slack reaction, you will see it again.

The playbook

• Every Monday, pull 5 resolved, 5 escalated, 5 reopened AI tickets.
• Score each: fact correct, policy correct, action allowed, tone, should-escalate.
• Write one change: article, cap, or escalation trigger.
• Ignore stylistic nits unless they are off-brand enough to notice in a screenshot.
• Once a month, compare week 1 vs week 4 on the same ticket types.

Rules the agent should follow

• The person who writes the policy should not be the only QA. Founders miss their own ambiguity.
• Do not punish the agent for a bad policy page. Fix the page.
• Reopened tickets are the richest sample. Start there if you are short on time.

Mistakes that keep the queue full

• Scoring 'empathy' on a 1–5 scale nobody can reproduce.
• QA only on chat, never on email.
• No owner for the weekly change.

What to measure

QA fail rate and the count of policy/rule changes shipped from QA. If the second number is zero, QA is theater.

FAQ

Who should own QA? The support lead. Not the intern. Not the agency that never sees Shopify admin.

Is 15 tickets enough? Yes if you pick the painful ones. Random 15 from 2,000 will miss the damage.

Should we tell customers it was AI? Be honest if asked. Do not watermark every WISMO. The work is the facts.

Can Relay help? You can filter AI-handled and reopened tickets and run this review on the real queue.

Run one Monday QA on Relay tickets after the first week of the trial.