How to Measure Your Own Shopify AI Support Results After 30 Days: Resolution Rate, CSAT and Cost per Ticket

This guide is about measuring your own store. It gives a definition and a formula for each metric, and a way to read the results. Your own tickets are the only data that matters for your decision.

Why 30 days is the right window

Thirty days covers a full order cycle for most stores: purchase, shipping, delivery, and the return or refund window that follows. You see the "where is my order" tickets, the exchange requests and the odd edge cases. Week one is mostly setup. The agent meets questions your knowledge base does not answer yet. Weeks two to four show the real pattern. Compare week one with week four, not with the 30-day average.

Before you start, write down your baseline from the previous 30 days:

• Tickets received per month
• Average first response time
• Hours your team spent on support
• CSAT, if you collect it
• Total support cost for the month

Without this, you cannot say whether anything improved.

The five metrics to track

• Resolution rate — how much does the AI finish alone? Tickets resolved by AI without a human ÷ tickets the AI received.
• Reopen rate — were those resolutions real? AI-resolved tickets reopened or followed up on the same issue ÷ AI-resolved tickets.
• CSAT on AI tickets — were customers satisfied? Positive ratings ÷ total ratings on AI-handled tickets.
• Cost per resolved ticket — what does each resolution cost? Total support spend for the period ÷ tickets resolved.
• Time saved — how many hours came back to you? Resolved tickets × your average handling time per ticket.

Resolution rate

Resolution rate is the share of tickets the AI closes without a human stepping in. It is the headline number, and the easiest one to inflate. Define "resolved" strictly. A ticket counts only if the customer got an answer or an action that solved the problem. An unanswered ticket that went quiet is not resolved. A ticket handed to a human is not resolved by the AI, even if the AI did part of the work.

Also split it by ticket type. A single global rate hides the useful detail. A store might see most order status questions resolved and few complaints about damaged goods. That tells you where to improve the knowledge base or which permissions to grant.

Reopen rate

Reopen rate catches false resolutions. If a customer writes back about the same issue within a few days, the first answer did not solve it. A high resolution rate with a high reopen rate means the agent is closing tickets, not solving them. Read a sample of reopened tickets. The cause is usually one of three things: a missing rule (for example, your return window), an unclear policy page, or an action the agent was not allowed to take.

CSAT on AI-handled tickets

CSAT is customer satisfaction, usually a one-question rating after the ticket closes. Measure it only on tickets the AI handled, then compare with the same score on tickets your team handled before. Two cautions. Response rates on CSAT surveys are often low, so read the comments, not just the score. And compare like with like: a refund request and a shipping question do not produce the same mood. Compare by ticket type when you have enough volume.

Check that your survey tool actually collects this rating on AI tickets before you rely on it. If it does not, use reopen rate and a manual read of a sample of conversations as your satisfaction check.

Cost per resolved ticket

Cost per resolved ticket is your total support spend for the period divided by the tickets resolved. Include the software subscription, any per-resolution fees, and the human time still spent on escalated tickets. Two mistakes to avoid. First, dividing only the software bill by AI resolutions. That ignores the human hours still going into the tickets the AI could not close. Second, comparing monthly spend without comparing volume. If tickets doubled in a busy month, a higher bill is not a worse result.

Time saved

Time saved turns resolutions into hours. Multiply AI-resolved tickets by the average minutes your team used to spend per ticket, taken from your baseline. Then convert hours into payroll cost using what you actually pay. This is the number that answers the real question for most merchants: did I avoid a hire? Relay's statistics show resolution rate, time saved and payroll saved. Reopen rate and cost per resolved ticket you calculate yourself from your own data, using the formulas above.

How to set your own reference points

Every store has a different catalog, return policy, shipping time and ticket mix. A store selling made-to-order furniture has very different tickets from one selling t-shirts. So your most useful reference is your own history.

• Your baseline. The 30 days before the AI went live.
• Your week-four result. After the agent has learned your rules.
• Your target. The share of tickets you want handled without you. Treat this as an ambition to work toward, not a promise from any tool.

Then review monthly. Each month, look at the tickets the AI escalated, find the most common reason, and fix it. That loop is what moves the numbers, not the first setup.

A simple scorecard for day 30

Answer these questions with your data:

• Is resolution rate rising from week one to week four?
• Is reopen rate stable or falling?
• Is CSAT on AI tickets close to, or better than, your human baseline?
• Is cost per resolved ticket lower than your baseline?
• Did you recover enough hours to delay or avoid a hire?

If you answer yes to most of them, keep going and expand the permissions. If resolution is low but reopen is also low, the agent is careful but under-equipped: add knowledge base articles and rules. If resolution is high but reopen is high, tighten the rules before you give the agent more power.

What to check in the tickets themselves

Numbers tell you where to look. Reading tickets tells you why. Each week, read a small sample: five tickets the AI resolved, five tickets it escalated, and five tickets that were reopened. Look for wrong policy answers, actions that should have been taken but were not, and tone that does not fit your brand. Fix the rule or the article, then watch the next week's numbers.

FAQ

How do I know if my resolution rate is good? Compare it with your own numbers. Your ticket mix decides what is realistic: stores with many order status and return questions can automate more than stores with complex complaints. Use your week-four rate as the reference and aim to raise it each month.

How do I calculate cost per resolved ticket? Divide your total support cost for the period by the number of tickets resolved. Include software fees and the human time still spent on escalated tickets. Compare the result with the same calculation for your previous 30 days.

What is the difference between resolution rate and reopen rate? Resolution rate shows how many tickets the AI closed. Reopen rate shows how many of those came back for the same issue. Together they tell you whether the closures were real.

How long should I test an AI support agent before deciding? Thirty days is a practical minimum because it covers shipping and return cycles. Compare week four with week one to see the trend. Relay offers a 14-day trial, so you can start measuring with your own tickets before committing.