How to Run Human-in-the-Loop AI Support on Shopify Without a New Bottleneck

If a human must click Send on every WISMO, you bought a typing assistant, not an agent. Hold the risky 15%. Release the boring 85%.

Who this is for

Risk-averse brands who want AI drafts in week one and autonomy in week four.

The actual problem

Full review feels safe for two days, then the queue of 'AI waiting for approve' is longer than the old inbox.

Human-in-the-loop is a temporary training wheel and a permanent control for a few ticket types — not a lifestyle.

The playbook

• Week 1: auto-send tracking lookups. Hold refunds and apologies.
• Week 2: auto-send in-policy refunds under the cap. Hold exceptions.
• Week 3: auto-send exchanges in stock. Hold out-of-stock swaps.
• Keep a permanent hold on legal, wholesale, and medical-adjacent products.
• SLA the review queue like you SLA email. A held draft older than 30 minutes is a failed design.

Rules the agent should follow

• The reviewer sees the rule that fired, not just the draft.
• Edits should become new rules, not one-off rewrites that teach nothing.
• Do not let reviewers rewrite tone for sport. Pick a voice and stop.

Mistakes that keep the queue full

• Holding everything 'until we trust it' with no date.
• Three reviewers with three tastes.
• No auto-send path at all.

What to measure

Median time from AI draft to send, plus the share of drafts sent unchanged. Unchanged-and-held is waste.

FAQ

Who should review? One lead, not the whole team. Consistency matters more than democracy.

Can customers tell? They can tell when the reply is late. They cannot tell who clicked send if the facts are right.

Is this required for compliance? For most DTC, no. For claims, medical, and regulated goods, yes — hold those forever.

Does Relay support holds? You choose which actions go live and which wait. The trial is where you tighten that list.

Start Relay with holds on refunds only. Release WISMO on day one.