January 29, 2025

Why Shopify AI Agents Invent Policies — and How to Shut That Down

The dangerous answer is not 'I don't know'. It is a made-up 60-day window when your site says 14. Customers screenshot that.

Who this is for

Anyone who has seen an AI promise free reships you do not offer.

Models complete sentences. If the retrieved policy is thin, they complete it with whatever sounded like a store policy on the internet.

You cannot prompt your way out of missing data. You constrain outputs and you starve the model of permission to guess.

Require a source: the agent must be able to point at a URL or a rule id for any numeric claim.
If retrieval is empty, the only allowed move is escalate or ask one clarifying question.
Ban phrases like 'typically we' and 'most brands allow'.
Keep one live policy URL per topic. Archive the rest.
Sample 10 AI replies a week and tag 'invented number'. That tag is a content bug.

No dates, windows, or dollar amounts unless they appear in the connected policy or Shopify field.
No carrier promises beyond the last scan.
No medical or safety advice on supplements and kids products — escalate.

Dumping the entire website into the brain, including blog posts that joke about 'lifetime returns'.
Letting the agent browse random pages at reply time.
Punishing 'I need a human' in QA so the model starts bluffing.

Invented-claim rate in QA. Target under 1% of sampled AI tickets after month one.

Is this unique to Shopify? No, but Shopify stores have lots of numeric rules — windows, weights, duties — so hallucinations are expensive.

Should I use a smaller model? Constraints beat model size. A disciplined agent on your URLs beats a clever one on the open web.

What if two pages disagree? The agent should escalate, not pick a winner. Then you delete the stale page.

How does Relay limit this? Rules plus your pages. If it cannot prove the number, it does not print the number.

Connect one clean returns URL in Relay and see the invented windows disappear.