Guides

How to test what AI shopping agents find in your Shopify store

By Pratik Poddar, developer of Botfound · Updated 10 November 2026

AI shopping agents that work with Shopify stores can call your store's own agent endpoint at /api/ucp/mcp, using a search_catalog tool. Running the questions shoppers ask through that same search shows which products an agent gets back, and in what order. Run a fixed set of queries, note the products you expect, fix the data on the ones that are missing, and re-run.

Why test the agent search, not your storefront search

Your theme's search box and an AI agent don't see the same thing. Shopify stores advertise an agent endpoint (look for the agent lines in your store's /robots.txt and /llms.txt) that answers structured catalog searches. That's the search to test.

A simple testing routine

  1. Write 10 to 20 shopper questions the way people ask an assistant: "linen shirt for summer", "waterproof hiking boots women", "gift for a coffee lover under $50".
  2. For each, list the products you'd expect to come back.
  3. Run them against the agent search and record whether each expected product appears and at what position.
  4. For each miss, check the product's data: category, attributes, description, title, stock and status. A missing category or attribute is the usual cause.
  5. Fix, then re-run the same queries. Keep the list fixed so changes mean something.

Doing it in Botfound

Botfound's Agent Search Tester runs your queries through your store's own agent search, shows what comes back, and lets you pin the products you expect, so you can see which tests pass after a fix. Free includes 3 saved tests; Pro includes 25.

Asking ChatGPT directly

Asking an assistant yourself is a useful sanity check, but answers change between runs and between assistants. Ask more than once, and treat a single answer as a hint, not a measurement.

Questions

Does a password-protected store work with agent search?

No. While the storefront password is on, outside agents can't search your catalog, and Botfound shows a warning.

Do I need to set up the agent endpoint?

No. Shopify provides it for your store. You only need product data worth finding.

Want to see where your store stands first? The free Botfound check scores your 50 newest public products and your discovery files in about 10 seconds, with no install and no sign-up. Botfound is on the Shopify App Store: free for your first 100 products; Pro is US$19/month with a 7-day free trial.

More guides

Botfound by Tinkerer (Pratik Poddar) · 76, Vrindavan Vihar, Ajmer Road, Jaipur, Rajasthan 302019, India · pratik.phodu@gmail.com
Home · Free check · Guides · Support · Privacy policy · Terms · Shopify App Store