About the Shelfproof crawler

If you’ve found this page in your server logs, it’s because a retailer is using Shelfproof to compare their prices against yours. Here’s what we do, and how to stop us if you want to.

How to identify us

Every request we make carries this User-Agent:

Mozilla/5.0 (compatible; Shelfproof/1.0; +https://shelfproof.app/crawler)

We don’t disguise this, rotate user agents or use residential proxies to make our traffic look like a person’s. If you block us, we stay blocked.

What we request

• Public product pages, at URLs a retailer has explicitly given us.
• Your robots.txt, so we can honour it.
• Occasionally your public site search, when a retailer asks us to find the page matching one of their products.

We don’t log in or try to reach anything behind a login, we don’t add things to carts, and we don’t place orders. We read the same page any shopper would.

How we behave

• We honour robots.txt, including Crawl-delay.
• We leave at least 1.2 seconds between requests to your site, and longer if your robots.txt asks for it. Requests are queued per site, so the number of retailers using us does not increase the rate you see.
• Each product page a retailer tracks is normally requested once a week.
• If you return 403, 429 or 503 we back off for several hours and honour Retry-After. We do not retry harder.

How to block us

Add this to your robots.txt and we will stop, usually within a day:

User-agent: shelfproof
Disallow: /

You can also block the User-Agent at your edge or firewall. Either is fine, and neither will be worked around.

If you’d rather we just stopped, email crawler@shelfproof.app with your domain and we’ll add it to a permanent exclusion list, no questions asked. That covers every retailer using Shelfproof, not only the one who added you.

Prices shown to shoppers

A retailer may show a price we read from your site on their own product pages, with the date we read it. If we can’t read a price cleanly we don’t show one, and we won’t show anything older than 30 days. Where a page lists several variants at different prices we leave it out rather than picking one.

If you believe a price of yours is being displayed inaccurately, email the address above and we will investigate and correct it.