{ "name": "Who blocks AI shopping agents", "description": "A weekly reading of public robots.txt files for major US retailers and marketplaces, plus policy limits already recorded on the Agent Shopping Tracker. robots.txt is a request, not enforcement.", "url": "https://agent-shopping-tracker.vercel.app/blocklist", "license": "CC BY 4.0", "licenseUrl": "https://creativecommons.org/licenses/by/4.0/", "attribution": "Free to use with a link back to https://agent-shopping-tracker.vercel.app/blocklist", "publisher": "Agent Shopping Tracker, a Digital Edge Marketing LLC project", "checkedOn": "2026-10-28", "methodology": "Each row is a reading of one public robots.txt, fetched from that site's host at /robots.txt. An agent matches when its product token appears, compared without regard to case, as RFC 9309 describes. GPTBot/1.0 still matches GPTBot. A different token, such as GPTBot-User, does not. User-agent: * is recorded on its own. If an agent is not named, the * rules are not copied onto that agent. Blocked fully means the agent's group disallows the whole site (Disallow: /, Disallow: /*, Disallow: /*$, or Disallow: *) and contains no Allow rule. Partly means the agent is named and the rules disallow only some paths, or they disallow the whole site and also allow something. The rules are listed so a reader can see the mix. A full disallow paired with an allow is not called a full block. Allowed means the agent is named and no path is disallowed. Crawl-delay is not a block and is not listed. Not mentioned means the file does not name that agent. That is not permission. The * group may still apply, and a site can block an agent without naming it. robots.txt is a request to well-behaved crawlers. It is not access control. This check does not browse stores, place carts, or test whether a block is enforced. It does not read robots meta tags or login walls. Google-Extended is not Google Search. Applebot-Extended is not Applebot. Amazonbot is Amazon's crawler, not a shopper. ChatGPT-User, Claude-User, and Perplexity-User are the tokens those companies publish for a fetch a person asked the product to make. An agent that sends an ordinary browser token will not appear here. Meta-ExternalAgent is matched case-insensitively, so a line written meta-externalagent counts. If the first request was refused, timed out, or came back as HTML, the check retried once with a common browser user agent. Some storefronts hide robots.txt from clients they do not recognize. The row says when that retry was used. A successful retry is still the public file. When a group has many rules, the stored list is a sample. The sample keeps any whole-site rule, one allow, and one specific disallow. ruleCount is the full number of rules. The label uses every rule.", "findings": [ "On October 28, 2026 we requested the public robots.txt file for 48 US retail or marketplace sites. We could read 44 files. 4 sites did not return a file we could read: Victoria's Secret, GameStop, Petco, and Saks Fifth Avenue. Those rows stay unread and are left out of the counts below.", "15 of the 44 readable files name at least one of the 14 agents we track. 6 readable files fully disallow at least one of them: Amazon (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent); Instacart (PerplexityBot, Perplexity-User, Meta-ExternalAgent); Chewy (Amazonbot); Temu (Bytespider); Lululemon (Meta-ExternalAgent); IKEA (Amazonbot). 29 readable files name none of them, including Walmart, Target, Etsy, The Home Depot, Lowe's, Costco, Sam's Club, and 22 others. Not named is not permission. A file can still set rules for User-agent: *, and a site can block an agent without naming it.", "GPTBot is fully disallowed in 1 readable file, ClaudeBot in 1, PerplexityBot in 2, Google-Extended in 1, CCBot in 1, and Bytespider in 2. ChatGPT-User is fully disallowed in 1, Claude-User in 1, and Perplexity-User in 2.", "1 readable file fully disallows all three user-triggered tokens (ChatGPT-User, Claude-User, and Perplexity-User): Amazon.", "1 readable file fully disallows User-agent: *: Instacart. That asks every unnamed crawler to stay off the whole site. Agents we marked not mentioned on that site can still fall under that * rule." ], "agents": [ { "token": "GPTBot", "vendor": "OpenAI", "kind": "Training crawler" }, { "token": "ChatGPT-User", "vendor": "OpenAI", "kind": "User-triggered fetch" }, { "token": "OAI-SearchBot", "vendor": "OpenAI", "kind": "Search crawler" }, { "token": "ClaudeBot", "vendor": "Anthropic", "kind": "Training crawler" }, { "token": "Claude-User", "vendor": "Anthropic", "kind": "User-triggered fetch" }, { "token": "Claude-SearchBot", "vendor": "Anthropic", "kind": "Search crawler" }, { "token": "PerplexityBot", "vendor": "Perplexity", "kind": "Crawler" }, { "token": "Perplexity-User", "vendor": "Perplexity", "kind": "User-triggered fetch" }, { "token": "Google-Extended", "vendor": "Google", "kind": "AI training opt-out, not Google Search" }, { "token": "Amazonbot", "vendor": "Amazon", "kind": "Crawler, not a shopper" }, { "token": "Applebot-Extended", "vendor": "Apple", "kind": "AI training opt-out, not Applebot search" }, { "token": "CCBot", "vendor": "Common Crawl", "kind": "Training corpus crawler" }, { "token": "Bytespider", "vendor": "ByteDance", "kind": "Crawler" }, { "token": "Meta-ExternalAgent", "vendor": "Meta", "kind": "External agent crawler" } ], "sites": [ { "id": "amazon", "name": "Amazon", "homepage": "https://www.amazon.com/", "robotsUrl": "https://www.amazon.com/robots.txt", "playerId": "amazon", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.amazon.com/robots.txt", "bytes": 7947, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /wishlist/universal*", "Disallow: /exec/obidos/account-access-login", "Disallow: /exec/obidos/change-style", "Disallow: /exec/obidos/flex-sign-in", "Disallow: /exec/obidos/handle-buy-box", "Disallow: /exec/obidos/tg/cm/member/", "Disallow: /gp/aw/help/id=sss", "Disallow: /gp/cart" ], "ruleCount": 137 }, "agents": { "GPTBot": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "GPTBot" ], "similarLines": [] }, "ChatGPT-User": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "ChatGPT-User" ], "similarLines": [] }, "OAI-SearchBot": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "OAI-SearchBot" ], "similarLines": [] }, "ClaudeBot": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 2, "userAgentLines": [ "ClaudeBot" ], "similarLines": [] }, "Claude-User": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "Claude-User" ], "similarLines": [] }, "Claude-SearchBot": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "Claude-SearchBot" ], "similarLines": [] }, "PerplexityBot": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "PerplexityBot" ], "similarLines": [] }, "Perplexity-User": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "Perplexity-User" ], "similarLines": [] }, "Google-Extended": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "Google-Extended" ], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "CCBot" ], "similarLines": [] }, "Bytespider": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "Bytespider" ], "similarLines": [] }, "Meta-ExternalAgent": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "meta-externalagent" ], "similarLines": [] } } }, { "id": "walmart", "name": "Walmart", "homepage": "https://www.walmart.com/", "robotsUrl": "https://www.walmart.com/robots.txt", "playerId": "walmart", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.walmart.com/robots.txt", "bytes": 3479, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /reviews/product/", "Disallow: /0/", "Disallow: /55875582/walmart-us/catalog/", "Disallow: /account/", "Disallow: /api/", "Disallow: /collection/api/logger", "Disallow: /cp/-201", "Disallow: /cp/-302" ], "ruleCount": 60 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "target", "name": "Target", "homepage": "https://www.target.com/", "robotsUrl": "https://www.target.com/robots.txt", "playerId": "target", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.target.com/robots.txt", "bytes": 3226, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Disallow: /*/Ntk", "Disallow: /*/Ntt", "Disallow: /*/Ntx", "Disallow: /*%7Cd_", "Disallow: /*/schoollist/", "Disallow: /*BTWN", "Disallow: /[path]/", "Disallow: /7078046/" ], "ruleCount": 114 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "ebay", "name": "eBay", "homepage": "https://www.ebay.com/", "robotsUrl": "https://www.ebay.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.ebay.com/robots.txt", "bytes": 24164, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /ads.txt", "Disallow: /*_kw", "Disallow: /*?maspect", "Disallow: /*modules=SEARCH_REFINEMENTS_MODEL_V2", "Disallow: /*redirect=mobile", "Disallow: /*redirect%3Dmobile", "Disallow: /*rt%3Dnc", "Disallow: /*rt=nc" ], "ruleCount": 234 }, "agents": { "GPTBot": { "status": "partly", "rules": [ "Disallow: /", "Allow: /adchoice", "Allow: /authenticity-guarantee", "Allow: /motors/blog", "Allow: /motors/diy", "Allow: /help", "Allow: /sellercenter", "Allow: /ebaylive/seller", "Allow: /sl/sell", "Allow: /sh/landing", "Allow: /giftcards", "Allow: /shp/calc/rates" ], "ruleCount": 12, "userAgentLines": [ "GPTBot" ], "similarLines": [] }, "ChatGPT-User": { "status": "partly", "rules": [ "Allow: /ads.txt", "Disallow: /*_kw", "Disallow: /*?maspect", "Disallow: /*modules=SEARCH_REFINEMENTS_MODEL_V2", "Disallow: /*redirect=mobile", "Disallow: /*redirect%3Dmobile", "Disallow: /*rt%3Dnc", "Disallow: /*rt=nc", "Disallow: /*src=urllib", "Disallow: /?SSOWebDispatcher&tg=web&ru=", "Disallow: /act/", "Disallow: /adchoice", "Disallow: /buyer-preferences", "Disallow: /motors/diy/blog/*", "Disallow: /motors/fnd/*", "Disallow: /ads" ], "ruleCount": 223, "userAgentLines": [ "ChatGPT-User" ], "similarLines": [] }, "OAI-SearchBot": { "status": "partly", "rules": [ "Allow: /ads.txt", "Disallow: /*_kw", "Disallow: /*?maspect", "Disallow: /*modules=SEARCH_REFINEMENTS_MODEL_V2", "Disallow: /*redirect=mobile", "Disallow: /*redirect%3Dmobile", "Disallow: /*rt%3Dnc", "Disallow: /*rt=nc", "Disallow: /*src=urllib", "Disallow: /?SSOWebDispatcher&tg=web&ru=", "Disallow: /act/", "Disallow: /adchoice", "Disallow: /buyer-preferences", "Disallow: /motors/diy/blog/*", "Disallow: /motors/fnd/*", "Disallow: /ads" ], "ruleCount": 223, "userAgentLines": [ "OAI-SearchBot" ], "similarLines": [] }, "ClaudeBot": { "status": "partly", "rules": [ "Disallow: /", "Allow: /adchoice", "Allow: /authenticity-guarantee", "Allow: /motors/blog", "Allow: /motors/diy", "Allow: /help", "Allow: /sellercenter", "Allow: /ebaylive/seller", "Allow: /sl/sell", "Allow: /sh/landing", "Allow: /giftcards", "Allow: /shp/calc/rates" ], "ruleCount": 12, "userAgentLines": [ "ClaudeBot" ], "similarLines": [] }, "Claude-User": { "status": "partly", "rules": [ "Allow: /ads.txt", "Disallow: /*_kw", "Disallow: /*?maspect", "Disallow: /*modules=SEARCH_REFINEMENTS_MODEL_V2", "Disallow: /*redirect=mobile", "Disallow: /*redirect%3Dmobile", "Disallow: /*rt%3Dnc", "Disallow: /*rt=nc", "Disallow: /*src=urllib", "Disallow: /?SSOWebDispatcher&tg=web&ru=", "Disallow: /act/", "Disallow: /adchoice", "Disallow: /buyer-preferences", "Disallow: /motors/diy/blog/*", "Disallow: /motors/fnd/*", "Disallow: /ads" ], "ruleCount": 223, "userAgentLines": [ "Claude-User" ], "similarLines": [] }, "Claude-SearchBot": { "status": "partly", "rules": [ "Allow: /ads.txt", "Disallow: /*_kw", "Disallow: /*?maspect", "Disallow: /*modules=SEARCH_REFINEMENTS_MODEL_V2", "Disallow: /*redirect=mobile", "Disallow: /*redirect%3Dmobile", "Disallow: /*rt%3Dnc", "Disallow: /*rt=nc", "Disallow: /*src=urllib", "Disallow: /?SSOWebDispatcher&tg=web&ru=", "Disallow: /act/", "Disallow: /adchoice", "Disallow: /buyer-preferences", "Disallow: /motors/diy/blog/*", "Disallow: /motors/fnd/*", "Disallow: /ads" ], "ruleCount": 223, "userAgentLines": [ "Claude-SearchBot" ], "similarLines": [] }, "PerplexityBot": { "status": "partly", "rules": [ "Disallow: /", "Allow: /adchoice", "Allow: /authenticity-guarantee", "Allow: /motors/blog", "Allow: /motors/diy", "Allow: /help", "Allow: /sellercenter", "Allow: /ebaylive/seller", "Allow: /sl/sell", "Allow: /sh/landing", "Allow: /giftcards", "Allow: /shp/calc/rates" ], "ruleCount": 12, "userAgentLines": [ "PerplexityBot" ], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "partly", "rules": [ "Disallow: /", "Allow: /adchoice", "Allow: /authenticity-guarantee", "Allow: /motors/blog", "Allow: /motors/diy", "Allow: /help", "Allow: /sellercenter", "Allow: /ebaylive/seller", "Allow: /sl/sell", "Allow: /sh/landing", "Allow: /giftcards", "Allow: /shp/calc/rates" ], "ruleCount": 12, "userAgentLines": [ "AmazonBot" ], "similarLines": [] }, "Applebot-Extended": { "status": "partly", "rules": [ "Disallow: /", "Allow: /adchoice", "Allow: /authenticity-guarantee", "Allow: /motors/blog", "Allow: /motors/diy", "Allow: /help", "Allow: /sellercenter", "Allow: /ebaylive/seller", "Allow: /sl/sell", "Allow: /sh/landing", "Allow: /giftcards", "Allow: /shp/calc/rates" ], "ruleCount": 12, "userAgentLines": [ "Applebot-Extended" ], "similarLines": [] }, "CCBot": { "status": "partly", "rules": [ "Disallow: /", "Allow: /adchoice", "Allow: /authenticity-guarantee", "Allow: /motors/blog", "Allow: /motors/diy", "Allow: /help", "Allow: /sellercenter", "Allow: /ebaylive/seller", "Allow: /sl/sell", "Allow: /sh/landing", "Allow: /giftcards", "Allow: /shp/calc/rates" ], "ruleCount": 12, "userAgentLines": [ "CCBot" ], "similarLines": [] }, "Bytespider": { "status": "partly", "rules": [ "Disallow: /", "Allow: /adchoice", "Allow: /authenticity-guarantee", "Allow: /motors/blog", "Allow: /motors/diy", "Allow: /help", "Allow: /sellercenter", "Allow: /ebaylive/seller", "Allow: /sl/sell", "Allow: /sh/landing", "Allow: /giftcards", "Allow: /shp/calc/rates" ], "ruleCount": 12, "userAgentLines": [ "Bytespider" ], "similarLines": [] }, "Meta-ExternalAgent": { "status": "partly", "rules": [ "Disallow: /", "Allow: /adchoice", "Allow: /authenticity-guarantee", "Allow: /motors/blog", "Allow: /motors/diy", "Allow: /help", "Allow: /sellercenter", "Allow: /ebaylive/seller", "Allow: /sl/sell", "Allow: /sh/landing", "Allow: /giftcards", "Allow: /shp/calc/rates" ], "ruleCount": 12, "userAgentLines": [ "meta-externalagent" ], "similarLines": [] } } }, { "id": "etsy", "name": "Etsy", "homepage": "https://www.etsy.com/", "robotsUrl": "https://www.etsy.com/robots.txt", "playerId": "etsy", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.etsy.com/robots.txt", "bytes": 52821, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /mailinglist/email/", "Disallow: /api/", "Disallow: /your/", "Disallow: /r/group/", "Disallow: /appeals/", "Disallow: /transactions/", "Disallow: /treasury", "Disallow: /teams" ], "ruleCount": 1770 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "instacart", "name": "Instacart", "homepage": "https://www.instacart.com/", "robotsUrl": "https://www.instacart.com/robots.txt", "playerId": "instacart", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.instacart.com/robots.txt", "bytes": 13851, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "partly", "rules": [ "Allow: /", "Allow: /store/brands/l/pads-tampons-liners-wipes", "Disallow: /accounts/unlock", "Disallow: /api/", "Disallow: /ex/", "Disallow: /orders/", "Disallow: /store/orders/", "Disallow: /redeem_coupon_token", "Disallow: /store/order_changes/", "Disallow: /store/*/browse_departments*", "Disallow: /store/*/ng/l/*", "Disallow: /store/referrals*", "Disallow: /store/item*", "Disallow: /store/meals/*", "Disallow: /store/pickup/status/*", "Disallow: /store/*/buy_it_again*" ], "ruleCount": 32, "userAgentLines": [ "ChatGPT-User" ], "similarLines": [] }, "OAI-SearchBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /accounts/unlock", "Disallow: /api/", "Disallow: /ex/", "Disallow: /orders/", "Disallow: /store/orders/", "Disallow: /redeem_coupon_token", "Disallow: /store/order_changes/", "Disallow: /store/*/browse_departments*", "Disallow: /store/*/ng/l/*", "Disallow: /store/referrals*", "Disallow: /store/item*", "Disallow: /store/meals/*", "Disallow: /store/pickup/status/*", "Disallow: /store/*/buy_it_again*", "Disallow: /store/*/l/*" ], "ruleCount": 31, "userAgentLines": [ "OAI-SearchBot" ], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "PerplexityBot" ], "similarLines": [] }, "Perplexity-User": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "Perplexity-User" ], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 2, "userAgentLines": [ "meta-externalagent", "Meta-ExternalAgent" ], "similarLines": [] } } }, { "id": "best-buy", "name": "Best Buy", "homepage": "https://www.bestbuy.com/", "robotsUrl": "https://www.bestbuy.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.bestbuy.com/robots.txt", "bytes": 15016, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /~assets/", "Disallow: /nex/", "Allow: /discover-learn/", "Disallow: /m/e/*", "Disallow: /site/builder/*", "Disallow: /browse-api/*", "Disallow: /api/*.json", "Disallow: /availability/*" ], "ruleCount": 184 }, "agents": { "GPTBot": { "status": "partly", "rules": [ "Allow: /site/", "Disallow: /nex/", "Allow: /discover-learn/", "Disallow: /m/e/*", "Disallow: /browse-api/*", "Disallow: /api/*.json", "Disallow: /availability/*", "Disallow: /cart", "Disallow: /cart/*", "Disallow: /cartitems/*", "Disallow: /fulfillment/*", "Disallow: /location/*", "Disallow: /pricing/*", "Disallow: /productfulfillment/*", "Disallow: /profile/*", "Disallow: /services/plans/*" ], "ruleCount": 139, "userAgentLines": [ "GPTBot" ], "similarLines": [ "GPTBot-Web" ] }, "ChatGPT-User": { "status": "partly", "rules": [ "Allow: /site/", "Disallow: /nex/", "Allow: /discover-learn/", "Disallow: /m/e/*", "Disallow: /browse-api/*", "Disallow: /api/*.json", "Disallow: /availability/*", "Disallow: /cart", "Disallow: /cart/*", "Disallow: /cartitems/*", "Disallow: /fulfillment/*", "Disallow: /location/*", "Disallow: /pricing/*", "Disallow: /productfulfillment/*", "Disallow: /profile/*", "Disallow: /services/plans/*" ], "ruleCount": 139, "userAgentLines": [ "ChatGPT-User" ], "similarLines": [] }, "OAI-SearchBot": { "status": "partly", "rules": [ "Allow: /site/", "Disallow: /nex/", "Allow: /discover-learn/", "Disallow: /m/e/*", "Disallow: /browse-api/*", "Disallow: /api/*.json", "Disallow: /availability/*", "Disallow: /cart", "Disallow: /cart/*", "Disallow: /cartitems/*", "Disallow: /fulfillment/*", "Disallow: /location/*", "Disallow: /pricing/*", "Disallow: /productfulfillment/*", "Disallow: /profile/*", "Disallow: /services/plans/*" ], "ruleCount": 139, "userAgentLines": [ "OAI-SearchBot" ], "similarLines": [] }, "ClaudeBot": { "status": "partly", "rules": [ "Allow: /site/", "Disallow: /nex/", "Allow: /discover-learn/", "Disallow: /m/e/*", "Disallow: /browse-api/*", "Disallow: /api/*.json", "Disallow: /availability/*", "Disallow: /cart", "Disallow: /cart/*", "Disallow: /cartitems/*", "Disallow: /fulfillment/*", "Disallow: /location/*", "Disallow: /pricing/*", "Disallow: /productfulfillment/*", "Disallow: /profile/*", "Disallow: /services/plans/*" ], "ruleCount": 139, "userAgentLines": [ "Claudebot" ], "similarLines": [] }, "Claude-User": { "status": "partly", "rules": [ "Allow: /site/", "Disallow: /nex/", "Allow: /discover-learn/", "Disallow: /m/e/*", "Disallow: /browse-api/*", "Disallow: /api/*.json", "Disallow: /availability/*", "Disallow: /cart", "Disallow: /cart/*", "Disallow: /cartitems/*", "Disallow: /fulfillment/*", "Disallow: /location/*", "Disallow: /pricing/*", "Disallow: /productfulfillment/*", "Disallow: /profile/*", "Disallow: /services/plans/*" ], "ruleCount": 139, "userAgentLines": [ "Claude-User" ], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "partly", "rules": [ "Allow: /site/", "Disallow: /nex/", "Allow: /discover-learn/", "Disallow: /m/e/*", "Disallow: /browse-api/*", "Disallow: /api/*.json", "Disallow: /availability/*", "Disallow: /cart", "Disallow: /cart/*", "Disallow: /cartitems/*", "Disallow: /fulfillment/*", "Disallow: /location/*", "Disallow: /pricing/*", "Disallow: /productfulfillment/*", "Disallow: /profile/*", "Disallow: /services/plans/*" ], "ruleCount": 139, "userAgentLines": [ "PerplexityBot" ], "similarLines": [] }, "Perplexity-User": { "status": "partly", "rules": [ "Allow: /site/", "Disallow: /nex/", "Allow: /discover-learn/", "Disallow: /m/e/*", "Disallow: /browse-api/*", "Disallow: /api/*.json", "Disallow: /availability/*", "Disallow: /cart", "Disallow: /cart/*", "Disallow: /cartitems/*", "Disallow: /fulfillment/*", "Disallow: /location/*", "Disallow: /pricing/*", "Disallow: /productfulfillment/*", "Disallow: /profile/*", "Disallow: /services/plans/*" ], "ruleCount": 139, "userAgentLines": [ "Perplexity-User" ], "similarLines": [] }, "Google-Extended": { "status": "partly", "rules": [ "Allow: /site/", "Disallow: /nex/", "Allow: /discover-learn/", "Disallow: /m/e/*", "Disallow: /browse-api/*", "Disallow: /api/*.json", "Disallow: /availability/*", "Disallow: /cart", "Disallow: /cart/*", "Disallow: /cartitems/*", "Disallow: /fulfillment/*", "Disallow: /location/*", "Disallow: /pricing/*", "Disallow: /productfulfillment/*", "Disallow: /profile/*", "Disallow: /services/plans/*" ], "ruleCount": 139, "userAgentLines": [ "Google-Extended" ], "similarLines": [] }, "Amazonbot": { "status": "partly", "rules": [ "Allow: /site/", "Disallow: /nex/", "Allow: /discover-learn/", "Disallow: /m/e/*", "Disallow: /browse-api/*", "Disallow: /api/*.json", "Disallow: /availability/*", "Disallow: /cart", "Disallow: /cart/*", "Disallow: /cartitems/*", "Disallow: /fulfillment/*", "Disallow: /location/*", "Disallow: /pricing/*", "Disallow: /productfulfillment/*", "Disallow: /profile/*", "Disallow: /services/plans/*" ], "ruleCount": 139, "userAgentLines": [ "AmazonBot" ], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "partly", "rules": [ "Disallow: /[1.1]" ], "ruleCount": 1, "userAgentLines": [ "CCBot" ], "similarLines": [] }, "Bytespider": { "status": "partly", "rules": [ "Disallow: /[1.1]" ], "ruleCount": 1, "userAgentLines": [ "Bytespider" ], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "home-depot", "name": "The Home Depot", "homepage": "https://www.homedepot.com/", "robotsUrl": "https://www.homedepot.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.homedepot.com/robots.txt", "bytes": 1394, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /services/*/reviews/page/1$", "Disallow: /*SiteMapView*", "Disallow: /*Navigation?Ns=P_Topseller_Sort|style=List*", "Disallow: /*Navigation?Ns=P_Topseller_Sort|style=A*", "Disallow: /*CheckoutSignIn*", "Disallow: /*OrderItemUpdate*", "Disallow: /*THDLogon*", "Disallow: /*ShippingMethod*" ], "ruleCount": 46 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "lowes", "name": "Lowe's", "homepage": "https://www.lowes.com/", "robotsUrl": "https://www.lowes.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.lowes.com/robots.txt", "bytes": 11048, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /content/lowes/desktop/en_us/landing-pages.xml", "Disallow: /search?*", "Disallow: /*_/N-*", "Disallow: /*lowes/mobile/*", "Disallow: /*search-intg-svc/*", "Disallow: /*_pxhc*", "Disallow: /*No=*", "Disallow: /HC-*" ], "ruleCount": 341 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "costco", "name": "Costco", "homepage": "https://www.costco.com/", "robotsUrl": "https://www.costco.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.costco.com/robots.txt", "bytes": 3997, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /*?currentPage=*pageSize=*", "Disallow: /*WarehouseLocatorBrowseView*", "Disallow: /*AddressBookDisplayView*", "Disallow: /*ChangePassword*", "Disallow: /*CommunicationPreferenceView*", "Disallow: /*NonAjaxAddressBook*", "Disallow: /*OrderStatusCmd*", "Disallow: /*PaymentMethodDisplayView*" ], "ruleCount": 55 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "sams-club", "name": "Sam's Club", "homepage": "https://www.samsclub.com/", "robotsUrl": "https://www.samsclub.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.samsclub.com/robots.txt", "bytes": 448, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /ads.txt", "Disallow: *SCDC=1*", "Disallow: /*?bvroute", "Disallow: /lists/", "Disallow: */samsclub/0", "Disallow: /checkout/", "Disallow: /cart", "Disallow: /search" ], "ruleCount": 17 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "kroger", "name": "Kroger", "homepage": "https://www.kroger.com/", "robotsUrl": "https://www.kroger.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.kroger.com/robots.txt", "bytes": 618, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Disallow: /forgot-password", "Disallow: /checkout$", "Disallow: /checkout?", "Disallow: /checkout/*", "Disallow: /checkout-mobile$", "Disallow: /checkout-mobile?", "Disallow: /checkout-mobile/*", "Disallow: /scheduling$" ], "ruleCount": 18 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "wayfair", "name": "Wayfair", "homepage": "https://www.wayfair.com/", "robotsUrl": "https://www.wayfair.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.wayfair.com/robots.txt", "bytes": 3224, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /*/sb0/", "Disallow: /*/sb*/", "Allow: /*/sb1/", "Allow: /*/sb2/", "Allow: /v/business_account/trade_program/home", "Disallow: /ajax/", "Disallow: /filter/", "Disallow: /filters/" ], "ruleCount": 63 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "chewy", "name": "Chewy", "homepage": "https://www.chewy.com/", "robotsUrl": "https://www.chewy.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.chewy.com/robots.txt", "bytes": 2536, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Disallow: /app/buy", "Disallow: /app/deals?", "Disallow: /app/account", "Disallow: /app/ajax", "Disallow: /app/checkout", "Disallow: /*/dp/*?features", "Disallow: /*/dp/*?hybrid", "Disallow: /app/catalog/product-dimensions" ], "ruleCount": 45 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "Amazonbot" ], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "newegg", "name": "Newegg", "homepage": "https://www.newegg.com/", "robotsUrl": "https://www.newegg.com/robots.txt", "playerId": "newegg", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.newegg.com/robots.txt", "bytes": 4596, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /info/newsroomdetail.aspx", "Disallow: /Common/BML/", "Disallow: /Common/ThirdParty/", "Disallow: /App/", "Disallow: /Application/", "Disallow: /Configuration/", "Disallow: /NewMyAccount/", "Disallow: /MyNewegg/" ], "ruleCount": 72 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "apple", "name": "Apple", "homepage": "https://www.apple.com/", "robotsUrl": "https://www.apple.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.apple.com/robots.txt", "bytes": 1018, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /ac/globalnav/2.0/*/images/ac-globalnav/globalnav/search/*", "Disallow: /*shop/browse/overlay/*", "Disallow: /*shop/iphone/payments/overlay/*", "Disallow: /cn/*/aow/*", "Disallow: /tmall*" ], "ruleCount": 5 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "nike", "name": "Nike", "homepage": "https://www.nike.com/", "robotsUrl": "https://www.nike.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.nike.com/robots.txt", "bytes": 3503, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Disallow: */member/inbox", "Disallow: */member/settings", "Disallow: */p/", "Disallow: */checkout/", "Disallow: /*.swf$", "Disallow: /*.pdf$", "Disallow: /pdf/", "Disallow: /ar/help/" ], "ruleCount": 18 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "sephora", "name": "Sephora", "homepage": "https://www.sephora.com/", "robotsUrl": "https://www.sephora.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.sephora.com/robots.txt", "bytes": 922, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Disallow: /sephoratv", "Disallow: /basket", "Disallow: /browse/", "Disallow: /checkout", "Disallow: /contentStore/", "Disallow: /error/", "Disallow: /lovelist/", "Disallow: /profile/MyAccount" ], "ruleCount": 27 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "ulta", "name": "Ulta Beauty", "homepage": "https://www.ulta.com/", "robotsUrl": "https://www.ulta.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.ulta.com/robots.txt", "bytes": 2483, "outcome": "ok", "fetchNote": null, "catchAll": { "present": false, "status": "no group", "rules": [], "ruleCount": 0 }, "agents": { "GPTBot": { "status": "partly", "rules": [ "Disallow: /wishlists/", "Disallow: /curbside-alert/", "Disallow: /metrics*", "Disallow: /community/groups?user=", "Disallow: /auth/", "Disallow: /authorize", "Disallow: /u/", "Disallow: /promotion/8462937141" ], "ruleCount": 8, "userAgentLines": [ "GPTBot" ], "similarLines": [] }, "ChatGPT-User": { "status": "partly", "rules": [ "Disallow: /authorize", "Disallow: /u/", "Disallow: /auth/", "Disallow: /metrics*", "Disallow: /community/groups?user=", "Disallow: /promotion/8462937141" ], "ruleCount": 6, "userAgentLines": [ "ChatGPT-User" ], "similarLines": [] }, "OAI-SearchBot": { "status": "partly", "rules": [ "Disallow: /authorize", "Disallow: /u/", "Disallow: /auth/", "Disallow: /metrics*", "Disallow: /community/groups?user=", "Disallow: /promotion/8462937141" ], "ruleCount": 6, "userAgentLines": [ "OAI-SearchBot" ], "similarLines": [] }, "ClaudeBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /wishlists/", "Disallow: /curbside-alert/", "Disallow: /metrics*", "Disallow: /community/groups?user=", "Disallow: /auth/", "Disallow: /authorize", "Disallow: /u/", "Disallow: /promotion/8462937141" ], "ruleCount": 9, "userAgentLines": [ "ClaudeBot" ], "similarLines": [] }, "Claude-User": { "status": "partly", "rules": [ "Allow: /", "Disallow: /wishlists/", "Disallow: /curbside-alert/", "Disallow: /metrics*", "Disallow: /community/groups?user=", "Disallow: /auth/", "Disallow: /authorize", "Disallow: /u/", "Disallow: /promotion/8462937141" ], "ruleCount": 9, "userAgentLines": [ "Claude-User" ], "similarLines": [] }, "Claude-SearchBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /wishlists/", "Disallow: /curbside-alert/", "Disallow: /metrics*", "Disallow: /community/groups?user=", "Disallow: /auth/", "Disallow: /authorize", "Disallow: /u/", "Disallow: /promotion/8462937141" ], "ruleCount": 9, "userAgentLines": [ "Claude-SearchBot" ], "similarLines": [] }, "PerplexityBot": { "status": "partly", "rules": [ "Disallow: /wishlists/", "Disallow: /curbside-alert/", "Disallow: /metrics*", "Disallow: /community/groups?user=", "Disallow: /auth/", "Disallow: /authorize", "Disallow: /u/", "Disallow: /promotion/8462937141" ], "ruleCount": 8, "userAgentLines": [ "PerplexityBot" ], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "partly", "rules": [ "Disallow: /wishlists/", "Disallow: /curbside-alert/", "Disallow: /metrics*", "Disallow: /community/groups?user=", "Disallow: /auth/", "Disallow: /authorize", "Disallow: /u/", "Disallow: /promotion/8462937141" ], "ruleCount": 8, "userAgentLines": [ "Google-Extended" ], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "partly", "rules": [ "Disallow: /wishlists/", "Disallow: /curbside-alert/", "Disallow: /metrics*", "Disallow: /community/groups?user=", "Disallow: /auth/", "Disallow: /authorize", "Disallow: /u/", "Disallow: /promotion/8462937141" ], "ruleCount": 8, "userAgentLines": [ "CCBot" ], "similarLines": [] }, "Bytespider": { "status": "partly", "rules": [ "Disallow: /account/login/", "Disallow: /auth/", "Disallow: /authorize", "Disallow: /u/", "Disallow: /metrics*", "Disallow: /community/groups/*", "Disallow: /community/groups?*", "Disallow: /promotion/8462937141" ], "ruleCount": 8, "userAgentLines": [ "Bytespider" ], "similarLines": [] }, "Meta-ExternalAgent": { "status": "partly", "rules": [ "Disallow: /wishlists/", "Disallow: /curbside-alert/", "Disallow: /metrics*", "Disallow: /community/groups?user=", "Disallow: /auth/", "Disallow: /authorize", "Disallow: /u/", "Disallow: /promotion/8462937141" ], "ruleCount": 8, "userAgentLines": [ "Meta-ExternalAgent" ], "similarLines": [] } } }, { "id": "macys", "name": "Macy's", "homepage": "https://www.macys.com/", "robotsUrl": "https://www.macys.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.macys.com/robots.txt", "bytes": 2491, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /shop/*/*,*,Pageindex/*,*", "Disallow: /compare", "Disallow: /registry/wedding/compare", "Disallow: /catalog/product/zoom.jsp", "Disallow: /search*", "Disallow: /shop/search*", "Disallow: /shop/registry/wedding/search*", "Disallow: *natuzzi*" ], "ruleCount": 75 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "allowed", "rules": [], "ruleCount": 0, "userAgentLines": [ "ClaudeBot" ], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "nordstrom", "name": "Nordstrom", "homepage": "https://www.nordstrom.com/", "robotsUrl": "https://www.nordstrom.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.nordstrom.com/robots.txt", "bytes": 2685, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /api/ng-looks/", "Disallow: /s/*/*/full*", "Disallow: /s/*/*/lite*", "Disallow: /s/*/write-review*", "Disallow: /s/*/sizeAndFitInfo*", "Disallow: /s/-/*", "Disallow: /s/*/libs/*", "Disallow: /s/an-test*" ], "ruleCount": 48 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "partly", "rules": [ "Disallow: /api/checkout/web/checkout/v1/shopping-bag" ], "ruleCount": 1, "userAgentLines": [ "meta-externalagent/1.1" ], "similarLines": [] } } }, { "id": "kohls", "name": "Kohl's", "homepage": "https://www.kohls.com/", "robotsUrl": "https://www.kohls.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.kohls.com/robots.txt", "bytes": 768, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Disallow: /*&sortfilter=true", "Disallow: */checkout/*", "Disallow: /checkout/shopping_cart.jsp", "Disallow: */myaccount/*", "Disallow: /pickuppass", "Disallow: /product_videos/", "Disallow: */null", "Disallow: /*?BL=" ], "ruleCount": 26 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "jcpenney", "name": "JCPenney", "homepage": "https://www.jcpenney.com/", "robotsUrl": "https://www.jcpenney.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.jcpenney.com/robots.txt", "bytes": 1128, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /signin", "Disallow: /jsp/browse/pp/print/", "Disallow: /jsp/profile/", "Disallow: /dotcom/jsp/profile/", "Disallow: /5705/", "Disallow: /cam/", "Disallow: /dotcom/jsp/cart", "Disallow: /dotcom/jsp/checkout" ], "ruleCount": 15 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "gap", "name": "Gap", "homepage": "https://www.gap.com/", "robotsUrl": "https://www.gap.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.gap.com/robots.txt", "bytes": 871, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do" ], "ruleCount": 17 }, "agents": { "GPTBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "GPTBot" ], "similarLines": [] }, "ChatGPT-User": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "ChatGPT-User" ], "similarLines": [] }, "OAI-SearchBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "OAI-SearchBot" ], "similarLines": [] }, "ClaudeBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "ClaudeBot" ], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "PerplexityBot" ], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "Amazonbot" ], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "CCBot" ], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "Meta-ExternalAgent" ], "similarLines": [] } } }, { "id": "shein", "name": "SHEIN", "homepage": "https://us.shein.com/", "robotsUrl": "https://us.shein.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://us.shein.com/robots.txt", "bytes": 651, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /user/auth/login", "Disallow: /user/", "Disallow: /cart/", "Disallow: /atomic/", "Disallow: /geetest/", "Disallow: /abt/userinfo", "Disallow: /aimtell-worker.js", "Disallow: /coupon/getCouponListForOrder" ], "ruleCount": 21 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "temu", "name": "Temu", "homepage": "https://www.temu.com/", "robotsUrl": "https://www.temu.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.temu.com/robots.txt", "bytes": 8239, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "allowed", "rules": [ "Allow: /.well-known/" ], "ruleCount": 1 }, "agents": { "GPTBot": { "status": "partly", "rules": [ "Disallow: /*oauth_code=", "Disallow: /*oauth_token=", "Disallow: /bgp_user_profile.html", "Disallow: /bgp_personal.html", "Disallow: /bgp_collect.html", "Disallow: /bgp_footprint.html", "Disallow: /bgst_order_success.html", "Disallow: /bgst_payment_detail.html", "Disallow: /bgst_express.html", "Disallow: /personal.html", "Disallow: /collect.html", "Disallow: /shopping_cart.html", "Disallow: /bgp_similar_rec.html", "Disallow: /bgcp_shipping_add_items.html", "Disallow: /bgcp_coupon.html", "Disallow: /bgcp_cart_coupon.html" ], "ruleCount": 103, "userAgentLines": [ "GPTBot" ], "similarLines": [] }, "ChatGPT-User": { "status": "partly", "rules": [ "Disallow: /*oauth_code=", "Disallow: /*oauth_token=", "Disallow: /bgp_user_profile.html", "Disallow: /bgp_personal.html", "Disallow: /bgp_collect.html", "Disallow: /bgp_footprint.html", "Disallow: /bgst_order_success.html", "Disallow: /bgst_payment_detail.html", "Disallow: /bgst_express.html", "Disallow: /personal.html", "Disallow: /collect.html", "Disallow: /shopping_cart.html", "Disallow: /bgp_similar_rec.html", "Disallow: /bgcp_shipping_add_items.html", "Disallow: /bgcp_coupon.html", "Disallow: /bgcp_cart_coupon.html" ], "ruleCount": 103, "userAgentLines": [ "ChatGPT-User" ], "similarLines": [] }, "OAI-SearchBot": { "status": "partly", "rules": [ "Disallow: /*oauth_code=", "Disallow: /*oauth_token=", "Disallow: /bgp_user_profile.html", "Disallow: /bgp_personal.html", "Disallow: /bgp_collect.html", "Disallow: /bgp_footprint.html", "Disallow: /bgst_order_success.html", "Disallow: /bgst_payment_detail.html", "Disallow: /bgst_express.html", "Disallow: /personal.html", "Disallow: /collect.html", "Disallow: /shopping_cart.html", "Disallow: /bgp_similar_rec.html", "Disallow: /bgcp_shipping_add_items.html", "Disallow: /bgcp_coupon.html", "Disallow: /bgcp_cart_coupon.html" ], "ruleCount": 103, "userAgentLines": [ "OAI-SearchBot" ], "similarLines": [] }, "ClaudeBot": { "status": "partly", "rules": [ "Disallow: /*oauth_code=", "Disallow: /*oauth_token=", "Disallow: /bgp_user_profile.html", "Disallow: /bgp_personal.html", "Disallow: /bgp_collect.html", "Disallow: /bgp_footprint.html", "Disallow: /bgst_order_success.html", "Disallow: /bgst_payment_detail.html", "Disallow: /bgst_express.html", "Disallow: /personal.html", "Disallow: /collect.html", "Disallow: /shopping_cart.html", "Disallow: /bgp_similar_rec.html", "Disallow: /bgcp_shipping_add_items.html", "Disallow: /bgcp_coupon.html", "Disallow: /bgcp_cart_coupon.html" ], "ruleCount": 103, "userAgentLines": [ "ClaudeBot" ], "similarLines": [] }, "Claude-User": { "status": "partly", "rules": [ "Disallow: /*oauth_code=", "Disallow: /*oauth_token=", "Disallow: /bgp_user_profile.html", "Disallow: /bgp_personal.html", "Disallow: /bgp_collect.html", "Disallow: /bgp_footprint.html", "Disallow: /bgst_order_success.html", "Disallow: /bgst_payment_detail.html", "Disallow: /bgst_express.html", "Disallow: /personal.html", "Disallow: /collect.html", "Disallow: /shopping_cart.html", "Disallow: /bgp_similar_rec.html", "Disallow: /bgcp_shipping_add_items.html", "Disallow: /bgcp_coupon.html", "Disallow: /bgcp_cart_coupon.html" ], "ruleCount": 103, "userAgentLines": [ "Claude-User" ], "similarLines": [] }, "Claude-SearchBot": { "status": "partly", "rules": [ "Disallow: /*oauth_code=", "Disallow: /*oauth_token=", "Disallow: /bgp_user_profile.html", "Disallow: /bgp_personal.html", "Disallow: /bgp_collect.html", "Disallow: /bgp_footprint.html", "Disallow: /bgst_order_success.html", "Disallow: /bgst_payment_detail.html", "Disallow: /bgst_express.html", "Disallow: /personal.html", "Disallow: /collect.html", "Disallow: /shopping_cart.html", "Disallow: /bgp_similar_rec.html", "Disallow: /bgcp_shipping_add_items.html", "Disallow: /bgcp_coupon.html", "Disallow: /bgcp_cart_coupon.html" ], "ruleCount": 103, "userAgentLines": [ "Claude-SearchBot" ], "similarLines": [] }, "PerplexityBot": { "status": "partly", "rules": [ "Disallow: /*oauth_code=", "Disallow: /*oauth_token=", "Disallow: /bgp_user_profile.html", "Disallow: /bgp_personal.html", "Disallow: /bgp_collect.html", "Disallow: /bgp_footprint.html", "Disallow: /bgst_order_success.html", "Disallow: /bgst_payment_detail.html", "Disallow: /bgst_express.html", "Disallow: /personal.html", "Disallow: /collect.html", "Disallow: /shopping_cart.html", "Disallow: /bgp_similar_rec.html", "Disallow: /bgcp_shipping_add_items.html", "Disallow: /bgcp_coupon.html", "Disallow: /bgcp_cart_coupon.html" ], "ruleCount": 103, "userAgentLines": [ "PerplexityBot" ], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "partly", "rules": [ "Disallow: /*oauth_code=", "Disallow: /*oauth_token=", "Disallow: /bgp_user_profile.html", "Disallow: /bgp_personal.html", "Disallow: /bgp_collect.html", "Disallow: /bgp_footprint.html", "Disallow: /bgst_order_success.html", "Disallow: /bgst_payment_detail.html", "Disallow: /bgst_express.html", "Disallow: /personal.html", "Disallow: /collect.html", "Disallow: /shopping_cart.html", "Disallow: /bgp_similar_rec.html", "Disallow: /bgcp_shipping_add_items.html", "Disallow: /bgcp_coupon.html", "Disallow: /bgcp_cart_coupon.html" ], "ruleCount": 103, "userAgentLines": [ "Google-Extended" ], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "blocked fully", "rules": [ "Disallow: /*" ], "ruleCount": 1, "userAgentLines": [ "Bytespider" ], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "lululemon", "name": "Lululemon", "homepage": "https://shop.lululemon.com/", "robotsUrl": "https://shop.lululemon.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://shop.lululemon.com/robots.txt", "bytes": 488, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /c/*/*?*icid=*", "Disallow: /search?Ntt=", "Disallow: /*/search?Ntt=", "Disallow: /shop/gadgets/", "Disallow: /cartridges/FeaturesResultsList/", "Disallow: /search/", "Disallow: /*?Ns=price*", "Disallow: /*?Ns=RATINGS*" ], "ruleCount": 14 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "meta-externalagent" ], "similarLines": [] } } }, { "id": "victorias-secret", "name": "Victoria's Secret", "homepage": "https://www.victoriassecret.com/", "robotsUrl": "https://www.victoriassecret.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 403, "finalUrl": "https://www.victoriassecret.com/robots.txt", "bytes": 4554, "outcome": "unreachable", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. The retry also failed. HTTP 403.", "catchAll": { "present": false, "status": "no group", "rules": [], "ruleCount": 0 }, "agents": { "GPTBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "bath-and-body-works", "name": "Bath & Body Works", "homepage": "https://www.bathandbodyworks.com/", "robotsUrl": "https://www.bathandbodyworks.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.bathandbodyworks.com/robots.txt", "bytes": 717, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: *Sites-BathAndBodyWorks-Library*/experiences/*", "Disallow: *Sites-BathAndBodyWorks-Library*", "Allow: *Sites-BathAndBodyWorks-Library*/images/*", "Disallow: *Sites-BathAndBodyWorks-Site*", "Disallow: /s?q*", "Disallow: /my-account*", "Disallow: /my-account/wishlist*", "Disallow: /cart*" ], "ruleCount": 14 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "dicks", "name": "Dick's Sporting Goods", "homepage": "https://www.dickssportinggoods.com/", "robotsUrl": "https://www.dickssportinggoods.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.dickssportinggoods.com/robots.txt", "bytes": 1739, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Disallow: /webapp/wcs/stores/servlet/", "Disallow: /Password", "Disallow: /Register", "Disallow: /Registration", "Disallow: /Address", "Disallow: /Cart", "Disallow: /Order", "Disallow: /Checkout" ], "ruleCount": 56 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "rei", "name": "REI", "homepage": "https://www.rei.com/", "robotsUrl": "https://www.rei.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.rei.com/robots.txt", "bytes": 2227, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /content/dam/*.gif", "Disallow: /AddItemToCart*", "Disallow: /AddSessionToCart*", "Disallow: /adventures/*departure=*", "Disallow: /adventures/*sort=*", "Disallow: /adventures/rs/trips/*", "Disallow: /adventures/search/keyword*", "Disallow: /asset/upload/*" ], "ruleCount": 62 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "gamestop", "name": "GameStop", "homepage": "https://www.gamestop.com/", "robotsUrl": "https://www.gamestop.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 403, "finalUrl": "https://www.gamestop.com/robots.txt", "bytes": 4547, "outcome": "unreachable", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. The retry also failed. HTTP 403.", "catchAll": { "present": false, "status": "no group", "rules": [], "ruleCount": 0 }, "agents": { "GPTBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "qvc", "name": "QVC", "homepage": "https://www.qvc.com/", "robotsUrl": "https://www.qvc.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.qvc.com/robots.txt", "bytes": 17192, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /", "Allow: /scripts/reference.pl*ref=GBA*", "Disallow: /test-do-not-publish/", "Disallow: /FAQPrivacy.content.html", "Disallow: /GeneralTermsandConditions.content.html", "Disallow: /legalinfo.content.html", "Disallow: /pagenotfound.content.html", "Disallow: /ProductRecall.content.html" ], "ruleCount": 94 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "partly", "rules": [ "Allow: /", "Allow: /scripts/reference.pl*ref=GBA*", "Disallow: /scripts/", "Disallow: /test-do-not-publish/", "Disallow: /AddressBookForm", "Disallow: /ChangePassword", "Disallow: /CreditCardDisplay", "Disallow: /EditPermanentShippingAddressView", "Disallow: /EmailServicesForm", "Disallow: /Logoff", "Disallow: /OrderStatus", "Disallow: /UserRegistrationUpdateFormView", "Disallow: /webapp/wcs/stores/servlet/OrderReceiptView", "Disallow: /webapp/wcs/stores/servlet/OrderStatus", "Disallow: /webapp/wcs/stores/servlet/ProdNotFoundView", "Disallow: /CatalogSearch" ], "ruleCount": 83, "userAgentLines": [ "OAI-SearchBot" ], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "partly", "rules": [ "Allow: /", "Allow: /scripts/reference.pl*ref=GBA*", "Disallow: /scripts/", "Disallow: /test-do-not-publish/", "Disallow: /AddressBookForm", "Disallow: /ChangePassword", "Disallow: /CreditCardDisplay", "Disallow: /EditPermanentShippingAddressView", "Disallow: /EmailServicesForm", "Disallow: /Logoff", "Disallow: /OrderStatus", "Disallow: /UserRegistrationUpdateFormView", "Disallow: /webapp/wcs/stores/servlet/OrderReceiptView", "Disallow: /webapp/wcs/stores/servlet/OrderStatus", "Disallow: /webapp/wcs/stores/servlet/ProdNotFoundView", "Disallow: /CatalogSearch" ], "ruleCount": 83, "userAgentLines": [ "PerplexityBot" ], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "zappos", "name": "Zappos", "homepage": "https://www.zappos.com/", "robotsUrl": "https://www.zappos.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.zappos.com/robots.txt", "bytes": 1230, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Disallow: /bin/", "Disallow: /logout", "Disallow: /register", "Disallow: /account", "Disallow: /cart$", "Disallow: /marty/account", "Disallow: /multiview/", "Disallow: /productNotifyMe.do*" ], "ruleCount": 30 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "petco", "name": "Petco", "homepage": "https://www.petco.com/", "robotsUrl": "https://www.petco.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 403, "finalUrl": "https://www.petco.com/robots.txt", "bytes": 4767, "outcome": "unreachable", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. The retry also failed. HTTP 403.", "catchAll": { "present": false, "status": "no group", "rules": [], "ruleCount": 0 }, "agents": { "GPTBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "cvs", "name": "CVS", "homepage": "https://www.cvs.com/", "robotsUrl": "https://www.cvs.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.cvs.com/robots.txt", "bytes": 7836, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /*.css", "Disallow: /shop/*DNC-9", "Disallow: /shop/*/SELECT_A_VALUE/*", "Disallow: /shop/merch/*/c1*", "Disallow: /shop/*?skuId=*?page=*", "Disallow: *?scrollToReview=true", "Disallow: *?sort=", "Disallow: /shop/review-submission?" ], "ruleCount": 124 }, "agents": { "GPTBot": { "status": "partly", "rules": [ "Disallow: /druginfo" ], "ruleCount": 1, "userAgentLines": [ "GPTBot" ], "similarLines": [] }, "ChatGPT-User": { "status": "partly", "rules": [ "Disallow: /druginfo" ], "ruleCount": 1, "userAgentLines": [ "ChatGPT-User" ], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "partly", "rules": [ "Disallow: /druginfo" ], "ruleCount": 1, "userAgentLines": [ "ClaudeBot" ], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "partly", "rules": [ "Disallow: /druginfo" ], "ruleCount": 1, "userAgentLines": [ "Google-Extended" ], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "partly", "rules": [ "Disallow: /druginfo" ], "ruleCount": 1, "userAgentLines": [ "CCBot" ], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "walgreens", "name": "Walgreens", "homepage": "https://www.walgreens.com/", "robotsUrl": "https://www.walgreens.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.walgreens.com/robots.txt", "bytes": 2593, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /common/react/assets/*.js$", "Disallow: /emailsignup/", "Disallow: /includes/", "Disallow: /retailsearch/", "Disallow: /buyagain/product/", "Disallow: /productrectransformer/v1/", "Disallow: /digitalretailbff/v1/sdui/", "Disallow: /library/checkdrug/" ], "ruleCount": 62 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "ikea", "name": "IKEA", "homepage": "https://www.ikea.com/us/en/", "robotsUrl": "https://www.ikea.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.ikea.com/robots.txt", "bytes": 4089, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: */recommendations/*.js", "Disallow: */search/?q=*", "Disallow: */search/products/?q=*", "Disallow: */search/content/?q=*", "Disallow: */search/?category=*", "Disallow: */search/all/?q=*", "Disallow: /compare*", "Disallow: *?filter=*" ], "ruleCount": 95 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "blocked fully", "rules": [ "Disallow: /" ], "ruleCount": 1, "userAgentLines": [ "Amazonbot" ], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "williams-sonoma", "name": "Williams-Sonoma", "homepage": "https://www.williams-sonoma.com/", "robotsUrl": "https://www.williams-sonoma.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.williams-sonoma.com/robots.txt", "bytes": 1175, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /llms.txt", "Disallow: /account/", "Disallow: /checkout/", "Disallow: /shoppingcart/", "Disallow: /services/", "Disallow: /shop_g/", "Disallow: /shop_r/", "Disallow: /products_g/" ], "ruleCount": 18 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "saks", "name": "Saks Fifth Avenue", "homepage": "https://www.saksfifthavenue.com/", "robotsUrl": "https://www.saksfifthavenue.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 403, "finalUrl": "https://www.saksfifthavenue.com/robots.txt", "bytes": 782, "outcome": "unreachable", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. The retry also failed. HTTP 403.", "catchAll": { "present": false, "status": "no group", "rules": [], "ruleCount": 0 }, "agents": { "GPTBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "unknown", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "tractor-supply", "name": "Tractor Supply", "homepage": "https://www.tractorsupply.com/", "robotsUrl": "https://www.tractorsupply.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.tractorsupply.com/robots.txt", "bytes": 6319, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /tsc/search/*?isItmUnpub=Y$", "Disallow: */ajaxlogonform*", "Disallow: */article*", "Disallow: */bayAisleWcs/*", "Disallow: */pdplite/*", "Disallow: */wayfinding/getAccessToken*", "Disallow: */wcs/resources/store/*", "Disallow: */wcsstore/*" ], "ruleCount": 167 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "autozone", "name": "AutoZone", "homepage": "https://www.autozone.com/", "robotsUrl": "https://www.autozone.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.autozone.com/robots.txt", "bytes": 3952, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /signin$", "Disallow: /atg/", "Disallow: /dyn/", "Disallow: /cart", "Disallow: /rest/", "Disallow: /checkout", "Disallow: /error/", "Disallow: /ymme/" ], "ruleCount": 48 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "bjs", "name": "BJ's Wholesale Club", "homepage": "https://www.bjs.com/", "robotsUrl": "https://www.bjs.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.bjs.com/robots.txt", "bytes": 682, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Disallow: /webapp/wcs/stores/servlet/", "Disallow: /orderLogin", "Disallow: /membershipForgotEmail", "Disallow: /membershipForgotPassword", "Disallow: /cart", "Disallow: /search", "Disallow: /signIn", "Disallow: /rmp/" ], "ruleCount": 27 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "staples", "name": "Staples", "homepage": "https://www.staples.com/", "robotsUrl": "https://www.staples.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.staples.com/robots.txt", "bytes": 3900, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /services/printing/Account/DefaultLogin*", "Disallow: /page/", "Disallow: /builder2/", "Disallow: /api/builder/", "Disallow: /cc/api/", "Disallow: /cc/common/api", "Disallow: /cc/session/api", "Disallow: /rik/" ], "ruleCount": 77 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "bh-photo", "name": "B&H Photo", "homepage": "https://www.bhphotovideo.com/", "robotsUrl": "https://www.bhphotovideo.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.bhphotovideo.com/robots.txt", "bytes": 4747, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /find/mobile-new/services.jsp", "Disallow: /upload/", "Disallow: *c/search*", "Disallow: *c/l-search*", "Disallow: ?searchWithin", "Disallow: */search?q*", "Disallow: c/search?q=navigation", "Disallow: *?origSearch*" ], "ruleCount": 157 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "albertsons", "name": "Albertsons", "homepage": "https://www.albertsons.com/", "robotsUrl": "https://www.albertsons.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.albertsons.com/robots.txt", "bytes": 595, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /", "Disallow: /account/*", "Disallow: /erums/*", "Disallow: /mylist/*", "Disallow: /delivery-subscription/*", "Disallow: /justforu/coupons-deals.html", "Disallow: /justforu/coupons-deals/*", "Disallow: /justforu/rewards.html" ], "ruleCount": 10 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "office-depot", "name": "Office Depot", "homepage": "https://www.officedepot.com/", "robotsUrl": "https://www.officedepot.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://www.officedepot.com/robots.txt", "bytes": 1279, "outcome": "ok", "fetchNote": "The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent.", "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /b/clearance/Featured_Items--Clearance/clearance", "Disallow: /account/", "Disallow: /adRedir.do*", "Disallow: /ads/", "Disallow: /b2b/", "Disallow: /billboard/", "Disallow: /cart/", "Disallow: /catalog/browseCatalog.do*" ], "ruleCount": 48 }, "agents": { "GPTBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ChatGPT-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "OAI-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "ClaudeBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] } } }, { "id": "old-navy", "name": "Old Navy", "homepage": "https://oldnavy.gap.com/", "robotsUrl": "https://oldnavy.gap.com/robots.txt", "fetchedOn": "2026-10-28", "httpStatus": 200, "finalUrl": "https://oldnavy.gap.com/robots.txt", "bytes": 883, "outcome": "ok", "fetchNote": null, "catchAll": { "present": true, "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do" ], "ruleCount": 17 }, "agents": { "GPTBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "GPTBot" ], "similarLines": [] }, "ChatGPT-User": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "ChatGPT-User" ], "similarLines": [] }, "OAI-SearchBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "OAI-SearchBot" ], "similarLines": [] }, "ClaudeBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "ClaudeBot" ], "similarLines": [] }, "Claude-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Claude-SearchBot": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "PerplexityBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "PerplexityBot" ], "similarLines": [] }, "Perplexity-User": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Google-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Amazonbot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "Amazonbot" ], "similarLines": [] }, "Applebot-Extended": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "CCBot": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "CCBot" ], "similarLines": [] }, "Bytespider": { "status": "not mentioned", "rules": [], "ruleCount": 0, "userAgentLines": [], "similarLines": [] }, "Meta-ExternalAgent": { "status": "partly", "rules": [ "Allow: /", "Disallow: /*InternalServerError.do", "Disallow: /*GeneralNoResults.do", "Disallow: /*OutOfStockNoResults.do", "Disallow: /*InvalidIdNoResults.do", "Disallow: /*PageNotFound.do", "Disallow: /*BadRequest.do", "Disallow: /*SystemError.do", "Disallow: /*productData.do", "Disallow: /browse/search.do", "Disallow: /*cookieFailure.do", "Disallow: /buy/", "Disallow: /checkout/", "Disallow: /profile/", "Disallow: /shopping-bag", "Disallow: /my-account/" ], "ruleCount": 17, "userAgentLines": [ "Meta-ExternalAgent" ], "similarLines": [] } } } ], "policy": [ { "name": "JPMorgan Chase", "stance": "blocks", "statement": "A spokeswoman told CNBC the bank blocks those agents from operating for retail customers, because logging in with the customer's own credentials is not safe. Open-banking connections can still share data such as a balance, but they cannot send money or pay a bill.", "date": "2026-10-13", "sourceName": "CNBC", "sourceUrl": "https://www.cnbc.com/2026/10/13/jpmorgan-ai-agents-account-access.html", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#jpmorgan-blocks-agents-2026-10-13" }, { "name": "Morgan Stanley", "stance": "blocks", "statement": "Axios reported that Morgan Stanley has begun blocking third-party AI agents from its platforms, and that a Morgan Stanley spokesperson said the bank had not seen rogue agents there.", "date": "2026-10-22", "sourceName": "Axios", "sourceUrl": "https://www.axios.com/2026/10/22/ai-agents-banks-unauthorized-accounts-access", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#morgan-stanley-blocks-agents-2026-10-22" }, { "name": "Wells Fargo", "stance": "blocks", "statement": "USA Today reported that a Wells Fargo spokesperson said the bank has taken steps to block unauthorized agents from customer accounts. Its updated online access agreement, effective November 24, 2026, says a customer who permits an AI agent is solely responsible for supervising it, and that the agent's transactions, including ones the customer did not want, count as authorized.", "date": "2026-10-23", "sourceName": "USA Today", "sourceUrl": "https://www.usatoday.com/story/money/2026/10/23/banks-move-to-limit-ai-agent-access/92385119007/", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#wells-fargo-blocks-agents-2026-10-23" }, { "name": "Citi", "stance": "blocks", "statement": "Citi does not allow a third-party AI agent to log into a customer's account or make a transaction in Citi's digital banking. Customers can still share limited read-only account data with some open-banking providers.", "date": "2026-10-19", "sourceName": "The Paypers", "sourceUrl": "https://thepaypers.com/fintech/news/citi-and-bank-of-america-restrict-ai-agents-from-accessing-accounts", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#citi-bofa-agent-accounts-2026-10-19" }, { "name": "Bank of America", "stance": "limits", "statement": "Bank of America is limiting AI agents' access to consumer deposits while it decides who is liable for a payment an agent authorizes.", "date": "2026-10-19", "sourceName": "The Paypers", "sourceUrl": "https://thepaypers.com/fintech/news/citi-and-bank-of-america-restrict-ai-agents-from-accessing-accounts", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#citi-bofa-agent-accounts-2026-10-19" }, { "name": "Charles Schwab", "stance": "blocks", "statement": "The Verge reported that Charles Schwab is blocking AI agents from client accounts. CEO Rick Wurster said the firm will not allow a client to give Meta's Muse a password and let Muse act for that client.", "date": "2026-10-16", "sourceName": "The Verge", "sourceUrl": "https://www.theverge.com/ai-artificial-intelligence/1013303/charles-schwab-is-blocking-ai-agents-too", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#schwab-blocks-agents-2026-10-16" }, { "name": "Royal Bank of Canada", "stance": "blocks", "statement": "block third-party personal AI agents from direct access to authenticated RBC digital properties, including online banking.", "date": "2026-10-27", "sourceName": "Royal Bank of Canada", "sourceUrl": "https://www.rbcroyalbank.com/en-ca/my-money-matters/money-academy/cyber-security/understanding-cyber-security/understanding-third-party-ai-agents-and-your-rbc-banking/", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#rbc-blocks-agents-2026-10-27" }, { "name": "Bank of Montreal", "stance": "blocks", "statement": "Bank of Montreal has blocked third-party AI agents from accessing its digital properties.", "date": "2026-10-28", "sourceName": "The Globe and Mail", "sourceUrl": "https://www.theglobeandmail.com/business/article-bmo-blocks-third-party-ai-agents-rbc-muse-october-28/", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#bmo-blocks-agents-2026-10-28" }, { "name": "TD", "stance": "blocks", "statement": "TD will block all third-party personal AI agents from direct access to banking information where authentication is required.", "date": "2026-10-29", "sourceName": "CBC News", "sourceUrl": "https://www.cbc.ca/news/business/canadian-banks-block-ai-9.7400777", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#canadian-banks-agent-limits-2026-10-29" }, { "name": "CIBC", "stance": "limits", "statement": "CIBC said it does not \"support personal AI agents directly opening or accessing accounts.\"", "note": "CIBC said it does not support this access. The Globe and Mail said the bank did not confirm a full block.", "date": "2026-10-29", "sourceName": "CBC News", "sourceUrl": "https://www.cbc.ca/news/business/canadian-banks-block-ai-9.7400777", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#canadian-banks-agent-limits-2026-10-29" }, { "name": "Scotiabank", "stance": "limits", "statement": "Scotiabank said it will limit third-party AI agents from its banking platforms while it monitors concerns.", "date": "2026-10-29", "sourceName": "CBC News", "sourceUrl": "https://www.cbc.ca/news/business/canadian-banks-block-ai-9.7400777", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#canadian-banks-agent-limits-2026-10-29" }, { "name": "Amazon", "stance": "blocks", "statement": "GeekWire reported that Muse shoppers on Amazon saw a notice that continued access by an unauthorized AI agent violates Amazon's Conditions of Use, and that Amazon is suing Perplexity over its Comet browser and moving to block shopping agents from Google and OpenAI.", "date": "2026-09-20", "sourceName": "GeekWire", "sourceUrl": "https://www.geekwire.com/2026/amazon-blocks-metas-muse-ai-assistant-in-new-standoff-over-agentic-shopping/", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#amazon-blocks-muse-2026-09-20", "trackerPlayerUrl": "https://agent-shopping-tracker.vercel.app/players/amazon" }, { "name": "Walmart", "stance": "limits", "statement": "Walmart said Agent Profiles, launching in the next month, let a customer authorize each AI agent with its own profile and either a spending limit or approval on every purchase, and that agents never receive payment-card data.", "note": "This is a limit on an authorized agent. The tracker does not record a Walmart terms-of-use ban on every agent.", "date": "2026-10-21", "sourceName": "Walmart newsroom", "sourceUrl": "https://corporate.walmart.com/news/2026/10/Making-It-Possible-for-Agents-to-Shop-on-Behalf-of-Customers", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#walmart-agent-gateway-2026-10-21", "trackerPlayerUrl": "https://agent-shopping-tracker.vercel.app/players/walmart" }, { "name": "Delta", "stance": "blocks", "statement": "Delta told TechCrunch it has no partnership or integration that lets a third-party AI agent shop for or book Delta flights on Delta's sites. Delta said it is protecting customers from unauthorized automated activity, and that any opening to outside agents would have to keep security and the customer experience in mind.", "date": "2026-10-06", "sourceName": "TechCrunch", "sourceUrl": "https://techcrunch.com/2026/10/06/the-next-hurdle-for-ai-agents-getting-websites-to-let-them-in/", "trackerNewsUrl": "https://agent-shopping-tracker.vercel.app/news#delta-no-agent-booking-2026-10-06" } ] }