Last checked
Who blocks AI shopping agents (updated weekly)
This is a weekly reading of the public robots.txt file at major US retailers and marketplaces. The file asks some crawlers to stay away. It is a request, not a lock. A site can stop an agent without saying so here, and an agent can ignore the file.
Key findings
On October 28, 2026 we requested the public robots.txt file for 48 US retail or marketplace sites. We could read 44 files. 4 sites did not return a file we could read: Victoria's Secret, GameStop, Petco, and Saks Fifth Avenue. Those rows stay unread and are left out of the counts below.
15 of the 44 readable files name at least one of the 14 agents we track. 6 readable files fully disallow at least one of them: Amazon (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent); Instacart (PerplexityBot, Perplexity-User, Meta-ExternalAgent); Chewy (Amazonbot); Temu (Bytespider); Lululemon (Meta-ExternalAgent); IKEA (Amazonbot). 29 readable files name none of them, including Walmart, Target, Etsy, The Home Depot, Lowe's, Costco, Sam's Club, and 22 others. Not named is not permission. A file can still set rules for User-agent: *, and a site can block an agent without naming it.
GPTBot is fully disallowed in 1 readable file, ClaudeBot in 1, PerplexityBot in 2, Google-Extended in 1, CCBot in 1, and Bytespider in 2. ChatGPT-User is fully disallowed in 1, Claude-User in 1, and Perplexity-User in 2.
1 readable file fully disallows all three user-triggered tokens (ChatGPT-User, Claude-User, and Perplexity-User): Amazon.
1 readable file fully disallows User-agent: *: Instacart. That asks every unnamed crawler to stay off the whole site. Agents we marked not mentioned on that site can still fall under that * rule.
Free downloads
The JSON and CSV are the same check as this page. Free to use with a link back. CC BY 4.0.
How to read a cell
- Blocked means blocked fully: the file disallows the whole site for that agent and does not also allow a path.
- Partly means the agent is named and the rules cover only some paths, or they disallow the whole site and also allow something. Read the rules. This is not a full block.
- Allowed means the file names the agent and disallows no path.
- Not named means not mentioned. That is not permission. Rules for User-agent: * can still apply.
- Unread means we did not get a robots.txt we could read. It is not counted as allowed.
ChatGPT-User, Claude-User, and Perplexity-User are the tokens those products publish for a fetch a person asked for. GPTBot, ClaudeBot, and PerplexityBot are crawlers. Blocking one is not the same as blocking the other. Google-Extended is not Google Search. Applebot-Extended is not Applebot. Amazonbot is Amazon's crawler, not a shopper.
Counts by agent
Totals use the 44 sites we could classify. 4 unread sites are in the table below and not in these totals.
| Agent | What it is | Blocked fully | Partly | Allowed | Not mentioned |
|---|---|---|---|---|---|
| GPTBotOpenAI | Training crawler | 1 | 7 | 0 | 36 |
| ChatGPT-UserOpenAI | User-triggered fetch | 1 | 8 | 0 | 35 |
| OAI-SearchBotOpenAI | Search crawler | 1 | 8 | 0 | 35 |
| ClaudeBotAnthropic | Training crawler | 1 | 7 | 1 | 35 |
| Claude-UserAnthropic | User-triggered fetch | 1 | 4 | 0 | 39 |
| Claude-SearchBotAnthropic | Search crawler | 1 | 3 | 0 | 40 |
| PerplexityBotPerplexity | Crawler | 2 | 7 | 0 | 35 |
| Perplexity-UserPerplexity | User-triggered fetch | 2 | 1 | 0 | 41 |
| Google-ExtendedGoogle | AI training opt-out, not Google Search | 1 | 4 | 0 | 39 |
| AmazonbotAmazon | Crawler, not a shopper | 2 | 4 | 0 | 38 |
| Applebot-ExtendedApple | AI training opt-out, not Applebot search | 0 | 1 | 0 | 43 |
| CCBotCommon Crawl | Training corpus crawler | 1 | 6 | 0 | 37 |
| BytespiderByteDance | Crawler | 2 | 3 | 0 | 39 |
| Meta-ExternalAgentMeta | External agent crawler | 3 | 5 | 0 | 36 |
Every site
Swipe sideways for every agent. Open Rules for the lines we stored. A long group is sampled. The label still uses every rule.
| Site | GPTBot | ChatGPT-User | OAI-SearchBot | ClaudeBot | Claude-User | Claude-SearchBot | PerplexityBot | Perplexity-User | Google-Extended | Amazonbot | Applebot-Extended | CCBot | Bytespider | Meta-ExternalAgent |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Amazon · TrackerRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /wishlist/universal*; Disallow: /exec/obidos/account-access-login; Disallow: /exec/obidos/change-style; Disallow: /exec/obidos/flex-sign-in; Disallow: /exec/obidos/handle-buy-box; Disallow: /exec/obidos/tg/cm/member/; Disallow: /gp/aw/help/id=sss; Disallow: /gp/cart (137 rules, sample shown)
| Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Not named | Not named | Blocked | Blocked | Blocked |
Walmart · TrackerRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /reviews/product/; Disallow: /0/; Disallow: /55875582/walmart-us/catalog/; Disallow: /account/; Disallow: /api/; Disallow: /collection/api/logger; Disallow: /cp/-201; Disallow: /cp/-302 (60 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
Target · TrackerRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Disallow: /*/Ntk; Disallow: /*/Ntt; Disallow: /*/Ntx; Disallow: /*%7Cd_; Disallow: /*/schoollist/; Disallow: /*BTWN; Disallow: /[path]/; Disallow: /7078046/ (114 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
eBayRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /ads.txt; Disallow: /*_kw; Disallow: /*?maspect; Disallow: /*modules=SEARCH_REFINEMENTS_MODEL_V2; Disallow: /*redirect=mobile; Disallow: /*redirect%3Dmobile; Disallow: /*rt%3Dnc; Disallow: /*rt=nc (234 rules, sample shown)
| Partly | Partly | Partly | Partly | Partly | Partly | Partly | Not named | Not named | Partly | Partly | Partly | Partly | Partly |
Etsy · TrackerRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /mailinglist/email/; Disallow: /api/; Disallow: /your/; Disallow: /r/group/; Disallow: /appeals/; Disallow: /transactions/; Disallow: /treasury; Disallow: /teams (1770 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
Instacart · TrackerRulesrobots.txt · HTTP 200 · ok User-agent: *: blocked fully — Disallow: /
| Not named | Partly | Partly | Not named | Not named | Not named | Blocked | Blocked | Not named | Not named | Not named | Not named | Not named | Blocked |
Best BuyRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /~assets/; Disallow: /nex/; Allow: /discover-learn/; Disallow: /m/e/*; Disallow: /site/builder/*; Disallow: /browse-api/*; Disallow: /api/*.json; Disallow: /availability/* (184 rules, sample shown)
| Partly | Partly | Partly | Partly | Partly | Not named | Partly | Partly | Partly | Partly | Not named | Partly | Partly | Not named |
The Home DepotRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /services/*/reviews/page/1$; Disallow: /*SiteMapView*; Disallow: /*Navigation?Ns=P_Topseller_Sort|style=List*; Disallow: /*Navigation?Ns=P_Topseller_Sort|style=A*; Disallow: /*CheckoutSignIn*; Disallow: /*OrderItemUpdate*; Disallow: /*THDLogon*; Disallow: /*ShippingMethod* (46 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
Lowe'sRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Allow: /content/lowes/desktop/en_us/landing-pages.xml; Disallow: /search?*; Disallow: /*_/N-*; Disallow: /*lowes/mobile/*; Disallow: /*search-intg-svc/*; Disallow: /*_pxhc*; Disallow: /*No=*; Disallow: /HC-* (341 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
CostcoRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /*?currentPage=*pageSize=*; Disallow: /*WarehouseLocatorBrowseView*; Disallow: /*AddressBookDisplayView*; Disallow: /*ChangePassword*; Disallow: /*CommunicationPreferenceView*; Disallow: /*NonAjaxAddressBook*; Disallow: /*OrderStatusCmd*; Disallow: /*PaymentMethodDisplayView* (55 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
Sam's ClubRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /ads.txt; Disallow: *SCDC=1*; Disallow: /*?bvroute; Disallow: /lists/; Disallow: */samsclub/0; Disallow: /checkout/; Disallow: /cart; Disallow: /search (17 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
KrogerRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Disallow: /forgot-password; Disallow: /checkout$; Disallow: /checkout?; Disallow: /checkout/*; Disallow: /checkout-mobile$; Disallow: /checkout-mobile?; Disallow: /checkout-mobile/*; Disallow: /scheduling$ (18 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
WayfairRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /*/sb0/; Disallow: /*/sb*/; Allow: /*/sb1/; Allow: /*/sb2/; Allow: /v/business_account/trade_program/home; Disallow: /ajax/; Disallow: /filter/; Disallow: /filters/ (63 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
ChewyRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Disallow: /app/buy; Disallow: /app/deals?; Disallow: /app/account; Disallow: /app/ajax; Disallow: /app/checkout; Disallow: /*/dp/*?features; Disallow: /*/dp/*?hybrid; Disallow: /app/catalog/product-dimensions (45 rules, sample shown)
| Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Blocked | Not named | Not named | Not named | Not named |
Newegg · TrackerRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /info/newsroomdetail.aspx; Disallow: /Common/BML/; Disallow: /Common/ThirdParty/; Disallow: /App/; Disallow: /Application/; Disallow: /Configuration/; Disallow: /NewMyAccount/; Disallow: /MyNewegg/ (72 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
AppleRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /ac/globalnav/2.0/*/images/ac-globalnav/globalnav/search/*; Disallow: /*shop/browse/overlay/*; Disallow: /*shop/iphone/payments/overlay/*; Disallow: /cn/*/aow/*; Disallow: /tmall* None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
NikeRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Disallow: */member/inbox; Disallow: */member/settings; Disallow: */p/; Disallow: */checkout/; Disallow: /*.swf$; Disallow: /*.pdf$; Disallow: /pdf/; Disallow: /ar/help/ (18 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
SephoraRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Disallow: /sephoratv; Disallow: /basket; Disallow: /browse/; Disallow: /checkout; Disallow: /contentStore/; Disallow: /error/; Disallow: /lovelist/; Disallow: /profile/MyAccount (27 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
Ulta BeautyRulesrobots.txt · HTTP 200 · ok User-agent: *: no group
| Partly | Partly | Partly | Partly | Partly | Partly | Partly | Not named | Partly | Not named | Not named | Partly | Partly | Partly |
Macy'sRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Allow: /shop/*/*,*,Pageindex/*,*; Disallow: /compare; Disallow: /registry/wedding/compare; Disallow: /catalog/product/zoom.jsp; Disallow: /search*; Disallow: /shop/search*; Disallow: /shop/registry/wedding/search*; Disallow: *natuzzi* (75 rules, sample shown)
| Not named | Not named | Not named | Allowed | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
NordstromRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /api/ng-looks/; Disallow: /s/*/*/full*; Disallow: /s/*/*/lite*; Disallow: /s/*/write-review*; Disallow: /s/*/sizeAndFitInfo*; Disallow: /s/-/*; Disallow: /s/*/libs/*; Disallow: /s/an-test* (48 rules, sample shown)
| Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Partly |
Kohl'sRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Disallow: /*&sortfilter=true; Disallow: */checkout/*; Disallow: /checkout/shopping_cart.jsp; Disallow: */myaccount/*; Disallow: /pickuppass; Disallow: /product_videos/; Disallow: */null; Disallow: /*?BL= (26 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
JCPenneyRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Allow: /signin; Disallow: /jsp/browse/pp/print/; Disallow: /jsp/profile/; Disallow: /dotcom/jsp/profile/; Disallow: /5705/; Disallow: /cam/; Disallow: /dotcom/jsp/cart; Disallow: /dotcom/jsp/checkout (15 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
GapRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /; Disallow: /*InternalServerError.do; Disallow: /*GeneralNoResults.do; Disallow: /*OutOfStockNoResults.do; Disallow: /*InvalidIdNoResults.do; Disallow: /*PageNotFound.do; Disallow: /*BadRequest.do; Disallow: /*SystemError.do (17 rules, sample shown)
| Partly | Partly | Partly | Partly | Not named | Not named | Partly | Not named | Not named | Partly | Not named | Partly | Not named | Partly |
SHEINRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /user/auth/login; Disallow: /user/; Disallow: /cart/; Disallow: /atomic/; Disallow: /geetest/; Disallow: /abt/userinfo; Disallow: /aimtell-worker.js; Disallow: /coupon/getCouponListForOrder (21 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
TemuRulesrobots.txt · HTTP 200 · ok User-agent: *: allowed — Allow: /.well-known/
| Partly | Partly | Partly | Partly | Partly | Partly | Partly | Not named | Partly | Not named | Not named | Not named | Blocked | Not named |
LululemonRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Allow: /c/*/*?*icid=*; Disallow: /search?Ntt=; Disallow: /*/search?Ntt=; Disallow: /shop/gadgets/; Disallow: /cartridges/FeaturesResultsList/; Disallow: /search/; Disallow: /*?Ns=price*; Disallow: /*?Ns=RATINGS* (14 rules, sample shown)
| Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Blocked |
Victoria's SecretRulesrobots.txt · HTTP 403 · unreachable The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. The retry also failed. HTTP 403. User-agent: *: no group
| Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread |
Bath & Body WorksRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: *Sites-BathAndBodyWorks-Library*/experiences/*; Disallow: *Sites-BathAndBodyWorks-Library*; Allow: *Sites-BathAndBodyWorks-Library*/images/*; Disallow: *Sites-BathAndBodyWorks-Site*; Disallow: /s?q*; Disallow: /my-account*; Disallow: /my-account/wishlist*; Disallow: /cart* (14 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
Dick's Sporting GoodsRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Disallow: /webapp/wcs/stores/servlet/; Disallow: /Password; Disallow: /Register; Disallow: /Registration; Disallow: /Address; Disallow: /Cart; Disallow: /Order; Disallow: /Checkout (56 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
REIRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Allow: /content/dam/*.gif; Disallow: /AddItemToCart*; Disallow: /AddSessionToCart*; Disallow: /adventures/*departure=*; Disallow: /adventures/*sort=*; Disallow: /adventures/rs/trips/*; Disallow: /adventures/search/keyword*; Disallow: /asset/upload/* (62 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
GameStopRulesrobots.txt · HTTP 403 · unreachable The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. The retry also failed. HTTP 403. User-agent: *: no group
| Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread |
QVCRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Allow: /; Allow: /scripts/reference.pl*ref=GBA*; Disallow: /test-do-not-publish/; Disallow: /FAQPrivacy.content.html; Disallow: /GeneralTermsandConditions.content.html; Disallow: /legalinfo.content.html; Disallow: /pagenotfound.content.html; Disallow: /ProductRecall.content.html (94 rules, sample shown)
| Not named | Not named | Partly | Not named | Not named | Not named | Partly | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
ZapposRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Disallow: /bin/; Disallow: /logout; Disallow: /register; Disallow: /account; Disallow: /cart$; Disallow: /marty/account; Disallow: /multiview/; Disallow: /productNotifyMe.do* (30 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
PetcoRulesrobots.txt · HTTP 403 · unreachable The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. The retry also failed. HTTP 403. User-agent: *: no group
| Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread |
CVSRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /*.css; Disallow: /shop/*DNC-9; Disallow: /shop/*/SELECT_A_VALUE/*; Disallow: /shop/merch/*/c1*; Disallow: /shop/*?skuId=*?page=*; Disallow: *?scrollToReview=true; Disallow: *?sort=; Disallow: /shop/review-submission? (124 rules, sample shown)
| Partly | Partly | Not named | Partly | Not named | Not named | Not named | Not named | Partly | Not named | Not named | Partly | Not named | Not named |
WalgreensRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /common/react/assets/*.js$; Disallow: /emailsignup/; Disallow: /includes/; Disallow: /retailsearch/; Disallow: /buyagain/product/; Disallow: /productrectransformer/v1/; Disallow: /digitalretailbff/v1/sdui/; Disallow: /library/checkdrug/ (62 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
IKEARulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: */recommendations/*.js; Disallow: */search/?q=*; Disallow: */search/products/?q=*; Disallow: */search/content/?q=*; Disallow: */search/?category=*; Disallow: */search/all/?q=*; Disallow: /compare*; Disallow: *?filter=* (95 rules, sample shown)
| Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Blocked | Not named | Not named | Not named | Not named |
Williams-SonomaRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /llms.txt; Disallow: /account/; Disallow: /checkout/; Disallow: /shoppingcart/; Disallow: /services/; Disallow: /shop_g/; Disallow: /shop_r/; Disallow: /products_g/ (18 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
Saks Fifth AvenueRulesrobots.txt · HTTP 403 · unreachable The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. The retry also failed. HTTP 403. User-agent: *: no group
| Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread | Unread |
Tractor SupplyRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Allow: /tsc/search/*?isItmUnpub=Y$; Disallow: */ajaxlogonform*; Disallow: */article*; Disallow: */bayAisleWcs/*; Disallow: */pdplite/*; Disallow: */wayfinding/getAccessToken*; Disallow: */wcs/resources/store/*; Disallow: */wcsstore/* (167 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
AutoZoneRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /signin$; Disallow: /atg/; Disallow: /dyn/; Disallow: /cart; Disallow: /rest/; Disallow: /checkout; Disallow: /error/; Disallow: /ymme/ (48 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
BJ's Wholesale ClubRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Disallow: /webapp/wcs/stores/servlet/; Disallow: /orderLogin; Disallow: /membershipForgotEmail; Disallow: /membershipForgotPassword; Disallow: /cart; Disallow: /search; Disallow: /signIn; Disallow: /rmp/ (27 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
StaplesRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Allow: /services/printing/Account/DefaultLogin*; Disallow: /page/; Disallow: /builder2/; Disallow: /api/builder/; Disallow: /cc/api/; Disallow: /cc/common/api; Disallow: /cc/session/api; Disallow: /rik/ (77 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
B&H PhotoRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /find/mobile-new/services.jsp; Disallow: /upload/; Disallow: *c/search*; Disallow: *c/l-search*; Disallow: ?searchWithin; Disallow: */search?q*; Disallow: c/search?q=navigation; Disallow: *?origSearch* (157 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
AlbertsonsRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /; Disallow: /account/*; Disallow: /erums/*; Disallow: /mylist/*; Disallow: /delivery-subscription/*; Disallow: /justforu/coupons-deals.html; Disallow: /justforu/coupons-deals/*; Disallow: /justforu/rewards.html (10 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
Office DepotRulesrobots.txt · HTTP 200 · ok The identified checker was refused, timed out, or got HTML. Retried once with a common browser user agent. User-agent: *: partly — Allow: /b/clearance/Featured_Items--Clearance/clearance; Disallow: /account/; Disallow: /adRedir.do*; Disallow: /ads/; Disallow: /b2b/; Disallow: /billboard/; Disallow: /cart/; Disallow: /catalog/browseCatalog.do* (48 rules, sample shown) None of the tracked agents are named. | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named | Not named |
Old NavyRulesrobots.txt · HTTP 200 · ok User-agent: *: partly — Allow: /; Disallow: /*InternalServerError.do; Disallow: /*GeneralNoResults.do; Disallow: /*OutOfStockNoResults.do; Disallow: /*InvalidIdNoResults.do; Disallow: /*PageNotFound.do; Disallow: /*BadRequest.do; Disallow: /*SystemError.do (17 rules, sample shown)
| Partly | Partly | Partly | Partly | Not named | Not named | Partly | Not named | Not named | Partly | Not named | Partly | Not named | Partly |
Companies that block or limit AI agents by policy
These rows are not robots.txt results. They repeat facts already published on this tracker, with the same dates and sources. A bank rule about account access is a different kind of limit from a store's crawler file.
JPMorgan Chase
BlocksA spokeswoman told CNBC the bank blocks those agents from operating for retail customers, because logging in with the customer's own credentials is not safe. Open-banking connections can still share data such as a balance, but they cannot send money or pay a bill.
Morgan Stanley
BlocksAxios reported that Morgan Stanley has begun blocking third-party AI agents from its platforms, and that a Morgan Stanley spokesperson said the bank had not seen rogue agents there.
Wells Fargo
BlocksUSA Today reported that a Wells Fargo spokesperson said the bank has taken steps to block unauthorized agents from customer accounts. Its updated online access agreement, effective November 24, 2026, says a customer who permits an AI agent is solely responsible for supervising it, and that the agent's transactions, including ones the customer did not want, count as authorized.
Citi
BlocksCiti does not allow a third-party AI agent to log into a customer's account or make a transaction in Citi's digital banking. Customers can still share limited read-only account data with some open-banking providers.
Bank of America
LimitsBank of America is limiting AI agents' access to consumer deposits while it decides who is liable for a payment an agent authorizes.
Charles Schwab
BlocksThe Verge reported that Charles Schwab is blocking AI agents from client accounts. CEO Rick Wurster said the firm will not allow a client to give Meta's Muse a password and let Muse act for that client.
Royal Bank of Canada
Blocksblock third-party personal AI agents from direct access to authenticated RBC digital properties, including online banking.
Bank of Montreal
BlocksBank of Montreal has blocked third-party AI agents from accessing its digital properties.
TD
BlocksTD will block all third-party personal AI agents from direct access to banking information where authentication is required.
CIBC
LimitsCIBC said it does not "support personal AI agents directly opening or accessing accounts."
CIBC said it does not support this access. The Globe and Mail said the bank did not confirm a full block.
Scotiabank
LimitsScotiabank said it will limit third-party AI agents from its banking platforms while it monitors concerns.
Amazon
BlocksGeekWire reported that Muse shoppers on Amazon saw a notice that continued access by an unauthorized AI agent violates Amazon's Conditions of Use, and that Amazon is suing Perplexity over its Comet browser and moving to block shopping agents from Google and OpenAI.
Walmart
LimitsWalmart said Agent Profiles, launching in the next month, let a customer authorize each AI agent with its own profile and either a spending limit or approval on every purchase, and that agents never receive payment-card data.
This is a limit on an authorized agent. The tracker does not record a Walmart terms-of-use ban on every agent.
Delta
BlocksDelta told TechCrunch it has no partnership or integration that lets a third-party AI agent shop for or book Delta flights on Delta's sites. Delta said it is protecting customers from unauthorized automated activity, and that any opening to outside agents would have to keep security and the customer experience in mind.
Questions people ask
Which big retailers block AI shopping agents?
On October 28, 2026, 6 readable files fully disallow at least one tracked agent: Amazon (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent); Instacart (PerplexityBot, Perplexity-User, Meta-ExternalAgent); Chewy (Amazonbot); Temu (Bytespider); Lululemon (Meta-ExternalAgent); IKEA (Amazonbot). Amazon is the only readable file that fully disallows all three user-triggered tokens (ChatGPT-User, Claude-User, and Perplexity-User). ChatGPT-User, Claude-User, and Perplexity-User are the tokens those products publish for a fetch a person asked for. Amazonbot is Amazon's crawler, not a shopper. A full disallow is a request, not enforcement. An agent can ignore the file, and a site can stop an agent without naming it here. Partly means the file names the agent and the rules are not a full-site disallow. The table shows those cells. That is not a full block. 29 readable files name none of them, including Walmart, Target, Etsy, and 26 others. Not named is not permission. The policy section records separate limits that companies published about agents. Those lines are not robots.txt results.
Every site · Policy limits · Amazon robots.txt · Instacart robots.txt · Chewy robots.txt · Temu robots.txt · Lululemon robots.txt · IKEA robots.txt
Does robots.txt actually stop an AI agent?
No. robots.txt is a request to well-behaved crawlers, not enforcement and not a lock. A site can stop an agent without saying so in the file, and an agent can ignore the file. This check does not browse stores, place carts, or test whether a disallow is enforced.
Which banks block AI agents from logging in?
These are account-access limits already published on this tracker, with the date of each news item. They are not robots.txt results. Wells Fargo (October 23, 2026): USA Today reported that a Wells Fargo spokesperson said the bank has taken steps to block unauthorized agents from customer accounts. Its updated online access agreement, effective November 24, 2026, says a customer who permits an AI agent is solely responsible for supervising it, and that the agent's transactions, including ones the customer did not want, count as authorized. Morgan Stanley (October 22, 2026): Axios reported that Morgan Stanley has begun blocking third-party AI agents from its platforms, and that a Morgan Stanley spokesperson said the bank had not seen rogue agents there. Bank of America (October 19, 2026): Bank of America is limiting AI agents' access to consumer deposits while it decides who is liable for a payment an agent authorizes. Citi (October 19, 2026): Citi does not allow a third-party AI agent to log into a customer's account or make a transaction in Citi's digital banking. Customers can still share limited read-only account data with some open-banking providers. Charles Schwab (October 16, 2026): The Verge reported that Charles Schwab is blocking AI agents from client accounts. CEO Rick Wurster said the firm will not allow a client to give Meta's Muse a password and let Muse act for that client. JPMorgan Chase (October 13, 2026): A spokeswoman told CNBC the bank blocks those agents from operating for retail customers, because logging in with the customer's own credentials is not safe. Open-banking connections can still share data such as a balance, but they cannot send money or pay a bill.
Wells Fargo, October 23, 2026 · Source: USA Today · Morgan Stanley, October 22, 2026 · Source: Axios · Bank of America, October 19, 2026 · Source: The Paypers · Citi, October 19, 2026 · Source: The Paypers · Charles Schwab, October 16, 2026 · Source: The Verge · JPMorgan Chase, October 13, 2026 · Source: CNBC
How this check works
- Each row is a reading of one public robots.txt, fetched from that site's host at /robots.txt.
- An agent matches when its product token appears, compared without regard to case, as RFC 9309 describes. GPTBot/1.0 still matches GPTBot. A different token, such as GPTBot-User, does not.
- User-agent: * is recorded on its own. If an agent is not named, the * rules are not copied onto that agent.
- Blocked fully means the agent's group disallows the whole site (Disallow: /, Disallow: /*, Disallow: /*$, or Disallow: *) and contains no Allow rule.
- Partly means the agent is named and the rules disallow only some paths, or they disallow the whole site and also allow something. The rules are listed so a reader can see the mix. A full disallow paired with an allow is not called a full block.
- Allowed means the agent is named and no path is disallowed. Crawl-delay is not a block and is not listed.
- Not mentioned means the file does not name that agent. That is not permission. The * group may still apply, and a site can block an agent without naming it.
- robots.txt is a request to well-behaved crawlers. It is not access control. This check does not browse stores, place carts, or test whether a block is enforced. It does not read robots meta tags or login walls.
- Google-Extended is not Google Search. Applebot-Extended is not Applebot. Amazonbot is Amazon's crawler, not a shopper. ChatGPT-User, Claude-User, and Perplexity-User are the tokens those companies publish for a fetch a person asked the product to make. An agent that sends an ordinary browser token will not appear here.
- Meta-ExternalAgent is matched case-insensitively, so a line written meta-externalagent counts.
- If the first request was refused, timed out, or came back as HTML, the check retried once with a common browser user agent. Some storefronts hide robots.txt from clients they do not recognize. The row says when that retry was used. A successful retry is still the public file.
- When a group has many rules, the stored list is a sample. The sample keeps any whole-site rule, one allow, and one specific disallow. ruleCount is the full number of rules. The label uses every rule.
The check is rerun weekly. This page, the JSON, and the CSV all read that result. Free to use with a link back. CC BY 4.0.