robots.txt — any AI bot blocked
per site 47.4% — 238 of 502 publishers.
Sites whose robots.txt disallows at least one of the 63 tracked AI/scraper user-agents at the site root. The "UAs blocked" column shows how many — see the per-row list for which.
| Publisher | Country | UAs blocked | Blocked UAs |
|---|---|---|---|
| Times Group Malawi | Malawi (MW) | 63 | AI2Bot, AliyunSecBot, Amazonbot, Applebot-Extended, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, GoogleOther, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, bingbot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Mwebantu | Zambia (ZM) | 61 | AI2Bot, AliyunSecBot, Amazonbot, Applebot-Extended, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, GoogleOther, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Ouest-France | France (FR) | 60 | AI2Bot, AliyunSecBot, Amazonbot, Applebot-Extended, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Adelaide Advertiser | Australia (AU) | 59 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Courier-Mail | Australia (AU) | 59 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Daily Telegraph | Australia (AU) | 59 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Herald Sun | Australia (AU) | 59 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| news.com.au | Australia (AU) | 59 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Reuters Top News | United Kingdom (GB) | 58 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| The Globe and Mail | Canada (CA) | 58 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, GoogleOther, Grok, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| The Sun | United Kingdom (GB) | 57 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| The Times | United Kingdom (GB) | 57 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Diario | Mexico (MX) | 56 | AI2Bot, AliyunSecBot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| The Australian | Australia (AU) | 56 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, Gemini-Deep-Research, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| The Wall Street Journal | United States (US) | 56 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, Gemini-Deep-Research, Google-Extended, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Iltalehti | Finland (FI) | 54 | AI2Bot, AliyunSecBot, Amazonbot, Applebot-Extended, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, cohere-ai, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| USA Today | United States (US) | 54 | AI2Bot, Amazonbot, Applebot-Extended, AudigentAdBot, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, GoogleOther, Grok, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Scrapy, SeekrBot, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Folha de S.Paulo | Brazil (BR) | 52 | AI2Bot, AliyunSecBot, Amazonbot, Applebot-Extended, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, FriendlyCrawler, Grok, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, bingbot, cohere-ai, cohere-training-data-crawler, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| NY Times | United States (US) | 50 | AliyunSecBot, Applebot-Extended, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| CNN | United States (US) | 49 | AI2Bot, AliyunSecBot, Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, ImagesiftBot, MistralAI-User, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, img2dataset, magpie-crawler, news-please, omgili, omgilibot, quillbot.com |
| El Debate | Mexico (MX) | 49 | AI2Bot, AliyunSecBot, Amazonbot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, Claude-User, Claude-Web, DataForSeoBot, DeepSeekBot, Diffbot, EchoboxBot, Feedfetcher-Google, FriendlyCrawler, Gemini-Deep-Research, Google-CloudVertexBot, Grok, ImagesiftBot, Jetslide, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, NewsNow, PanguBot, Perplexity-User, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Sydney Morning Herald | Australia (AU) | 48 | AliyunSecBot, Amazonbot, Applebot-Extended, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, cohere-ai, ia_archiver, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| The Age | Australia (AU) | 48 | AliyunSecBot, Amazonbot, Applebot-Extended, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, cohere-ai, ia_archiver, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| VnExpress (English) | Vietnam (VN) | 48 | AliyunSecBot, Applebot-Extended, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Jetslide, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, archive.org_bot, bingbot, cohere-ai, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler, quillbot.com |
| Haaretz | Israel (IL) | 46 | AliyunSecBot, Amazonbot, Applebot-Extended, AudigentAdBot, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, EchoboxBot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MyCentralAIScraperBot, NewsNow, OAI-SearchBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Quora-Bot, Scrapy, SeekrBot, SeznamHomepageCrawler, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, ViennaTinyBot, YouBot, anthropic-ai, cohere-ai, news-please, omgili, omgilibot, peer39_crawler |
| The Conversation (AU) | Australia (AU) | 43 | AI2Bot, Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, GoogleOther, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Scrapy, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot, quillbot.com |
| i | United Kingdom (GB) | 42 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, GoogleOther, Grok, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Scrapy, SeekrBot, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, img2dataset, omgili, omgilibot |
| Les Echos | France (FR) | 42 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, DuckAssistBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, SeekrBot, Timpibot, TurnitinBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, omgili, omgilibot |
| Metro | United Kingdom (GB) | 42 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, GoogleOther, Grok, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Scrapy, SeekrBot, TaraGroup Intelligent Bot, Timpibot, TurnitinBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, img2dataset, omgili, omgilibot |
| Heise Online | Germany (DE) | 39 | AI2Bot, Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, GoogleOther, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, NewsNow, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Timpibot, TurnitinBot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, magpie-crawler, omgili, omgilibot |
| Le Monde | France (FR) | 37 | AI2Bot, Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, DuckAssistBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, GoogleOther, ImagesiftBot, Meta-ExternalFetcher, MistralAI-User, NewsNow, PanguBot, Timpibot, TurnitinBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, magpie-crawler, omgili, omgilibot |
| Toronto Star | Canada (CA) | 37 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, DuckAssistBot, FacebookBot, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, GoogleOther, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Scrapy, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| Deutsche Welle | Germany (DE) | 36 | AI2Bot, Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, NewsNow, OAI-SearchBot, Perplexity-User, PerplexityBot, Scrapy, Timpibot, TurnitinBot, YouBot, anthropic-ai, cohere-ai, ia_archiver, img2dataset, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler |
| Iza | Japan (JP) | 36 | AI2Bot, Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, NewsNow, OAI-SearchBot, PanguBot, PerplexityBot, Scrapy, SeekrBot, Timpibot, TurnitinBot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, magpie-crawler, omgili, omgilibot |
| Nettavisen | Norway (NO) | 36 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, FriendlyCrawler, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, GoogleOther, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Scrapy, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| The Telegraph | United Kingdom (GB) | 36 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, DuckAssistBot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, GoogleOther, Grok, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Scrapy, SeekrBot, Timpibot, anthropic-ai, cohere-ai, img2dataset, omgili, omgilibot |
| zakzak | Japan (JP) | 36 | AI2Bot, Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, NewsNow, OAI-SearchBot, PanguBot, PerplexityBot, Scrapy, SeekrBot, Timpibot, TurnitinBot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, magpie-crawler, omgili, omgilibot |
| Frankfurter Rundschau | Germany (DE) | 35 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, Google-CloudVertexBot, Google-Extended, GoogleOther, Grok, ImagesiftBot, Meta-ExternalAgent, PanguBot, PerplexityBot, Scrapy, Timpibot, TurnitinBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, omgili, omgilibot, peer39_crawler |
| NRK | Norway (NO) | 35 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, DuckAssistBot, EchoboxBot, FacebookBot, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, NewsNow, OAI-SearchBot, PanguBot, PerplexityBot, Scrapy, SeekrBot, anthropic-ai, cohere-ai, news-please, omgili, omgilibot, peer39_crawler |
| Vox | United States (US) | 35 | AI2Bot, Applebot-Extended, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, DuckAssistBot, FacebookBot, FriendlyCrawler, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Jetslide, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, Perplexity-User, PerplexityBot, Scrapy, Timpibot, TurnitinBot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, news-please, omgili, omgilibot |
| Helsingin Sanomat | Finland (FI) | 34 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Quora-Bot, Timpibot, YouBot, anthropic-ai, cohere-ai, img2dataset, omgili, omgilibot |
| Ilta-Sanomat | Finland (FI) | 34 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Quora-Bot, Timpibot, YouBot, anthropic-ai, cohere-ai, img2dataset, omgili, omgilibot |
| Neue Zürcher Zeitung | Switzerland (CH) | 34 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, NewsNow, OAI-SearchBot, Perplexity-User, PerplexityBot, Quora-Bot, Scrapy, Timpibot, TurnitinBot, anthropic-ai, cohere-ai, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler |
| New York Post | United States (US) | 32 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, NewsNow, PerplexityBot, Scrapy, SeekrBot, TurnitinBot, anthropic-ai, archive.org_bot, cohere-ai, ia_archiver, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler |
| Nikkei Shimbun (日本経済新聞) | Japan (JP) | 32 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, Scrapy, SeekrBot, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, magpie-crawler, omgili, omgilibot |
| UOL | Brazil (BR) | 32 | AI2Bot, Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, FacebookBot, Gemini-Deep-Research, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, PanguBot, Perplexity-User, PerplexityBot, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, omgili, omgilibot |
| Õhtuleht | Estonia (EE) | 32 | AI2Bot, Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, GPTBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, NewsNow, OAI-SearchBot, PanguBot, PerplexityBot, SeekrBot, Timpibot, TurnitinBot, YouBot, anthropic-ai, bingbot, cohere-ai, cohere-training-data-crawler, omgili, omgilibot |
| Bild | Germany (DE) | 31 | AI2Bot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, FriendlyCrawler, ImagesiftBot, MistralAI-User, MyCentralAIScraperBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Scrapy, Timpibot, TurnitinBot, YouBot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, omgili, omgilibot, quillbot.com |
| Die Welt | Germany (DE) | 31 | AI2Bot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, FriendlyCrawler, ImagesiftBot, MistralAI-User, MyCentralAIScraperBot, PanguBot, Perplexity-User, PerplexityBot, Poseidon Research Crawler, Scrapy, Timpibot, TurnitinBot, YouBot, cohere-ai, cohere-training-data-crawler, ia_archiver, img2dataset, magpie-crawler, omgili, omgilibot, quillbot.com |
| New Zealand Herald | New Zealand (NZ) | 31 | AI2Bot, AliyunSecBot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, Feedfetcher-Google, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, Scrapy, TurnitinBot, anthropic-ai, archive.org_bot, cohere-ai, ia_archiver, img2dataset, omgili, omgilibot, quillbot.com |
| Sankei Shimbun (産経新聞) | Japan (JP) | 31 | AI2Bot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Scrapy, Timpibot, TurnitinBot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, magpie-crawler, omgili, omgilibot |
| La Repubblica | Italy (IT) | 30 | Applebot-Extended, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, DataForSeoBot, Diffbot, DuckAssistBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Google-Extended, ImagesiftBot, Meta-ExternalFetcher, PanguBot, PerplexityBot, Quora-Bot, Scrapy, Timpibot, TurnitinBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, ia_archiver, magpie-crawler, omgilibot, peer39_crawler |
| La Stampa | Italy (IT) | 30 | Applebot-Extended, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, DataForSeoBot, Diffbot, DuckAssistBot, FacebookBot, Feedfetcher-Google, FriendlyCrawler, GPTBot, Google-Extended, ImagesiftBot, Meta-ExternalFetcher, PanguBot, PerplexityBot, Quora-Bot, Scrapy, Timpibot, TurnitinBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, ia_archiver, magpie-crawler, omgilibot, peer39_crawler |
| The Atlantic | United States (US) | 30 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, NewsNow, Perplexity-User, PerplexityBot, Quora-Bot, Scrapy, Timpibot, TurnitinBot, anthropic-ai, archive.org_bot, cohere-ai, magpie-crawler, news-please, omgili, omgilibot |
| France24 | France (FR) | 29 | AI2Bot, BLEXBot, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, DuckAssistBot, Feedfetcher-Google, GPTBot, Google-Extended, Grok, Meta-ExternalFetcher, MistralAI-User, MyCentralAIScraperBot, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot, quillbot.com |
| Frankfurter Allgemeine | Germany (DE) | 29 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, DuckAssistBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, MistralAI-User, NewsNow, OAI-SearchBot, PanguBot, PerplexityBot, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, omgili, omgilibot |
| Asahi Shimbun (朝日新聞) | Japan (JP) | 28 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, Scrapy, Timpibot, anthropic-ai, cohere-ai, cohere-training-data-crawler, magpie-crawler, omgili, omgilibot |
| Chicago Tribune | United States (US) | 28 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, MistralAI-User, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, Timpibot, YouBot, anthropic-ai, archive.org_bot, cohere-training-data-crawler, omgili, omgilibot |
| CNBC | United States (US) | 28 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, NewsNow, OAI-SearchBot, PerplexityBot, Scrapy, TurnitinBot, anthropic-ai, cohere-ai, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler |
| Le Soir | Belgium (BE) | 28 | AI2Bot, Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, Feedfetcher-Google, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, NewsNow, OAI-SearchBot, Perplexity-User, PerplexityBot, TurnitinBot, anthropic-ai, cohere-ai, ia_archiver |
| Yomiuri Shimbun (読売新聞) | Japan (JP) | 28 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, GoogleOther, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, Scrapy, Timpibot, anthropic-ai, cohere-ai, cohere-training-data-crawler, ia_archiver, magpie-crawler, omgili, omgilibot |
| Dagens Nyheter | Sweden (SE) | 27 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, Perplexity-User, PerplexityBot, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, omgili, omgilibot |
| De Telegraaf | Netherlands (NL) | 27 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, FacebookBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, Perplexity-User, PerplexityBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, magpie-crawler, omgili, omgilibot |
| Expressen | Sweden (SE) | 27 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, Perplexity-User, PerplexityBot, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, omgili, omgilibot |
| Independent | Ireland (IE) | 27 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, FacebookBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, Perplexity-User, PerplexityBot, YouBot, anthropic-ai, archive.org_bot, cohere-ai, magpie-crawler, omgili, omgilibot |
| Postimees | Estonia (EE) | 27 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, DuckAssistBot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, PanguBot, PerplexityBot, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, omgili, omgilibot |
| Yahoo! News | United States (US) | 27 | AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, ImagesiftBot, NewsNow, PerplexityBot, Scrapy, SeekrBot, TurnitinBot, YouBot, anthropic-ai, cohere-ai, img2dataset, magpie-crawler, news-please, omgili, omgilibot |
| Dainik Bhaskar | India (IN) | 26 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, GoogleOther, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| El Universo | Ecuador (EC) | 26 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, Google-Extended, NewsNow, PerplexityBot, Quora-Bot, Scrapy, TurnitinBot, anthropic-ai, cohere-ai, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler |
| Jiji Press | Japan (JP) | 26 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, Scrapy, Timpibot, anthropic-ai, cohere-ai, cohere-training-data-crawler, magpie-crawler, omgili, omgilibot |
| Kyodo News (English) | Japan (JP) | 26 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, Scrapy, Timpibot, anthropic-ai, cohere-ai, cohere-training-data-crawler, magpie-crawler, omgili, omgilibot |
| Mainichi Shimbun (毎日新聞) | Japan (JP) | 26 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, PerplexityBot, Scrapy, Timpibot, anthropic-ai, cohere-ai, cohere-training-data-crawler, magpie-crawler, omgili, omgilibot |
| Yahoo! News Japan | Japan (JP) | 26 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, Scrapy, Timpibot, anthropic-ai, cohere-ai, cohere-training-data-crawler, magpie-crawler, omgili, omgilibot |
| Yle (Finnish Broadcasting Company) | Finland (FI) | 26 | AI2Bot, Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, Grok, Meta-ExternalAgent, PanguBot, Scrapy, Timpibot, TurnitinBot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| Yonhap News Agency | South Korea (KR) | 26 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, ImagesiftBot, Meta-ExternalAgent, OAI-SearchBot, PerplexityBot, Scrapy, Timpibot, YouBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| Aftenposten | Norway (NO) | 25 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, Claude-SearchBot, Claude-Web, ClaudeBot, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, ImagesiftBot, Meta-ExternalAgent, PanguBot, PerplexityBot, Scrapy, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| Aftonbladet | Sweden (SE) | 25 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, Claude-SearchBot, Claude-Web, ClaudeBot, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, ImagesiftBot, Meta-ExternalAgent, PanguBot, PerplexityBot, Scrapy, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| Delfi | Estonia (EE) | 25 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, PanguBot, PerplexityBot, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, omgili, omgilibot |
| Huffington Post | United States (US) | 25 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, Timpibot, TurnitinBot, anthropic-ai, cohere-ai, magpie-crawler, omgilibot |
| La Presse | Canada (CA) | 25 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, TurnitinBot, YouBot, anthropic-ai, archive.org_bot, ia_archiver, magpie-crawler |
| Le Figaro | France (FR) | 25 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, Feedfetcher-Google, GPTBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, Meta-ExternalFetcher, MistralAI-User, NewsNow, OAI-SearchBot, Scrapy, TurnitinBot, YouBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| O Globo | Brazil (BR) | 25 | AI2Bot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DeepSeekBot, DuckAssistBot, FacebookBot, GPTBot, Google-Extended, Grok, MistralAI-User, MyCentralAIScraperBot, OAI-SearchBot, PanguBot, Perplexity-User, PerplexityBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgilibot |
| Sudan Akhbar | Sudan (SD) | 25 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, Meta-ExternalAgent, OAI-SearchBot, Perplexity-User, PerplexityBot, Scrapy, TurnitinBot, YouBot, anthropic-ai, cohere-ai, magpie-crawler, omgili, omgilibot |
| Svenska Dagbladet | Sweden (SE) | 25 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, Claude-SearchBot, Claude-Web, ClaudeBot, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, ImagesiftBot, Meta-ExternalAgent, PanguBot, PerplexityBot, Scrapy, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| VG | Norway (NO) | 25 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, Claude-SearchBot, Claude-Web, ClaudeBot, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, ImagesiftBot, Meta-ExternalAgent, PanguBot, PerplexityBot, Scrapy, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| Egemen Qazaqstan | Kazakhstan (KZ) | 24 | Amazonbot, Applebot-Extended, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, OAI-SearchBot, PerplexityBot, Timpibot, YouBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| Ekstra Bladet | Denmark (DK) | 24 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DataForSeoBot, DeepSeekBot, Diffbot, DuckAssistBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, Timpibot, YouBot, anthropic-ai |
| Gazeta Wyborcza | Poland (PL) | 24 | Applebot-Extended, BLEXBot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, DataForSeoBot, FriendlyCrawler, GPTBot, Gemini-Deep-Research, ImagesiftBot, PerplexityBot, Poseidon Research Crawler, Scrapy, SeznamHomepageCrawler, TurnitinBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, ia_archiver, magpie-crawler, omgili, omgilibot, peer39_crawler |
| La Nación (Costa Rica) | Costa Rica (CR) | 24 | AliyunSecBot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, CCBot, Claude-SearchBot, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, DuckAssistBot, GPTBot, ImagesiftBot, OAI-SearchBot, Scrapy, SeekrBot, anthropic-ai, archive.org_bot, cohere-ai, news-please, omgili, omgilibot |
| RAI News | Italy (IT) | 24 | Amazonbot, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, Google-Extended, NewsNow, PerplexityBot, Scrapy, TurnitinBot, anthropic-ai, cohere-ai, magpie-crawler, news-please, omgili, omgilibot, peer39_crawler |
| Sky News | United Kingdom (GB) | 24 | AI2Bot, Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, NewsNow, PanguBot, PerplexityBot, Scrapy, TurnitinBot, anthropic-ai, cohere-ai, magpie-crawler, news-please, omgili, omgilibot |
| Spiegel Online (German) | Germany (DE) | 24 | Applebot-Extended, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, ImagesiftBot, Meta-ExternalAgent, OAI-SearchBot, Timpibot, TurnitinBot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, ia_archiver, magpie-crawler, omgili, omgilibot |
| SVT Nyheter | Sweden (SE) | 24 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, DeepSeekBot, Diffbot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, PanguBot, Timpibot, YouBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| Η Καθημερινή | Greece (GR) | 24 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, FacebookBot, GPTBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, NewsNow, PerplexityBot, YouBot, anthropic-ai, cohere-ai, magpie-crawler, omgili, omgilibot, peer39_crawler |
| BBC News | United Kingdom (GB) | 23 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, OAI-SearchBot, Perplexity-User, PerplexityBot, Scrapy, TurnitinBot, YouBot, anthropic-ai, cohere-ai, magpie-crawler, omgili, omgilibot |
| El Tiempo | Colombia (CO) | 23 | AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, Feedfetcher-Google, GPTBot, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, PerplexityBot, TurnitinBot, anthropic-ai, archive.org_bot, magpie-crawler, news-please, omgili, omgilibot |
| L'Express | France (FR) | 23 | AI2Bot, Amazonbot, Applebot-Extended, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, DuckAssistBot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, NewsNow, OAI-SearchBot, PerplexityBot, YouBot, anthropic-ai, cohere-ai, ia_archiver |
| La Stampa | Italy (IT) | 23 | Applebot-Extended, AwarioRssBot, AwarioSmartBot, BLEXBot, Bytespider, DataForSeoBot, Diffbot, DuckAssistBot, Feedfetcher-Google, FriendlyCrawler, ImagesiftBot, Meta-ExternalFetcher, PanguBot, PerplexityBot, Quora-Bot, Scrapy, Timpibot, TurnitinBot, YouBot, archive.org_bot, ia_archiver, magpie-crawler, peer39_crawler |
| Nederlands Dagblad | Netherlands (NL) | 23 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, MyCentralAIScraperBot, OAI-SearchBot, PerplexityBot, Scrapy, anthropic-ai, cohere-ai, img2dataset, omgili, omgilibot |
| The Phnom Penh Post | Cambodia (KH) | 23 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, Google-Extended, GoogleOther, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, PerplexityBot, Timpibot, TurnitinBot, YouBot, anthropic-ai, cohere-ai, omgilibot |
| Wired | United States (US) | 23 | Amazonbot, Applebot-Extended, Bytespider, CCBot, Claude-SearchBot, Claude-User, ClaudeBot, Diffbot, DuckAssistBot, Google-CloudVertexBot, Google-Extended, GoogleOther, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User, PanguBot, Perplexity-User, PerplexityBot, Timpibot, archive.org_bot, cohere-ai, cohere-training-data-crawler, ia_archiver |
| Bloomberg | United States (US) | 22 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, PerplexityBot, YouBot, anthropic-ai, cohere-ai, peer39_crawler |
| Die Presse | Austria (AT) | 22 | Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, Google-Extended, ImagesiftBot, PerplexityBot, YouBot, anthropic-ai, cohere-ai, magpie-crawler, omgili, omgilibot, peer39_crawler |
| 15min | Lithuania (LT) | 21 | AI2Bot, Amazonbot, Applebot-Extended, BLEXBot, Bytespider, CCBot, Diffbot, DuckAssistBot, FriendlyCrawler, ImagesiftBot, PanguBot, Perplexity-User, PerplexityBot, Scrapy, Timpibot, YouBot, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| Buzzfeed | United States (US) | 21 | Amazonbot, Applebot-Extended, Bytespider, CCBot, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, Perplexity-User, PerplexityBot, Timpibot, TurnitinBot, anthropic-ai, cohere-ai, magpie-crawler, omgilibot |
| RPP Noticias | Peru (PE) | 21 | Amazonbot, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, Google-Extended, NewsNow, Scrapy, TurnitinBot, YouBot, anthropic-ai, cohere-ai, magpie-crawler, news-please, omgili, omgilibot |
| Washington Post | United States (US) | 21 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, PerplexityBot, anthropic-ai, archive.org_bot, cohere-training-data-crawler, ia_archiver, omgili, omgilibot |
| de Volkskrant | Netherlands (NL) | 20 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, GPTBot, Google-Extended, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, Perplexity-User, PerplexityBot, Timpibot, YouBot, anthropic-ai, cohere-ai |
| France Télévisions (franceinfo) | France (FR) | 20 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, MistralAI-User, OAI-SearchBot, Perplexity-User, PerplexityBot, anthropic-ai, cohere-ai, cohere-training-data-crawler |
| La Razón | Spain (ES) | 20 | AI2Bot, Applebot-Extended, BLEXBot, Bytespider, CCBot, ClaudeBot, Diffbot, FacebookBot, Feedfetcher-Google, GPTBot, Meta-ExternalAgent, Meta-ExternalFetcher, PanguBot, Timpibot, TurnitinBot, anthropic-ai, cohere-training-data-crawler, magpie-crawler, omgili, omgilibot |
| News24 | South Africa (ZA) | 20 | Amazonbot, Applebot-Extended, BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FacebookBot, GPTBot, Google-Extended, PerplexityBot, Scrapy, YouBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| The Guardian | United Kingdom (GB) | 20 | Amazonbot, Applebot-Extended, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, ClaudeBot, DuckAssistBot, FacebookBot, Google-CloudVertexBot, ImagesiftBot, Meta-ExternalAgent, NewsNow, PerplexityBot, SeekrBot, TurnitinBot, YouBot, anthropic-ai |
| Trouw | Netherlands (NL) | 20 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DeepSeekBot, Diffbot, GPTBot, Google-Extended, Meta-ExternalFetcher, MistralAI-User, OAI-SearchBot, Perplexity-User, PerplexityBot, Timpibot, YouBot, anthropic-ai, cohere-ai |
| ANSA | Italy (IT) | 19 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, GPTBot, Google-CloudVertexBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, PerplexityBot, SeekrBot, YouBot, anthropic-ai, cohere-ai, omgilibot |
| De Standaard | Belgium (BE) | 19 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, PerplexityBot, anthropic-ai, cohere-ai, magpie-crawler, omgili, omgilibot |
| Die Zeit | Germany (DE) | 19 | Applebot-Extended, Bytespider, CCBot, ClaudeBot, DeepSeekBot, Diffbot, FacebookBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, PanguBot, Perplexity-User, PerplexityBot, Timpibot, anthropic-ai, cohere-training-data-crawler, img2dataset, quillbot.com |
| El Pais | Spain (ES) | 19 | Amazonbot, BLEXBot, Bytespider, CCBot, Claude-SearchBot, Claude-User, ClaudeBot, DeepSeekBot, DuckAssistBot, Feedfetcher-Google, Meta-ExternalFetcher, MistralAI-User, Perplexity-User, PerplexityBot, TurnitinBot, archive.org_bot, ia_archiver, magpie-crawler, omgilibot |
| Financial Times | United Kingdom (GB) | 19 | Applebot-Extended, Bytespider, CCBot, Claude-Web, ClaudeBot, Diffbot, FacebookBot, Google-Extended, GoogleOther, Meta-ExternalAgent, Meta-ExternalFetcher, NewsNow, Perplexity-User, PerplexityBot, anthropic-ai, cohere-ai, magpie-crawler, omgili, omgilibot |
| Het Nieuwsblad | Belgium (BE) | 19 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, PerplexityBot, anthropic-ai, cohere-ai, magpie-crawler, omgili, omgilibot |
| Il Messaggero | Italy (IT) | 19 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Diffbot, GPTBot, Google-Extended, ImagesiftBot, OAI-SearchBot, PerplexityBot, SeekrBot, YouBot, anthropic-ai, cohere-ai, omgilibot |
| Luxemburger Wort | Luxembourg (LU) | 19 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, PerplexityBot, anthropic-ai, cohere-ai, magpie-crawler, omgili, omgilibot |
| ABC News Australia | Australia (AU) | 18 | BLEXBot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, GPTBot, Google-Extended, Meta-ExternalAgent, PerplexityBot, Timpibot, TurnitinBot, YouBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| Channel 4 News | United Kingdom (GB) | 18 | Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DuckAssistBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, OAI-SearchBot, PerplexityBot, TurnitinBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| Liberation | France (FR) | 18 | AI2Bot, Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, DuckAssistBot, FacebookBot, GPTBot, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, PerplexityBot, anthropic-ai, cohere-ai |
| News18 | India (IN) | 18 | Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, FriendlyCrawler, GPTBot, Google-Extended, ImagesiftBot, OAI-SearchBot, Scrapy, Timpibot, cohere-ai, img2dataset, omgili, omgilibot |
| Stuff.co.nz | New Zealand (NZ) | 18 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, PerplexityBot, Timpibot, YouBot, omgili, omgilibot |
| 20minutes | France (FR) | 17 | Applebot-Extended, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Feedfetcher-Google, GPTBot, Google-Extended, Meta-ExternalFetcher, MistralAI-User, NewsNow, OAI-SearchBot, PerplexityBot, TurnitinBot, anthropic-ai, cohere-ai, ia_archiver |
| Asia-Plus | Tajikistan (TJ) | 17 | Amazonbot, Applebot-Extended, BLEXBot, Bytespider, CCBot, Claude-Web, ClaudeBot, DataForSeoBot, Diffbot, GPTBot, Google-Extended, ImagesiftBot, Meta-ExternalAgent, Timpibot, anthropic-ai, cohere-ai, omgilibot |
| Huffington Post UK | United Kingdom (GB) | 17 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, ClaudeBot, DuckAssistBot, FacebookBot, GPTBot, Meta-ExternalAgent, Meta-ExternalFetcher, OAI-SearchBot, Perplexity-User, PerplexityBot, Timpibot, omgilibot |
| RTV SLO | Slovenia (SI) | 17 | Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, GPTBot, Google-CloudVertexBot, OAI-SearchBot, Perplexity-User, PerplexityBot, Scrapy, anthropic-ai, cohere-ai, magpie-crawler, omgili, omgilibot |
| Forbes | United States (US) | 16 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, FacebookBot, FriendlyCrawler, GPTBot, ImagesiftBot, Meta-ExternalAgent, PerplexityBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| Gazeta.PL | Poland (PL) | 16 | BLEXBot, Bytespider, CCBot, DataForSeoBot, FriendlyCrawler, GPTBot, Gemini-Deep-Research, ImagesiftBot, Poseidon Research Crawler, Scrapy, SeznamHomepageCrawler, TurnitinBot, anthropic-ai, cohere-training-data-crawler, magpie-crawler, peer39_crawler |
| Le Parisien | France (FR) | 16 | Amazonbot, CCBot, Claude-Web, ClaudeBot, FacebookBot, Feedfetcher-Google, GPTBot, Meta-ExternalFetcher, NewsNow, PerplexityBot, TurnitinBot, anthropic-ai, cohere-ai, ia_archiver, omgili, omgilibot |
| NPR | United States (US) | 16 | Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, OAI-SearchBot, PerplexityBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| Times Live | South Africa (ZA) | 16 | Amazonbot, Applebot-Extended, Bytespider, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, ImagesiftBot, PerplexityBot, YouBot, anthropic-ai, omgili, omgilibot |
| Indian Express | India (IN) | 15 | Applebot-Extended, Bytespider, Claude-Web, ClaudeBot, Diffbot, FriendlyCrawler, Meta-ExternalAgent, Meta-ExternalFetcher, PerplexityBot, Timpibot, anthropic-ai, cohere-ai, img2dataset, omgili, omgilibot |
| Lidové noviny | Czechia (CZ) | 15 | Amazonbot, Applebot-Extended, Bytespider, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, ImagesiftBot, PerplexityBot, YouBot, anthropic-ai, omgili, omgilibot |
| Mladá fronta DNES (MF DNES) | Czechia (CZ) | 15 | Amazonbot, Applebot-Extended, Bytespider, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, ImagesiftBot, PerplexityBot, YouBot, anthropic-ai, omgili, omgilibot |
| ORF Nachrichten/News | Austria (AT) | 15 | Bytespider, CCBot, Claude-Web, ClaudeBot, Diffbot, FacebookBot, GPTBot, Google-Extended, PerplexityBot, YouBot, anthropic-ai, cohere-ai, magpie-crawler, omgili, omgilibot |
| Politiken | Denmark (DK) | 15 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-Web, DeepSeekBot, DuckAssistBot, GPTBot, Google-Extended, Timpibot, YouBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| tagesschau.de | Germany (DE) | 15 | Applebot-Extended, Bytespider, CCBot, ClaudeBot, DeepSeekBot, Diffbot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, PanguBot, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| Westdeutscher Rundfunk (WDR) | Germany (DE) | 15 | Applebot-Extended, Bytespider, CCBot, ClaudeBot, DeepSeekBot, Diffbot, FacebookBot, GPTBot, Google-Extended, Meta-ExternalAgent, PanguBot, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| Amar Ujala | India (IN) | 14 | Amazonbot, Applebot-Extended, Bytespider, CCBot, Claude-Web, ClaudeBot, Diffbot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, anthropic-ai, cohere-ai, omgili, omgilibot |
| El Mundo | Spain (ES) | 14 | BLEXBot, CCBot, ChatGPT-User, Feedfetcher-Google, GPTBot, Google-Extended, Meta-ExternalFetcher, OAI-SearchBot, PerplexityBot, TurnitinBot, anthropic-ai, ia_archiver, magpie-crawler, omgilibot |
| Radio Canada FR | Canada (CA) | 14 | Bytespider, CCBot, ChatGPT-User, Claude-Web, DeepSeekBot, GPTBot, Google-Extended, OAI-SearchBot, Perplexity-User, PerplexityBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| The Hindu | India (IN) | 14 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, FacebookBot, GPTBot, Meta-ExternalAgent, OAI-SearchBot, PerplexityBot, YouBot, anthropic-ai, ia_archiver |
| TV2 | Norway (NO) | 14 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, Diffbot, GPTBot, PerplexityBot, anthropic-ai, cohere-ai, omgili, omgilibot |
| ABC | Spain (ES) | 13 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, ClaudeBot, GPTBot, Google-Extended, OAI-SearchBot, PerplexityBot, archive.org_bot, ia_archiver |
| ABC.es | Spain (ES) | 13 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, ClaudeBot, GPTBot, Google-Extended, OAI-SearchBot, PerplexityBot, archive.org_bot, ia_archiver |
| Corriere della Sera | Italy (IT) | 13 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-CloudVertexBot, Google-Extended, Meta-ExternalAgent, Scrapy, Timpibot, YouBot, anthropic-ai |
| Der Standard | Austria (AT) | 13 | CCBot, ChatGPT-User, Claude-Web, ClaudeBot, FacebookBot, GPTBot, Google-Extended, OAI-SearchBot, PerplexityBot, anthropic-ai, ia_archiver, omgili, omgilibot |
| El Correo | Spain (ES) | 13 | Amazonbot, Applebot-Extended, Bytespider, CCBot, Claude-SearchBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent, OAI-SearchBot, PerplexityBot, archive.org_bot, ia_archiver |
| Islamic Emirate of Afghanistan - Alemarah | Afghanistan (AF) | 13 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent, anthropic-ai |
| Liberty Times (自由時報) | Taiwan (TW) | 13 | AI2Bot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| Nový Čas | Slovakia (SK) | 13 | Amazonbot, Bytespider, CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, DeepSeekBot, GPTBot, Google-Extended, MistralAI-User, OAI-SearchBot, Perplexity-User, PerplexityBot |
| Taipei Times | Taiwan (TW) | 13 | AI2Bot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, anthropic-ai, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| The Economist | United Kingdom (GB) | 13 | Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, GPTBot, Google-Extended, Perplexity-User, PerplexityBot, TurnitinBot, anthropic-ai, ia_archiver, magpie-crawler |
| Actu Cameroun | Cameroon (CM) | 12 | BLEXBot, Bytespider, CCBot, DataForSeoBot, Feedfetcher-Google, Meta-ExternalFetcher, NewsNow, Scrapy, TurnitinBot, img2dataset, omgili, omgilibot |
| Associated Press | United States (US) | 12 | Amazonbot, Applebot-Extended, CCBot, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot, PerplexityBot, Timpibot, anthropic-ai, cohere-ai |
| El Nacional | Dominican Republic (DO) | 12 | BLEXBot, CCBot, Claude-Web, ClaudeBot, Feedfetcher-Google, GPTBot, Meta-ExternalFetcher, TurnitinBot, anthropic-ai, ia_archiver, magpie-crawler, omgilibot |
| NHK News Web | Japan (JP) | 12 | Applebot-Extended, Bytespider, CCBot, ChatGPT-User, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent, OAI-SearchBot, PerplexityBot, anthropic-ai, cohere-ai |
| NHK World English | Japan (JP) | 12 | Applebot-Extended, Bytespider, CCBot, ChatGPT-User, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent, OAI-SearchBot, PerplexityBot, anthropic-ai, cohere-ai |
| The Week | United Kingdom (GB) | 12 | AI2Bot, Amazonbot, Bytespider, Diffbot, Meta-ExternalAgent, MistralAI-User, YouBot, cohere-ai, cohere-training-data-crawler, img2dataset, omgili, omgilibot |
| aktuality.sk | Slovakia (SK) | 11 | Amazonbot, Bytespider, DeepSeekBot, DuckAssistBot, Scrapy, Timpibot, YouBot, cohere-ai, img2dataset, omgili, omgilibot |
| Dagbladet | Norway (NO) | 11 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent, PerplexityBot, anthropic-ai, cohere-ai |
| FOCUS Online | Germany (DE) | 10 | Amazonbot, Bytespider, CCBot, ClaudeBot, FacebookBot, FriendlyCrawler, GPTBot, Google-CloudVertexBot, Meta-ExternalAgent, Timpibot |
| Hürriyet | Türkiye (TR) | 10 | ChatGPT-User, Claude-Web, ClaudeBot, GPTBot, OAI-SearchBot, PerplexityBot, TurnitinBot, anthropic-ai, cohere-ai, quillbot.com |
| ITV News | United Kingdom (GB) | 10 | Bytespider, CCBot, Claude-Web, ClaudeBot, Scrapy, anthropic-ai, cohere-ai, magpie-crawler, omgili, omgilibot |
| La Vanguardia | Spain (ES) | 10 | Bytespider, CCBot, Claude-Web, ClaudeBot, Feedfetcher-Google, GPTBot, Meta-ExternalFetcher, YouBot, anthropic-ai, ia_archiver |
| La Vanguardia | Spain (ES) | 10 | Bytespider, CCBot, Claude-Web, ClaudeBot, Feedfetcher-Google, GPTBot, Meta-ExternalFetcher, YouBot, anthropic-ai, ia_archiver |
| The Daily Mirror / Sunday Mirror | United Kingdom (GB) | 10 | Applebot-Extended, CCBot, Claude-Web, ClaudeBot, GPTBot, Meta-ExternalAgent, OAI-SearchBot, PerplexityBot, YouBot, anthropic-ai |
| The Scotsman | United Kingdom (GB) | 10 | Applebot-Extended, Bytespider, Claude-Web, ClaudeBot, GPTBot, Google-Extended, OAI-SearchBot, Perplexity-User, PerplexityBot, anthropic-ai |
| CBC News | Canada (CA) | 9 | CCBot, ChatGPT-User, Claude-Web, DeepSeekBot, GPTBot, OAI-SearchBot, PerplexityBot, anthropic-ai, cohere-ai |
| Süddeutsche Zeitung | Germany (DE) | 9 | ChatGPT-User, Claude-Web, ClaudeBot, GPTBot, Google-Extended, ImagesiftBot, PerplexityBot, anthropic-ai, cohere-ai |
| The Korea Times | South Korea (KR) | 9 | Amazonbot, AwarioRssBot, AwarioSmartBot, Bytespider, DataForSeoBot, magpie-crawler, omgili, omgilibot, peer39_crawler |
| Al Bawaba | Jordan (JO) | 8 | Amazonbot, CCBot, GPTBot, TurnitinBot, archive.org_bot, ia_archiver, omgili, omgilibot |
| Al Jazeera English | Qatar (QA) | 8 | Bytespider, ChatGPT-User, Claude-Web, ClaudeBot, GPTBot, PerplexityBot, anthropic-ai, cohere-ai |
| Brújula Digital | Bolivia (BO) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| CamboJA News | Cambodia (KH) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| CTV News | Canada (CA) | 8 | CCBot, ChatGPT-User, Claude-Web, DeepSeekBot, GPTBot, OAI-SearchBot, PerplexityBot, anthropic-ai |
| Dagblad Suriname | Suriname (SR) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| De Ware Tijd | Suriname (SR) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| DVB (Democratic Voice of Burma, English) | Myanmar (MM) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| Enab Baladi (English) | Syria (SY) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| Estado de São Paulo | Brazil (BR) | 8 | Bytespider, CCBot, ChatGPT-User, ClaudeBot, GPTBot, OAI-SearchBot, PerplexityBot, anthropic-ai |
| Expreso | Ecuador (EC) | 8 | BLEXBot, CCBot, Feedfetcher-Google, Meta-ExternalFetcher, TurnitinBot, ia_archiver, magpie-crawler, omgilibot |
| Iraqi News | Iraq (IQ) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| Islamic Emirate of Afghanistan - Alemarah (English) | Afghanistan (AF) | 8 | CCBot, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot, anthropic-ai |
| New Telegraph | Nigeria (NG) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| Novinite (Sofia News Agency) | Bulgaria (BG) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| Petra (Jordan News Agency) | Jordan (JO) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| Tchadinfos | Chad (TD) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| The Daily Star (Bangladesh) | Bangladesh (BD) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| Xalq So'zi (Narodnoe Slovo) | Uzbekistan (UZ) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| Äripäev | Estonia (EE) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| Дневник | Bulgaria (BG) | 8 | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, Meta-ExternalAgent |
| Axios | United States (US) | 7 | Amazonbot, Bytespider, CCBot, Diffbot, FacebookBot, ImagesiftBot, Scrapy |
| stern.de | Germany (DE) | 7 | Applebot-Extended, CCBot, ChatGPT-User, Diffbot, GPTBot, Google-Extended, Scrapy |
| Arzuw News | Turkmenistan (TM) | 6 | BLEXBot, Claude-Web, ClaudeBot, DataForSeoBot, FacebookBot, Meta-ExternalAgent |
| Neue Vorarlberger Tageszeitung (NEUE) | Austria (AT) | 6 | BLEXBot, CCBot, ChatGPT-User, GPTBot, Google-Extended, ia_archiver |
| Khaleej Times | United Arab Emirates (AE) | 5 | Applebot-Extended, ClaudeBot, Gemini-Deep-Research, Google-Extended, anthropic-ai |
| 即時/娛樂 (United Daily News) | Taiwan (TW) | 5 | Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot |
| Correio da Manhã | Portugal (PT) | 4 | CCBot, ChatGPT-User, GPTBot, Google-Extended |
| Jornal de Negócios | Portugal (PT) | 4 | CCBot, ChatGPT-User, GPTBot, Google-Extended |
| PBS NewsHour | United States (US) | 4 | Bytespider, CCBot, GPTBot, PerplexityBot |
| RTVE | Spain (ES) | 4 | Feedfetcher-Google, ImagesiftBot, Meta-ExternalFetcher, TurnitinBot |
| Blesk.cz | Czechia (CZ) | 3 | BLEXBot, Bytespider, DataForSeoBot |
| Euronews | France (FR) | 3 | CCBot, GPTBot, Google-Extended |
| Free Malaysia Today | Malaysia (MY) | 3 | Amazonbot, Bytespider, CCBot |
| n-tv | Germany (DE) | 3 | CCBot, GPTBot, Google-Extended |
| RTL News | Germany (DE) | 3 | CCBot, GPTBot, Google-Extended |
| Tageblatt | Luxembourg (LU) | 3 | CCBot, ChatGPT-User, GPTBot |
| ThePrint | India (IN) | 3 | CCBot, GPTBot, Google-Extended |
| Verslo žinios | Lithuania (LT) | 3 | CCBot, GPTBot, Google-Extended |
| Berlingske | Denmark (DK) | 2 | CCBot, GPTBot |
| BT | Denmark (DK) | 2 | CCBot, GPTBot |
| Chosun Ilbo | South Korea (KR) | 2 | DeepSeekBot, GPTBot |
| Criterio.hn | Honduras (HN) | 2 | Diffbot, SeznamHomepageCrawler |
| Daily Graphic | Ghana (GH) | 2 | archive.org_bot, ia_archiver |
| El Universal (Spanish) | Mexico (MX) | 2 | Feedfetcher-Google, Meta-ExternalFetcher |
| Irish Times | Ireland (IE) | 2 | ChatGPT-User, GPTBot |
| La Patilla | Venezuela (VE) | 2 | ClaudeBot, GPTBot |
| Lusa Agência de Notícias de Portugal | Portugal (PT) | 2 | ChatGPT-User, GPTBot |
| National Post | Canada (CA) | 2 | omgili, omgilibot |
| taz | Germany (DE) | 2 | Bytespider, GPTBot |
| 24 часа | Bulgaria (BG) | 1 | Scrapy |
| Antara News | Indonesia (ID) | 1 | ClaudeBot |
| ERR | Estonia (EE) | 1 | GPTBot |
| Granma (English) | Cuba (CU) | 1 | CCBot |
| Hospodářské noviny | Czechia (CZ) | 1 | GPTBot |
| Japan Today | Japan (JP) | 1 | ia_archiver |
| L'Internaute | France (FR) | 1 | DataForSeoBot |
| Philippine Daily Inquirer | Philippines (PH) | 1 | GPTBot |
| Prensa Libre | Guatemala (GT) | 1 | ia_archiver |
| Rio Times | Brazil (BR) | 1 | CCBot |
| Stirile Pro TV | Romania (RO) | 1 | GPTBot |
| The Star | Malaysia (MY) | 1 | Diffbot |
| Times of India | India (IN) | 1 | Meta-ExternalAgent |