Crawl Activity
SEO Compare — seocompare.co.uk
Bing crawl ran 3m ago · 21 Jun, 06:01 · runs daily 06:00
Cloudflare AI Crawl Control (14d)
Edge-level bot counts, true hits — not sampled
6,623
13 bots tracked
06-05 06-12
ClaudeBot 1,445
Bingbot 1,125
Amazonbot 985
ChatGPT-User 845
Googlebot 649
4315
Total Bing Crawls
396
Pages Indexed (Bing)
13
AI Bots Allowed
0
AI Bots Blocked
Crawler Activity
Bing (bars, left axis) vs AI bots (lines, right axis)
AI Crawler Access
GPTBot allowed
ChatGPT-User allowed
Google-Extended allowed
GoogleOther allowed
ClaudeBot allowed
Claude-Web not_mentioned
PerplexityBot allowed
Bytespider allowed
CCBot allowed
cohere-ai allowed
Amazonbot allowed
anthropic-ai not_mentioned
FacebookBot allowed
Applebot-Extended allowed
Bingbot allowed
Server-Side Bot Hits
Real crawler visits detected at the server level (last 30 days)
17897
Total Bot Hits
4211
AI Crawler Hits
5395
Search Crawler Hits
22
Unique Bots
Bot Breakdown
Other Bot 3374
Bingbot 2363
SemrushBot 2162
ChatGPT-User 1388
YandexBot 1382
ClaudeBot 1229
Googlebot 1163
Other Crawler 806
MajesticBot 644
Other Spider 631
AhrefsBot 588
Meta-Agent 527
GPTBot 468
Applebot 460
PerplexityBot 430
Daily Hits
21 Jun
28 AI 36 search 3 seo 134
20 Jun
48 AI 107 search 8 social 201 seo 664
19 Jun
149 AI 131 search 3 social 30 seo 477
18 Jun
297 AI 154 search 3 social 166 seo 1092
17 Jun
153 AI 127 search 4 social 20 seo 1009
16 Jun
53 AI 197 search 5 social 32 seo 833
15 Jun
65 AI 162 search 8 social 37 seo 425
14 Jun
124 AI 137 search 1 social 401 seo 690
13 Jun
124 AI 142 search 2 social 58 seo 463
12 Jun
177 AI 96 search 2 social 20 seo 391
11 Jun
193 AI 111 search 2 social 26 seo 425
10 Jun
228 AI 128 search 18 seo 782
9 Jun
345 AI 137 search 1 social 34 seo 656
8 Jun
104 AI 110 search 7 social 337 seo 720
Live Crawler Access Test (HTTP Response)
Tests if each bot actually gets a 200 response when visiting the homepage
GPTBot
ClaudeBot
PerplexityBot
Googlebot
Bingbot
Topic Clusters (220 detected)
/research/
4
pages in cluster
/ai-search-questions/
1
pages in cluster
/what-is-ai-search-optimisation/
1
pages in cluster
/ai-seo-solicitors/
1
pages in cluster
/ai-seo-private-healthcare/
1
pages in cluster
/ai-seo-law-firms-uk/
1
pages in cluster
/insights/
8
pages in cluster
/pure-ai-search-agencies/
1
pages in cluster
/best-ai-seo-audit-services/
1
pages in cluster
/ai-search-visibility/
1
pages in cluster
/ai-search-agencies-london/
1
pages in cluster
/ai-seo-audit-birmingham/
1
pages in cluster
/best-ai-website-builders-for-ecommerce-uk/
1
pages in cluster
/how-to-get-cited-ai-overviews/
1
pages in cluster
/affordable-ai-seo-uk/
1
pages in cluster
/ai-search-agencies-chelmsford/
1
pages in cluster
/ai-search-agencies-enterprise/
1
pages in cluster
/best-geo-agencies-b2b/
1
pages in cluster
/ai-seo-b2b-guide/
1
pages in cluster
/ai-seo-tools-saas/
1
pages in cluster
/b2b-ai-overviews/
1
pages in cluster
/chatgpt-vs-perplexity-vs-gemini/
1
pages in cluster
/contact/
1
pages in cluster
/ai-seo-law-firms/
1
pages in cluster
/tools/
5
pages in cluster
/best-llmo-agencies/
1
pages in cluster
/best-ai-seo-agencies-healthcare/
1
pages in cluster
/ai-seo-agencies-usa/
1
pages in cluster
/ai-seo-recruitment-agencies/
1
pages in cluster
/ai-seo-audit-newcastle/
1
pages in cluster
/ai-search-agencies-cardiff/
1
pages in cluster
/ai-search-agencies-edinburgh/
1
pages in cluster
/best-seo-agencies-uk-2026/
1
pages in cluster
/ai-search-agencies-canada/
1
pages in cluster
/show-on-ai-search/
1
pages in cluster
/privacy/
1
pages in cluster
/ai-search-agencies-saas/
1
pages in cluster
/getcito-review/
1
pages in cluster
/ai-seo-audit-sheffield/
1
pages in cluster
/otterly-ai-review/
1
pages in cluster
/disclaimer/
1
pages in cluster
/ai-search-agencies-birmingham/
1
pages in cluster
/agency-comparison-matrix/
1
pages in cluster
/how-to-rank-b2b-chatgpt/
1
pages in cluster
/ai-seo-audit-chelmsford/
1
pages in cluster
/chatgpt-seo-healthcare/
1
pages in cluster
/best-perplexity-seo-agencies/
1
pages in cluster
/ai-seo-ecommerce-guide/
1
pages in cluster
/ai-seo-tools-uk/
1
pages in cluster
/geo-for-healthcare/
1
pages in cluster
/best-ai-seo-agencies-saas/
1
pages in cluster
/ai-seo-audit-oxford/
1
pages in cluster
/best-ai-search-website-builders-uk/
1
pages in cluster
/geo-for-law-firms/
1
pages in cluster
/authors/
1
pages in cluster
/agencies-with-case-studies/
1
pages in cluster
/ai-seo-audit-brighton/
1
pages in cluster
/ai-seo-vs-geo-vs-aeo-agencies/
1
pages in cluster
/find-an-agency/
1
pages in cluster
/chatgpt-seo-financial/
1
pages in cluster
/chatgpt-seo-law-firms/
1
pages in cluster
/ai-seo-audit-cambridge/
1
pages in cluster
/saas-ai-overviews/
1
pages in cluster
/ai-seo-audit-cardiff/
1
pages in cluster
/ai-search-agencies-europe/
1
pages in cluster
/ai-seo-tools-ecommerce/
1
pages in cluster
/ai-seo-tools-b2b/
1
pages in cluster
/how-we-are-funded/
1
pages in cluster
/ai-seo-dentists-uk/
1
pages in cluster
/ai-search-agencies-brighton/
1
pages in cluster
/best-ai-visibility-agencies/
1
pages in cluster
/best-ai-seo-agencies-b2b/
1
pages in cluster
/methodology/
1
pages in cluster
/best-geo-agencies/
1
pages in cluster
/ai-seo-ifa-financial-advisors/
1
pages in cluster
/best-free-ai-website-builders-uk/
1
pages in cluster
/ai-search-agencies-southampton/
1
pages in cluster
/site-index/
1
pages in cluster
/chatgpt-seo-ecommerce/
1
pages in cluster
/best-aeo-agencies-ecommerce/
1
pages in cluster
/ai-seo-agencies-saas-uk/
1
pages in cluster
/schema-markup-ai-search/
1
pages in cluster
/terms/
1
pages in cluster
/ai-seo-audit-southampton/
1
pages in cluster
/peec-ai-review/
1
pages in cluster
/best-aeo-agencies-uk/
1
pages in cluster
/ai-seo-saas-vs-b2b/
1
pages in cluster
/ai-seo-healthcare/
1
pages in cluster
/ai-search-hospitality/
1
pages in cluster
/best/
16
pages in cluster
/ai-seo-professional-services/
1
pages in cluster
/ai-seo-nhs/
1
pages in cluster
/agencies-offering-audits/
1
pages in cluster
/ai-seo-audit-leeds/
1
pages in cluster
/about/
1
pages in cluster
/get-found-on-chatgpt/
1
pages in cluster
/best-chatgpt-seo-agencies/
1
pages in cluster
/ai-search-statistics-2026/
1
pages in cluster
/ai-search-agencies-liverpool/
1
pages in cluster
/best-ai-seo-agencies-uk/
1
pages in cluster
/ai-seo-banking/
1
pages in cluster
/ai-citation-gaps/
1
pages in cluster
/ai-search-agencies-leeds/
1
pages in cluster
/blog/
45
pages in cluster
/ai-search-agencies-colchester/
1
pages in cluster
/semrush-vs-ahrefs-ai-visibility/
1
pages in cluster
/how-to-rank-ecommerce-chatgpt/
1
pages in cluster
/chatgpt-seo-agency-uk/
1
pages in cluster
/ai-seo-agency-manchester/
1
pages in cluster
/ai-seo-vs-traditional-seo/
1
pages in cluster
/ai-seo-shopify/
1
pages in cluster
/best-ai-seo-companies/
1
pages in cluster
/ai-overviews-agencies/
1
pages in cluster
/ai-seo-audit-glasgow/
1
pages in cluster
/ai-search-agencies-oxford/
1
pages in cluster
/ai-seo-audit-nottingham/
1
pages in cluster
/best-aeo-agencies/
1
pages in cluster
/uk-ai-overviews-agencies/
1
pages in cluster
/how-to-get-cited-gemini/
1
pages in cluster
/chatgpt-optimisation-agencies/
1
pages in cluster
/ai-visibility-agency-uk/
1
pages in cluster
/ai-seo-agencies-ecommerce-uk/
1
pages in cluster
/chatgpt-visibility/
1
pages in cluster
/best-ai-search-tools/
1
pages in cluster
/agency-directory/
1
pages in cluster
/ai-instructions/
1
pages in cluster
/aeo-vs-geo-vs-llmo/
1
pages in cluster
/se-ranking-ai-tracking-review/
1
pages in cluster
/ai-search-agencies-middle-east/
1
pages in cluster
/ai-seo-barristers/
1
pages in cluster
/ai-seo-audit/
1
pages in cluster
/ai-search-agencies-uk/
1
pages in cluster
/best-llmo-agencies-2026/
1
pages in cluster
/build-brand-ai-trust/
1
pages in cluster
/ai-seo-dental/
1
pages in cluster
/guides/
1
pages in cluster
/ai-seo-audit-london/
1
pages in cluster
/best-geo-agencies-2026/
1
pages in cluster
/editorial-policy/
1
pages in cluster
/ai-search-agencies-reading/
1
pages in cluster
/structure-website-ai-crawlers/
1
pages in cluster
/ai-seo-audit-colchester/
1
pages in cluster
/affordable-ai-seo-ecommerce/
1
pages in cluster
/ai-seo-manufacturing/
1
pages in cluster
/best-geo-agencies-ecommerce/
1
pages in cluster
/ai-search-agencies-norwich/
1
pages in cluster
/ai-search-agencies-asia/
1
pages in cluster
/ai-visibility-check/
1
pages in cluster
/ai-search-local-business/
1
pages in cluster
/ai-search-agencies-nottingham/
1
pages in cluster
/how-to-get-cited-perplexity/
1
pages in cluster
/best-ai-search-agencies/
1
pages in cluster
/ai-search-agencies-ecommerce/
1
pages in cluster
/ai-seo-financial-uk/
1
pages in cluster
/compare/
67
pages in cluster
/agency-vs-tool/
1
pages in cluster
/how-to-rank-law-firm-chatgpt/
1
pages in cluster
/ai-search-agencies-sheffield/
1
pages in cluster
/ai-seo-travel-tourism-uk/
1
pages in cluster
/affordable-ai-seo-agencies/
1
pages in cluster
/best-geo-agencies-uk/
1
pages in cluster
/ai-seo-linkedin-b2b/
1
pages in cluster
/ai-seo-audit-bristol/
1
pages in cluster
/ai-search-agencies-newcastle/
1
pages in cluster
/chatgpt-seo-b2b/
1
pages in cluster
/ai-seo-london/
1
pages in cluster
/ai-seo-audit-edinburgh/
1
pages in cluster
/perplexity-optimisation-agencies/
1
pages in cluster
/best-ai-overviews-agencies-2026/
1
pages in cluster
/ai-search-agencies-small-business/
1
pages in cluster
/ai-search-agencies-b2b/
1
pages in cluster
/best-geo-agencies-saas/
1
pages in cluster
/ai-seo-audit-norwich/
1
pages in cluster
/uk-ai-seo-agency-market-benchmarks-2026/
1
pages in cluster
/ai-seo-audit-reading/
1
pages in cluster
/restaurant-hospitality-ai-search-guide/
1
pages in cluster
/ai-seo-agency-london/
1
pages in cluster
/ai-search-agencies-australia/
1
pages in cluster
/ai-seo-agencies-b2b-uk/
1
pages in cluster
/best-ai-seo-agencies/
1
pages in cluster
/how-to-get-cited-chatgpt/
1
pages in cluster
/ai-seo-insurance/
1
pages in cluster
/best-gemini-seo-agencies/
1
pages in cluster
/ai-seo-saas-guide/
1
pages in cluster
/ai-search-agencies-bristol/
1
pages in cluster
/ai-seo-healthcare-uk/
1
pages in cluster
/ai-seo-agency-uk/
1
pages in cluster
/alternatives/
13
pages in cluster
/geo-for-fintech/
1
pages in cluster
/geo-vs-seo/
1
pages in cluster
/ai-visibility-score/
1
pages in cluster
/ai-seo-audit-manchester/
1
pages in cluster
/ai-search-agencies-cambridge/
1
pages in cluster
/accessibility/
1
pages in cluster
/data/
4
pages in cluster
/ai-seo-hotels-accommodation/
1
pages in cluster
/best-ai-seo-agencies-ecommerce/
1
pages in cluster
/ai-search-agencies-glasgow/
1
pages in cluster
/ai-search-glossary/
1
pages in cluster
/ai-seo-manchester/
1
pages in cluster
/ai-search-optimisation-cost/
1
pages in cluster
/ai-seo-recruitment/
1
pages in cluster
/ai-seo-manufacturing-companies/
1
pages in cluster
/ai-seo-audit-liverpool/
1
pages in cluster
/free-ai-visibility-audit/
1
pages in cluster
/ai-seo-financial-services/
1
pages in cluster
/how-to-get-cited-claude/
1
pages in cluster
/stats/
23
pages in cluster
/chatgpt-seo-agencies-uk/
1
pages in cluster
/chatgpt-seo-saas/
1
pages in cluster
/ai-search-restaurants-uk/
1
pages in cluster
/ai-search-manufacturing-companies/
1
pages in cluster
/ecommerce-ai-overviews/
1
pages in cluster
/questions/
69
pages in cluster
/ai-seo-product-led-growth/
1
pages in cluster
/best-ai-seo-agencies-financial/
1
pages in cluster
/best-ai-seo-agencies-law-firms/
1
pages in cluster
/ai-search-agencies-manchester/
1
pages in cluster
/best-aeo-agencies-2026/
1
pages in cluster
/gemini-optimisation-agencies/
1
pages in cluster
Terminology Consistency (H1 Analysis)
2026
x (%)
search
x (%)
agencies
x (%)
seocompare
x (%)
best
x (%)
chatgpt
x (%)
agency
x (%)
visibility
x (%)
side
x (%)
platform
x (%)
perplexity
x (%)
coverage
x (%)
optimisation
x (%)
compare
x (%)
compared
x (%)