Table of Contents
Je vais créer le guide complet sur la détection de bots, les logs et les exports pour AISEOmatic AI.
AISEOmatic Bot Detection — Logs, KPIs & Exports Guide #
🤖 Complete Guide to Bot Activity Monitoring & Analysis #
Master bot detection, understand AI crawler behavior, and leverage bot activity data for SEO insights.
This comprehensive guide covers bot detection technology, activity logs, analytics dashboards, KPI tracking, and how to use bot data to optimize your AI-first SEO strategy.
📋 Table of Contents #
- Bot Detection Overview
- Supported Bot Types
- Bot Detection Dashboard
- Activity Logs & Monitoring
- Bot Analytics & KPIs
- Export Formats
- Bot Behavior Patterns
- Crawler Management
- Troubleshooting & Verification
- Advanced Bot Intelligence
1. Bot Detection Overview #
What is Bot Detection? #
AISEOmatic AI’s bot detection identifies and logs visits from search engine crawlers and AI system bots.
Purpose:
- 📊 Track which AI engines are discovering your content
- 📈 Measure AI-first SEO effectiveness
- 🎯 Optimize content for specific AI platforms
- 📉 Monitor crawler behavior and patterns
- 🔍 Verify sitemap effectiveness
How Bot Detection Works #
Detection Process:
Incoming Request
↓
┌─────────────────────────────────────┐
│ 1. User Agent Analysis │
│ Check HTTP User-Agent string │
└─────────────────────────────────────┘
↓
┌─────────────────────────────────────┐
│ 2. IP Verification │
│ Validate against known bot ranges │
└─────────────────────────────────────┘
↓
┌─────────────────────────────────────┐
│ 3. Reverse DNS Lookup │
│ Confirm bot identity │
└─────────────────────────────────────┘
↓
┌─────────────────────────────────────┐
│ 4. Behavioral Analysis │
│ Check crawl patterns │
└─────────────────────────────────────┘
↓
┌─────────────────────────────────────┐
│ 5. Log & Classify │
│ Store visit data in database │
└─────────────────────────────────────┘Lite vs. Pro Detection #
Lite Version:
✅ Basic bot detection (real-time)
✅ Live bot counter in dashboard
❌ No historical logs
❌ No detailed analytics
❌ No exports
❌ Limited to last 24 hoursPro Version:
✅ Full bot detection & logging
✅ 1,000+ entry historical logs
✅ Detailed analytics dashboard
✅ CSV/JSON exports
✅ Bot behavior analysis
✅ Crawler ETA predictions
✅ Advanced filtering
✅ Unlimited retention2. Supported Bot Types #
AI Search Engine Bots #
Actively Detected AI Crawlers:
🤖 GPTBot (OpenAI/ChatGPT)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: GPTBot/1.0
Purpose: Content indexing for ChatGPT
Frequency: Daily to weekly
Respect: robots.txt, AI Sitemap
🤖 ClaudeBot (Anthropic/Claude)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: ClaudeBot/1.0
Purpose: Content discovery for Claude
Frequency: 2-3 times per week
Respect: robots.txt, AI Sitemap
🤖 PerplexityBot (Perplexity AI)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: PerplexityBot/1.0
Purpose: Real-time search indexing
Frequency: Very active (multiple daily)
Respect: robots.txt
🤖 Google-Extended (Google Bard/Gemini)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: Google-Extended
Purpose: AI model training & responses
Frequency: Varies with Googlebot
Respect: robots.txt
🤖 Amazonbot (Amazon Alexa)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: Amazonbot/1.0
Purpose: Alexa voice responses
Frequency: Weekly
Respect: robots.txt
🤖 Applebot-Extended (Apple Intelligence)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: Applebot-Extended/1.0
Purpose: Apple AI services
Frequency: Moderate
Respect: robots.txt
🤖 Bytespider (TikTok)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: Bytespider/1.0
Purpose: Content discovery
Frequency: Variable
Respect: robots.txt
🤖 Meta-ExternalAgent (Meta AI)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: Meta-ExternalAgent/1.0
Purpose: Meta AI training
Frequency: Occasional
Respect: robots.txtTraditional Search Bots #
Classic Search Engine Crawlers:
🔍 Googlebot
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: Googlebot/2.1
Purpose: Google Search indexing
Frequency: Very high (multiple times daily)
Priority: Critical for SEO
🔍 Bingbot
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: bingbot/2.0
Purpose: Bing Search indexing
Frequency: Daily to weekly
Priority: Important
🔍 Yandexbot
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: YandexBot/3.0
Purpose: Yandex Search (Russia)
Frequency: Moderate
Priority: Regional
🔍 DuckDuckBot
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: DuckDuckBot/1.0
Purpose: DuckDuckGo indexing
Frequency: Weekly
Priority: Privacy-focused users
🔍 Baiduspider
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User Agent: Baiduspider/2.0
Purpose: Baidu Search (China)
Frequency: Moderate
Priority: Chinese market3. Bot Detection Dashboard #
Dashboard Layout #
┌─────────────────────────────────────────────────────────┐
│ 🤖 AISEOmatic AI Bot Detection │
├─────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Total Bots │ │ AI Crawlers │ │ Last 24h │ │
│ │ Detected │ │ Active │ │ Activity │ │
│ │ │ │ │ │ │ │
│ │ 2,847 │ │ 8 │ │ 234 │ │
│ │ All Time │ │ Different │ │ Visits │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ 📊 Bot Activity by Type (Last 30 Days) │ │
│ │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │ │
│ │ │ │
│ │ GPTBot ████████████ 456 visits │ │
│ │ Googlebot █████████████████ 892 visits │ │
│ │ ClaudeBot ████████ 287 visits │ │
│ │ PerplexityBot ███████ 234 visits │ │
│ │ Bingbot ██████ 178 visits │ │
│ │ Google-Ext. ████ 123 visits │ │
│ │ Amazonbot ██ 67 visits │ │
│ │ Other ███ 89 visits │ │
│ │ │ │
│ └───────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ 🎯 Most Visited Pages (AI Bots) │ │
│ │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │ │
│ │ 1. /wordpress-guide/ 89 visits (Score:97)│ │
│ │ 2. /seo-checklist/ 76 visits (Score:96)│ │
│ │ 3. /ai-optimization/ 67 visits (Score:95)│ │
│ │ 4. /local-seo-guide/ 54 visits (Score:94)│ │
│ │ 5. /content-strategy/ 45 visits (Score:92)│ │
│ └───────────────────────────────────────────────────┘ │
│ │
│ [View Full Logs] [Export CSV] [Export JSON] │
│ │
└─────────────────────────────────────────────────────────┘Real-Time Bot Counter #
Live Activity Widget:
┌─────────────────────────────────────┐
│ 🔴 Live Bot Activity │
├─────────────────────────────────────┤
│ │
│ Last 10 Minutes: │
│ │
│ 🤖 GPTBot │
│ /wordpress-guide/ │
│ 2 minutes ago │
│ │
│ 🔍 Googlebot │
│ /seo-checklist/ │
│ 4 minutes ago │
│ │
│ 🤖 ClaudeBot │
│ /ai-optimization/ │
│ 7 minutes ago │
│ │
│ [View All Recent Activity] │
│ │
└─────────────────────────────────────┘4. Activity Logs & Monitoring #
Bot Visit Log Table #
┌──────────────────────────────────────────────────────────────────┐
│ Bot Activity Log (Last 1,000 Entries) │
├──────────────────────────────────────────────────────────────────┤
│ Date/Time Bot Page IP Code │
├──────────────────────────────────────────────────────────────────┤
│ 2025-12-13 10:47 GPTBot /guide/ 66.249.*.* 200 │
│ 2025-12-13 10:45 Googlebot /home/ 66.249.*.* 200 │
│ 2025-12-13 10:42 ClaudeBot /about/ 52.24.*.* 200 │
│ 2025-12-13 10:38 PerplexBot /products/ 3.16.*.* 200 │
│ 2025-12-13 10:35 Bingbot /blog/ 157.55.*.* 200 │
│ 2025-12-13 10:31 GPTBot /faq/ 66.249.*.* 200 │
│ 2025-12-13 10:28 Google-Ext /services/ 66.249.*.* 200 │
│ 2025-12-13 10:24 ClaudeBot /contact/ 52.24.*.* 200 │
│ ... │
├──────────────────────────────────────────────────────────────────┤
│ Showing 1-10 of 1,000 entries │
│ [Previous] [1] [2] [3] ... [100] [Next] │
└──────────────────────────────────────────────────────────────────┘
[Filter] [Search] [Export] [Clear Old Logs]Log Entry Details #
Complete Log Entry Structure:
┌─────────────────────────────────────────┐
│ Bot Visit Details │
├─────────────────────────────────────────┤
│ │
│ 🤖 Bot Information │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │
│ Name: GPTBot │
│ Type: AI Crawler │
│ Category: AI Search Engine │
│ Owner: OpenAI │
│ User Agent: GPTBot/1.0 (+https://...) │
│ │
│ 📍 Visit Details │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │
│ Date/Time: 2025-12-13 10:47:23 UTC │
│ URL: /wordpress-guide/ │
│ Page Title: Complete WordPress Guide │
│ AI Score: 97/100 │
│ Response Code: 200 OK │
│ Response Time: 234ms │
│ │
│ 🌐 Technical Details │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │
│ IP Address: 66.249.64.12 │
│ Reverse DNS: crawl.openai.com │
│ Request Method: GET │
│ HTTP Version: HTTP/2.0 │
│ Referrer: (Direct) │
│ │
│ 📊 Context │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │
│ Previous Visit: 3 days ago │
│ Total Visits: 47 (all time) │
│ Pages Visited: 23 different pages │
│ Avg. Crawl Interval: 2.8 days │
│ │
└─────────────────────────────────────────┘Filtering & Search #
Filter Options:
┌─────────────────────────────────────┐
│ Filter Bot Logs │
├─────────────────────────────────────┤
│ │
│ Bot Type: │
│ ☐ All │
│ ☑ AI Crawlers Only │
│ ☐ Traditional Bots Only │
│ │
│ Specific Bots: │
│ ☑ GPTBot │
│ ☑ ClaudeBot │
│ ☑ PerplexityBot │
│ ☐ Googlebot │
│ ☐ Bingbot │
│ ☐ Other │
│ │
│ Date Range: │
│ From: [2025-12-01] │
│ To: [2025-12-13] │
│ │
│ Page/URL: │
│ [/wordpress-guide/ ] │
│ │
│ Response Code: │
│ [All Codes ▼] │
│ │
│ [Apply Filters] [Reset] │
│ │
└─────────────────────────────────────┘5. Bot Analytics & KPIs #
Key Performance Indicators #
Primary Bot KPIs:
1. AI Bot Discovery Rate
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Current: 8 different AI bots detected
Target: 10+ (comprehensive coverage)
Trend: ↗ +2 new bots this quarter
Metric Breakdown:
├─ GPTBot: ✅ Active
├─ ClaudeBot: ✅ Active
├─ PerplexityBot: ✅ Active
├─ Google-Extended: ✅ Active
├─ Amazonbot: ✅ Active
├─ AppleBot-Extended: ⚠️ Rare
├─ Bytespider: ⚠️ Occasional
└─ Meta-External: ❌ Not yet detected
2. Crawl Frequency
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
AI Bots Average: 2.3 visits/week/bot
Traditional: 5.7 visits/week/bot
Combined: 456 total visits/month
By Bot:
├─ Googlebot: Daily (1.2/day)
├─ GPTBot: 2.1/week
├─ ClaudeBot: 1.8/week
├─ PerplexityBot: 3.4/week (very active!)
└─ Bingbot: 1.1/week
3. Page Coverage
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total Pages: 150
Pages Visited by AI Bots: 127 (85%)
Never Visited: 23 (15%)
Avg. Visits per Page: 3.2
Priority Pages Coverage:
├─ High-scoring (90+): 100% ✅
├─ Medium-scoring (70-89): 87% ✅
└─ Low-scoring (<70): 45% ⚠️
4. Response Quality
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Successful Crawls (200): 98.7%
Not Found (404): 0.8%
Server Errors (5xx): 0.5%
Average Response Time: 187ms
Target: >99% success rate
Status: ⚠️ Minor optimization neededBot Activity Trends #
Weekly Trend Analysis:
┌─────────────────────────────────────────┐
│ Bot Activity Trend (12 Weeks) │
├─────────────────────────────────────────┤
│ 500│ ╱─ │
│ 450│ ╱───╯ │
│ 400│ ╱───╯ │
│ 350│ ╱───╯ │
│ 300│ ╱───╯ │
│ 250│ ╱───╯ │
│ 200│──╯ │
│ └────────────────────────────────── │
│ W1 W3 W5 W7 W9 W11 │
│ │
│ Total Growth: +156% in 12 weeks │
│ AI Bots: +234% (outpacing traditional)│
│ Conclusion: Strong AI adoption ✅ │
└─────────────────────────────────────────┘Bot-to-Score Correlation #
Correlation Analysis:
┌─────────────────────────────────────────────┐
│ AI Score vs Bot Visits │
├─────────────────────────────────────────────┤
│ │
│ Pages Score 90-100: Avg 23 bot visits/mo │
│ Pages Score 80-89: Avg 12 bot visits/mo │
│ Pages Score 70-79: Avg 7 bot visits/mo │
│ Pages Score 60-69: Avg 3 bot visits/mo │
│ Pages Score <60: Avg 1 bot visit/mo │
│ │
│ Correlation Coefficient: 0.87 │
│ Conclusion: Strong positive correlation │
│ │
│ 💡 Insight: +10 points in AI Score = │
│ ~8.5 additional bot visits per month │
│ │
└─────────────────────────────────────────────┘6. Export Formats #
CSV Export Format #
Bot Activity CSV Structure:
csv
Timestamp,Bot_Name,Bot_Type,Bot_Category,User_Agent,IP_Address,Reverse_DNS,URL,Page_Title,AI_Score,Response_Code,Response_Time_ms
2025-12-13T10:47:23Z,GPTBot,AI,OpenAI,"GPTBot/1.0",66.249.64.12,crawl.openai.com,/wordpress-guide/,"Complete WordPress Guide",97,200,234
2025-12-13T10:45:12Z,Googlebot,Traditional,Google,"Googlebot/2.1",66.249.73.15,crawl-66-249-73-15.googlebot.com,/home/,"Homepage",85,200,156
2025-12-13T10:42:08Z,ClaudeBot,AI,Anthropic,"ClaudeBot/1.0",52.24.187.93,ec2-52-24-187-93.compute-1.amazonaws.com,/about/,"About Us",78,200,198Use Cases:
- ✅ Import to Excel/Google Sheets
- ✅ Business intelligence tools
- ✅ Custom analytics dashboards
- ✅ Historical tracking
- ✅ Client reporting
JSON Export Format #
Bot Activity JSON Structure:
json
{
"export_metadata": {
"export_date": "2025-12-13T11:00:00Z",
"site_url": "https://yoursite.com",
"total_entries": 1000,
"date_range": {
"start": "2025-11-13T00:00:00Z",
"end": "2025-12-13T11:00:00Z"
}
},
"summary": {
"total_visits": 2847,
"unique_bots": 8,
"ai_bots": 5,
"traditional_bots": 3,
"pages_visited": 127,
"average_response_time_ms": 187
},
"bot_breakdown": {
"GPTBot": {
"visits": 456,
"pages": 89,
"avg_interval_hours": 67.2,
"success_rate": 99.1
},
"ClaudeBot": {
"visits": 287,
"pages": 76,
"avg_interval_hours": 84.5,
"success_rate": 98.9
}
},
"visits": [
{
"timestamp": "2025-12-13T10:47:23Z",
"bot": {
"name": "GPTBot",
"type": "AI",
"category": "OpenAI",
"user_agent": "GPTBot/1.0",
"verified": true
},
"request": {
"url": "/wordpress-guide/",
"page_title": "Complete WordPress Guide",
"ai_score": 97,
"method": "GET",
"protocol": "HTTP/2.0"
},
"response": {
"code": 200,
"time_ms": 234,
"size_bytes": 45678
},
"network": {
"ip": "66.249.64.12",
"reverse_dns": "crawl.openai.com",
"verified_origin": true
}
}
]
}
```
**Use Cases:**
- ✅ API integration
- ✅ Automated workflows
- ✅ Machine learning analysis
- ✅ Advanced analytics
- ✅ Third-party tools
---
### Export Options
**Customizable Export Settings:**
```
┌─────────────────────────────────────┐
│ Export Bot Activity │
├─────────────────────────────────────┤
│ │
│ Format: │
│ ○ CSV │
│ ● JSON │
│ ○ XML (Advanced) │
│ │
│ Date Range: │
│ From: [2025-11-01] │
│ To: [2025-12-13] │
│ │
│ Bot Filter: │
│ ☑ AI Bots │
│ ☑ Traditional Bots │
│ ☐ Unknown Bots │
│ │
│ Include: │
│ ☑ Visit details │
│ ☑ Page information │
│ ☑ AI Scores │
│ ☑ Response metrics │
│ ☑ Network data │
│ ☐ User agent strings │
│ │
│ Limit: [1000 ▼] entries │
│ │
│ [Export Now] [Schedule Export] │
│ │
└─────────────────────────────────────┘
```
---
## 7. Bot Behavior Patterns
### Crawl Pattern Analysis
**Typical Bot Behaviors:**
```
GPTBot (OpenAI)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Pattern: Systematic, depth-first
Frequency: 2-3 times per week
Preference: High AI Score pages (90+)
Behavior: Follows internal links
Time: Distributed throughout day
Pages/Visit: 5-12 pages average
Observation: Strongly favors:
├─ FAQ pages
├─ How-to guides
├─ Long-form content (2000+ words)
└─ Pages with schema markup
ClaudeBot (Anthropic)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Pattern: Selective, quality-focused
Frequency: 1-2 times per week
Preference: Comprehensive content
Behavior: Deep reading (long dwell time)
Time: Business hours (US)
Pages/Visit: 3-8 pages average
Observation: Prefers:
├─ In-depth articles
├─ Technical documentation
├─ Well-structured content
└─ Recent updates
PerplexityBot
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Pattern: Aggressive, broad coverage
Frequency: Multiple times daily
Preference: Real-time content
Behavior: Fast, breadth-first
Time: 24/7 activity
Pages/Visit: 10-20 pages average
Observation: Targets:
├─ News/blog posts
├─ Recent updates
├─ Trending topics
└─ Time-sensitive content
```
---
### Peak Activity Times
**Bot Activity Heatmap:**
```
┌───────────────────────────────────────┐
│ Bot Activity by Hour (UTC) │
├───────────────────────────────────────┤
│ │
│ 00:00 ▓░░░░░░░ Low │
│ 03:00 ▓░░░░░░░ Low │
│ 06:00 ▓▓▓░░░░░ Moderate │
│ 09:00 ▓▓▓▓▓▓░░ High │
│ 12:00 ▓▓▓▓▓▓▓▓ Very High ← Peak │
│ 15:00 ▓▓▓▓▓▓▓░ High │
│ 18:00 ▓▓▓▓░░░░ Moderate │
│ 21:00 ▓▓░░░░░░ Low-Moderate │
│ │
│ Peak Hours: 09:00-17:00 UTC │
│ Recommendation: Update content │
│ during 06:00-09:00 UTC for │
│ maximum same-day indexing │
│ │
└───────────────────────────────────────┘
```
---
## 8. Crawler Management
### Robots.txt Configuration
**AISEOmatic Auto-Generated Directives:**
```
# Added by AISEOmatic AI
# AI Search Engine Bots
User-agent: GPTBot
Allow: /
Crawl-delay: 1
Sitemap: https://yoursite.com/ai-sitemap.xml
User-agent: ClaudeBot
Allow: /
Crawl-delay: 1
User-agent: PerplexityBot
Allow: /
Crawl-delay: 1
User-agent: Google-Extended
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: Bytespider
Allow: /
User-agent: Meta-ExternalAgent
Allow: /
# Sitemap Declaration
Sitemap: https://yoursite.com/ai-sitemap.xml
Sitemap: https://yoursite.com/sitemap.xml
```
---
### Bot Throttling (Pro)
**Gentle Throttling Settings:**
```
┌─────────────────────────────────────┐
│ Bot Throttling Configuration │
├─────────────────────────────────────┤
│ │
│ ☑ Enable intelligent throttling │
│ │
│ Mode: [Gentle ▼] │
│ • Gentle (recommended) │
│ • Moderate │
│ • Aggressive │
│ • Off │
│ │
│ Rules: │
│ Max requests/minute: [30] │
│ Crawl delay: [1] second │
│ │
│ Whitelist (Never Throttle): │
│ ☑ Googlebot │
│ ☑ Bingbot │
│ ☑ GPTBot │
│ ☑ ClaudeBot │
│ ☐ Unknown bots │
│ │
│ [Save Settings] │
│ │
└─────────────────────────────────────┘
```
---
### Crawler ETA Predictions (Pro)
**Next Visit Estimates:**
```
┌─────────────────────────────────────┐
│ 📅 Predicted Next Bot Visits │
├─────────────────────────────────────┤
│ │
│ GPTBot │
│ Last Visit: 2 days ago │
│ Avg Interval: 2.8 days │
│ Next Visit: ~20 hours │
│ Confidence: 78% │
│ │
│ ClaudeBot │
│ Last Visit: 5 days ago │
│ Avg Interval: 6.2 days │
│ Next Visit: ~1-2 days │
│ Confidence: 65% │
│ │
│ PerplexityBot │
│ Last Visit: 4 hours ago │
│ Avg Interval: 8.5 hours │
│ Next Visit: ~4 hours │
│ Confidence: 85% │
│ │
│ 💡 Tip: Publish updates before │
│ predicted visits for faster │
│ indexing │
│ │
└─────────────────────────────────────┘
```
---
## 9. Troubleshooting & Verification
### Common Issues
**Issue #1: No Bot Activity Detected**
```
Symptoms:
❌ Zero bot visits logged
❌ Empty bot detection dashboard
Possible Causes:
→ New website (< 2 weeks old)
→ No sitemap submitted
→ Robots.txt blocking crawlers
→ Low-quality/thin content
→ Server blocking bot IPs
Solutions:
1. Verify AI Sitemap accessible at /ai-sitemap.xml
2. Check robots.txt allows AI bots
3. Submit sitemap to Google Search Console
4. Ensure content quality (AI Score 70+)
5. Check server firewall settings
6. Wait 7-14 days for initial discovery
```
---
**Issue #2: Only Traditional Bots, No AI Bots**
```
Symptoms:
✅ Googlebot, Bingbot detected
❌ No GPTBot, ClaudeBot, etc.
Possible Causes:
→ AI Sitemap not accessible
→ Content not AI-optimized
→ Low AI Scores across site
→ Geo-restrictions on site
Solutions:
1. Verify /ai-sitemap.xml accessible
2. Improve AI Scores (target 80+)
3. Add data-llm attributes
4. Add schema markup
5. Check no country blocks on bots
6. Be patient (AI bots slower to discover)
```
---
**Issue #3: Bot Verification Failures**
```
Symptoms:
⚠️ "Unverified bot" warnings
⚠️ Suspicious activity flags
Causes:
→ IP spoofing attempts
→ Fake user agents
→ Malicious crawlers
AISEOmatic Protection:
✅ Reverse DNS verification
✅ IP range validation
✅ Behavioral analysis
✅ Known bot database
Action:
→ Automatic blocking of fake bots
→ Only real bots logged
→ Security maintained
```
---
### Bot Verification Methods
**How AISEOmatic Verifies Bots:**
```
1. User Agent Check
├─ Parse UA string
├─ Match known patterns
└─ Initial classification
2. IP Validation
├─ Check against known ranges
├─ Google: 66.249.0.0/16
├─ OpenAI: Various AWS ranges
└─ Verify ownership
3. Reverse DNS Lookup
├─ Query PTR record
├─ Verify domain ownership
├─ googlebot.com
├─ crawl.openai.com
└─ Must match
4. Behavioral Analysis
├─ Crawl patterns
├─ Request frequency
├─ Header consistency
└─ Known behavior match
Result: High-confidence bot identification
```
---
## 10. Advanced Bot Intelligence
### Bot Segmentation
**Intelligent Bot Categorization:**
```
AI Search Bots (Primary Focus)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Priority: Critical
Optimization: Maximum
Tracking: Detailed
Impact: High SEO value
Traditional Search Bots
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Priority: High
Optimization: Standard
Tracking: Standard
Impact: Essential baseline
Social Media Bots
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Priority: Medium
Optimization: Moderate
Tracking: Basic
Impact: Share optimization
Research/Academic Bots
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Priority: Low
Optimization: Minimal
Tracking: Logged only
Impact: Negligible
Monitoring/Uptime Bots
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Priority: None
Optimization: None
Tracking: Excluded
Impact: Zero
```
---
### Custom Bot Rules (Pro)
**Advanced Bot Management:**
```
┌─────────────────────────────────────┐
│ Custom Bot Rules │
├─────────────────────────────────────┤
│ │
│ Rule 1: Priority Bots │
│ If bot = [GPTBot, ClaudeBot] │
│ Then: │
│ • Allow all pages │
│ • No rate limiting │
│ • Priority in logs │
│ │
│ Rule 2: Heavy Crawlers │
│ If requests > 100/hour │
│ Then: │
│ • Gentle throttling │
│ • Crawl-delay: 2 seconds │
│ • Monitor closely │
│ │
│ Rule 3: Unknown Bots │
│ If bot = unverified │
│ Then: │
│ • Standard rate limit │
│ • Log but don't prioritize │
│ • Flag for review │
│ │
│ [Add Rule] [Save All] │
│ │
└─────────────────────────────────────┘AISEOmatic Bot Detection Guide v1.0
Last Updated: December 13, 2025
