arsam
26e86f9af2
Import the existing local project snapshot. This commit organizes existing files; it does not represent newly developed functionality.
55 lines
714 B
Plaintext
55 lines
714 B
Plaintext
User-agent: *
|
|
Allow: /
|
|
Crawl-delay: 1
|
|
|
|
# Major search engines
|
|
User-agent: Googlebot
|
|
Allow: /
|
|
Crawl-delay: 0
|
|
|
|
User-agent: Bingbot
|
|
Allow: /
|
|
Crawl-delay: 1
|
|
|
|
User-agent: Slurp
|
|
Allow: /
|
|
|
|
User-agent: DuckDuckBot
|
|
Allow: /
|
|
|
|
# AI crawlers — explicitly welcome
|
|
User-agent: GPTBot
|
|
Allow: /
|
|
|
|
User-agent: ChatGPT-User
|
|
Allow: /
|
|
|
|
User-agent: ClaudeBot
|
|
Allow: /
|
|
|
|
User-agent: Claude-Web
|
|
Allow: /
|
|
|
|
User-agent: anthropic-ai
|
|
Allow: /
|
|
|
|
User-agent: PerplexityBot
|
|
Allow: /
|
|
|
|
User-agent: Amazonbot
|
|
Allow: /
|
|
|
|
User-agent: cohere-ai
|
|
Allow: /
|
|
|
|
User-agent: YouBot
|
|
Allow: /
|
|
|
|
User-agent: Meta-ExternalAgent
|
|
Allow: /
|
|
|
|
Sitemap: https://arsamsabbagh.ir/sitemap.xml
|
|
|
|
# AI-readable plain-text summary of this site
|
|
# https://arsamsabbagh.ir/llms.txt
|