abonnel-siteweb/robots.txt

14 lines
145 B
Plaintext
Executable File

User-agent: *
Allow: /
User-agent: JobSpider
Disallow: /
User-agent: JobSpider_BA
Disallow: /
#User-agent: *
#Crawl-delay: 5
#Disallow: /lib