|
|
禁止所有爬虫搜索引擎蜘蛛收录网站任何页面的robots终极写法:
- User-agent: Baiduspider
- Disallow: /
- User-agent: Googlebot
- Disallow: /
- User-agent: MSNBot
- Disallow: /
- User-agent: Baiduspider-image
- Disallow: /
- User-agent: YoudaoBot
- Disallow: /
- User-agent: Sogou web spider
- Disallow: /
- User-agent: Sogou inst spider
- Disallow: /
- User-agent: Sogou spider2
- Disallow: /
- User-agent: Sogou blog
- Disallow: /
- User-agent: Sogou News Spider
- Disallow: /
- User-agent: Sogou Orion spider
- Disallow: /
- User-agent: ChinasoSpider
- Disallow: /
- User-agent: Sosospider
- Disallow: /
- User-agent: yisouspider
- Disallow: /
- User-agent: EasouSpider
- Disallow: /
- User-agent: *
- Disallow: /
复制代码 懒人包:直接上传覆盖网站根目录的robots.txt:
robots.txt
(651 Bytes, 下载次数: 1)
阅读权限: 10
|
|