七牛云存储robots.txt搜索引擎规则

七牛云存储设置了自定义域名后,可能会造成重复收录的问题,所以需要设置 robots.txt 来禁止搜索引擎对自定义域名的收录。

七牛默认的robots.txt内容如下:

# robots.txt generated at http://portal.qiniu.com
User-agent: Baiduspider
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: YodaoBot
Disallow: /
User-agent: Googlebot
Disallow: /
User-agent: Bingbot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: MSNBot
Disallow: /
User-agent: googlebot-image
Disallow: /
User-agent: googlebot-mobile
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
User-agent: psbot
Disallow: /
User-agent: *
Disallow: /

这个 robots.txt 规则同样也阻止了对图片等必要资源的收录,可以稍作修改后再上传到七牛云存储即可。

# robots.txt generated at http://portal.qiniu.com
# edit for statice domain
User-agent: Baiduspider
Allow: /wp-content/uploads/sites/2/*.jpg$
Allow: /wp-content/uploads/sites/2/*.png$
Disallow: /
User-agent: Baiduspider-image
Allow: /wp-content/uploads/sites/2/*.jpg$
Allow: /wp-content/uploads/sites/2/*.png$
Disallow: /
User-agent: 360Spider
Allow: /wp-content/uploads/sites/2/*.jpg$
Allow: /wp-content/uploads/sites/2/*.png$
Disallow: /
User-agent: 360Spider-Image
Allow: /wp-content/uploads/sites/2/*.jpg$
Allow: /wp-content/uploads/sites/2/*.png$
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: YodaoBot
Disallow: /
User-agent: Googlebot
Disallow: /
User-agent: Bingbot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: MSNBot
Disallow: /
User-agent: googlebot-image
Disallow: /
User-agent: googlebot-mobile
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
User-agent: psbot
Disallow: /
User-agent: *
Disallow: /

参与评论

游客评论不支持回复他人评论内容,如需回复他人评论内容请