• Register

七牛云存储robots.txt搜索引擎规则

七牛云存储设置了自定义域名后,可能会造成重复收录的问题,所以需要设置 robots.txt 来禁止搜索引擎对自定义域名的收录。

七牛默认的robots.txt内容如下:

# robots.txt generated at http://portal.qiniu.com
User-agent: Baiduspider
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: YodaoBot
Disallow: /
User-agent: Googlebot
Disallow: /
User-agent: Bingbot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: MSNBot
Disallow: /
User-agent: googlebot-image
Disallow: /
User-agent: googlebot-mobile
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
User-agent: psbot
Disallow: /
User-agent: *
Disallow: /

这个 robots.txt 规则同样也阻止了对图片等必要资源的收录,可以稍作修改后再上传到七牛云存储即可。

# robots.txt generated at http://portal.qiniu.com
# edit for statice domain
User-agent: Baiduspider
Allow: /wp-content/uploads/*.jpg$
Allow: /wp-content/uploads/*.png$
Disallow: /
User-agent: Baiduspider-image
Allow: /wp-content/uploads/*.jpg$
Allow: /wp-content/uploads/*.png$
Disallow: /
User-agent: 360Spider
Allow: /wp-content/uploads/*.jpg$
Allow: /wp-content/uploads/*.png$
Disallow: /
User-agent: 360Spider-Image
Allow: /wp-content/uploads/*.jpg$
Allow: /wp-content/uploads/*.png$
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: YodaoBot
Disallow: /
User-agent: Googlebot
Disallow: /
User-agent: Bingbot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: MSNBot
Disallow: /
User-agent: googlebot-image
Disallow: /
User-agent: googlebot-mobile
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
User-agent: psbot
Disallow: /
User-agent: *
Disallow: /
版权声明:转载请注明来自杨俊伟博客,本文地址:http://yangjunwei.com/a/1808.html
除非注明,杨俊伟博客文章均为原创,转载请注明出处和链接!
 

发表评论 

我还是想用邮箱注册

登录

忘记密码 ?

用第三方帐号快捷登录

已有账户?前往登录吧~

注册