User-agent: Baiduspider Disallow: / #20091108 User-agent: dotbot Crawl-Delay: 30 #20091202 # Naughty dotbot - crawl-delay was set to 10 a month ago, and you're still fetching at about every 4-5. Tut tut!!! User-agent: MJ12bot Crawl-Delay: 10 #20091013 User-agent: ia_archiver Disallow: / #20091021 User-agent: Sosospider Disallow: / User-agent: Yeti Disallow: / User-agent: YodaoBot Disallow: / User-agent: Yandex Disallow: /