Yandex is an over zealous Russian search engine that will scrupulously index as many pages as your site can offer up and it’s spider enjoys it’s work. Yandex may be industrious but unless you have pages of interest to Russian speakers you may want to restrict it’s spidering of your site because it can use a lot of your valuable bandwidth allowance.
To prevent Yandex from crawling simply add the following to your robots.txt file:
User-agent: yandex bot
Disallow: /
Friday, 4 December 2009
Subscribe to:
Post Comments (Atom)
0 comments:
Post a Comment