WebProNews

Yahoo Slurp, the portal company's search crawler, has been migrated to its new domain at crawl.yahoo.net and is coming to a web server near you. There's a new crawler in town from Yahoo. Sort of. It's the same crawler webmasters know and love already, but it will look different in their access logs, according to the Yahoo Search crew:

As of today, the transition is complete and all machines crawling as Slurp are now in crawl.yahoo.net. You can see this change in your web server logs, where the page accesses from inktomisearch.com are being fully replaced by crawl.yahoo.net contacts. Note that this does not cover other Yahoo! crawlers, such Yahoo! China, and other verticals, like Yahoo! Shopping, Yahoo! Travel, etc., which have their own user-agent.

Webmasters won't have to tweak their robots.txt files or any IP-based filtering they perform, since the agent is still called Yahoo Slurp and the IP addresses originating the crawl will remain the same.

However, security administrators will want to permit crawl.yahoo.net to access pertinent systems within their networks, or risk blocking the crawlers and incurring the screaming wrath of the webmaster.

Priyank Garg and Yoram Arnon of Yahoo recommend webmasters set up reverse DNS-based authentication of the crawler for security reasons. This keeps rogue bots from masquerading as Yahoo Slurp.