WebProNews

New crawler in rollout phase to Internet

Site publishers will see a new Yahoo bot crawling through their pages, and webmasters should be aware of a couple of little issues that may affect them.

Yahoo's search team needed the past few weeks to prep their latest crawlers for its sojourns onto the World Wide Web. This new mutli-legged beast began its official rollout, as noted on the Yahoo Search blog.

"The new Yahoo! Slurp 3.0 recognizes the same user-agent and all robots.txt directives for 'Yahoo! Slurp,' though it'll identify itself as Slurp 3.0 in your web logs," the blog said. A handful of webmasters may notice a couple of changes.

"Please note that if you're using IP-based recognition of our crawlers, you might see a drop in crawl/coverage from Yahoo! We strongly recommend that you move to reverse DNS-based identification of Yahoo! Slurp if you're using any other method to avoid this problem," they said. The IP ranges for Yahoo's crawlers will change to a smaller set, too.

The other concept applies to robots.txt files and directives. Yahoo Slurp 3.0 will recognize existing entries for 'Slurp' or 'Yahoo Slurp', but not a reference to 'Slurp/2.0.' Yahoo thinks those entries are rare enough anyway, but just in case, it was prudent for them to reference it.