All Posts Tagged Tag: ‘Web Crawlers’

New “Bingbot” Will Crawl Non-optimized Sites More Easily

Microsoft has announced that it will be bringing the Bing web crawler out of beta on October 1st. It will be rebranded as "the Bingbot" and replace the existing msnbot. "It will still honor robots.txt directives written for msnbot, so no change is required to robots.txt file(s)," a Bing representative tells WebProNews.

"Improvements to the bot enable more efficient crawling, and increase the ability to crawl content on sites not optimized for search," he says.

Read More

Google States Case for Online News in WSJ

Update: The Wall Street Journal is running a piece from Google CEO Eric Schmidt on how Google can help newspapers. It’s an interesting read.

Read More

Yahoo Slurps Somewhere Else

The migration of the Slurp is complete, says Yahoo. Over the past few weeks, the search engine has been transitioning its crawler, dubbed (disgustingly) "Slurp," to a new address at Adjust your server logs as necessary and join the curmudgeons who are unimpressed.

Read More

Webmaster Claims Spider Entered Contract In Suit

The Web and artificial intelligence have brought about some surreal, science fiction like questions. The most recent mind-bending concept is whether or not robots can enter into contracts – that is, is a Web crawler implicitly entering a contract posted on a website announcing copyright conditions?

Read More