WebProNews

Yahoo Slurp 3.0: Crawler Revised And Rolling Out

Site publishers will see a new Yahoo bot crawling through their pages, and webmasters should be aware of a couple of little issues that may affect them.

Google Starts Controversial Form Crawling Program

Googlebot received an update that permits it to complete certain forms, and learn more about the site hosting them.

Unvalidated Robots.Txt Risks Google Banishment

The web crawling Googlebot may find a forgotten line in robots.txt that causes it to de-index a site from the search engine.

Avenue A / Razorfish Clarifies Crawler

Rob Aronson, Vice President of Search Engine Marketing for Avenue A Razorfish spoke to WebProNews for an interview about their Super Intelligent Link Crawler. (SiLC)

Avenue A | Razorfish Launches Link Crawler

Interactive marketing and technology company Avenue A | Razorfish has launched Super-intelligent Link Crawler (SiLC), a tool that crawls Web sites to find errors such as broken links and "404 Error Messages" that inform users the Web page is not loading.

Wikia Acquires Grub from LookSmart

Wikipedia founder Jimmy Wales would like you to help him build the revenue for his new “for profit” venture Wikia. Wikia has acquired the distributed crawler Grub from LookSmart and Wales plans to make it open source. He’d like to invite the community to line his coffers.

The Name Is Crawl. Yahoo Crawl

Yahoo Slurp, the portal company's search crawler, has been migrated to its new domain at crawl.yahoo.net and is coming to a web server near you.

Webmaster Settles With IA, Goes After Teenager

One of the most bizarre Internet stories this year gets more bizarre. Internet Archive and Suzanne Shell have resolved their lawsuits against each other "amicably," a word that follows words like "theft" and "racketeering" about as well as toothpaste follows orange juice. Next on her list: a 15-year-old Canadian jokester. 

The Search Engines Agree on an XML Sitemap Standard

In great news, Google, Yahoo and MSN have agreed upon a XML sitemap standard.

Yahoo Unleashes New Search Crawler

Yahoo Slurp may be showing up heavily in access logs around the web, as the company gave the new spider a workout alongside the old spider before retiring the previous Slurp.