Yahoo Slurp 3.0: Crawler Revised And Rolling Out

Yahoo Slurp 3.0: Crawler Revised And Rolling Out

By WebProNews Staff April 14, 2008

Site publishers will see a new Yahoo bot crawling through their pages, and webmasters should be aware of a couple of little issues that may affect them.

Wikia Acquires Grub from LookSmart

Wikipedia founder Jimmy Wales would like you to help him build the revenue for his new “for profit” venture Wikia. Wikia has acquired the distributed crawler Grub from LookSmart and Wales plans to make it open source. He’d like to invite the community to line his coffers.

The Name Is Crawl. Yahoo Crawl

Yahoo Slurp, the portal company’s search crawler, has been migrated to its new domain at crawl.yahoo.net and is coming to a web server near you.

Webmaster Settles With IA, Goes After Teenager

One of the most bizarre Internet stories this year gets more bizarre. Internet Archive and Suzanne Shell have resolved their lawsuits against each other "amicably," a word that follows words like "theft" and "racketeering" about as well as toothpaste follows orange juice. Next on her list: a 15-year-old Canadian jokester. 

The Search Engines Agree on an XML Sitemap Standard

In great news, Google, Yahoo and MSN have agreed upon a XML sitemap standard.

Yahoo Unleashes New Search Crawler

Yahoo Slurp may be showing up heavily in access logs around the web, as the company gave the new spider a workout alongside the old spider before retiring the previous Slurp.

What Happens If You Block The Google Crawler

Although it’s a move few would consider, what would happen to a site if the webmaster decided to block the Google crawler from indexing their sites by stipulating such in the site’s robot.txt file?

Become Crawler Info For Webmasters

A few days ago I told you about the new Become shopping search engine and gave you a glimpse at what you can expect from it.

MSN Joins Battle Against Blog Spam

Ken Moss, the General Manger of MSN Search Dev & Test announced in the MSN Search WebLog their support of the new NoFollow tag.

Crawler Alert: Instant Notification When Google Visits

If you want to know the instant Google’s bot visits your site then you’re a severe algoholic. And Crawler alert will help you get your fix. Crawler alert will send you an email when any of the major search bots visit the page you specify.

Search Optimization With Hyphenated URLs
· 1

It’s widely believed that hyphenated urls that include your primary keywords can help you rank higher for those keywords. For my entomological supply site I might create a url like this: Butterfly-Nets.com, especially if I specialize in butterfly nets.

Search Bots Behaving Badly

In a classic bit of Internet history (1998/12/30), a newsgroup member explained the GoogleBot to a poster complaining of its rude behavior.

Crawler Insights From Google and Yahoo!
· 1

My favorite sessions at the SES conference were those where Google and Yahoo appeared on the same panels. You could almost always count on some crackling tension between the two search giants.

The “Meet The Crawlers” session was no exception.

Creating a File Content Crawler with ColdFusion

This tutorial will show you how to create a local file crawler that will enable you to find a specified document type (i.e. PDF files) within a directory (and it’s children directories).