Yahoo’s search team needed the past few weeks to prep their latest crawlers for its sojourns onto the World Wide Web. This new mutli-legged beast began its official rollout, as noted on the Yahoo Search blog.

“The new Yahoo! Slurp 3.0 recognizes the same user-agent and all robots.txt directives for ‘Yahoo! Slurp,’ though it’ll identify itself as Slurp 3.0 in your web logs,” the blog said. A handful of webmasters may notice a couple of changes.

“Please note that if you’re using IP-based recognition of our crawlers, you might see a drop in crawl/coverage from Yahoo! We strongly recommend that you move to reverse DNS-based identification of Yahoo! Slurp if you’re using any other method to avoid this problem,” they said. The IP ranges for Yahoo’s crawlers will change to a smaller set, too.

The other concept applies to robots.txt files and directives. Yahoo Slurp 3.0 will recognize existing entries for ‘Slurp’ or ‘Yahoo Slurp’, but not a reference to ‘Slurp/2.0.’ Yahoo thinks those entries are rare enough anyway, but just in case, it was prudent for them to reference it.

Websites place content behind forms for the purpose of collecting information from a visitor requesting access to it. The site publisher might want those details for demographic details to improve marketing campaigns, for example.

Google thinks it can present better results to searchers by having access to the URLs behind forms, improving the site’s exposure in the process. The Google Webmaster Central blog promised their crawls will be well-behaved:

Only a small number of particularly useful sites receive this treatment, and our crawl agent, the ever-friendly Googlebot, always adheres to robots.txt, nofollow, and noindex directives. That means that if a search form is forbidden in robots.txt, we won’t crawl any of the URLs that a form would generate. Similarly, we only retrieve GET forms and avoid forms that require any kind of user information.

However, concerns have been raised about Google crawling forms not marked as forbidden. Kevin Heisler complained at Search Engine Watch the practice could violate the privacy of corporate data.

Though confident in Google’s intentions, Heisler thinks potential backlash from corporate interests could be a problem. “The costs to CEOs, CIOs and CTOs at corporations far outweigh the benefits to consumers,” he said.

Unvalidated Robots.Txt Risks Google Banishment

Webmasters welcome being dropped out of Google about as much as they enjoy flossing with barbed wire. Making it easier for Google to do that would be anathema to being a webmaster. Why willingly exclude one’s site from Google?

That could happen with an unvalidated robots.txt file. Robots.txt allows webmasters to provide standing instructions to visiting spiders, which contributes to having a site indexed faster and more accurately.

Google has been considering new syntax to recognize within robots.txt. The Sebastians-Pamphlets blog said Google confirmed recognizing experimental syntax like Noindex in the robots.txt file.

This poses a danger to webmasters who have not validated their robots.txt. A line reading Noindex: / could lead to one’s site being completely de-indexed.

The surname-less Sebastian recommended Google’s robots.txt analyzer, part of Google’s Webmaster Tools, and only using the Disallow, Allow, and Sitemaps crawler directives in the Googlebot section of robots.txt.

follow me on Twitter

Listen to the audio I spoke to Rob Aronson, Vice President of Search Engine Marketing for Avenue A/ Razorfish about their proprietary tool Super Intelligent Link Crawler (SiLC) Listen to the audio here.

He described the tool as a simulated spider that is basically a Web crawler that helps them identify how search engines will interact with their client’s Web sites and also improves user’s experience on the site.

The tool itself is a proprietary tool for their clients that want to optimize sites for search results. The tool is not a stand alone for
public use and there is no link to the tool available.

When asked if it would be offered to the public, he said not at this point but that there was the potential for it to be made public in the future.

As for the cost to use the tool, it depends on the size of the site, how much work is being done and how much Web development needs to be done. Those factors determine a wide range of pricing.

He said the tool is aimed at mainly larger Web sites such as U.S. News and World Reports, and the larger the site the more benefits the technology will have.

Avenue A | Razorfish Launches Link Crawler

The tool examines why errors occur and can measure a Web site’s performance against competing sites. The company has combined the tool with its search engine optimization and Web design to improve search rankings for brands like U.S. News & World Report.

U.S. News & World Report recently redesigned its Web site to improve traffic and search capability for the site. After using the SiLC tool, Avenue A | Razorfish found search engines were not ranking many of its Web pages because they were flagging pages as duplicate content.

U.S. News & World Report created the main content of a health page article and a "printer friendly" version. The web crawlers were tracking both and not ranking the pages in search engine results. After the relaunch organic visits increased 24 percent and organic visits from Google increased 45 percent.

William Flaiz, vice president, Search Engine Optimization (SEO) and Web Analytics for Avenue A | Razorfish said, "SiLC is like a Swiss Army Knife that serves many purposes beyond web crawling."

"It is different than other web crawlers because it can intelligently process why certain web pages aren’t performing well and integrate with other SEO tools to improve efficiencies not only within a web site, but to other linked Web sites, portals and blogs."

In case you need a better understanding of Wales’ plans, he explains his motives:

“Search is part of the fundamental infrastructure of the Internet. And, it is currently broken,” Wales said back in December 2006, when Wikia launched Search Wikia effort. “Why is it broken? It is broken for the same reason that proprietary software is always broken: lack of freedom, lack of community, lack of accountability, lack of transparency.”

Of course, Wales forgets to mention that Wikipedia wouldn’t have anywhere near the success if its content didn’t appear in the top 5 of Google for just about every generic keyword you can imagine. It also slipped his mind that he has a track record of not exactly giving back to those that helped make it a success – they’ll happily “assimilate” your web content, just don’t expect a link back in the process.

I wish Jimmy Wales the best of luck, but if he really wants to change the world, why not improve the non-profit Wikipedia? If his plans involve Wikia – which means he’s in this for the money – spare us the sanctimonious spin.

Comments

Tag:

As of today, the transition is complete and all machines crawling as Slurp are now in crawl.yahoo.net. You can see this change in your web server logs, where the page accesses from inktomisearch.com are being fully replaced by crawl.yahoo.net contacts. Note that this does not cover other Yahoo! crawlers, such Yahoo! China, and other verticals, like Yahoo! Shopping, Yahoo! Travel, etc., which have their own user-agent.

Webmasters won’t have to tweak their robots.txt files or any IP-based filtering they perform, since the agent is still called Yahoo Slurp and the IP addresses originating the crawl will remain the same.

However, security administrators will want to permit crawl.yahoo.net to access pertinent systems within their networks, or risk blocking the crawlers and incurring the screaming wrath of the webmaster.

Priyank Garg and Yoram Arnon of Yahoo recommend webmasters set up reverse DNS-based authentication of the crawler for security reasons. This keeps rogue bots from masquerading as Yahoo Slurp.

Shell, who claimed IA’s web crawler entered into a contract while indexing, responded (quite reasonably, I might add) with a countersuit accusing IA of conversion, civil theft, breach of contract, and violations of both federal racketeering and Colorado organized crime legislation.

It was an important case, not necessarily because of its overall merit (all but one claim was immediately dismissed), but because the judge would have to consider whether or not spiders could enter a contract, and the world was listening.

Seems the judge won’t be able to make that declaration any time soon, as IA and Shell seem to have kissed and made up, issuing a joint statement about the settlement.

“Internet Archive has no interest in including materials in the Wayback Machine of persons who do not wish to have their Web content archived," said an IA spokesperson in a statement.

"We recognize that Ms. Shell has a valid and enforceable copyright in her Web site and we regret that the inclusion of her Web site in the Wayback Machine resulted in this litigation. We are happy to have this case behind us.”

Shell seems to have gotten over it, too. “I respect the historical value of Internet Archive’s goal," she said. "I never intended to interfere with that goal nor cause it any harm,” said Ms. Shell.

That’s interesting…could have sworn she sued them for theft, racketeering, and organized criminal activity. But if both say they settled it amicably, then who am I to question? Maybe IA just didn’t want to give the judge a chance to rule that web crawlers could actually enter a contract.

The statement ends with this gem:

Both parties sincerely regret any turmoil that the lawsuit may have caused for the other. Neither Internet Archive nor Ms. Shell condones any conduct which may have caused harm to either party arising out of the public attention to this lawsuit. The parties have not engaged in such conduct and request that the public response to the amicable resolution of this litigation be consistent with their wishes that no further harm or turmoil be caused to either party.

The turmoil they’re talking about may include 15-year-old Canadian and Digg.com user Jeff Veillette, who issued a kind of online challenge to Shell by framing her entire site, daring her to take action against him. While brazen, Shell claims it went well beyond framing.

If you visit her website, profane-justice.org (careful, you’re entering into a contract with an anti-child protective services activist), Shell posts Veillette’s name, home address, emails, and phone number after accusing him of hacking her website and setting up a pop-up spam campaign, viewable by 25 million people and defaming her. Shell says he expressly admitted to the hacking and spam campaign.

She includes blogger Billy Wiseman (a guy she really, really doesn’t like) in the accusation, listing his address as well. The FBI Computer Crime Center was contacted.

However, in an interview with WebProNews, hacking and spamming came as news to Veillette, who says his personal information listed is outdated (remember, at 15, he’s a minor  — is it legal to post a minor’s information online? Maybe the FBI needs to give her a call). Jeff denies hacking her site or spamming anyone, but does admit to framing her site elsewhere, an act we reported when it happened.

As for Wiseman, Veillette says he had nothing to do with any of it. "Billy Wiseman wasn’t part of the attacks," he said. "He was a blogger that was watching the Shell case and interviewed me."

"What happened was I put the site in a frame. How that’s hacking I’ll never know."

So Shell never lost control of her site?

"Nope," said Veillette. "She ran out of bandwidth for her hosting."

So, has the FBI been in touch?

"Nope. They probably laughed at this, too. What I’m wondering overall, though, is if she’s for supporting kids, why is she going out of her way to try and ruin a 15-year-old’s life?"

Good question, Jeff. I’m sure, though, she never meant any harm. 

Financial terms of the settlement between Shell and Internet Archive were not disclosed.

What’s this mean to you?

Well, you just need to create one XML sitemap and you can submit it to all three engines.

I’ve been doing that for a while and didn’t seem to have any problem but now it’s official.

Why should you create an XML sitemap? An XML sitemap outlines all the URLs on a site so that search engines can find them all easily.

Much like a normal sitemap, it just lists all the pages however, an XML sitemap is optimized for search engines, not for people. By giving search engines a list of URLs on your site, you are increasing the number of pages they know about and can refer traffic to.

If you posted once a day for a year, you’d have 365 post pages. That doesn’t include categories, archives and any other blog related pages.

Since the numbers add up so quickly, it’s hard for a search engine to keep up and even harder for a user to keep them organized so that an engine can find them all easily.

An XML sitemap gives engines one file full of all the URLs on your site, optimized just for them.

So, how do you create an XML site map? There are variety of ways but here are my top three choices:

1. Google Sitemap Generator Plugin for WordPress. Once installed, it automatically updates and pings Google when you have new content. Since all engines now use the same XML formatting, the name is a little outdated. It should be updated to Search Engine Sitemap Generator.

2. GSite Crawler. This is a free desktop application for PCs that will crawl your site and create the XML site map for you to upload. It doesn’t automatically update but it is easy to use and you could re-generate an XML map ever month or so. If anyone knows of a free Mac app like this, I’d love to hear about it.

3. XML-Sitemaps.com. This is a web based XML sitemap creation tool that is free for small sites. Again, it’s not automatic, but it’s free and easy to use. Plus it works on any platform.

If you have a blog, you should be creating an XML sitemap. Even if you do it only once every six months, it’s better than none at all.

Do you use a different tool for creating XML sitemaps? If so, do share.

Comments

Bookmark WebProNews:

Thomas McMahon is a SEO Designer for TopRank Online Marketing in Minneapolis, MN. His specialities inlude technical optimization of existing web sites, creating search engine friendly web designs, and blog optimization. He has also created a number of blog marketing tools, WordPress plug-ins and FireFox add-ons.

Blog: http://bloggerdesign.com

Yahoo Unleashes New Search Crawler

It has taken a few weeks for Yahoo to ensure its new version of Slurp will work properly, which explains why sites have probably seen a lot more Yahoo requests trawling their sites.

On the Yahoo! Search blog entry about the new Slurp crawler, David Simpson and Priyank Garg discussed the switch in their Weather Report post:

As always, ranking changes and page shuffling around the index may have happened for websites, so publishers will want to take a look at where they show up now in Yahoo! Search.

The efficiency as reported by Yahoo should be welcome by publishers when their new bandwidth bills arrive.

Discuss this article with your fellow WebPros at WebProWorld.
—
Tag:

Add to Del.icio.us | Digg | Yahoo! My Web | Furl

Bookmark WebProNews:

David Utter is a staff writer for WebProNews covering technology and business.