Crawler Insights From Google and Yahoo!

Get the WebProNews Newsletter:

[ Search]

My favorite sessions at the SES conference were those where Google and Yahoo appeared on the same panels. You could almost always count on some crackling tension between the two search giants.

The “Meet The Crawlers” session was no exception.

Join the discussion on web crawlers here.

Weaving the Web
Weaving the Web

Craig Neville-Manning, senior research scientist at Google, had some great advice for webmasters (and a pointed barb for Yahoo – we’ll get to that though).

“We don’t like pointing users to pages that you change,” said Craig, describing the practice of cloaking. You should avoid cloaking at all costs. As Tim Mayer of Yahoo pointed out during our lunch chat, there’s a legitimate use for every potential spam technique.

For example, you can use cloaking to show search engines an optimized page and then, say, a Flash intensive page when users arrive. Despite cloaking’s legitimate uses though, authorities recommend that you don’t do it.

Concerned you might be cloaking? Read this page cloaking article.

Craig revealed a good rule of thumb for optimizers – Google’s algorithm values text and links that your site visitors can see more highly than anything they can’t see. This means focus your efforts more on explicit, helpful, and keyword-focused links, as well as copy that informs your visitors.

For those concerned that the Google bot uses too much bandwidth he mentioned that it can detect when your server is slowing down and it will back off. Also, the Google bot follows the robots.txt file to the letter.

If you have content you don’t want the bot to find be sure to put the robots.txt file up to keep it out. The bot, says Craig, can find content that’s unlinked. That’s right, the Google bot can find single pages dangling unlinked in space. He didn’t explain how this happens.

The question and answer session revealed a bit of how the Google looks at keywords in the url. Someone from the audience asked about how Google views words in the url, whether you should hyphenate them or not for added relevancy and ranking.

Craig said that Google does index words from the url, but they don’t have as much weight as text links. He added quickly though that you should not engineer your links for the algorithm – it’s better to have your url meaningful to your visitors than use it to affect your ranking.

Tim Mayer of Yahoo seconded this. He said focusing too much on url engineering can get you into the realm of over-optimization. “As a user,” said Tim, “if I see a domain with lots of hyphens it’s usually a low quality site.” He advised that you not push your filenames too hard, and that you have intuitive directory structures.

In the Link Building Strategies session, prominent seo guru Greg Boser said “hyphenated domains have come and gone.”

The big talk at this conference was the new Yahoo paid inclusion program, which allows webmasters to pay to show up in Yahoo’s primary search results. At the close of Google employee Craig’s presentation he declared, in a comment obviously leveled at fellow presenter Tim Mayer of Yahoo, “our search results are not for sale.”

Tim, ever the gentleman, let it slide.

Garrett French is the editor of iEntry’s eBusiness channel. You can talk to him directly at WebProWorld, the eBusiness Community Forum.

Crawler Insights From Google and Yahoo!
Top Rated White Papers and Resources
  • http://seethrureviews.com/Rocket-Spanish-Download-Reviews.html Rocket Spanish Download

    I personally find cloaking useful when the url looks real messy which sometimes puts off visitors.

  • Join for Access to Our Exclusive Web Tools
  • Sidebar Top
  • Sidebar Middle
  • Sign Up For The Free Newsletter
  • Sidebar Bottom