Google Helps Websites Get Crawled – for Free!

    June 30, 2005

Known for its regular introduction of new products as much as for its regularly increasing stock prices, Google is continually evolving.

Recently the Internet giant unveiled another worthwhile product, Google Sitemaps, aimed at what it calls an attempt to expand its coverage of the World Wide Web and increase the freshness of its indexed Web pages.

The free service allows webmasters to place an XML-formatted file on their webservers so that Googlebot can better crawl all Web pages on a site. The new offering also allows webmasters to make updates to the Sitemap when changes to the website have been made so that the Google spiders will know to crawl the new content, as well. At its best this would mean that no Web page is left behind and possibly lead to increased search relevancy – a benefit for searchers and the search industry, in general – and better search placement for participating websites.

Google could be using this feature to play catch-up after a recent “study” of 10,000 rated searches, conducted by RustyBrick to test which search engine (Google, Yahoo! Search, MSN Search, and Ask Jeeves) returned the most relevant results. So far Yahoo! is beating Google by a small margin. But when it comes to the biggest search engine in the world losing out to its main competitor, even a small margin can become a big problem.

Google Sitemaps became part of the discussion at the WebmasterWorld Search and Marketing Conference held in New Orleans last week. One thousand conference attendees – including myself – were present to witness a not-so-friendly exchange of words between Google (Matt Cutts) and Yahoo! (Tim Mayer) representatives over – what else? – search engine rankings. One good point made by Google engineer Cutts was that the “Google Sitemaps is like paid inclusion for free”. Yahoo!, the only search engine to offer paid inclusion, allows a website to pay for spiders to conduct regular visits to the site better ensuring all, or, at least the most important pages on a website will be scanned and have a better chance of appearing higher in the search engine results. Yahoo! has received some negative feedback for offering this service, with many claiming that paid inclusion basically allows websites to pay for placement in regular search results, and therefore skewing the whole purpose of a search engine – to provide the most relevant, unbiased search results.

And while some could argue that the new Google Sitemaps could potentially skew search results, because it is being offered free of charge to anyone, someone else could argue equally well that the new service will actually help to ensure better search relevancy because it allows any website to have its pages regularly updated, even if just for the purpose of “keeping up with the Jones'”.

Chris Winfield is the President and Co-founder of 10e20. He has written for various organizations in the past and frequently speaks with the media.