One of the most common problems many large web sites have with rankings is that portions of their web sites are not included in the search engines.

The cause can be any of a variety of reasons that range from complicated URL strings to slow server response or lack of a proper internal linking structure. If Google can’t find your pages, they won’t get into the search results.

Search engines benefit from having as many good quality pages in their databases as possible because it improves user experience. Better user experience equals more traffic and that means more advertising revenue. As a result, several efforts have been made to make it easier for webmasters to have their web sites indexed more effectively. This article discusses efforts Google and Yahoo have made to augment current methods of discovering as many URLs of a web site as possible.

First there was Google Sitemaps

Earlier this summer Google announced their Google Sitemaps tool that allows web site owners to create an XML or plain text file listing all or some of the URLs of a web site for inclusion in the Google index. Up to 50,000 are allowed per document and if you’re over that, you can create another site map file.

Google offers this tool as a compliment to what Googlebot, Google’s search engine spider, can find on it’s own by crawling the web. In cases where a site might have some difficulty getting indexed, providing a sitemap to Google may assist in getting missing pages into Google’s search results.

Google does not charge for this beta service and it’s pretty easy to do in three easy steps. However, as a beta, there are no guarantees. More information is available on Google Sitemaps at: https://www.google.com/webmasters/sitemaps/

What about Yahoo?

Yahoo has offered a free web site submission form for quite a while where, as a registered user, you can submit one URL at a time. There is no guarantee of inclusion and no way to know if your site is accepted except to check the Yahoo search results for your web site.

For companies that want to provide a large number of URLs to Yahoo, the Yahoo Search Submit paid inclusion program offers the opportunity to have guaranteed inclusion in Yahoo’s organic index. ( http://searchmarketing.yahoo.com/srchsb/ ) Search results display intermingled with URLs that are discovered naturally by Slurp, Yahoo’s web crawler. There is a cost per URL for inclusion and also a cost per click based on the category of your business. Unlike sponsored listings, Yahoo claims there is no effect on ranking to use the Search Submit paid inclusion program, only the guarantee that your web site will be included in the Yahoo search engine index and that all URLs will be revisited every 48 hours.

You can also pay Yahoo $299 per year for your domain name to be included in the Yahoo Directory. With this option, there is no guarantee; even in you pay, that you will be accepted or that your suggested title, description and category will be used. Submissions that are accepted into the Yahoo Directory are also automatically submitted to the organic index. There is plenty of speculation in the SEO/SEM community about the value of a Yahoo Directory link, both pro and con.

And then there was Yahoo Sitemaps

A free, bulk URL submission option for Yahoo was first presented at the recent Search Engine Strategies conference in San Jose from Yahoo’s Tim Mayer during a session on indexing. Previously, web sites were discovered for the organic search results by the Slurp web crawler or by submitting one URL at a time. Now Yahoo offers the option of submitting many URLs at once using Yahoo’s free submission service at http://submit.search.yahoo.com/free/request.

Registration for a Yahoo account is required, but free.

Here’s how it works: Create a list of all the URLs you want indexed in a text file and name it something like, urllist.txt (Yahoo’s suggestion). You can also provide a compressed file and call it urllist.gz. Example: http://www.domainname.com/urllist.txt .

If you are providing a plain text list of URLs to Google Sitemaps, then you can use that same file for Yahoo.

Place the file in the main directory of your web site and submit the URL to the Yahoo free submission form at: http://submit.search.yahoo.com/free/request.

Some sites that pay for inclusion into the Yahoo index may gain that benefit from a natural submission, provided their URLs are crawlable. The tradeoff is, it’s there is no guarantee of inclusion and if your site changes often, those changes will not be reflected in the search results until the site is update by the Yahoo web crawler. With Search Submit paid inclusion, your URLs will be updated every 48 hours.

It’s important to note that inclusion is not search engine optimization. But it is important, because if none or only part of your site is included in a search engine’s index, then you could be missing out on significant amounts of natural search traffic. Submitting a sitemap to Google and Yahoo has no known side effects and may give your site a boost in overall rankings since you’ll have more pages available to search results.

Lee Odden is President and Founder of
TopRank Online Marketing, specializing in organic SEO, blog
marketing and online public relations. He’s been cited as a search
marketing expert by publications including U.S. News & World Report and
The Economist and has implemented successful search marketing programs
with top BtoB companies of all sizes. Odden shares his marketing
expertise at Online Marketing Blog offering
daily news, interviews and best practices.