Thanks Jason for providing such nice information. I would also like to give some information about Indexing of Google

Today nearly 50% of the traffic is generated by the Google. This is a greatest revolution in the internet scenario. This revolution is done by two PhD researcher Lary Page & Sergey Brin & make Google a company on the international scene.

Google becomes popular due to his algorithm which is made out by the two founders & also by making his presentation very simpler, because "the simplest things are sometimes most effective".Google has a very simple interface, without doing any advertising & only concentrating only on services of searching & nothing else.

In addition of these search results, Google also succeeded to indexing of large number of webpages approximatelt 2 billion of pages. Recently it also includes new type of documents like: Word, Excel, PDF files, PowerPoint slides, Worpad.

The algorithm is based on two systems

1. a precise analysis of the contents of the indexed pages (keywords, occurrences, positions in the document, type of HTML tag, etc.)
2. a classification of the pages according to their popularity (PageRank), calculated from the topology of the Web (i.e. the whole structure of the documents and the links between them).

One way to get all your pages index by google is to create an RSS feed for all you menu items. Second, display new articles on your home page via RSS. Third, create a sitemap with all of your pages. Break down site map into catergories with not more than 100 links

Great article! Worth noting that it is a good idea to several "paths" for spiders accessing your site. I always favour 2 sitemaps – both HTML and XML to be certain that all bots can get around the site.

And don’t forget to check for broken links occasionally, if the pages of your site are being crawled via maps this can often mask problems with your site navigation, conversely a page with no visitors in your stats could point to the same issue.

Don’t overlook the basics of good page and site structure — building your pages so they’re not only user-friendly, but search engine friendly too…

First of all – here’s the most important thing(s) to remember:

The search engines want your content. Period. Your content is their primary resource for business and revenues. …But if (a.) your website cannot be traversed by automated, text-reading spiders, and/or (b.) your pages have no distinguishing (descriptive) features, then you’re putting up barriers that impact the spiders’ ability (or in some cases, "desire") to index your site and content.

Here are some tips on page and site structure:

1. Make sure EVERY PAGE has a unique HTML TITLE — Use a descriptive title that actually relates to the content of that page, and which also provides proper context to readers who first see this title on on Google and other search results – NOT on your website… For example, if you’re a political commentary site, writing another political article about President Bush, DON’T assume this title is viewed by your existing readers (such as by using a less formal title like "About Bush and his <whatever topic>"). Instead, make sure every title (and Description, noted below) provides a stand-alone and accurate summary of your content (like "President George Bush and his <whatever topic>" (or "George W. Bush", to avoid confusion with his father)), so that (a.) the title will make more sense to people scanning Google search results, and (b.) so that Google et al can more accurately index AND catalog your content, and find better matches between your pages and their users’ search phrases.

First, if you don’t provide any META description tag, but your page content is displayed in a fairly straight-forward manner, the search engines will make a "guess" as to the topic of the page (using complex keyword weight algorythms), and then they’ll extract a bit of text from around the sentence(s) or paragraph that is most weighted towards the topic they *think* you’ve written about. That’s bad. Computers are dumb, and rarely get the topic correct.

3. Use good old HTML hierarchical conventions. The H1 tag must be the first, main visible title, followed by your normal paragraph text, and then also use H2, H3, etc as needed for any subheaders.

4. Remove (never use) META KEYWORDS tags. The search engines have pretty much ignored them for many years now. Or, I should be more accurate: KEYWORDS tags do nothing to add to your favorable scores with the search engines, but they are used as a factor of your negative ratings. It’s best to just remove them, and be safe. No reason to temp fate and mess with something that has clearly been abused since the very start of the tag.

5. Dynamic drop-down menus, fancy Flash animations and javascript or form-based navigation are not spiderable, and therefore none of that content and links would be found or read by the search engines…

Search engines generally only follow text (or standard HREF) links, and don’t read inside of javascripts or DHTML menu scripts. You can use CSS visible/hidden based menus, which load all text and links into the source code where it can be read by spiders, or at the very least, add a "Site Map" link into your page header and footer, which links to a page where you have plain, simple HTML HREF links to every single page on your site.

6. Yes, provide a "Site Map" — either in your own format, or Google’s XML format), so Google and others can more easily find all pages on your website (but even this is no guarantee of indexing, because again, if Google and others can’t figure out the topic or any descriptions for your pages, they will not index or properly catalog them).

7. Dynamic URLs are NOT A PROBLEM for any search engine – **UNLESS** (a.) your URLs and site are designed such that following links might create "spider-traps" for the crawlers (where search engine spiders get caught in an infinite looping of links within your site – such as with calendar links that go to infinite number of future and past months – at which point they will simply abandon your site), or (b.) when dynamic variables are appended to URLs (such as datestamps or session IDs) which would create the problem of duplicate content (where the same page is reachable through different URLs).

And that’s all… Hope this helps!
TK

I submitted my sitemap to google but received an error message stating the format was not valid.  What next?  My site is pretty simple in format so I wasn’t expecting any problems. . .

Great article!  The comments are as informative as the article itself!

i think the speed  of  website update is very important.

Hi

I’ve recently added a lot of pages to my site. Once I did this I generated a new Google Sitemap and resubmitted it. Within a couple of days, my indexed pages increased from 43 to 86.

A couple of days ago, I added another lot of pages to the site. Once again, I generated a new Google Sitemap and resubmitted it. Webmaster Tools now shows under Sitemap Statistics that there are a total of 180 urls and 86 indexed urls. I’m expecting all 180 pages to be indexed within the next few days. I’ll let you know what happens!

Peter

Thanks for the great tips, the page load thing is something that I had never heard of, might have to look into that.

I have a very high ranking web site under the search term Cozumel Mexico which is very targeted for my purposes.

I write web sites that are content driven and revolve around Cozumel Mexico, activities, tours, points of interest, etc.

When I want a web site to be indexed quickly I put a link on if with relevent anchor text and of course it has to be optimized for, and with similar content that applies to Cozumel Mexico. Car rentals, fly fishing, snorkeling or whatever is appropriate gets a link on my index page for a short time and in no time at all I find that it is indexed. It is just that simple.

I hope that this helps!

Bob

I think everything is OK… but to me, what really matters is how many external links your web site have…

I’m still struggling to have my site climb up to the search engines, and ofcourse get all my pages indexed but my site is still at the last spot. How could I improve this?

Thanx for the tips! Very useful!

http://www.houseplanet.dj/

I like these tips, they make pretty good sense. I like this because it is not very common knowledge. Infact, I think I only knew about 2/3 of the topic on this.

I have a client that insists on releasing thousands and thousands of pages at time, and I have told them that its not a good idea.  So they launched over 40000 at once and it took a while for them to be indexed.  The best thing to do in this case is alot of internal linking from other pages to these pages to show they are not as spammy as they may look.

I think one of the best ways is to get links back.  Getting websites indexed is all find and good, but there is this notion of the supplemental index that is index purgatory.  The only way out is by getting quality links back to those pages to lift them out of the supplemental and into the primary index so they have a better chance of being found.