Subscribe to our RSS feed!
Popular » Botched Debate AdWords Case Funny Death By Blogging Wikipedia Rant Craigslist Money Link Bartering

10 Ways To Increase Pages Indexed


Or how to make Google pay more attention

For a while now webmasters have fretted over why all of the pages of their website are not indexed. As usual there doesn't seem to be any definite answer. But some things are definite, if not automatic, and some things seem like pretty darn good guesses.

So, we scoured the forums, blogs, and Google's own guidelines for increasing the number of pages Google indexes, and came up with our (and the community's) best guesses. The running consensus is that a webmaster shouldn't expect to get all of their pages crawled and indexed, but there are ways to increase the number.

PageRank

It depends a lot on PageRank. The higher your PageRank the more pages that will be indexed. PageRank isn't a blanket number for all your pages. Each page has its own PageRank. A high PageRank gives the Googlebot more of a reason to return. Matt Cutts confirms, too, that a higher PageRank means a deeper crawl.

Links

Give the Googlebot something to follow. Links (especially deep links) from a high PageRank site are golden as the trust is already established.

Internal links can help, too. Link to important pages from your homepage. On content pages link to relevant content on other pages.

Sitemap

A lot of buzz around this one. Some report that a clear, well-structured Sitemap helped get all of their pages indexed. Google's Webmaster guidelines recommends submitting a Sitemap file, too:

· Tell us all about your pages by submitting a Sitemap file; help us learn which pages are most important to you and how often those pages change.

That page has other advice for improving crawlability, like fixing violations and validating robots.txt.

Some recommend having a Sitemap for every category or section of a site.

Speed

A recent O'Reilly report indicated that page load time and the ease with which the Googlebot can crawl a page may affect how many pages are indexed. The logic is that the faster the Googlebot can crawl, the greater number of pages that can be indexed.

This could involve simplifying the structures and/or navigation of the site. The spiders have difficulty with Flash and Ajax. A text version should be added in those instances.


Google's crawl caching proxy

Matt Cutts provides diagrams of how Google's crawl caching proxy at his blog. This was part of the Big Daddy update to make the engine faster. Any one of three indexes may crawl a site and send the information to a remote server, which is accessed by the remaining indexes (like the blog index or the AdSense index) instead of the bots for those indexes physically visiting your site. They will all use the mirror instead.

Verify

Verify the site with Google using the Webmaster tools.

Content, content, content

Make sure content is original. If a verbatim copy of another page, the Googlebot may skip it. Update frequently. This will keep the content fresh. Pages with an older timestamp might be viewed as static, outdated, or already indexed.

Staggered launch

Launching a huge number of pages at once could send off spam signals. In one forum, it is suggested that a webmaster launch a maximum of 5,000 pages per week.

Size matters

If you want tens of millions of pages indexed, your site will probably have to be on an Amazon.com or Microsoft.com level.

Know how your site is found, and tell Google

Find the top queries that lead to your site and remember that anchor text helps in links. Use Google's tools to see which of your pages are indexed, and if there are violations of some kind. Specify your preferred domain so Google knows what to index

Digg This! StumbleUpon This!
AddThis Social Bookmark Widget

About the author:
Jason Lee Miller is a WebProNews editor and writer covering business and technology.

Comments

Page Rank Factor

Will having two sitemaps confuse Google?

Defintily Great Info

I have a client that insists on releasing thousands and thousands of pages at time, and I have told them that its not a good idea.  So they launched over 40000 at once and it took a while for them to be indexed.  The best thing to do in this case is alot of internal linking from other pages to these pages to show they are not as spammy as they may look.

Thanks for the tips.

Thanks for the tips. Backlinks are important indeed. A bit of patience and all will be well!

All the best,

David

Self-Help Ebooks

I like these tips, they make

I like these tips, they make pretty good sense. I like this because it is not very common knowledge. Infact, I think I only knew about 2/3 of the topic on this.

Increasing number of pages indexed

Thanks a lot Jason for this high quality aritcle!

 

free online Excel tutorial training

SEO for wordpress blogs and getting google to index your site-po

great info but there are still things you can do on your wordpress blog to attract the google bots...this seo wordpress plugin is critical if you want high pr and consistent targeted traffic. http://seowordpressninja.com

it wont give you pr 8 overnight but if you follow the info hear and have your blog working for you then the traffic will take care of itself...thats why its critical to make sure your blog is set up properly.

Problem of indexing my web pages

 I use google webmaster tools for my website. Before submission of sitemap of my Oracle website to google, almost 100 pages are indexed, after submission of sitemap it has reduced to 60. And i have included all the urls in the sitemap.

Thanx for the tips! Very

Thanx for the tips! Very useful!

http://www.houseplanet.dj/

Indexing

I have a site called www.newsgaze.com featuring current events and news. My problem is only my home page get indexed and not the inside pages. How can I include the other pages also in index as I will be changing the content on a daily basis. Can anybody help on this???

Page rank

I do agree. Using Google webmaster tools help. I have been using them from about 2 months, i have seen improvement in rank when i used Google's sitemap. (if anybody want google's sitemap help, let me know. Email: leatherbull@live.com) plz also visit my websites. Especially Bikers all over the world! http://www.leatherbull.com & http://www.frontiercycle.com Thanks, AM

Page rank

I do agree. Using Google webmaster tools help. I have been using them from about 2 months, i have seen improvement in rank when i used Google's sitemap. (if anybody want google's sitemap help, let me know. Email: leatherbull@live.com) plz also visit my websites. Especially Bikers all over the world! http://www.leatherbull.com & http://www.frontiercycle.com Thanks, AM

good tips

Thanks for the tips, and the comments.  I do pretty well with my site www.sourceoneinc.com , but have recently lauched a blog and only google has picked it up.  The tips about yahoo ping service were great.  

I agree though, no matter how fresh and relavant the content is, the key is to get more links to your site. 

Tried everything

I'm still struggling to have my site climb up to the search engines, and ofcourse get all my pages indexed but my site is still at the last spot. How could I improve this?

indexing webpages

One way that I found seems to have helped and that is indexing with other search engines.
Especially the older Search engines such as Alta Vista.
Not sure  how it works but the ranking with Google might be connected with some form of competition. If a website is popular on one search engine then this might trigger Google to also index it.
I can not see Google sitting on its laurels while other SE index websites.
If they did then they might not be worth the advertising revinue.
Especially if the website attracts 1,000's  or even millions of visits.

I did everything... but...

I think everything is OK... but to me, what really matters is how many external links your web site have...

Google is driving me mad!

Google is changing rules so frequently that I suspect nobody can guess how to definetely optimize his/her site.

I´m always tryng to follow the advises of this newsletter, you always publish great tips!!

 

regards

Rub

http://www.peninsulapm.com

 

Great

Great article! 

Ex...http://articlebankloan.blogspot.com/

How to get your pages indexed on google

I have a very high ranking web site under the search term Cozumel Mexico which is very targeted for my purposes.

I write web sites that are content driven and revolve around Cozumel Mexico, activities, tours, points of interest, etc.

When I want a web site to be indexed quickly I put a link on if with relevent anchor text and of course it has to be optimized for, and with similar content that applies to Cozumel Mexico. Car rentals, fly fishing, snorkeling or whatever is appropriate gets a link on my index page for a short time and in no time at all I find that it is indexed. It is just that simple.


I hope that this helps!

 

Bob

Link building

Link Building

There are countless methods to increase site ranking and a good search engine optimization company is definately recommended to point you in the right direction.

One area of importance is links back to your site. This was mentioned in the 10 ways - and is another way to bolster your position on the web.

links back

One thing that I've seen, is a photographer  I help others with their websites. I have a videographer who has a 1 google rating and yet by the number of links she has through industry weblistings, she shows up on the first page by googling her town and videographer. 

I was trying to help her out and now I'm listing myself on these same sites. This works for photographers, videographers and DJs.

I also link to strategic partners on my word press blog, Vistaphotography, where I chat about the clients I have and the weddings I shoot and I try to offer useful tutorials on how to use photo editing software.

These are all good ideas above to think about.

Jeff

PR0 get index more than 50%

Yes i agree to my web site American Insurance Information  get index over 50% of my site too.

Graet article!

Thanks for the great tips, the page load thing is something that I had never heard of, might have to look into that.

In my case I continue with

In my case I continue with my effort of having two sitemaps. But the main focus remains on the page content, quality inbound links to the pages and last but not the least the size of the page with proper title and description.

This helps me a lot to get my pages indexed in no time.

Indexing

Hi

I've recently added a lot of pages to my site. Once I did this I generated a new Google Sitemap and resubmitted it. Within a couple of days, my indexed pages increased from 43 to 86.

A couple of days ago, I added another lot of pages to the site. Once again, I generated a new Google Sitemap and resubmitted it. Webmaster Tools now shows under Sitemap Statistics that there are a total of 180 urls and 86 indexed urls. I'm expecting all 180 pages to be indexed within the next few days. I'll let you know what happens!

Peter

Getting pages index

If you want to get more new pages indexed within 24 hours into yahoo or google one of the best ways is to have a blog either on blogger or on your site. I have a blog updated weekly about sell house and rent back  Then you can ping the blog at yahoo and it will crawl the blog within hours. Make sure you put a link to your new pages in your blog though only 1 don't spam it because it will know. Blogs are their to keep fresh upto date content on the net. So if you had a new topic on your blog about sell house to rent back ; write the new content and ping it at yahoo.

3 Easy Steps

Yeah, everything that's been writen above is spot on and it all counts...

What I boil down to is the 3 easy steps to get indexed "FACTOR" hehe, sounds eery!

1. Good Hosting Company

2. Google Sitemap

3. Fresh new content daily (at least for the first week or two)

You do that, you'll not only have all your pages indexed but you'll have google coming back on a regular basis which means new content will get indexed alot quicker.

Hope that helps.

update speed

i think the speed  of  website update is very important.

thanks.

thanks.

Site Maps

I submitted my sitemap to google but received an error message stating the format was not valid.  What next?  My site is pretty simple in format so I wasn't expecting any problems. . .

Great article!  The comments are as informative as the article itself!

Google Indexing - my experience

I am no expert but i found from my experience that the following method works in combination:

1. There have to be at least 8 (eight) content rich pages - not contact, link to us, disclaimer etc.

2. Each content page to be submitted to Google for inclusion of url.

I found my 3rd or 6th page appearing on top in search results - may be because it had a better keyword density etc.

Make some new sites

My rankings on Casale went up when I developed some new sites with food information (e.g., Scrippelle and Fettucine) on Wordpress.  Then link these to your site of interest.     Takes a bit of time but costs nothing.

Don't forget good old HTML conventions and page/site structure

Don't overlook the basics of good page and site structure -- building your pages so they're not only user-friendly, but search engine friendly too...

First of all - here's the most important thing(s) to remember:

The search engines want your content. Period. Your content is their primary resource for business and revenues. ...But if (a.) your website cannot be traversed by automated, text-reading spiders, and/or (b.) your pages have no distinguishing (descriptive) features, then you're putting up barriers that impact the spiders' ability (or in some cases, "desire") to index your site and content.

Here are some tips on page and site structure:

1. Make sure EVERY PAGE has a unique HTML TITLE -- Use a descriptive title that actually relates to the content of that page, and which also provides proper context to readers who first see this title on on Google and other search results - NOT on your website... For example, if you're a political commentary site, writing another political article about President Bush, DON'T assume this title is viewed by your existing readers (such as by using a less formal title like "About Bush and his <whatever topic>"). Instead, make sure every title (and Description, noted below) provides a stand-alone and accurate summary of your content (like "President George Bush and his <whatever topic>" (or "George W. Bush", to avoid confusion with his father)), so that (a.) the title will make more sense to people scanning Google search results, and (b.) so that Google et al can more accurately index AND catalog your content, and find better matches between your pages and their users' search phrases.

2. Make sure EVERY PAGE has a unique, stand-alone summary of the page content in the form of a META description tag.
 
If your page is missing a description, then the search engines will try and figure out a summary for you. Read that last sentence again... Yes, it's true - and this is a significant indicator of how much the search engines want your content. They will do this two ways:

First, if you don't provide any META description tag, but your page content is displayed in a fairly straight-forward manner, the search engines will make a "guess" as to the topic of the page (using complex keyword weight algorythms), and then they'll extract a bit of text from around the sentence(s) or paragraph that is most weighted towards the topic they *think* you've written about. That's bad. Computers are dumb, and rarely get the topic correct.

Alternately, if there's no description and the topic of the page is too hard to figure out, search engines will see if your site has a DMOZ listing and will extract the summary of your website provided by the human editors there. Again, this is bad... Why force them through all this trouble, which will only result in a description that is not entirely accurate, when you could just write a few sentences yourself and put them into the META DESCRIPTION tag?
 
Remember, your META descriptions become the "teaser" text you see under the links to your articles or pages on Google and other engines' search results pages... so be sure that description is adequate to describe the page and your business to someone who has NO idea who you are, and who is NOT already on your website. The description must provide a stand-alone summary for potential visitors who are NOT on your website, and who have no other context with which to associate one article or page from another.
 
When a link to one of your pages appears on the search results of one of the major search engines, you had better make sure the title and description are interesting AND ACCURATE enough to make that link look better than the other 20 on that results page they're looking at.

Note: Never EVER try and "game" search engines by using descriptions or titles that don't actually relate to the content on your pages. They're smart enough to check that now, and it only takes one or two mistakes like this to get your entire website banned from all major search engines.

3. Use good old HTML hierarchical conventions. The H1 tag must be the first, main visible title, followed by your normal paragraph text, and then also use H2, H3, etc as needed for any subheaders.

4. Remove (never use) META KEYWORDS tags. The search engines have pretty much ignored them for many years now. Or, I should be more accurate: KEYWORDS tags do nothing to add to your favorable scores with the search engines, but they are used as a factor of your negative ratings. It's best to just remove them, and be safe. No reason to temp fate and mess with something that has clearly been abused since the very start of the tag.

5. Dynamic drop-down menus, fancy Flash animations and javascript or form-based navigation are not spiderable, and therefore none of that content and links would be found or read by the search engines...

Search engines generally only follow text (or standard HREF) links, and don't read inside of javascripts or DHTML menu scripts. You can use CSS visible/hidden based menus, which load all text and links into the source code where it can be read by spiders, or at the very least, add a "Site Map" link into your page header and footer, which links to a page where you have plain, simple HTML HREF links to every single page on your site.

6. Yes, provide a "Site Map" -- either in your own format, or Google's XML format), so Google and others can more easily find all pages on your website (but even this is no guarantee of indexing, because again, if Google and others can't figure out the topic or any descriptions for your pages, they will not index or properly catalog them).

7. Dynamic URLs are NOT A PROBLEM for any search engine -- **UNLESS** (a.) your URLs and site are designed such that following links might create "spider-traps" for the crawlers (where search engine spiders get caught in an infinite looping of links within your site - such as with calendar links that go to infinite number of future and past months - at which point they will simply abandon your site), or (b.) when dynamic variables are appended to URLs (such as datestamps or session IDs) which would create the problem of duplicate content (where the same page is reachable through different URLs).

And that's all... Hope this helps!
TK

 

Link Structure And Getting More Pages Indexed

Hi there, over the last few months I have been monitoring my site's new pages and notice very clearly that when I started restructuring my links by adding the full url meaning instead of /marketing.htm I started using  http://www.wealth-plus.com/marketing.htm that those pages seem to get  index as soon as the site was cached.  This all came about when I was creating a site map and the comments from the software kept saying that certain links were from another domain and those were pages I had in folders and as soon as I restructured to full links those comments  nolonger came up.

Mark Felix

googles

So is it true that google will only index half your site if your page rank is below 3.

Even Website have 0 PR still google indexes all pages

Hello,

"So is it true that google will only index half your site if your page rank is below 3"

I am not agree with the comment above because you will find lots of pages from diffrent sites indexed which have 0 Page Rank,

Just Check all pages indexed by google of http://www.3seo.com/  you will find many pages which have 0 Page Rank are also indexed. Total 452 pages indexed.

Now a days I find google gives more weightage to domain name, it means if your keyword comes in domain name it will achieve very good ranking even if that domain have 0 PR

for example if you search keyword "plc scada training" in google then you find http://www.plcscadatraining.com comes on 4 th Rank and I achieve this position within 8 days when that page have 0 PR

 

Dual Sitemaps

Great article! Worth noting that it is a good idea to several "paths" for spiders accessing your site. I always favour 2 sitemaps - both HTML and XML to be certain that all bots can get around the site.

And don't forget to check for broken links occasionally, if the pages of your site are being crawled via maps this can often mask problems with your site navigation, conversely a page with no visitors in your stats could point to the same issue.

Getting more pages indexed on Google;

Ive always thought the menu on your site itself is very important. There is no point having a Javascript menu that Google crawler cant read and no other way of a crawler getting from page to page on your site. But if you do have a Javascript or even a Flash menu, have a plain html version along the bottom of each page. This solves the problem, and ofcourse makes your site that bit more accessible.

Also if you are into SEO, and do a lot of linking, try getting links pointing back to all your important pages, and not just the homepage. That would ensure that pagerank will be spread more evenly throughout your website.

getting all pages index by google

One way to get all your pages index by google is to create an RSS feed for all you menu items. Second, display new articles on your home page via RSS. Third, create a sitemap with all of your pages. Break down site map into catergories with not more than 100 links

How can I improve my sites?

I´ll study Your comment

"10 Ways To Increase Pages Indexed "

Best Regards, H. Eckert

 

Indexing of Web Pages

Thanks Jason for providing such nice information. I would also like to give some information about Indexing of Google

Today nearly 50% of the traffic is generated by the Google. This is a greatest revolution in the internet scenario. This revolution is done by two PhD researcher Lary Page & Sergey Brin & make Google a company on the international scene.

Google becomes popular due to his algorithm which is made out by the two founders & also by making his presentation very simpler, because "the simplest things are sometimes most effective".Google has a very simple interface, without doing any advertising & only concentrating only on services of searching & nothing else.

In addition of these search results, Google also succeeded to indexing of large number of webpages approximatelt 2 billion of pages. Recently it also includes new type of documents like: Word, Excel, PDF files, PowerPoint slides, Worpad.

The algorithm is based on two systems

  1. a precise analysis of the contents of the indexed pages (keywords, occurrences, positions in the document, type of HTML tag, etc.)
  2. a classification of the pages according to their popularity (PageRank), calculated from the topology of the Web (i.e. the whole structure of the documents and the links between them).

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

The content of this field is kept private and will not be shown publicly.
CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
1 + 1 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.
Subscribe to WebProNews


Send me relevant info