Quantcast

10 Ways To Increase Pages Indexed

Or how to make Google pay more attention

Get the WebProNews Newsletter:


[ Search]

For a while now webmasters have fretted over why all of the pages of their website are not indexed. As usual there doesn’t seem to be any definite answer. But some things are definite, if not automatic, and some things seem like pretty darn good guesses.

So, we scoured the forums, blogs, and Google’s own guidelines for increasing the number of pages Google indexes, and came up with our (and the community’s) best guesses. The running consensus is that a webmaster shouldn’t expect to get all of their pages crawled and indexed, but there are ways to increase the number.

PageRank

It depends a lot on PageRank. The higher your PageRank the more pages that will be indexed. PageRank isn’t a blanket number for all your pages. Each page has its own PageRank. A high PageRank gives the Googlebot more of a reason to return. Matt Cutts confirms, too, that a higher PageRank means a deeper crawl.

Links

Give the Googlebot something to follow. Links (especially deep links) from a high PageRank site are golden as the trust is already established.

Internal links can help, too. Link to important pages from your homepage. On content pages link to relevant content on other pages.

Sitemap

A lot of buzz around this one. Some report that a clear, well-structured Sitemap helped get all of their pages indexed. Google’s Webmaster guidelines recommends submitting a Sitemap file, too:

· Tell us all about your pages by submitting a Sitemap file; help us learn which pages are most important to you and how often those pages change.

That page has other advice for improving crawlability, like fixing violations and validating robots.txt.

Some recommend having a Sitemap for every category or section of a site.

Speed

A recent O’Reilly report indicated that page load time and the ease with which the Googlebot can crawl a page may affect how many pages are indexed. The logic is that the faster the Googlebot can crawl, the greater number of pages that can be indexed.

This could involve simplifying the structures and/or navigation of the site. The spiders have difficulty with Flash and Ajax. A text version should be added in those instances.

Google’s crawl caching proxy

Matt Cutts provides diagrams of how Google’s crawl caching proxy at his blog. This was part of the Big Daddy update to make the engine faster. Any one of three indexes may crawl a site and send the information to a remote server, which is accessed by the remaining indexes (like the blog index or the AdSense index) instead of the bots for those indexes physically visiting your site. They will all use the mirror instead.

Verify

Verify the site with Google using the Webmaster tools.

Content, content, content

Make sure content is original. If a verbatim copy of another page, the Googlebot may skip it. Update frequently. This will keep the content fresh. Pages with an older timestamp might be viewed as static, outdated, or already indexed.

Staggered launch

Launching a huge number of pages at once could send off spam signals. In one forum, it is suggested that a webmaster launch a maximum of 5,000 pages per week.

Size matters

If you want tens of millions of pages indexed, your site will probably have to be on an Amazon.com or Microsoft.com level.

Know how your site is found, and tell Google

Find the top queries that lead to your site and remember that anchor text helps in links. Use Google’s tools to see which of your pages are indexed, and if there are violations of some kind. Specify your preferred domain so Google knows what to index

10 Ways To Increase Pages Indexed
Top Rated White Papers and Resources
  • http://www.demtron.com/Search-Engine-Optimization.aspx SEO Consultant

    The less junk that Google has to cut through in the HTML, the better.  From my experience, this is especially true of sites have been handled by multiple designers and not run through validators recently.  Too much clutter = inability to crawl.

  • http://www.efluencemarketing.com St. Louis Web Design

    Thanks for the useful information.
    I know that the use of stylesheets certainly helps cut down the clutter in the HTML

  • http://hitratesolutions.com/ SEO Outsourcing

    Launching a large number of links and content all at once doesn’t seem to have as large an impact as continuous, steady additions.

  • http://vb.maas1.com/t77254.html ???? ????

    thanks for post

  • simla

    I have uploaded sitemap for my website in google webmaster tool. my sitemap can be found at www.example.com/sitemap.xml. 1938 urls in my sitemap.Now sitemap status shows: indexed url is only 6. Why google not index my sitemap fully? Does anyone know how to solve this problem?

  • http://www.tinafountain.com/blog/ Tina

    So many people say the pagerank means nothing, yet it’s interesting that Google keeps it around. You may be correct that it’s used to determine the depth that they crawl a site, thanks!

  • http://www.palmsolutions.dk/ Guest

    “Some blogger claimed he could get a post indexed in 10 minutes, not sure how that works though!”

    In my experience new articles are indexed within 10 minutes with a WordPress blog. The blog software “pings out” (not sure on the correct term) that a new article has been posted -> Google arrives 10 minutes later.

  • http://www.rankbetterseo.com/seo-web-design.php charlotte web design

    That’s some great info Jason.

  • http://www.smartbusinessweb.com Minneapolis Web Design

    Good tips! Design a search engine friendly site is the key. A lot of people think that spiders can sort out what is content and what is html, javascript or css codes. It is true, but if you make the job easier for spiders, your site gets much better indexed. One very useful technique is to create good internal links that guide spiders from one page to another. This is often ignored by most web designers. A sitemap can help. However, if a site is well designed for spider to crawl, you may not need the sitemap if the site is relatively small and not a huge ecommerce site.

  • http://seo-newtechupdates.blogspot.com/ sri78

    As a newbie in SEO, there could be a lot of things to know more. It is easy to say doing that than actually do it. I am still in experiment how things work well in this category of page indexing. But thanks for this post. Got something new.

  • http://www.4uricinex.com Uricinex

    Jason,
    Great article. You are very right, at the end of the day it is all about content.

  • http://www.bocaexecutiverealty.com Boca Raton Realtor

    I can donfirm that PageRank will effect how many pages are indexed. The higher your pagerank the deeper googlebot will crawl. That means both on a site wide basis as well as on a per page basis.

  • http://swineflumasks.co.cc http://swineflumasks.co.cc

    useful info, thanks!

  • http://www.superbingoslots.com/ superbing

    I have learned a lot really very useful for newbies for reference. Thanks a lot….

  • http://www.mailinglistarchive.com mailing list archive

    Very helpful info.

  • http://www.bingofreeuk.com bingo freeuk

    Actually I am beginner to business marketing and frm me there are no words….Really helped me a lot and from now onwards its my fav blog to refer.

  • http://www.dcpweb.co.uk DCP Web Design

    Some good tips – thanks – I would also recommend updating your home page on a frequent basis, I use news items and articles.

  • http://www.myfri3nd.com myFri3nd

    Thank you. It’s very useful.

  • http://www.trafficstacker.com/links/1897 David

    I would like to be on the top 5 in Google, msn yahoo but i don’t know how to get there

  • http://www.atcomaart.com atco

    Hi,

    Its too informative post, I learn a lot from this post.
    I have b2b portal which has 60000 Pages but Google Can only crawl 1500 pages, I can’t able to find the reason, And After reading your post, I got the point is, it takes too much time for page loading. and Its is in really big size or heavy size portal. check it www.atcomaart.com

    thanks

  • http://www.atcomaart.com atco

    Its a helpful tips for improving indexing of web pages.

  • http://www.liverpoollegal.com Chirnese Liverpool

    Thanks so much for the information.

  • http://www.enginesearchseo.com/ Ben Acheson

    I read your comments with interest. Our SEO team find that nothing beats deep links from high ranking sites, as you suggest. However we find that links from deeper links from the originating website are less effective and the text on the originating page – especially the text around the link itself (and above all the anchor text) make a huge difference. We see a lot of clients who have large numbers of links with poor relevance that are damaging their search engine optimisation. It can be challenging and take a long time to get these bad links removed and even longer for your PageRank to recover.

  • http://www.everythingforcity.com Guest

    Hi!

    It was a very good tips for increasing number of indexed pages.
    But can you tell me that where google give relaxation in duplication of contents.
    Take the example of my site http://www.everythingforcity.com . It is an city portal site. It has sms, fact which may be duplicate.

    we cannot change the wording of sms as its rhym got changes.

    So can you tell me that can I publish sms messages, facts which are already present on other sites.
    Please reply.

    Regards
    Vikram

  • http://www.voucherfreebies.co.uk Edina

    My best ratio of pages indexed is on a small static site. Annoying as the blogs have rss feeds, sitemaps, new content added every day and new links added…

    I think I need some help finding a sitemap generator that will make an archive site map month by month for all the posts I have.

    Going back over incoming traffic keywords and reinforcing them will help sometimes, but when you want new pages indexed it’s harder. Frustrating!

  • http://www.evapress.org Eva foaming machine

    I want to learn seo. Thank you for sharing.

  • http://www.afrofabric.com Therese Coker

    The pieces of advice I gathered from the the two articles helped our overall site ranking after they were implemented. Thanks a million for the free advice.

  • http://www.queenbeeremoval.com LiveBeeRemoval

    As Im just beginning to learn about SEO, I found this extremely helpful. Thank you for the informative article. Just what I was looking for while surfing the net and I am already using the info I learned here.

  • http://www.topseo.in Rock

    Thanks for the info.

    Just a quick question: How does the below activity help?

    Verify the site with Google using the Webmaster tools

  • http://mflhomesforsale.com Orlando Homes for Sale

    Is there a negative ranking factor for age of site? or is it just neutral?

    • http://miami-realestate.net Miami Real Estate Agent

      Yes, If you are working on a competitive search term, the domain aging will be an important factor. I believe you can still cover it with some good SEO work.

  • http://pid=681841&profile_id=58977857&profile_name=rogeralvarez717&user_id=58977857&username=rogeralvarez717 Cyril Risinger

    I do not even know the way I ended up right here, however I believed this post was once great. I don’t realize who you are but definitely you’re going to a famous blogger should you aren’t already ;) Cheers!

  • http://www.yachtofworld.com yachtofworld

    thanks for those informations
    Yachtofworld.com

  • http://www.theleader.info/spanish-property/ Spanish Property

    Good article and interesting comment from Thomas which helps to clarify the use of META tags. One question: when “duplicate content” is referred to, does that just mean content which is indexed? If content appears on different sites can it be the case that it is only indexed for one website?

  • http://mp3begins.com/ jamez

    and what about blackhat seo ? i mean is it good to use those tools to boast our sites ?

  • http://nyomanmedia.com Nyoman Media Indonesia

    Thank u For Share and Really Need This Information..

  • http://www.ozel.co.in/ Naveen

    Our keywords are dancing now-a-days i couldn’t understand that what is happening with my website.If i check my one keyword is on first page just tomorrow when i8 check it out this this will be on other position.I couldn’t understand that what is happening with my site can you please tell me about what could i do for keyword stability.

  • http://kielichlawfirm.com Adam Kielich

    Good info, thanks

  • Join for Access to Our Exclusive Web Tools
  • Sidebar Top
  • Sidebar Middle
  • Sign Up For The Free Newsletter
  • Sidebar Bottom