Tips for Getting Crawled Faster by Google

Google Presents Examples

Get the WebProNews Newsletter:

[ Search]

Probably the most important step in getting your site found in a search engine is the one in which the search engine crawls it. There are things that can be done and things that can be avoided to make this process as painless as possible for the search engine, which will in turn, make it as painless as possible for the webmaster.

Since Google dominates the search market share by such a large market share, it is always a good idea to listen to what they have to say about such matters. So when they post a presentation with tips on optimizing crawling and indexing, you’ll probably want to pay attention.

Google has done just that, highlighting things to stay away from, and things you can do to enhance your site’s crawlability. Here is that presentation with specific examples of URLs.

"The Internet is a big place; new content is being created all the time," says Google Webmaster Trends Analyst Susan Moskwa. "Google has a finite number of resources, so when faced with the nearly-infinite quantity of content that’s available online, Googlebot is only able to find and crawl a percentage of that content. Then, of the content we’ve crawled, we’re only able to index a portion."

"URLs are like the bridges between your website and a search engine’s crawler: crawlers need to be able to find and cross those bridges (i.e., find and crawl your URLs) in order to get to your site’s content," continues Moskwa. "If your URLs are complicated or redundant, crawlers are going to spend time tracing and retracing their steps; if your URLs are organized and lead directly to distinct content, crawlers can spend their time accessing your content rather than crawling through empty pages, or crawling the same content over and over via different URLs."

If you want to get crawled faster by Google, you should remove user-specific details from URLs. Specifics of this can be viewed in the slideshow.  Basically, URL parameters that don’t change the content of the page, should be removed and put into a cookie. This will reduce the number of URLs that point to the same content, and speed up crawling.

Google says infinite spaces are a waste of time and bandwidth for all, which is why you should consider taking action when you have calendars that link to infinite numbers of past/future dates with unique URLs, or other paginated data.

Tell Google to ignore pages it can’t crawl. This includes things like log-in pages, contact forms, shopping carts, and other pages that require users to perform actions that crawlers can’t perform themselves. You can do this with the robots.txt file.

Finally, avoid duplicate content when possible. Google likes to have one URL for each piece of content. They do recognize that this is not always possible though (because of content management systems and what have you), which is why the canonical link element exists to let you specify the preferred URL for a particular piece of content.

Tips for Getting Crawled Faster by Google
Top Rated White Papers and Resources
  • http://www.lexolutionit.com Maneet Puri

    These are very generic tips. I guess all SEO professionals would know about it. I expected Google to shell out something more exclusive…

    • Chris Crum

      I think sometimes it helps to consider that not everyone is a seasoned pro. There are always going to be people who still have to learn. Lots of webmasters who can’t afford to hire SEOs may appreciate tips for how they can improve their crawlability on their own.

      • http://www.lexolutionit.com Maneet Puri

        Yeah… I suppose so. Just that when I read the title of the post, I got all excited about what would follow. And was disappointed after I read through all of it. Anyways, you are doing a good job with your posts :)

  • http://www.searchandhra.com/ Telugu Cinema News

    Yes google is lot of information resource. Great information Thank U

  • http://www.controldatainc.com agency collection

    Great tips. The cost of hiring someone for SEO has gone through the roof. Thanks for the tips on this I look for more. As mentioned not everyone is a pro so these type of releases are worth their weight in gold.

  • http://www.playingcardsandmore.com Tom

    What is the consensus opinion regarding hyphenated URLs? Good idea or not? Thanks in advance.

  • http://seodesktophelp.blogspot.com Liju Eliazer

    Chris what you meant by ” user-specific details “. Can you please tell me?

    Advance thanks

  • http://www.lerentech.com Web Design Syracuse

    I’m glad Google is trying to help and I’m sure as someone pointed out the tips are useful, but I agree they are generic and I was hoping for something more in depth. I don’t think we’ll ever get that from Google though. Thanks, though, Chris for keeping us up on the new information coming out on Google and search engines in general. I’m glad I signed up for the webpronews e-mail.

  • http://www.seoservicecompanybangalore.blogspot.com/ SEO Services Bangalore

    How to remove user-specific details from URLs?

  • Guest

    Thanks Chris !
    I Just tuned up my site page URL’s.
    Just what the Doctor Ordered :o)
    Now the Fireworks Begin ! ! ! !

  • http://www.homesweetdollars.com Home Workers

    Good advice. No need to drown by spending big money on SEO.
    With Google’s Webmaster Toolbox and a little of common sense you can keep your head above the water.

    Thank you.

  • http://www.nomoref.com/ Dayo

    What is user-specific details “. could you please tell us more about this?

  • http://www.ripbluray.us rip blu-ray

    I have used it and found that it is very good.blu-ray to mp4 converter, blu-ray burner

  • http://www.lancashireroseblinds.co.uk steve

    any thing that helps is good. Thanks for sharing

  • https://www.consumersdiscountrx.com/instameds/index Instameds

    sweet maximizATION through purification. right on !
    do we enjoy diseasses, and need a few simple cheap cures ?
    Look no further – https://www.consumersdiscountrx.com/instameds/index

  • http://www.smallbizexpress.com steve

    This document is really coool. I would love to download if u can share me the linkf for download.

  • http://www.health-avenue.co.cc HealthAvenue

    Eventhough is generic but I find it refreshing for a new webmaster. In my opinion having adsense do assist in our website/blog being crawled fast as it has to ascertain what ads to place there.

  • http://www.udjuni.com Simon

    You may think that these are old tips, but you will be amazed by how many webmasters do not know or follow them. Thanks for sharing!

  • http://westcoastvinyl.wordpress.com West Coast Vinyl

    Great Tips to know for not only new web pages but for updating web as well!

  • http://www.bikeshopcastlehill.com.au Home Solar Power Systems

    Thanks for sharing this article. Really a good one

  • http://www.work-at-home-online.com/ Maqsood

    So I learned some insight on GoogleBot and web crawling. Thanks for the post.

  • http://www.zygella.com Zygella

    Great tips but very basic. Thnaks

  • http://www.independentcomputerconsultant.com computer consultant

    Thanks for sharing. useful info.

  • http://www.videosplittermac.com/ Video Splitter for Mac

    Thanks for your useful article.

  • http://www.SpunkMarketing.com Kevin Webb

    I gave up on trying to “get indexed” in a certain period of time. I found that at the end of the day… what I do doesn’t really matter to a certain degree.

    I have had websites build structured and marketed exactly to a “T” and still found the search results (timing) to be on a different pace. Once site would get indexed within 48 hours and the other may take several weeks.

    Same blogs pointing, same SEO work done, same meta infrastructure, linking, submitting, etc.

    At the end of the day there are still too many factors that can’t be controlled.

  • http://www.plantinhamilton.com Hamilton – Free Marketing Tips

    I’m glad Google is trying to help and the tips are very useful, Yes Google has lot of info resource.

    Thanks for keeping us update from Google and search engines in general.

    Great information Thank U

  • http://www.livetrainingsession.com Live Training Session

    Anothe great post Chris. I have always believed in helping search engines find information easier in my web sites. The less time they spend going back and forth the same alleys the better they will use their resources to index valuable information.

    We all have the responsability to make the web better, a business place just like the offline world. A little education through Google Webmaster pages is helpful to all. Let’s help Search Engines help us.

  • http://www.socialterrain.com Guest

    Hi Chris

    Great article, would love you to help me figure a way to get them to crawl faster http://www.socialterrain.com is a new and up coming social network, with open blogs, and groups. These are crawlable, but because we have so many I need help. 😉

    Ping me back if you can


  • Join for Access to Our Exclusive Web Tools
  • Sidebar Top
  • Sidebar Middle
  • Sign Up For The Free Newsletter
  • Sidebar Bottom