Has Google Begun Changing How it Indexes the Web?

The Holidays are Over. Is it Caffeine Time?

Get the WebProNews Newsletter:

[ Search]

Last summer Google announced a new project called "Caffeine", which was described as a re-write of Google’s web search architecture. Around that time, Matt Cutts discussed Caffeine with WebProNews, comparing it to the "Big Daddy Update" of 2005, which consisted of changes to the way Google crawls and indexes websites. It appears that more people are now seeing the effects from Caffeine out in the wild.

Have you seen possible Caffeine effects in use? Tell us about it.

Back before the holidays, Google made it a point to assure everybody that Caffeine would not be rolled out (except for at one data center) until after the holidays were over – January at the earliest. The reason for this was that Google didn’t want to shake everything up during a key time for businesses (they didn’t want a repeat of the Florida update).

The company let everyone know about its intentions at PubCon in November. In fact, a few days ago, Google’s Matt Cutts posted a video running through his presentation from that event on his blog. He also provided the slideshow. It covers much more than just Caffeine, but if you missed it, you may want to consider watching it anyway (Caffeine discussion starts at about 22:10 in the video and at slide 29 in the presentation).

"It’s a re-write of our indexing infrastructure. It’s taking the old way that we used to index things that we’d crawled around the web, and we’re replacing that with new architecture that’s fresh and that had been written to be more scalable, more flexible, [with] the ability to attach different types of data, and in the process of indexing, the ability to do more documents for a more comprehensive version of the web, and the ability to do it faster," Cutts says of Caffeine.

But enough background. Barry Schwartz at Search Engine Roundtable points to a WebmasterWorld forum thread where administrator Tedster claimed to have seen Caffeine in action at a number of IP addresses. He wrote:

I’m seeing the Caffeine data-set being served via this set of IP addresses:,,,,,

It seems to take 5 IP addresses to build the complete SERP, where in the past it often took only 3.

Schwartz also pointed to another member’s post (Whitenight), who said:

Well, just tripled checked with offices/employees in Texas, Colorado, and Indiana. All 5 "control" keywords/sites showed live Caffeine.

That member’s latest post says that the Caffeine Dataset is also on and

We don’t know for sure if this is all really Caffeine in action though. Google hasn’t commented on it, and has not made any announcements regarding Caffeine since what Matt said above. Some people don’t believe this is Caffeine at all. As Schwartz notes, we’ll have to wait for Google to say something.

Still, January is almost over, and Google said it would wait until after the holidays, specifically mentioning the month of January. It’s about time for this to be rolling out to some extent. Speed has been emphasized a significant amount in Caffeine discussion, and Cutts told us that page speed would likely become a ranking factor. Regardless of whether or not you are witnessing Caffeine in action yet, rest assured that it will be here sooner or later, and any edge you can give yourself in the meantime is for the good of your own site’s performance. Speed will not only supposedly help you in search going forward, but it just makes for a better user experience.

Share your thoughts about Google’s Caffeine update.

Related Articles:

> Matt Cutts Talks Google Caffeine Update

> New Details on Google Caffeine Update

> A Markup That Could Have Big Implications for SEO

Has Google Begun Changing How it Indexes the Web?
Top Rated White Papers and Resources
  • http://www.appliancerepairboiseidaho.com Boise Appliance Repair

    I have seen a couple of my sites for no apparent reason dropping of the google map.I wonder if this is the effect of caffeine?These were hold high ranking google sites that dropped for no reason.

  • Guest

    Our indexed pages in google decreased dramatically, we keep same number of pages in yahoo, but in google, it dropped. Does it have a caffein effect?

    I appreciate the comments.

  • http://www.bigdvdsale.co.uk BIG Pete

    My sites dropped off too. Ive been on the first page of google, when all of a sudden i did find myself on the fifth page of results in Google? WTF? I mean really : WTF!!!

  • http://www.guiaserviciosproductos.com Guest

    Hi all

    I noticed, back in December, that to of my sites dropeed down drastically. These two sites web took more than 15 seconds to load-according with webmasters tools-. So I took some images off these sites and resized some others. Now these two sites take no longer than five seconds to load and they are back to the same search Google results back in november 2009.

    I guess, Google, as they announced, is taking very seriously our site load time.


  • http://www.netalyon.com Guest

    the way of google index my website seems different.
    3 o 4 days a month my position in relative keywords get down or disapear of the google’s index.
    and without do anything my website return in the previous position for the same keywords.
    I don’t know the reason.

  • http://www.writtenbysumer.com/blog Britt

    We have been trying to figure out why certain pages of our website aren’t being crawled. The site dropped in rankings tremendously a few weeks ago. And our blog is even seeing some very strange rankings. One day it will be sky high, the next day sight visits are at a major low. It’s unlike anything we’ve seen before. I guess this might be the answer to our problems….

  • http://www.thebookabyss.com.au Australian Bookshop

    We have found that our search results are all over the place. We can find ourselves anywhere from 1 to 15 in the search results for our organic keywords depending what time we look. Its really volatile at the moment.

  • http://www.dailymotion.com/video/xbxx4a_concrete-floor-coating_lifestyle Concrete Floor Coating

    I think the best practice is to be sure that your site loads quickly and to be sure that you’re up to snuff on your On site optimization & Off Site.. Of course pay attention to your competitors

  • http://officialsafetyandsecurity.com Official Safety and Security

    I’ve tried to figure this whole thing out and even paid people who said they knew what was going on but nothing ever seems to do much for my safety and security web site traffic. It’s pretty much a mystery to me. Thanks, Chris.

  • http://ilikeitdirty.com christopher

    I don’t know what they are waiting for. They need to roll-out caffeine and get it over with. I would like to see the results so I can take action from there.

  • http://mabdulbasit.blogspot.com Abdul Basit

    I am working on a website with millions of visitors daily. Although, PR update was positive for that website but from last few days I am observing declined traffic. Don’t know whether it is Caffeine or some other reason but I am pretty much sure that the effect is of Caffeine.

    Waiting for any update from Google regarding this.

    • http://www.imagemaker2.com John Smith

      Hey you are absolutely rite .. there are some changes lets see what happens.

  • http://www.cheapcharlieshotels.com/ Cheap charley

    I have been going through on my sites and makeing them lighter by 20% or more by eliminating stuff but what is a good weight? I have most around 20K to 25K.

    But then I am thinking that I did not take into effect the weight of the pictures does that count against the site as well? What size should a imiage be for the new mobile devices? 200×200? 300 x 300? most of my imiages are 500 x 375 at 72 dpi?

    Social well I wated a fair ammount of time on twitter and facebook and didn’t see any real value. You tube I did see value.

    Thanks Cheap Charley

  • http://www.submitwebsiteurl.com jack

    Google obviously has included page speed as a new factor in ranking. Recently Google webmaster tools have introduced page speed analysis. The report highlights web pages within your site being too slow to load. generally pages should load in 5 seconds.
    A faster web is better for searching and search results. Fast loading content is essential.
    Webmasters will have to adapt, not all searchers have time for fancy graphics and hi res pics.
    Faster is better, and you use less bandwidth.

  • http://www.submitwebsiteurl.com http://www.SubmitWebsiteURL.com

    Google indexing changes seem definate. I to have noticed in my google webmaster tools site speed reports have appeared, any page taking to long to load appears in these results. i am updating my pages that take too long to load.
    Webmasters you have to agree it is annoying when searching the web and the page takes forever to load, then its not specific enough to your search result and you feel you wasted your time waiting for it to load.
    faster sites with good content, Why not?

  • http://teakdoor.com Thai Girl

    I think google penalising slow sites is a great idea, most people don’t have the spare time for a slow page to load that may not contain the information they are looking for, also it may cause some to rethink there ideas on using free servers or cheap servers which are heavily overloaded, this will make a better experiance for internet users all round.
    1 question though, will google penalise itself for its slow loading google maps :-)

  • http://teakdoor.com Thai Girl

    November last year we had a massive increase in served pages and visitors, this increase has stayed at that level, obviously doing something right :-)

  • http://www.millonesdejuegosgratis.com victim

    On Nov 30 I lost 2 positions for my main keyword that meant a bunch of dollars, now i have dissapeared from the google index, the weird thing is that in spanish we have 5 or 6 important datacenters which are, .com.co, com.ar, .es, com.mx, and com.uy, my website has dissapeared on the .com but stands first on the .com.uy, second on .com.ar , and so on, could this mean theres a manual penalization instead of a massive algorithm??

  • http://www.article-online.com Article Online

    I think this is a very good idea to set speed itself as a standard. My personal experience is that people have never liked to wait for page download for more than five seconds. I have some doubts. What will happen to those sites where images are the backbone. There are flash sites that are also slow at times. At the same time, excessive use of javascripts can also make things slow. I am waiting to see the complete results of this experiment.

  • Guest

    Yes – Recently the number of visitors arriving from Google search results has dropped dramatically. I used to be on the first page of search results all the time, now usually a page or two in. It has made a huge difference.

    I think blogs with lots of images get penalized disproportionately.

    This change will really hurt the little site and reward the big boys with more money to do things the way Google wants them done. But this sort of change was inevitable as the web transitions from its open, even, roots to the gib business it is today.

    • Guest

      “gib business” Dyslexic today, I guess. Meant big business.

    • Guest

      Yes – Just checked the SE position of my top 5 keywords. All have dropped in Google while increasing in Yahoo & Live. I’m not happy about the current changes!

  • Peter Hinson

    All my sub-domains have lost their Google PageRank. They have dropped from 3 to “Unknown”

    It appears that Google doesn’t page Rank sub-domains anymore?

  • http://www.ugureskici.com Ugur Eskici

    is caffeine online? that ip address and search results are different than google.com.tr result. But idk caffeine is on?

  • http://www.waystoquitsmoking.com.au/ Shawn

    Truly nice post! Really helped me a lot, appreciated.

  • Join for Access to Our Exclusive Web Tools
  • Sidebar Top
  • Sidebar Middle
  • Sign Up For The Free Newsletter
  • Sidebar Bottom