White House Unblocks Google

Removes 2400 robots.txt lines from website

Get the WebProNews Newsletter:

[ Life]

If the outgoing Bush Administration was thought to run a secretive, bubble-icious type of White House, the Obama Administration so far is proving to be the opposite. The Whitehouse.gov redesign for greater transparency has already been widely noted—Presidential blog and all—but the website is now much more open to a new kind of visitor: the search engine spider.

On Monday, Whitehouse.gov was still blocking search engine access to a tremendous amount of website information. In all, the robots.txt file used the “Disallow” command 2,400 times, blocking search engine access to information on earmarks, African American history, photo essays from various places and events, first lady initiatives, the budget, defense, on and on.

White House Unblocks Google

Obviously, if posted on the White House website, none of this information would be considered classified, or even sensitive, so it’s unclear why Bush’s web crew felt the need to prevent the site from being searchable.

Regardless, all search crawler barriers were removed with the Bushes’ furniture, the “Disallow” command lines reduced from 2,400 to basically none.

Requests for comment and/or explanations from prior and current administrations were not returned. Meanwhile, it appears President Obama will be able to keep his Blackberry after all—with some super-encryption functionality added to it. 


White House Unblocks Google
Top Rated White Papers and Resources
  • http://www.twitter.com Jason E Miller

    Yo dude, you made a minor spelling mistake! :)

    You said, “fist lady initiatives” instead of “first lady initiatives”.

    If you made that mistake before yesterday, that would be seriously funny! And I’d probably comment further (use your imagination here)

    Now, I just have to be serious about it and tell you it was just a minor spelling mistake.


    • Jason Lee Miller

      thanks for the close reading.

      shooting my proofer, now.

      Oh, and that’s a fine name you got.

  • Chris

    That wasn’t a mistake. ;)

  • Nuno Lagoa

    Two ladies one fist. Sorry, first. First lady I mean.

  • Guest

    The old robots.txt file was for the most part just blocking text-only versions of the site, and thus prevented duplciate pages.

    In other words, this merely shows that Bush aparently had a ebtter web admin.

    Also: robots.txt is only respected out of politeness, so really making use of it counts as neither blocking search engine results nor obscuring facts…

  • http://www.foursquareinnovations.co.uk/ Chris

    With over 3million IBLs I bet the whitehouse.gov is going to be knocking a lot of websites off top places with this change to robots.txt

  • http://www.propdata.co.za/ Robert

    I don’t get what the big deal was. Half of the pages that have been indexed now return 404’s. So basically this is either:

    1. Another PR stunt
    2. Realization that robots.txt doesn’t really protect anything
    3. Obama is more of a rel=”nofollow” kinda guy
    4. Linkbait

    I’m going for option number one myself. But it’s a fantastic bit of linkbait if you think of it.

  • http://sape.co.ua/ SaPe

    google find

    121 000 from whitehouse.gov. (0,28 ???)

    let’s check in a week.

  • http://mdlotto.net/ MDLotto

    great notice

  • http://rowanlandscaping.com/ Henderson Landscaping company

    Thats amazing they had it blocked.

  • Guest

    Funny how so many people that have no study of history or reality other than what they read are so quick to judge and it somehow becomes fact.

    • http://www.roomfurniturechina.com wholesale bedroom furniture

      This service for sharing text updates with friends proved itself a veritable rapid-transit system for breaking news, and soon it was this micro-content production house, not the blogosphere, that became the check-digit for the established blogging pro.

  • snt

    Bush and the people around him were dumb enough to think that blocking search engine access from web pages will stop people from accessing it. Not surprising at all.

  • http://www.vipdistinct.com Atlanta Concierge

    How knows the real story behind it.

    I’ve notice all that’s in the robots.txt file is this now:

    User-agent: *
    Disallow: /includes/
    Disallow: /search/
    Disallow: /omb/search/

    Which makes sense.

  • Join for Access to Our Exclusive Web Tools
  • Sign Up For The Free Newsletter