Removes 2400 robots.txt lines from website

Text:

| Print

| Share:

14 commentsSaturday, January 24, 2009

White House Unblocks Google

Removes 2400 robots.txt lines from website

14 Comments

By Atlanta Concierge (WPN reader) - Thu, 03/05/2009 - 00:22

What in the Robots.txt file now

How knows the real story behind it.

I've notice all that's in the robots.txt file is this now:

User-agent: *
Disallow: /includes/
Disallow: /search/
Disallow: /omb/search/

Which makes sense.

By snt (WPN reader) - Mon, 01/26/2009 - 10:47

Stupid Bush

Bush and the people around him were dumb enough to think that blocking search engine access from web pages will stop people from accessing it. Not surprising at all.

By Guest (WPN reader) - Sun, 01/25/2009 - 12:01

Funny how so many people

Funny how so many people that have no study of history or reality other than what they read are so quick to judge and it somehow becomes fact.

By wholesale bedroom furniture (WPN reader) - Fri, 04/17/2009 - 01:52

This service for sharing text updates with friends proved itself a veritable rapid-transit system for breaking news, and soon it was this micro-content production house, not the blogosphere, that became the check-digit for the established blogging pro.

By Henderson Landscaping company (WPN reader) - Sun, 01/25/2009 - 08:21

Great aritcle

Thats amazing they had it blocked.

By MDLotto (WPN reader) - Fri, 01/23/2009 - 10:19

great notice

By SaPe (WPN reader) - Fri, 01/23/2009 - 07:43

121k

google find

121 000 from whitehouse.gov. (0,28 сек)

let's check in a week.

By Robert (WPN reader) - Fri, 01/23/2009 - 02:53

Um... big deal?

I don't get what the big deal was. Half of the pages that have been indexed now return 404's. So basically this is either:

1. Another PR stunt
2. Realization that robots.txt doesn't really protect anything
3. Obama is more of a rel="nofollow" kinda guy
4. Linkbait

I'm going for option number one myself. But it's a fantastic bit of linkbait if you think of it.

By Chris (WPN reader) - Fri, 01/23/2009 - 02:29

With over 3million IBLs I

With over 3million IBLs I bet the whitehouse.gov is going to be knocking a lot of websites off top places with this change to robots.txt

By Guest (WPN reader) - Thu, 01/22/2009 - 13:13

The old robots.txt file was

The old robots.txt file was for the most part just blocking text-only versions of the site, and thus prevented duplciate pages.

In other words, this merely shows that Bush aparently had a ebtter web admin.

Also: robots.txt is only respected out of politeness, so really making use of it counts as neither blocking search engine results nor obscuring facts...

By Nuno Lagoa (WPN reader) - Wed, 01/21/2009 - 19:01