iEntry 10th Anniversary
RSS
Newsletter
Advertising
| Advertisement |
| Popular WPN Business Resources |
 |
 |
|
|
|
-
Search + Social = Better ROI
Are you utilizing search and social media together? According to Lee... -
Yahoo Reveals SEM of Re-Brand
Near the end of September, Yahoo began a new branding campaign in an... -
Marketing in the Age of Google
Former Googler Vanessa Fox has written a book entitled Marketing in...
     |
3 Comments
Experimental robots.txt syntax
Thanks for the coverage Dave. :)
Joshua, folks not familiar with the REP syntax often leave garbage or experimental statements in robots.txt. As long as the crawlers ignore those, such forgotten stuff is not a big deal. It becomes risky when a search engine experiments itself, and the engine's interpretation doesn't match the webmaster's thoughts.
robots.txt
When you put noindex is is to be excluded, so this presents no threat. In fact it accomplishes your goal. It's those spiders which ignore your wishes that you need worry about.
Noindex:
Not to be a smart a**, but why would it be surprising if Google deindexed a site/page that include Noindex: in the robots.txt??
Why would you have it in there in the first place if you didn't want to be excluded?
Maybe I missed something?
Post new comment