Quantcast

Bing Indexing NoIndex/Nofollow Content

Microsoft Says It's Working to Fix the Problem

Get the WebProNews Newsletter:
[ Search]

Bing’s webmaster forum has a number of posts where people are complaining that Bing will not index their content. Some people, however, are having the opposite problem. Content that they do not want indexed by Bing is being indexed by Bing, despite the webmaster’s efforts to keep it out of the search engine. Are search engines ignoring your NoIndex/Nofollow attributes? Let us know.

One person posted about this, and a Microsoft employee responded, admitting that this is an issue on Bing’s side. Here’s how the conversation goes:

Webmaster:

Bing userI have a site containing pages that I don’t want their content being indexed by search engines so all of those pages have

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW" />

in their header and it has been like this for a while now.

Both Google and Yahoo respected the tag and those pages are not being indexed by them but today I checked and found all those pages even recent ones are indexed and cached on Bing.

Brett YountProgram Manager, Bing Webmaster Center:

This is a known issue we are working quickly to resolve.  If you have pages you would like permanently removed from our index, please send me a mail to bwmc@microsoft.com with your domain name and "MSNBot ignoring robots tags" in the subject line. Please also include the URLs in the body of the message. You may use an * wildcard for any directories such as:

http:example.com/ wrongdirectory/*

Normally, I would request that you fill out a content removal request, however, since this is a problem on our side, I’ll do the leg work for you.

Others have complained recently that Bing and (in some cases) even Yahoo have been ignoring NoIndex and Nofollow in meta tags. Have you experienced any similar issues? Please share.

Bing Indexing NoIndex/Nofollow Content
Top Rated White Papers and Resources
  • http://www.eporiaecommerce.com Ecommerce Design

    As much as some people don’t like to hear it, “if you don’t want something crawled by the search engines, don’t put it on the internet”.

  • http://www.homesolarpowersystems.org Home Solar Power Systems

    Of course this is an issue where needs to to resolved where Bing does not index their content. It’s better the webmaster’s efforts to keep it out of the search engine.

  • http://www.encodexindia.com/ .Net Application Development

    Yes, it’s true that Bing is indexing all the pages. Our all pages has been cached by Bing. even we block those pages to index for any search engines but Bing does.

    I thing this issue much be solve asap by Bing team

  • http://www.iblogzone.com DiTesco

    This nofollow/dofollow thing should just stop once and for all. It seems that it is haunting as since the day it was invented. The original purpose no longer applies and as one can see, it is actually being ignored by Bing. The question is, with so many engines out there, who else is or is not respecting these tags. I’m for what eCommerce Design said, if you don’t want it indexed or displayed at all, don’t put it inline:)

  • http://www.palatnikfactor.com Online Marketing Tips

    Well I can see why some people may not want a certain page indexed by the engines and given they are actually taking emails on Bing to help remove these indexed pages, that’s great.

    On the other hand, we’ll see how the no-follow attribute plays out in the next 2-6 months from now as there is a lot of speculation of where its going and how and when it should be used.

  • http://gardenbirdfeedersonline.com William

    I have been having issues with the BING search engine and I have contacted their support regarding the issue but have gotten no where with them. I currently run 3 websites that are all being indexed by Google and Yahoo. My site http://doglifejacketsonline.com has been going strong with BING ever since I launched it without any problems. But my site at http://gardenbirdfeedersonline.com used to show up in BING search results and then it just dropped from the search results. My newest site at http://pampereddogfurniture.com has never appeared within BING search results. The thing I find to be most confusing is BING is showing backlink results for all of my websites so it knows they exist. In my opinion BING definitely is having issues that make me feel that their search engine belongs on the fail list. You would think when you contacted support they would atleast try to find out what is causing the issues.

  • Guest

    I feel google and yahoo are ignoring my no follows,and i don’t feel msn is indexing my site or indexing it right.My site is shanesworldofproducts.Also don’t feel google,yahoo or msn are putting in the results right,because my site is extremely revelant,but yet i can’t find many of my advertisements.My analyzed my home page at a 287 score,but yet i still can’t get hardly any first page results.

  • http://www.laokay.com Steve

    How long have we all known that bots do not necessarily follow the robots.txt file or the robots meta tags as they should be? How about over a year that I’ve known this.

    They are supposed to, but it’s not a “bug” as Bing suggests it is.
    We’re talking about a tech center rep who only could assume it was a bug, because after all, the tags should be followed right?

    I mean how could a page that it’s robots meta tag says not to index, and not to follow all of a sudden get indexed by a major search engine? I believe they are simply ignoring the tag on purpose. After all if they don’t index it, some smaller less known search engines will, and when you’re talking about billions of pages, who cares if a million or so get indexed when they were not supposed to be. The more content indexed the better, and trust me, there is grey area stuff going on here that nobody will admit to, at least not in public.

    Google uses the nofollow tags for PR juice flow, and still follows the links, but doesn’t let the PR juice flow to those links. See what I mean. Hey Google, I said not to follow, you a-hole’s!

    It reminds me of when a liquor store got fined several times for selling cigarettes to minors, and the last time they revoked his tobacco license to be able to sell cigarettes and all the owner could say was “If I don’t sell it to the kids, my competitor down the street has and will”. Yeah, but he got caught and the other guy didn’t.

    I think that guy who had his content indexed should sue Bing, because it’s his work that now is displaying on Bing without his approval. If AP News is going to start seeking out websites that have a snippet of AP news with links to the AP article on the AP website and think that’s a good idea to sue them, then why couldn’t he sue Bing for indexing his content when the bot it was specifically told not to. That’s like me putting a notice that the material on the website is copyrighted and nobody is allowed to republish any materials and then sit back and watch as people do and do nothing about it.

    What good is a copyright when search engines can index the content as they please, but smaller publishers are threatened to get sued if they do?

  • http://www.jubejube.net Jubejubenet

    I found BING to work great and smooth. For years Google has not added my index page after numerous webmaster requests, but added all my hidden background pages, annoying and confusing for people getting to my site (especially when they are googlephiles and only use google to search not even knowing what the address bar is for) BING added my index page and I feel my keywords and everything setup for search was honoured by BING. Hats off to BING.com, they are great and look good, also Google deserves some props for a very powerful search engine but they should have more respect for index pages too.

  • http://seopardeep.wordpress.com pradeep chauhan

    Hi

    in starting i was facing a lot of problem with indexing my websites but now bing has imroved …..

    As we know everything take time to imrove their quality so hope soon we get good results from bing also….

    and Thanks to Bing also for their team
    Keep it up

  • Guest

    anyone getting the same issue using dissallow to stop bots crawling pages. I dont use the NOFOLLOW tag, i use dissallow /(folder name)/ in the robots.txt file.

    • Guest

      Correct me if I’m wrong, but the robots.txt file is supposed to overrule anything that’s set on a specific page as far as access. So let’s say even if you put “index,follow” on a page, but robots.txt said to dissallow access, the bot isn’t supposed to even look at that page.

      The fact that it is looking at the page is proof it’s not following the rules. Forget what meta tags are on the page when there is a robots.txt telling the bot “not this page/directory”. Why would the bots bother reading it anyways if it’s just not going to follow the rules? It’s almost like they want us to think it’s a bug somewhere and it’s going to take months to fix or whatever. In the meantime they’re infringing and telling us we’re that stupid to believe they didn’t intend it this way.

      • Jasee

        I think it’s the other way around. The page callout will overrule a robot.txt callout for NOFOLLOW. Also, Disallow does seem to be more stable than NOFOLLOW across the engines.

        I’d like to add that I’m sick of the mutliple browsers, multiple search engines, multiple this and that. Ugh. Stupid.

  • http://www.midlandsadviceandtraining.co.uk Karl Craig-West

    If you have a website, why on earth would you not want people to find it?

    Surely if you put pages up on the web (which is public space) then you’re going to run the risk of them being found and publicised by someone?

    If you don’t want the pages indexed then don’t make them public pages.

    I’m not trying to be contentious, I genuinely don’t understand the rationale of some people on this issue.

    Karl
    Training in the UK Midlands

    • http://www.lo-ve-media.com Lothar Velling

      I think it makes a big difference if a single person hints on my noindex-page or a mass media does so. It’s a standard and as web designer I have to rely on standards to be respected, what ever the reason maybe for noindexing my or my client’s page. It’s me who has to explain this failure to a client, not the global player company which is mostly estimated to be unfailable.

      Though I could make a big bet that most search engines have a database where they have our noindex pages listed for internal (ab)use.

      My personal habit is to noindex contact and address pages to limit the access just to interested site-visitors. I agree with you that if I really don’t want search engines or persons to know my true secrets it’s best not to publish or tell them anywhere, like in real life as well.

      For myself I couldn’t find any problem on this issue so far, though I could find some folder content being indexed at Google’s which wasn’t linked from my homesite or any other publication. After listing these folders in robots.txt the problem was solved during a week or two.

  • http://www.newegg.com Freddy C.

    This specific shopping search engine has it’s own robot, it does not crawl all pages… looks like it ignores meta tags and does not understand wildcard robots.txt yet. But it’s not a problem since it does not use any cache and tries to index only product pages, although it costed us a lot of time to ask them not to crawl our forums…

  • http://www.poppyspantry.com Buy British Food

    I’m curious, why would a webmaster not want content indexing ? Is this for government top secret stuff ?

    • http://www.controldatainc.com collection agencies

      Some people dont want pages of their website to index Im guessing for SEO reasons.

    • http://www.sandokai.co.uk Robin

      One reason why I use NoIndex files and folders is that when I build a site I put a lot of the original PhotoShop files, notes on the coding and bits and pieces from the original build on the site as well as the site itself.
      Should I get struck by lightening, hit by a bus or otherwise incapacitated the person who takes over the maintenance of the site has all he needs to keep the site going. Also, if my hard drives get wiped out in a fire or some other disaster I have a copy online of the site and the associated files used to build it.
      I also add new sites on the back end of current sites. Two of my sites are in the process of being rebuilt and I have a a folder tacked on the main site called “new site” so my client can see the development of the new site “live”. Obviously, while the new version of a site is in flux I don’t want it to be seen by the general public. Until the new site is ready to go live I’d prefer it not to be indexed.
      There are plenty of good reasons for web pages not to be indexed which aren’t illegal or immoral.

      • Guest

        That sounds very dumb. You don’t put this stuff in your website. You should store it in a non-browsable folder.

  • http://www.uk-house-rentals.co.uk SKETCH3D

    I’m of the opinion that if you don’t want it to be found, don’t put it on the web. Don’t quite get the point of no follow tags other than to prevent web crawlers from leaving your site too soon. (Perhaps that just my lack of knowledge on the subject)

    Just my opinion though.

  • http://articlesfind.com Articles

    Can you block the flow of page rank by using the robots.txt file or do you have to use both the robots.txt file and the noindex/nofollow meta tags?

  • http://www.byfchat.com jay

    Seems to me that’s a huge problem. One would think they would have at least tested the crawling before allowing it to cache and publish sites.

  • Guest

    Funny that, my site seems to struggle to get indexed by Bing, Yahoo does okay, and Google just picks all pages seems within a week. Even the very few that are indexed by Bing, dont seem to rate at all with kewords coming in first page on the others. Go figure, Bings still got a lot of work to do.

    • http://briarpatchgiftshop.com Joe Hayes

      When “Bing” was “Msn” I was indexed at about 98% that number dropped like a rock to 10% I check everyday it moves around 8-10% What is up with ‘Bing” I wanted to use “cashback” but I don”t have time to insert all the coding, I have over 525 url’s with the the coding required for each one I just don’t have the time. http:briarpatchgiftshop.com

  • Not the point

    The robot exclusion thing is not about hiding pages, it’s about ensuring your better or more relevant pages come up first in SERPS and avoiding hammering the search engine with every variation of the same page.

    This is especially true of auto-generated pages, where listing each would be a waste of space.

    If you’re hiding something you put it in a password protected directory or something.

  • http://www.mobile-phone-lcd.com Mobile-Phone-LCD.com

    I had this problem and tried to contact Bing to ask them to remove the pages. The pages they were indexing were all popup images of my products. So whenever someone clicked on the link in the Bing search engine all they got was the popup and a reduced sized browser cut to the defined size of the popup window. I have log files with 100′s of these click throughs and of course no orders as most people would blame my site, not Bing, and leave as fast as they can.

    So for all of these posts above questioning why you would have pages that have a NoIndex NoFollow tag, this is a perfect example where a dumb crawler snags everything regardless of what it is told in the header and produces a lousy experience for the search engine users.

  • http://www.khaye-welcometomylife.blogspot.com khaye

    I also experiencing this kind of situation. kind help too.

  • http://www.qsand.com/index.htm Bill Lilling

    For all you newbies out there… there are lots of legit reasons why a webmaster does not want certain pages indexed. For example… recently I put up a site for a client who is a medical doctor. Although I used BOTH meta and robots.txt directives NOT to index my \private folder, Google and MSN both did. This folder contained the form results from prospective clients, and published their personal information on the web. This was definitely not cool, and could have led to serious legal repercussions had I not noticed it right away. There is no reason in the world why a bot should go into the \private form results folder, then publish and cache what is there. In 13 years on the web, I’ve never encountered anything like this. That’s just plain sloppy, if not derelict in responsibility.

    • Guest

      Idexed or not unless you’ve got those files protected via password or encrypted somehow anyone with a mind to try can get into them and you’re opening yourself, and your client, up for some serious data protection legal backlash.

  • http://www.bookalimo.com Book a Limo

    Hi, every one, I saw this post and very confused … how can we secure the data from search engines ? even if we define the nofollow content? are the Bing ppl bothering anything about this problem??

  • Anon

    Yahoo indexes pages quickly and effectively – arguably better than Goog – Bing cannot index pages even when it’s spoon fed the data…
    Bing powered by Yahoo would be a step forward to compete with Goog; but MS has decided to go the other way… Goog wins again.

  • Guest J

    I used Bing to search for news articles and external views on a certain company. The first three pages Bing returned only had pages from that company

  • http://www.blogrotica.ca Sensual Articles and Stories

    Ever know Microsoft to get anything right?

  • http://toowoombawebdesign.com.au mark

    Bing has ignored my website after creating an account with them and jumping through the hoops of creating a XML site map etc;

    Bing has indexed another one of my sites – for a search of my business name, it has my business name AT THE BOTTOM of it?

    So according to Bing, my business name at the bottom of another website has more significance and relevance than MY OWN BUSINESS???

    Pathetic at best.

  • http://www.wedding-disco-hire.co.uk CEEWorld

    I can see why many are frustrated by the problem of pages being indexed which they don’t want to be indexed as the no follow tag has been ignored. I would like to offer up a suggestion that can alleviated the need for no follow tags to web pages, the way to do it is to use a .js file for the links bots cannot follow the links as there are none in the HTML code they reside within the .js file. You then only make links to pages you want indexed by the bots in HTML If you submit an .xml sitemap then leave out the pages that you do not want indexed. To protect files from being accessed they can be password protected and made hidden as far as I am aware a bot cannot crawl a password protected folder as it will not have the correct permissions. It is also possible to set up individual permissions for files and folders on a website at the root level, this also aids in security.

    Whilst we all like to find fault there are things that we can as well. Whilst I agree that bots can and often are at fault we all want our websites indexed and therefore should take a proactive roll in that happening to the pages we want it to happen to not a reactive one when things don’t go the way we like.

    When we build websites or have someone build one for us we should make sure that it does what we want it to, we cannot rely on others to do things for us. We have websites more often than not to complement a business, we run that business, we make the decisions that define it and we should do the same with our websites.

    On the subject of MSN not indexing sites I have had a similar problem, it appears that when changes are made to a site that site may be flagged by MSN, this has happened to myself as my business has a number of websites, it appears that potentially one of the links on the site MSN objected to and after discussion with them it was removed, we also added certain things that MSN requested and one of those sites is now page 1 on MSN search results, we are confidant that this will also prove to be the case in another instance.

    If you have a problem getting indexed contact MSN and if they require changes and they do not adversly affect your site make them. You can also make an appeal to MSN if your site is flagged. Sometimes you just have to pick up the phone and ask.

    All search engines want to index your sites, they exist for that reason alone. Make use of information supplied by them, try and co-operate and in turn you might just be surprised.

  • http://worldvitaminesonline.com World Vitamins Online

    I think Bing has a way to go before they ring out the bugs.

  • http://www.scriptbloggers.com Soroush

    Hi,
    I’m the person who originally reported this issue in the thread quoted above. I just want to first answer those who are questioning the reason why a webmaster may not want the content be indexed.
    1- The site in discussion is an article marketplace where people buy exclusive rights to an article and publish it as they want. When someone buys the article, that article gets removed from the store. The articles are unique but because Bing has indexed them if someone searches, they will find the old location instead of where it actually exist now and it takes forever till it gets de-indexed.
    2- As a basic SEO rule, you don’t want duplicated content on your site. If you have a blog and you don’t block some of your category archive pages, you may end up with many pages with same content which can lead your site to penalties.
    3- Some pages are for your site’s visitors and you don’t want visitors landing on them directly! If they are indexed, someone may find it and land on it from search engines.
    4- Staging servers, where you test your site’s changes before pushing it live. It doesn’t have to be passsword protected but it should not be indexed because of duplicated content which will happen after pushing the content to live server.
    5- Contact page. That has no useful content but can easily be abused by spammers.

    Now here is an update on my situation:
    I sent Brett (the program manager) and email with details of my problem and he kindly agreed to investigate and remove the already indexed content form Bing’s index. It was supposed to take 48-72 hours and this conversation happened Thursday Aug 6th. My pages are still indexed and MSNbot still hits my old and new pages daily.

    • http://www.scriptbloggers.com Soroush

      Still no news and my emails are being ignored while Bing is still indexing every single new page added to my site and despite their promise my old pages have not been de-indexed. I’m thinking about taking legal action.

  • http://www.attorneysokay.com Steve

    How about blocking Bing’s class B or class C IP range(s) from your server instead of worrying about robots.txt or meta tags? It’s not hard to find it’s ip range.

    I think that will send the high and mighty Microsoft a very clear message with each webmaster that blocks Bing’s access that we’re not going to put up with the stupid search engines making up the rules as they see fit. If we let this start with a small(er) fry, by the time the big guy starts dictating how things are going to be, it will be a lot harder to get them to realize we’re the customers and they are the here to serve us, not the other way around.

    What if Google came out and said “We don’t care about robots.txt or tags, we’re going to do how and what we please” ?????

    With some sites reporting over 80% of their traffic coming from search engines like Google (ours is like 91%, of which 70% of that is Google), how can they justify cutting a search engine entirely if it decides to not follow the standards of the rest of the world? If we don’t stop this now, it will just get more difficult later on.

    If somebody were to write up a petition site for meta tag/robot.txt reform, I’d gladly add my name to that list.

    • http://sylent9blue-2.webs.com Guest–Carl A. Helsing

      To;The Webmaster;Coinidence in”time”.Over the previous year or so I have been able to find my own web site thru Yahoo-Google-FireFox and so on.
      Today,none of those access it either by url or by title(SKYTRAINZASTRON)–But,Bing is now accessing it beautifully by the title.I’m thrilled with that.I just got a new Hewlett_Packard PC and it come’s with a Bing browser in windows vista.
      So with me,everything is fine.If it can’t be the”Robot Of Fate”,is it I who may be accused of cheating—Thank you—Carl A,. Helsing–

  • http://www.ripbluray.us rip blu-ray

    Enjoy the software and enjoy your life.blu-ray to mp4 converter, blu-ray burner

  • Spheric

    I don’t give a darn if they crawl any page I have publicized on the internet. But, I would like them to honor their promise to not index pages I have asked them not to index.

    I have an ecommerce site and Bing is ignoring both the robot.txt file and the robots meta tag.

    As others have tried to point out, there is some content that is only relevent in context to the site. I would prefer not to have searches that return my site cluttered up with this stuff.

    For example, why on earth would I want them to index empty cart pages, checkout pages (that redirect to the empty cart pages), ‘email a friend’ pages that don’t know what product to send, customer pages that all redirect to the login page, or even the login page itself, etc. etc. etc. And, why in God’s name would they want to waste their own resources indexing all this crap.

    They have indexed dozens of pages like this that I have asked them not to index (nearly half of the pages that occur in the index). This is garbage and they know it.

    Now, if they will just get off the dime and rectify the situation, I’ll calm down a little.

    < /rant > :D

  • http://www.mobile-phone-lcd.com mobile-phone-lcd.com

    I submitted my info regarding the pages I wanted removed to the listed email address for the Bing Webmaster Center as above. I even got a reply back requesting more details and how they could find the pages in their own index. I told them use site:MyWebsite.com and you will see lots of indexed popup pages.

    Since then nothing. No removal of the Popup image pages.

    I sent a couple of follow up emails. Got no reply.

    So i think the above reply and email address as far as I can see is just a public relations stunt to suggest they are on top of this.

    How difficult can it be to remove pages that I have requested.

    So far the pages that have been dropped out the Bing index are genuine product pages I want listed. The number of product pages listed actually dropped after I contacted them.

    So what is going on with these guys at Bing?

  • Murray

    This issue has not gone away as far as my site is concerned. What is more worrying is the popup image links are for the most popular products on my site. The products with the best back links from other sites that have recommended or linked to them.

    It seems very convenient that the best selling products link to popup images.

    Is this some kind of strategy by Bing to eliminate competition for their best paying corporate customers who have big advertising budgets?

  • http://blog-webmarketing.eu/?tag=e-reputation-2 blog webmarketing

    I used to be suggested this web site by way of my cousin. I am not sure whether this publish is written by way of him as no one else know such specific approximately my problem. You’re incredible! Thanks!

  • http://deltac0.com Whaaaat?

    I am not positive the place you’re getting your information, however good topic. I needs to spend some time learning much more or figuring out more. Thank you for great information I used to be in search of this information for my mission.

  • http://cotteg-stroi.blogspot.com/ коттедж| строительство| материал| сооружение| здание| стена| технология| участок| проект коттеджа| загородный коттедж

    Hi there, I discovered your website by means of Google even as searching for a related topic, your web site came up, it seems great. I’ve added to favourites|added to my bookmarks.