WebProNews

Why Your Robots.txt Blocked URLs May Show up in Google

Matt Cutts has appeared in yet another Google Webmaster Video, and this time he has a whiteboard with him so he can illustrate what he's talking about. What he's talking about this time are uncrawled URLs in search results. Cutts says Google gets a lot of complaints from webmasters who say the search engine is violating their robots.txt files, with which they intend to keep Google from crawling certain pages. Sometimes those URLs still end up in search results.