Quantcast

spider traps Articles

Webmasters: Googlebot Caught in Spider Trap, Ignoring Robots.txt Webmasters: Googlebot Caught in Spider Trap, Ignoring Robots.txt
· 4

Sometimes webmasters set up a spider trap or crawler trap to catch spambots or other crawlers that waste their bandwidth. If some webmasters are right, Googlebot (Google’s crawler) seems to be having some issues here. In the WebmasterWorld forum, member …