All Posts Tagged Tag: ‘spider traps’

Webmasters: Googlebot Caught in Spider Trap, Ignoring Robots.txt

Sometimes webmasters set up a spider trap or crawler trap to catch spambots or other crawlers that waste their bandwidth. If some webmasters are right, Googlebot (Google’s crawler) seems to be having some issues here. In the WebmasterWorld forum, member Starchild started a thread by saying, “I saw today that Googlebot got caught in a spider trap that it shouldn’t …

Read More