WebProNews

Unvalidated Robots.Txt Risks Google Banishment

The web crawling Googlebot may find a forgotten line in robots.txt that causes it to de-index a site from the search engine.

Review: Trackback Spider

What Does Trackback Spider Do? From what I have read on the pre-release materials, it sends trackback notifications to create one-way links to your website. That means:-

Webmaster Claims Spider Entered Contract In Suit

The Web and artificial intelligence have brought about some surreal, science fiction like questions. The most recent mind-bending concept is whether or not robots can enter into contracts – that is, is a Web crawler implicitly entering a contract posted on a website announcing copyright conditions?

Cloaking Is Bad... Unless It's Good

The concept of page cloaking has come under fire; again, because the idea is being used by a number of legitimate sites in order to protect or hide their content from users and/or search engine bots. The fact that these sites do not get punished for using cloaking techniques has become a sore spot with some bloggers.

The ROR Advantage: No Spider Discrimination!

Search engine optimization is a very complex science, but at its heart is the simple rule: to format your website in such a way that spiders can immediately recognize and index its content.

Let the Spider Crawl

For most sites, one of the first things we check is to make sure the site crawler friendly. "Crawler friendly" you say? What the heck does that mean?

GoogleBot the "Spider of Doom"

A funny story is circulating in tech circles about how Googlebot inadvertently destroyed the database of a content management system (CMS) based site that took months of work to build.

Could the New Google Spider be Causing Issues with Websites?

Around the time Google announced "Big Daddy," there was a new Googlebot roaming the web. Since then I've heard stories from clients of websites and servers going down and previously unindexed content getting indexed.

Assertivenet is Gigablast Spider (Gigabot)

The purpose of this article is to provide evidence and information to counteract the suggestion that Assertivenet is potentially used for malicious purposes.

LookSmart Builds Spider Bait

So LookSmart's creating content now, to ensure itself a higher-quality distribution network for the pay-per-click ads it sells.