Robots Articles

Google Flexes Robots Exclusion Protocol

Two new features added to the protocol will help webmasters govern when an item should stop showing up in Google’s web search, as well as providing some control over the indexing of other data types.

SES: Built By People, for People & Robots

Finally catching my breath from Search Engine Strategies Toronto 2007. As close to an unqualified success as I could have hoped. Many attendees including Pauline from HighRankings (who interviewed me for Jill’s newsletter) noticed that the freshness of the program spurred speakers into coming up with new, fresh, cutting-edge material.

Google May Ask You To Prove You’re Human
· 2

British digi-rag the Inquirer’s Fernando Cassia had a run in with the "GoogleMind," as he calls it, after entering a query that sent up red flags. To continue his search, Cassia had to prove he wasn’t a robot.

Technology Advancing To Replace People
· 1

The Internet is growing rapidly in many different formats. In an interview with WebProNews at the Web 2.0 Expo, Renat Khasanshyn of Apatar spoke with us about this growth and how it affects businesses.

AP Considers Selling News Ala Carte

From a story about AP by AP:

Rock Em Sock Em: MSN Versus Google
· 11

We know you’ve been waiting minutes, hours, and even days for the release of the Rock Em Sock Em Robot’s Fight from the SES conference. You can all rest better now because the much-anticipated rumble is up and ready for your enjoyment. Only WebProNews was there to bring you exclusive coverage of this event to find out which search engine is truly best.

Ensuring a Page is NOT Indexed by Search Engines

Dear Kalena…

I want to publish a private page on the web, that only I and a few other people will use. How can I ensure that this page is NOT picked up by search engines?

It will be a wiki style page, so there may be lots of content which could be indexed by the Search Engines. This is what I want to avoid.

Google On Making Use Of Robots Meta Tags

Owing to Danny Sullivan’s post on how search engines handle meta tags, Google’s Vanessa Fox has written a post on how Google handles these tags.

Robots for the Rest of Us

The robots.txt file.

Nothing can be more confusing to a website owner as the robots.txt file. Born out of technology in the programming world, the robot.txt file is nothing more than a server command for search engines. Unfortunately, while search engines understand the file, humans have a difficult time understanding machine language.

Google On Robots Exclusion Protocol

A post on official Google blog informs about Robots Exclusion Protocol. Sometime back we informed you about a previous post on Robots.txt file.

Korea Vows Online Robots In Every Home

A heaping dose of Battle Bots and movies like Terminator and I, Robot have instilled a (healthy?) paranoia of centrally controlled robot futures – at least in the States. In South Korea, not so much, as the country announced its plan to have a robot in every home – very, very soon.

Mobile Searchs Future Is Now

More and more, “sometime in the future” is yesterday, and the ubiquitous “New Year’s Tech Predictions” lists find themselves rendered obsolete by mid-January.

Anybody Care That The Robots Are Winning?

This shift to contextual advertising caused by intelligent software has privacy advocates going squirrelly, particularly because the larger public seems relatively unconcerned. After all, says the public, it’s all about the end-user experience, not whether a robot saw my iTunes list or my Gmail inbox and made a suggestion.

Japan Surrendering Moon To Robots

The Japan Aerospace Exploration Agency imagines a future where humanoid robots populate a lunar base in 2025.

UK Robots Will Hunt Underwater Mines

Britain’s Royal Navy should gain ten advanced unmanned mine-hunting craft next year.

Scientists Unleash Robot Horde On Humanity

Husband and wife scientists at the University of Wyoming receive a grant to develop their swarm of robots.

Cornell Takes Next Robotic Leap

Robots can now build more robots, at least on a simplistic level. Hod Lipson and his team at Cornell University developed a group of robots that can replicate themselves. It’s considered a major breakthrough in robotics and while it may seem simple now, the future certainly looks bright.

Robots Learning To Walk

A trio of robots that were shown off at the American Association for the Advancement of Science are demonstrating an ability to walk, as well as the ability to learn and improve its way of walking.

SEO’s Relationship With Website Architecture

Search engine optimization for today’s search engine robots requires that sites be well-designed and easy-to-navigate. To a great degree, organic search engine optimization is simply an extension of best practices in web page design. SEO’s relationship with web design is a natural one. By making sites simple and easily accessible, you are providing the easiest path for the search engine robots to index your site, at the same time that you are creating the optimum experience for your human visitors.

Disabling Google and Other Search Engines From Crawling a Site
· 17

Reader question: I have a online database of horror movies, and I have a good Google rank. In my traffic logs I noted the last month a really growing of the bandwidth: one of the most important browsers of the server logs is Googlebot, so this traffic was generated for the spidering engine of Google. I have the 20 Gb bandwidth limit and I don’t want to pay for excess, so I disable Google into my Web site. My question is:

Optimizing Dynamic Pages – Part II

The Widget Queen Revisited
You have the world’s finest collection of widgets. You created the world’s best widget website. You have no traffic.

You checked in the search engines and find that your site does not appear at all, even though all your competitors’ sites do. Perhaps the search engine robots cannot get to your pages to index them.

Search Engine Robots
Search engine robots are simple creatures. They can “read” text to add to their databases, and they can follow “normal” links–those links that are coded to look like

<a href=”bluewidgets.html”>blue widgets </a>

or the slight variation

<a href=”bluewidgets.html ><img src=”bluewidget.gif” ></a>