Quantcast

Break It Down: Search Engine Crawlers

Get the WebProNews Newsletter:
[ Search]

How do they find those wonderful sites? A question many interested in the internet search world often ponder is also answered fairly easy. It’s all in the way they crawl.

Rob Sullivan put together a delightful article at SearchEngineJournal on the background and current life of search engine crawlers. He covers their wee beginnings at MIT up to the hordes crawling the sprawling internet we know and love today.

He also tries to explain a few things regarding what people might see when the do searches based on their particular topic:

You may also notice, upon reviewing your reports, that crawlers like Googlebot will visit repeatedly and request the same page(s) repeatedly. This is common as crawlers also want to be sure the site is stable and also to measure the page’s change frequency.

He goes on to explain crawler behavior and how they work, including certain behavior patterns and even breaks it down somewhat based on individual companies like Yahoo or AskJeeves.

If you want some incite on how crawlers work, this is an excellent article.

John Stith is a staff writer for WebProNews covering technology and business.

Break It Down: Search Engine Crawlers
Comments Off
Top Rated White Papers and Resources

Comments are closed.