Initial Research

On Saturday, March 11, 2006, I received a somewhat urgent telephone call from a client of mine, Hibiscus Florals. The owner, Mark Morkowski, was concerned because he had been reviewing his website traffic statistics and had noticed that at numerous points throughout the day, a user or spider from “ASSERTIVENET” (IP 66.154.103.125) had visited the Hibiscus website.

Since this was rather unusual, Mark elected to investigate further by searching for more information “Assertivenet” via the Google search engine. The first three results that he found appear below:

• http://www.webmasterworld.com/forum10/11080.htm
• http://www.powerbasic.com/support/forums/Forum12/HTML/002979.html
• http://www.completewhois.com/cgi-bin/whois.cgi?query=66.154.111.40

From this information, Mark and I gathered that the owner of the spider in question appears to be a company called Assertive Networks, and hosted through a company called “BC Hosting.” More information wass not immediately available.

It is this lack of information that likely led some of the members of the PowerBASIC forums to block the IP range 66.154.* from accessing their various websites, and justifiably so. But this same lack of information led to additional questions:

1. What files was the Assertivenet spider accessing/trying to access? Was the spider crawling pages or, like some bots, was it looking for specific files that could be used for malicious purposes (e.g. files and scripts that could be manipulated for website attacks?)

2. Why is the apparent owner of the Assertivenet spider a web hosting company (BC Hosting)?

3. What is the intended purpose of the Assertivenet spider?

Additional Research – All Is Not As It Appears

At this point, I decided to look beyond what the website traffic statistics revealed, as well as the information that Mark’s initial search revealed. I needed to start by answering the questions I posed earlier, and in
order to do so, I needed to access the raw log files for the Hibiscus website.

I opened up the log files, searched for the particular IPs in question, and found a series of entries such as these:

2006-03-11 03:47:34 66.154.103.125 - 216.89.218.168 80 GET /robots.txt - 200 0 400 285 78 HTTP/1.0 www.hibiscusflorals.com Gigabot/2.0/gigablast.com/spider.html -
2006-03-11 03:47:34 66.154.103.119 - 216.89.218.168 80 GET /larger_image.asp PID=215 200 0 0 299 125 HTTP/1.0 www.hibiscusflorals.com Gigabot/2.0/gigablast.com/spider.html -
2006-03-11 03:50:37 66.154.103.119 - 216.89.218.168 80 GET /larger_image.asp PID=195 200 0 0 299 31 HTTP/1.0 www.hibiscusflorals.com Gigabot/2.0/gigablast.com/spider.html -
2006-03-11 07:47:05 66.154.103.125 - 216.89.218.168 80 GET /robots.txt - 200 0 400 285 78 HTTP/1.0 www.hibiscusflorals.com Gigabot/2.0/gigablast.com/spider.html -
2006-03-11 07:47:05 66.154.103.119 - 216.89.218.168 80 GET /larger_image.asp PID=219 200 0 0 299 109 HTTP/1.0 www.hibiscusflorals.com Gigabot/2.0/gigablast.com/spider.html -

The spider in this case actually belongs to a search engine called Gigablast, and is appropriately named the Gigabot. The Gigabot only crawled pages and files as other search engines have, and made no attempts whatsoever to access files and scripts of a known malicious nature.

Gigablast is a “Tier 2″ search engine that has over 1,000,000,000 pages indexed as of the date of this article (March 13, 2006.) While it is not on the same level in terms of popularity as the Big 3 of Yahoo!, MSN, and Google, it has indexed a significantly large portion of the web, and can be useful for some searches. In particular, Gigablast has implemented a “Giga bits” feature whereby alternate searches are suggested based on the user’s original query in order to help narrow the query down and provide greater relevancy.

I conducted additional research and discovered that some IP addresses from the 66.154.* IP block do resolve to gigablast.com e.g.:

Conclusion – The Gigabot is Safe

As you may well have gathered by now, the Gigabot is a perfectly safe spider that acts and operates in the same manner as other search engine spiders operate. There is no reason at this time to block the 66.154.* IP range that the bot uses; if anything, webmasters would gain from the potential free traffic that Gigablast would generate for their websites as the result of the Gigabot’s efforts.

*Previously published at www.searchenginefriendlylayouts.com

Add to | DiggThis | Yahoo! My Web

Technorati:

Adam Senour is the owner of ADAM Web Design, a leading web design and development company in Toronto, Ontario, Canada. Visit http://www.adamwebdesign.ca for more information on ADAM Web Design products and services.

FirstGov.gov, a web site focused on delivering government information to the public, has a new top-three list of Internet search engines eligible to compete for federal task orders through September of 2010. Gigablast, Vivisimo, and Norwegian-based FAST Search and Transfer received the news from the General Services Administration.

Gigablast congratulated Vivisimo, a longtime Gigablast client, in a statement today. Vivisimo, working with MSN Search, won the first task order awarded. That task order will see Vivisimo and MSN provide enhanced search results for the FirstGov web portal.

In the new arrangement, Vivisimo will use MSN Search Web Service to pull results from government documents in its index. Further, filtering operators supported by MSN Search will help users better query for documents. Vivisimo will use its Velocity software and Clusty search service to cluster MSN and Vivisimo government search results.

“Together with Microsoft, we will build the finest government search portal the world has seen, helping citizens learn and profit from the rich information offered across thousands of government Web sites,” said Raul Valdes-Perez, CEO and co-founder of Vivsimo, in a statement today.

David Utter is a staff writer for WebProNews covering technology and business. Email him here.

The toolbar is available for download from Gigablast’s site now. Some of the toolbar’s features as quoted from Gigblast’s site, include:

The company is not currently offering a version of the toolbar for the increasingly popular Mozilla Firefox, but it sounds like they will in the future. They’d be fools not to.

Gigablast does currently have its All-In-One Search button, which is a Firefox extension that “adds in-page highlighting and search functionality”. Gigablast also points to an extension available for the bulletin search box.

Chris is a staff writer for WebProNews. Visit WebProNews for the latest ebusiness news.

And they’re just not good enough. Either they don’t track enough blogs or their services aren’t reliable enough.

You could say that this is just the “nerdy” part of blogging. Desperat bloggers staring at their screens hoping that someone will pick up their ideas. You could say that the relevant contacts will be taken anyway (MSN Messenger, e-mail, Skype or whatever).

But I don’t agree.

There are different levels of conversation. The basis of it all is link tracking. It’s an important tool to find new blogs, to measure how an idea spreads, to find out what others are saying in areas that are important for us (that is, the things we write about). What we write in our own blog is usually a lot more interesting than what most of us write in blog comments.

I think there’s a market for a really good link tracking service. Money to be made. Since I have a problem with companies that owns to much of a market, in this case services in relation to the web, I hope someone beats Google to it. But I’m getting to the point where I might accept even Google.

By the way, have I told you that Gigablast’s site search is way better than Google’s?

Fredrik Wacka is the author and founder of the popular CorporateBlogging.Info blog which is a guide to business and corporate blogging.

Visit Fredrik Wacka’s blog: CorporateBlogging.Info.