Netcraft looks at the number of sites hosted on the different top servers. The following graph represents market share for top servers across all domains from August 1995 to October 2008:

Netcraft offers a more detailed look at the numbers of sites coming from different servers in this list. Amit Agarwal at Digital Inspiration compares NetCraft’s data to that of another Firm:

Now Domain Tools also tracks all the different web domain names registered on the Internet and their count puts the total number of active domains as 106 million.

This is different from the Netcraft survey as Domain Tools won’t count the sub-domains since they are all registered to the same owner (like Google’s blogspot) but may technically be a different website and hence adds to the Netcraft count.

I wonder how many of theses millions of sites are blogs. Of those, I wonder how many are still being updated. Those are some numbers that might fuel the debate about the life of blogs.

Google was found to be behind a total of 10.5 million sites in October.  Microsoft’s name was attached to 62.8 million, and Apache supported 91.9 million sites.

 Netcraft’s Webserver Stats

The space between these numbers then narrows a bit if you only look at what Netcraft considers "active" sites.  Google, Microsoft, and Apache are tied to 7.6 million, 25.6 million, and 33.3 million of them, respectively.  This gives Google a market share of 10.62 percent.

As for month-to-month changes, Netcraft found that after Apache, "Google shows the next largest growth and boosts its total by 411 thousand sites."

So for being in a very distant third place, Google’s in pretty good shape.  The company gave Stephen Shankland a tiny insight into its position by stating, "The Google Web server is a custom-built server that runs on Linux."

Following its April 2008 Web Server Survey, Netcraft revealed, "Largely due to the growth at Blogger, Google’s developer share increases by 0.54% to 6.08% this month, while Apache falls by 0.27% and Microsoft by 0.14%."  lighttpd and Sun are due to lose share, too, making Google the only developer to move ahead.

 Market Share for Top Servers

However, both Apache and Microsoft are still running far more sites, so, "[i]n absolute terms, this still means that Apache has gained slightly more sites than Google’s 1.07 million, while Microsoft gains 847 thousand sites."

What will happen next month is anybody’s guess; Netcraft goes out of its way to state that Google is not guaranteed to repeat this performance.  The data pool should grow larger, at least, as Netcraft recorded a growth of 3.1 million sites for this month alone.

Netcraft’s survey involved almost 168 million sites, by the way, and a hat tip goes to Alistair Croll for being the first to cover it.

The 100 million site milestone caps an extraordinary year in which the Internet has already added 27.4 million sites, easily topping the previous full-year growth record of 17 million from 2005. The Internet has doubled in size since May 2004, when the survey hit 50 million.

Blogs and small business web sites have driven the explosive growth this year, with huge increases at free blogging services at Google and Microsoft. Domain industry juggernauts Go Daddy (U.S.) and 1&1 Internet (Germany) have also seen strong demand for low-priced domain names and shared hosting accounts.

Via: A Welsh View

Comments

Tag:

Add to Del.icio.us | Digg | Yahoo! My Web | Furl

Bookmark WebProNews:

Darren Straight a Microsoft Student Partner (MSP) and Windows Live Butterfly
Expert is a University of Kent Student who is an avid blogger and technology
enthusiast who loves the passion of beta testing new products and services
from Microsoft.

While it may be true that you can file GoLexa.com under the “you can’t judge a book by its cover” category, that does not reduce the usability of their service. Nick at Threadwatch described GoLexa as an SEO Swiss Army Knife, and he’s not wrong.

GoLexa offers users the ability to perform a complete page analysis, SEO-wise, to any website entered into the search box. Specifically, GoLexa checks for the following:

Screenshot of a GoLexa result page:

Click For Larger Image

Surprisingly, these are not the only options available to those using GoLexa. As you can imagine, GoLexa is powered by the Google API, a script that gets results for the above analyses, and Alexa’s Thumbnail program.

With the number of tools available on GoLexa, it’s easy to see why they came up with the catch phrase: “The Search Tool with Complete Page Analysis for each Result,” because the engine certainly provides users with complete SEO analysis capability. So while GoLexa may be a little difficult on the eyes, it is quite functional and provides the SEO curious everything they need to perform analyses of all kinds.

Chris Richardson is a search engine writer and editor for WebProNews. Visit WebProNews for the latest search news.

Via Netcraft:

Google is now an ICANN-accredited registrar of domain names, providing it with yet another potential line of expansion. The fast-growing search provider is approved to sell names in seven top-level domains (TLDs) including .com, .net, .org, .biz., info, .name and .pro.

Cheap or free domain names could prove useful to Google in the notoriously price-sensitive blog hosting sector, where most bloggers use subdomains (i.e. myblog.bloghost.com) rather than full domain names (www.myblog.com).

Andy Beal is an internet marketing consultant and considered one of the world’s most respected and interactive search engine marketing experts. Andy has worked with many Fortune 1000 companies such as Motorola, CitiFinancial, Lowes, Alaska Air, DeWALT, NBC and Experian.

You can read his internet marketing blog at Marketing Pilgrim and reach him at andy.beal@gmail.com.

A report by Internet monitoring company Netcraft revealed sites having to do with Apple’s new products (apple.com, store.apple.com, and macworldexpo.com) have all experienced a degree of slowdown and lag because of large numbers of visitors. Netcraft’s research indicated:

The Apple web site, which runs on Mac OS X, experienced some slowdowns but was largely available. Apple’s online store (also on Mac OS X) struggled, however, experiencing outages and lengthy response times. Faring even worse was the official site for MacWorld Expo, which runs on Windows Server 2003, and was offline for hours following the show’s keynote address by Apple CEO Steve Jobs.

Chris Richardson is a search engine writer and editor for WebProNews. Visit WebProNews for the latest search news.

The Port80 Top 1000 Corporations’ Web Server Survey can be found at: http://www.port80software.com/surveys/top1000webservers

Netcraft’s Web Server Survey can be accessed at: http://news.netcraft.com/archives/web_server_survey.html

Since its inception in 1995, Netcraft’s Web server survey has received widespread attention in the technology industry press to the point that their surveys are accepted by many as the standard picture of Web server market share. However, as part of its November 2003 Web server survey, Port80 Software has raised serious doubts as to the business relevance of the Netcraft survey. A close examination of Netcraft’s methodology reveals systemic biases that may inflate Apache’s market share while under-representing Microsoft IIS. More importantly, understanding the specific methodologies of both the Netcraft and Port80 surveys reveal that they are, in fact, asking very different questions.

Netcraft Casts a Wide Net (With a Few Significant Holes)

In compiling its most recent survey, Netcraft reports responses from 44,946,965 sites, “[collected and collated] from as many hostnames as can be found providing an HTTP service.” Exactly 30,298,060 are running Apache, while 9,449,180 are running IIS. Precisely what does this mean? It goes almost without saying that no one has a definitive picture of what the Web looks like — how many sites there are, where they are located, who is running them, and who is visiting them. Even compiling a reliable list of the most popular sites on the Web is the province of market research firms commanding high prices for their educated best guesses. No company has the resources or the means to hand-count every site on the Web, checking for mirrored content, dead links, private sites, or numerous other variables — and any site administrator who readily reveals to any visitor exactly how his/her site works has serious security issues!

Netcraft’s solution to making sense of this muddy picture is to cast as wide a net as possible and then sift out what they believe to be live sites with unique content. To do this, their survey does not count Web servers per se, but sites, or more precisely, hostnames (see sidebar). This is a sensible step, since a single server may run numerous sites or a single site may be hosted on multiple servers. For example, a visitor to Yahoo! may be connected to any number of different servers depending on his/her location, traffic loads, and which servers are active at that moment. But, if that visitor then navigates to a friend’s GeoCities homepage, he/she is likely directed to a single server hosting hundreds of other such sites.

Netcraft’s focus on sampling as many sites as possible introduces a systematic overrepresentation of servers that are used to host high numbers of sites, including parked domains (hostnames pointed to a physical server but not actively used as Web sites) and sites of extremely variable quality and traffic. As an extreme example of this sampling bias, the single largest gain for Apache and loss for IIS in the November Netcraft survey came from the migration of Register.com, a domain parking service that accounts for over 1.4 million Web server responses alone in their survey.

Answers and Qualifications in Netcraft’s Methodology?

Netcraft does acknowledge the problem of templated, mirrored, and parked domains in their methodology, however one must dig deeply into their site to the July 2000 survey to discover their approach to this problem (http://www.netcraft.com/survey/index-200007.html). Here they address the radical changes in the Web in the first five years of their survey: “[W]hereas in the early days of the Web, hostnames were a good indication of actively managed content providing information and services to the internet community, the situation now is considerably more blurred.” Conspicuous in its absence is any discussion of changes over the past three years in either the Web or Netcraft’s survey methodology.

Netcraft utilizes a logarithmic formula to “correct” for parked domains in their survey. For example, Netcraft reported that futuresite.register.com’s 1,414,626 sites were whittled down with their formula to only 515 “active” sites in the July 2000 report. Similarly, the 44.9 million sites found in the November 2003 survey are reduced to less than 20 million “active” sites. The obvious problem is that, even in Netcraft’s “corrected” numbers, Register.com’s choice of Apache is still being counted 515 times as opposed to Disney’s choice of IIS being counted only once. Less obvious, but also worth considering, are the questions of which companies or organizations are choosing Apache, IIS, or other server platforms and why. By their own admission, Netcraft’s method of reducing the number of parked domains leaves in place a sampling bias in favor of “the cheap or free bulk hosters.” The fact that a bulk hoster chose to revert to Apache to run 1.4 million domains may have more to do with its lower up-front cost than with its performance, security, or features.

Headlines Have Been Promoting the Wrong Numbers

Port80′s analysis of the Netcraft survey reveals other problems. First of all, Netcraft’s November 2003 survey headline uses the “uncorrected” figure of 44.9 million sites, numbers that their own methodology acknowledges as inaccurate. They go on to discuss the recent migrations of Register.com, Network Solutions, and several other domain parking services and bulk hosting providers as signaling a “significant” change in Web server market share: A 2.8 percent increase in Web sites running Apache is set against a 2.44 percent drop for Microsoft IIS. Scrolling past the splashy lead to look at the “corrected” November results reveals a much more modest gain of 1.25 percent for Apache and loss of only 1.06 percent for IIS. Of course, given the extremely rapid changes in Web technology and business, one must also be wary of calling one-month variations of a couple percentage points “significant.” Port80′s monthly surveys have shown contrary fluctuations in market share — according to the Port80 Web server survey, Apache’s market share has actually decreased by 2.2 percent since January 2003, while IIS has surrendered a paltry 0.1 percent over the same period.

Returning briefly to Netcraft’s choice of data, the company does possess a means to estimate numbers of physical servers at a given ISP as opposed to hostnames, but they choose not to apply this technique to the Web server survey released to the public (http://news.netcraft.com/archives/2003/04/14/ netcraft_hosting_provider_server_count_available.html). Netcraft also conducts a survey of secure Web sites, which might be expected to provide market share data more relevant to e-business decision makers, but they do not generally highlight the results of that survey or provide them without a cost (http://news.netcraft.com/archives/2003/04/09/netcraft_ssl_survey.html). The last publicly available, free results from the Netcraft SSL survey are from January 2001 (http://www.netcraft.com/surveys/analysis/https/2001/Jan/CMatch/dv_all.html). These results showed IIS in the lead with 47.4 percent to Apache’s 28.1 percent in a sample of 121,542 secure sites.

Unfortunately, these qualifications and other data supporting alternate conclusions are not a focus for Netcraft and are not the metrics that make headlines in the technology community. Netcraft’s survey may be an accurate representation of “Web server software usage on Internet connected computers,” as their methodology states, but such statistics are not synonymous with Web server market share.

Port80′s Alternate Methodology Has More Relevance — and Significance

In contrast, Port80 Software’s Top 1000 Corporations Web Server Survey focuses exclusively on the corporate sites of Fortune 1000 companies. Technically, the two surveys are very similar — each uses a header check to determine which software is being used to serve a particular hostname, and are therefore surveys of domains rather than of actual servers. What distinguishes the two is their choice of data sample. While Netcraft’s wide net captures far too many hostnames to be individually verified, and therefore includes a great deal of undesirable data points, Port80′s small net captures exactly 1000 live sites with unique content. A metaphor would be two fishing boats: one using an acres-wide drift net, the other fishing with individual lines. While the first boat goes home at the end of the day overflowing with tuna of all sizes, shrimp, dolphins, and garbage to be sorted out later, the second boat ends the day with nothing but prime adult tuna — if anything else bites, the fisherman can easily determine “I don’t want this” and throw it back.

Beyond the question of scale, the two surveys also differ dramatically in the quality of their data sample. As opposed to asking which Web server software is most common across the whole of the Internet, the Port80 survey attempts to determine the Web server of choice among large corporations with high-volume sites and demanding business requirements. Each Fortune 1000 Web site administrator needs a server that will effectively manage high volumes of traffic and each business decision maker must carefully invest large amounts of money in Web technology. To return to the fishing metaphor, a researcher interested in maintaining healthy tuna stocks is probably more interested in what is sustaining the big fish than the scrawny ones.

With this more focused approach, Port80′s monthly surveys represent a clearer picture of what technology large businesses are choosing to run their Web sites. Among Port80′s Fortune 1000 sample base, Microsoft IIS can accurately claim a much more impressive market share of 53.8 percent, including such heavily trafficked industry leaders as BankOne, Walt Disney and the Gap, compared to 15.4 percent for Apache. Port80′s survey also reports that many Fortune 1000 companies running IIS are upgrading from Windows NT to Windows 2000 and Windows Server 2003, including Intel, Martin Marrieta, and Goody’s Online, reflecting a long-term commitment to the Microsoft platform. Port80 Software’s Top 1000 Web Servers Survey demonstrates that Apache’s market share is likely much smaller in dedicated hosting and corporate environments.

Clearly, there are numerous ways of slicing up the Web, and therefore defining server market share. While Netcraft’s survey of hostnames is touted and accepted in the industry press as representative of the actual Web server market, one should be cautious of using Netcraft data to make claims of this kind. Instead of truly examining market share, Netcraft’s Web server survey is ultimately limited to the question “which server software is most common?” and thus their conclusions are of questionable relevance to business decision makers concerned with which Web server technology to deploy for e-commerce or Web-based applications. In contrast, Port80′s Fortune 1000 Web server survey seeks to answer a far more pressing question: “Which Web server software should I trust with my business?”

Related: How do I mask an Apache or IIS Server header?

Click here to sign up for FREE tech newsletters from iEntry!

Joseph Lima
COO and Director of Product Development, Port80 Software

Joe Lima has led the product development and support teams at Port80
Software since its inception. He has worked for a variety of Internet,
wireless and software development companies, specializing in research
and development for server-centric technologies. A lecturer at the
University of California, San Diego and a published author on Web server
technologies, Joe Lima brings a depth of knowledge on HTTP and server
systems coupled with an everyday understanding of the challenges faced
by Web administrators.

Then it invariably moves to a discussion about how Netcraft and other stats sites show Apache as the dominant server on the Web, or how a certain big site uses Apache, or how there are so many cool modules to add to Apache. Pointing out that scads of non-identified corporate in-house servers run IIS, or that it too is a free server (since it comes with the operating system), or that there are in fact plenty of cool add-ons for IIS (including many that provide source code) — all this does little to dissuade these server chauvinists of their opinion. Rather than whining about rude Apache admins, however, I thought it would be a more useful response simply to write down some of the ways I’ve found of improving IIS. So without further delay here are my top ten tips for making the most of your IIS.

Tip 10: Customize Your Error Pages
Although this is quite simple to do, few people seem to take advantage of it. Just select the “Custom Errors” tab in MMC and map each error, such as 404, to the appropriate HTML or ASP template. Full details can be found here. If you want an even easier solution — or if you want to let developers handle the mapping without giving them access to the MMC — use a product like CustomError.

Tip 9: Dive into the MetaBase
If you think Apache is powerful because it has a config file, then take a look at the MetaBase. You can do just about anything you want with IIS by editing the MetaBase. For example, you can create virtual directories and servers; stop, start and pause Web sites; and create, delete, enable and disable applications.

Microsoft provides a GUI utility called MetaEdit, somewhat similar to RegEdit, to help you read from and write to the MetaBase. Download the latest version here. But to really impress those UNIX admins — and to take full advantage of the MetaBase by learning how to manipulate it programmatically — you’ll want to try out the command-line interface, officially called the IIS Administration Script Utility. Its short name is adsutil.vbs and you’ll find it in C:inetpubadminscripts, or else in %SystemRoot%system32inetsrvadminsamples, together with a host of other useful administrative scripts.

A word of caution though: Just like Apache conf files, the MetaBase is pretty crucial to the functioning of your Web server, so don’t ruin it. Back it up first.

Tip 8: Add spell checking to your URLs
Apache folks always brag about cool little tricks that Apache is capable of — especially because of the wealth of modules that can extend the server’s basic functionality. One of the coolest of these is the ability to fix URL typos using a module called mod_speling. Well, thanks to the folks at Port80 Software, it now appears that IIS admins can do this trick too, using an ISAPI filter called URLSpellCheck. You can check it out right on their site, by trying URLs like www.urlspellcheck.com/fak.htm, www.urlspellcheck.com/faq1.htm — or any other simple typo you care to make.

Tip 7: Rewrite your URLs
Cleaning your URLs has all sorts of benefits — it can improve the security of your site, ease migration woes, and provide an extra layer of abstraction to your Web applications. Moving from a ColdFusion to an ASP based site, for example, is no big deal if you can remap the URLs. Apache users have long bragged about the huge power of mod_rewrite — the standard Apache module for URL rewriting. Well, there are now literally a dozen versions of this type of product for IIS — many of them quite a bit easier to use than mod_rewrite, which tends to presume familiarity with regular expression arcana. Check out, for example, IIS ReWrite or ISAPI ReWrite. So brag no more, Apache partisans.

Tip 6: Add browser detection
There are a lot of ways to build Web sites, but assuming everybody has a certain browser or screen size is just plain stupid. Simple JavaScript sniff-scripts exist for client-side browser detection, but if you are an IIS user you can do better with a product called BrowserHawk from CyScape. The Apache world doesn’t really have something comparable to this popular, mature and well-supported product. Speaking of CyScape, they’ve recently added an interesting-looking related product called CountryHawk that helps with location detection, but so far I haven’t had the language- or location-sensitive content to warrant trying it out.

Tip 5: Gzip site content
Browsers can handle Gzipped and deflated content and decompress it on the fly. While IIS 5 had a gzip feature built-in, it is pretty much broken. Enter products like Pipeboost to give us better functionality — similar to what Apache users have enjoyed with mod_gzip. Don’t waste your bandwidth — even Google encodes its content, and their pages are tiny.

Tip 4: Cache your content
While I’m on the topic of improving performance, remember to make your site cache friendly. You can set expiration headers for different files or directories right from the MMC. Just right click on an item via the IIS MMC, flip to the “HTTP Headers” tab, and away you go. If you want to set cache control headers programmatically — or even better, let your site developers do it — use something like CacheRight. If you want to go further and add reverse proxy caching, particularly for generated content, use a product like XCache — which also throws in compression.

It might involve more time and expense to take full advantage of caching, but when you watch your logs shrink because they don’t contain tons of pointless 304 responses, and your bandwidth consumption drop like a stone, even while your total page views increase over the same period, you’ll start to understand why this particular tip was so important. Cache friendly sites are quite rare, but there is plenty of information available online about the enormous benefits to be had by doing it right: Check out Brian Davidson’s page, this nifty tutorial from Mark Nottingham, and what AOL has to say on the subject.

Tip 3: Tune your server
Tuning IIS is no small topic — whole books and courses are dedicated to it. But some good basic help is available online, such as this piece from IIS guru Brett Hill, or this Knowledge Base article from Microsoft itself. However, if you don’t feel like getting your hands dirty — or can’t afford the time and expense of turning yourself into an expert — take a look at XTune, from the makers of XCache. It’s performance tuning wizards step you through the process of tuning your IIS environment, making expert recommendations along the way.

Tip 2: Secure your server with simple fixes
Sure people are going to attack sites, but you don’t have to be a sitting duck if you’re willing to make even a small effort. First off, don’t advertise the fact that you are running IIS by showing your HTTP server header. Remove or replace it using something like ServerMask — probably the best twenty-five bucks you’ll ever spend. You can go farther than this by removing unnecessary file extensions to further camouflage your server environment, and scanning request URLs for signs of exploits. There are number of commercial products that do user input scanning, and Microsoft offers a free tool called URLScan which does the job. URLScan runs in conjunction with IISLockDown, a standard security package which should probably be installed on every IIS server on the planet. These are simple fixes that could pay off big, so do them now.

Tip 1: Patch, patch, patch!
Okay, we in the IIS world do have to patch our systems and make hotfixes. However, as a former Solaris admin I had to do the same thing there, so I am not sure why this is a big surprise. You really need to keep up with the patches, Microsoft is of course the definitive source, but if you can also use the highly-regarded www.cert.org. Simply search on “IIS”.

Well there you have it: 10 tips for IIS admins to improve their servers. Some of the tips might become obsolete once IIS 6 is gold, but, for now at least, W2K and NT IIS admins should apply a few of these today and sleep a little better at night.

Reprinted with permission from Port80 Software.

Matt Foley is a former Solaris sysadmin who was turned to the “darkside” and is now works for a large southern California hosting and Web agency. He quite likes Windows now in spite of himself.