Spider silk is a versatile material. For example, the single thread the spider uses to drop down, pull it and it stretches, but with no outside forces acting on it, it stiffens to its original state.

How does this work? You may ask, well, According to one source, during the elastic stage of the web, the proteins in spider silk are pressed into finite folded structures. So when the spider decides to drop, the structures unfurl, when the folds are gone, the proteins reconfigure into the tough structures referred to as beta-sheet nanocrystals.

Scientists performed a study to see how molecular properties impact the entire web. So, a group of researchers sought out a a basic web built by your every day casual spider. With the spider still on the web, the team hung tiny metal wires on the web to simulate an insect caught in the web. When they pulled the wires, the individual spokes of the web broke, but the other threads, acting as almost support structures, didn’t break.

The results of the experiment really show off the unique ecology of these arachnids. Spider silk is one of the most scientifically interesting natural materials. Other creatures that spin silk, such as silk worms, their silk is made to be more stretchy than anything. A spider’s silk, however, can be elastic and incredibly strong at different times.

Needless to say spider webs are incredible structures and a useful hunting tool for the creatures. The architecture and pure aesthetic beauty of the web is just an added bonus. Spiders may be creepy to you, but they sure are smart arachnids.

"It’s important to realize that it’s not really actual robots or actual spiders out there…instead, it’s banks of machines …at Google’s data centers who open up an HTTP connection and request a page and then get it back," he says. "So any bank of machines (even 50 machines) could easily be requesting a bunch of different content."

"We try to refresh a large fraction of the web every few days," he adds. "So it turns out you really don’t need a ton of machines. Even a relatively small amount of machines operating in parallel and fetching pages in parallel can really be able to crawl of find new pages on the web in a very quick way."

Matt says that Google doesn’t give out the exact number, but that it’s somewhere between 25 and 1,000. I’m not sure what you can really do with that information, but it’s worth hearing a quick rundown of how it works for those who aren’t real familiar with how Google indexes content.

Related Articles:

> Google Rolls Out Breadcrumb Display in SERPs

> Google Makes it Easier to Tell Where Results Originate From

> Get More Links in Your Actual Google Results

I had a lot of comments about whether or not Adwords was necessary, so I thought I’d try it again without running Adwords this time. Here’s how it all played out:

1) I created 5 pages of content (Home, FAQ, About Us, etc.).

2) I put them in a simple template with site-wide links. I also linked to it from one of my other sites (it’s very relevant so it makes sense).

3) I added tagged the site on only 2 social bookmarking sites.

4) Commented in 1 forum, put the URL in one directory (niche specific), and submitted it to Digg.

5) Installed Google Analytics

6) Created a sitemap, pinged Google, and put the sitemap in my Robots.txt. Logged into Google Webmaster Central and submitted my sitemap there.

When I checked exactly 24 hours later I was sitting in the index and had already begun to get a few visitors from Google.

I had previously done Google adwords out of both necessity (get quick traffic) but also because of the trust factor I believe it gives to Google, and the fact that Google integrates a quality factor into their quality score (so they come to your site and look at it). Obviously this is just one test compared to several others I’ve done with Adwords, but it seems its very possible without running some ads.

Anyone else seeing 24 hour indexing for new sites?

Comments

Nearly all search engines utilize spiders (which are also known by their original name, robots) to go out and scour the web looking for web pages. These search engine spiders then bring the data back to be indexed by the engine.

 Since roughly 1996, individual meta commands have existed that can be used on individual web pages to modify how these search engine spiders behave. The most useful of these commands are fairly universal and respected by almost all search engines. What follows is a list of some of the more popular spider commands and instances in which you might want to use them.

 <meta name="robots" content="index">

 This meta command is one of the most common ones used – and it is also the least necessary. It tells search engine spiders to come on in and put the page in their index. However, all search engines do this by default anyway. Basically, if you want to put it in there for fun, be my guest, but this command is not giving you any special treatment. All search engines are going to index your page, unless you specifically tell them otherwise.

 <meta name="robots" content="follow">

 The follow command is different from the index command. It basically requests that the search engine spiders follow the links that are on a particular page. Again, however, this piece of code is completely unnecessary because all search engines are going to follow the links on a page, unless otherwise directed.

 <meta name="robots" content="noindex">

 The noindex command, the opposite of the index command, tells search engine spiders not to index the content of a page. It’s important to note however that search engine spiders will still follow the links on a page that uses only this command.

 When not used for legitimate purposes, this tag can be dangerous because it can put you at risk for penalization by most, if not all search engines. This is because you can use a noindex tag to hide pages with multiple links that you don’t want visitors to see but that you do want all search engines to index.

 There are however some legitimate uses for the noindex command. For example, if you have a dynamic site and you’ve created static pages to replace some of your dynamic pages, which can make them easier for search engine spiders to access, you could put a noindex tag on the dynamic version.

 As Google mentions in its Webmaster Help Center:

 "Consider creating static copies of dynamic pages. Although the Google index includes dynamic pages, they comprise a small portion of our index. If you suspect that your dynamically generated pages (such as URLs containing question marks) are causing problems for our crawler, you might create static copies of these pages."

 In cases like these, it is acceptable to use the "no index" command on the dynamic version of the page, so that your content will not be treated as duplicate. You are not tricking all search engines, you’re just redirecting them.

 <meta name="robots" content="nofollow">

 This tag tells search engine spiders that it’s OK to go ahead and index a page and list it but that they shouldn’t follow any of the links that are on the page. This can be useful if, for example, you had some partners that requested a link on your site that you felt obligated to give, but you wanted to hold onto as much Page Rank as possible. Now this is of course between you and your own personal god, but you would be able to in effect have a partners page, add the nofollow attribute to the meta tags, and basically not pass on any of your Page Rank to any of the sites to which you are linking. The nofollow command in effect tells all search engines that this is the end of the line.

 <meta name="robots" content="noindex,nofollow">

 Obviously, noindex and nofollow are powerful tags – and in combination, they can make a page and the subsequent pages to which it links invisible to nearly all search engines.  This combination command tells search engine spiders, "Do not read this page; do not follow any of the links on this page; do not include this page in your index."

 This command has its beneficial uses. For example, it can be placed on pages on a site that have duplicate content for legitimate reasons. A website might have both a page for the United States and a page for England that cover the same product with exactly the same content. However, nearly all search engines would see this as duplicate content and could devalue both pages. So placing this command on one of them means that search engine spiders will walk on by and you won’t be penalized.

 <meta name="robots" content="noarchive">

 Finally, almost all search engines today, including Google and Yahoo, offer a cached version of a site alongside its listing that provides a snapshot of what the page used to look like. The noarchive tag, therefore, is available to be used in circumstances where there is content on your website that is of a timely nature and therefore that you might not necessarily want search engine spiders to cache for people to have access to moving forward.

 For example, a business might run a one-time special that has a ridiculously low price to drum up some business while things are slow. The business will want to be able to shut that sale down as soon as sales are back up to a solid level. However, it is conceivable that someone could click on the cached version of the business’s site, see the old deal that was out there, and insist on getting it for themselves. By using the noarchive tag, you are telling search engine spiders, in effect, "This page is subject to frequent changes, and I don’t want my visitors to have access to some of this content at a later time."

 Conclusion

 The commands discussed above are just a few of the ones in existence, and new ones are being added frequently. While nearly all search engines support these commands, there are still some that don’t.  The ones in this article, however, are fairly universally understood by search engine spiders, no matter from where they originate. As more universal commands are introduced, I will write about them in future articles.