[Thanks Markus!]

Comments

Tag:

Google’s paper titled ‘Failure Trends in a Large Disk Drive Population’ was crafted for the Usenix conference on File and Storage Technologies.

With little published formal research on why drives fail available, Eduardo Pinheiro, Wolf-Dietrich Weber and Luiz André Barroso did what engineers do and researched it themselves.

They suggest in the paper’s abstract that 90 percent of the new information produced around the globe has been tucked into storage on electronic media somewhere; that’s based on 2002 research performed at UC-Berkeley.

In most cases, that storage means a hard drive. Since the engineers just happen to have access to "a large disk drive population in a production Internet services deployment," they decided to look at that and see what they could learn about drive failures.

That research made it into their report (PDF). It’s what they didn’t find that proves the most interesting.

Modeling based on the self-monitoring facility, called SMART, of a drive, looking at parameters that tend to match failing drives, was dubbed "unlikely to be useful" for predicting other drive failures. "Surprisingly, we found that temperature and activity levels were much less correlated with drive failures than previously reported," they wrote.

Spinning and overheated disks have long been considered an early warning sign of drive failure, but to the engineers these symptoms were not a red flag of imminent failure.

Drives failed in their analysis without tripping any SMART indicators.

They also noted that after a drive suffered its first scan error, it was 39 times more likely to fail within a 60-day period than drives lacking those errors.

Predicting failures based on SMART alone looks like a very inaccurate endeavor, and SMART may be more useful for determining trends in a drive population instead.

 —

Add to Del.icio.us | Digg | Reddit | Furl

Bookmark WebProNews:

That’s great for performance, but sometimes you want to know that data really has gotten to the disk drive. This could be because you want to test the performance of the drive, but could also be when you suspect a drive is malfunctioning: if you just write and read back, you’ll be reading from cache, not from actual disk platters.

So how can you be sure you are reading data from the disk? The answer actually gets a little complicated, particularly if you are testing for integrity, so bear with me.

Obviously the first thing you need to do is get the data in the cache sent on its way to the disk. That’s “sync”, which tells the kernel that you want the data written. But that doesn’t mean that a subsequent read comes from disk: if the requested data is still in cache, that’s where it will be fetched from. It also doesn’t necessarily mean that the kernel actually has sent the data along to the disk controller: a “sync” is a request, not a command that says “stop everything else you are doing and write your whole buffer cache to disk right now!”. No, “sync” just means that the cache will be written, as and when the kernel has time to do so.

Traditonally, the only way to be sure you were not reading back from the cache was to overwrite the cache with other data. That required two things: knowing how big the cache is at this moment, and having unrelated data of sufficient size to overwrite with. On older Unixes with fixed sized buffer caches, the first part was easy enough, and since memory was often expensive and in shorter supply than it is now, the cache wasn’t apt to be all that large anyway. That’s changed radically: modern systems allocate cache memory dynamically and while the total cache is still small compared to disk drives, it can now be gigabytes of data that you need to overwrite.

Well, that’s not always so hard: for a large filesystem and relatively small memory, a simple “ls -lR” might be enough. If not, a “dd” redirected to /dev/null can fill it up. Just make sure that you are looking at different disk blocks than what you first wrote. Note that you really didn’t even need the “sync” if this is what you are doing: the overwrite forces the sync itself.

Modern Linux kernels make this a bit easier: in /proc/sys/vm/ you’ll find “drop_caches”. You simply echo a number to that to free caches.

From http://linux.inet.hr/proc_sys_vm_drop_caches.html:

To free pagecache:

But that didn’t behave as I thought it would either. Possibly VM caching is throwing this off? Nope: I tried the same thing on a real system; the file doesn’t lose its updates. So I’m not sure you can trust drop_caches.

However, if testing for integrity, and perhaps even if doing serious performance testing, this isn’t enough: disk drives almost always do their own caching. If we really need to be certain that our reads came directly from the platters and not from ram on the controller, we still need to go back to the idea of knowing how big that cache is and writing enough data to force it to be flushed. So, we are still going to do “dd”‘s or “ls -lR”‘s or something like that.

If you are examining integrity and suspect corruption, keep in mind that aging can affect your results: you might need data to sit in cache (kernel or disk hardware) for some period before the problem occurs. Quick overwrites might mask it. Tracking down this kind of problem can be very difficult.

See also Caches and cache data corruption

By the way, if your aim is simply to bypass cache buffering, you can do that: Raw Disk I/O is what you want. See http://aplawrence.com/Bofcusm/2658.html also. And (as some databases do) you could simply write data to a raw partition (no filesystem).

*Originally published at APLawrence.com

Reddit | Furl

Bookmark WebProNews:

A.P. Lawrence provides SCO Unix and Linux consulting services http://www.pcunix.com

Onyx

Onyx is free software. It’s the only one of this group that includes S.M.A.R.T monitoring (turn that on under Preferences; actually see it under Info-> Disk). As all these do, Onyx reads and sets preferences (it calls them “Parameters”) for the Finder, Dock and others. Onyx includes Safari, allowing you to control such things as the size of your History menu and other settings that aren’t in the Safari Preferences panes. As most such utilities do, it includes running and rescheduling of system cron tasks, verification of disk permissions and rebuilding of Spotlight and the Launch Services database. It can clean out caches and log files and empty your trash. It also has an “Automation” section that does similar tasks and more at the cick of the “Execute” button.

A Log file viewer conveniently gives centralized access to all system and application logs; Unix Utilities include a man page viewer, plutil, locate, and system_profiler. “Info” shows similar informaton to system_profiler in a more compact form.

Cocktail

Cocktail is $14.95, the freely downloadable demo is apparently limited in some features, though it wasn’t clear to me just what those limitations are. Coctail controls preferences under its “Interface” tab. Like Xupport, the Login section includes a Kiosk mode which I did not test. The Misc. section includes the ability to set the screenshot file format; I didn’t find that in the other utilities. It includes setting MTU and TCP window sizes under Network. It has control of locked files and the usual cleaning of logs and caches under “Files”. “System” lets you run the system cron jobs, rebuild the ocate and whatis databases as well as Launch Services, empty the trash and so on. “Disks” can enable or disable journaling, repair permissions and set the power saving Spindown times.

Xupport

Xupport is $19.90. The free download seems to be time limited, but I don’t know when it expires. Xupport has a large number of preference items under “Settings” (but not as many as Onyx). Like Cocktail, it includes control of some TCP settings and adds control of virtual memory files. Xupport includes a “Backup” utility that can clone a bootable disk or create an image for Apple Software Restore. The “Browser” tab is a Finder-like file browser; I really don’t see the point of that. The “Unix” tab displays man pages, and “Info” is a convenient overview of your machine’s hardware and software.

MainMenu runs as a menu bar item – you may not notice the little plus sign in a rectangle that appears when you run it. You need to click on that to actually use the program. MainMenu includes the typical running of system cron jobs, repairing disk permissions, cleaning caches, rebuilding Spotlight and other databases, and setting some Finder settings. Under “Other Tasks” it has log cleaning and flushing of some caches, but also includes disabling Dashboard (I had done that long ago manually).

Summary

There is plenty of overlap between these, but there are also unique features not found in the others. None of this is anything you cannot do yourself through terminal commands or editing preference files manually, but these do offer the graphical interface for ease of use. Onyx is proably the most full featured of the group.

There are lots more of a similar nature: see System/Disk Utilities at Apple for more.

*Originaly published at APLawrence.com

Tag:

Add to Del.icio.us | Digg | Reddit | Furl

Bookmark WebProNews:

A.P. Lawrence provides SCO Unix and Linux consulting services http://www.pcunix.com

But do you really need all that disk space? Is it really more important to puchase 20 GB of space than it is to purchase 10 GB?

Disk Space And Bandwidth

Disk Space is the amount of space you’re given to store your files. The common misperception is that you need all the space being offered and so should always go with the host that offers the most.

A small static website typical for many small businesses really doesn’t need a lot of space. Let’s assume your site is 100 pages and that each page has a file size of 100KB. Further let’s assume each of your pages has another 100 KB of images for a total of 200 KB per page.

100 pages at 200 KB each is approximately 20 MB of space. So does it really matter much if you purchase a hosting plan that comes with 5 GB space or 10 GB space? You could fill 5 GB with about 250 copies of each of your files.

It’s still a good idea to get more space than you currently need, but having 500 times as much space as you need isn’t a much better deal than having 250 times the space you need.

Bandwidth or disk transfer can be thought of as the amount of disk space you can trasfer across the web. Using the above 200 MB site as an example, if one person accesses your site over the course of a month and views every page then you’ve used 20 MB of disk trasfer that month. If 10 people accessed each of your pages you’ve transferred 200 MB across the web during the month.

More bandwidth is usually a good idea, but how much you need depends on how many visitors you get to your site and how many pages they attempt to view. Just as with disk space there’s a good chance you won’t need all the Bandwidth a web host offers.

You’d sooner have more of both disk space and bandwidth, but do keep in mind that most hosting plans will likely offer more than enough. A good idea is to determine as best you can what your needs will be. If you estimate you will be needing 10 GB of space two years from now then make sure the plan you sign up for has at least 10 GB, but don’t jump to another host simply because they offer 30 GB.

Real World Examples

I’ll use my own site as a real world example. At present I have close to 300 pages on my site. Not overly large, by any means, but probably within range of many small business sites. In addtion to HTML related files and images I have several databases installed to run among other things my blog, TheVanBlog, and client management software to help run my business.

Even with all those pages and databases my site currently consumes about 70 MB of disk space. Bandwidth, while consistently growing as traffic to the site grows, was still just under 1.5 GB for last month. Admittedly I know how to develop pages to keep their file sizes smal and most of the site is text, with very few images. Still it should provide an indication that the multi-gigabyte offerings with many hosting packages aren’t always necessary.

When You May Need More Space

It is possible you’re space requirements will be higher. Video and audio files will take up a lot of space and if your site is about either then you will probably want to purchase more space.

You may decide to run a forum on your site. If you do and if you build a stong commuity you may find your database and your disk space growing pretty fast. Those forum posts are generally text only and not needing a lot of space, but space may be more of a consideration with any fast growing site.

Most web hosts will let you upgrade your plan or purchase more space if your needs change at a later date so even if you do find yourself needing more space it shouldn’t be an issue. Just make sure to keep an eye on how much pace you are using and if necessary purchase more when the time comes.

Overselling, What Is It And Should You Be Concerned

Many web hosts practice something called overselling. The idea behind overselling is to sell a little more space than the host can really offer, because they know most people won’t ever use all they purchase.

Back to the example above let’s say you chose a 5 GB hosting plan to host your 200 MB website. Your site is using less than 5% of the space your currently paying for. That leaves a lot of empty space on the server. Again webhosting is a very competitive field and knowing that you’ll likely never us all the space with your plan many web hosts will sell that space as part of another hosting account.

In the end they may sell 150 GB of space on a 100 GB server. Obviously there would be trouble if every account decided to make use of all the space in their plans. But reality tells us this probably won’t happen and even if it did your host would simply move some sites to another server before it became an issue.

Overselling may seem rather dishonest and unethical, but unless abused there’s really nothing wrong with it. While it is possible that everyone would max out their disk space, it’s very unlikely. Also servers can be set up easily to send a warning when they are getting full. Your host might have a threshhold set and once a server reaches 80% of capacity some of the accounts are moved to another server to assure that there is always enough space.

If abused though, overselling can be risky. While you and every other account on a server may only be using 5% the space alloted, it would be very risky to sell the space on that server 20 times over. However it’s not uncommon to see server space sold five times over.

Summary

If you’ve ever wondered how some hosts can offer so much more than any other at much lower rates it’s probably because they are overselling their server capacity more than they should. Their offer of unlimited space for $1.99 may sound great, but you really aren’t getting as a good a deal as you might think. That oversold server may not have your space when you need it and may run slower because it’s being filled to greater capacity.

And while it may be tempting to purchase the hosting plan that offers the most space keep in mind you may only need a small amount of that space.

Related posts:

Interview with Steven Bradley (vangogh)
Guest blogging
How to use word of mouth

Reddit | Furl

Bookmark WebProNews:

Steven Bradley is a web designer and search engine optimization
specialist. Known to many in the webmaster/seo community by the username
vangogh, he is the author of TheVanBlog, which focuses on how to build
and optimize websites and market them online.

Let’s say you have a system with a few filesystems. One of those systems is getting tight on space, but the other has plenty of room. You want to add a large amount of data, but it has to be in the file system that’s low on space. For example, it’s /home that is low on space, and it’s /home/fred/drwaings that needs more room.

There are several ways to handle that. First you could just move that “drawings” directory to /bigdrive/drawings and leave it at that. Any scripts that use /home/fred/drawings would need to be updated and any users who need access would have to get used to looking on /bigdrive. You could also just move all of fred: ~fred could now be /bigdrive/users/fred or whatever. If only Fred used these, that might be an easy solution: move him over, edit /etc/passwd to change his home directory (or use “usermod” to do it for you). Or..

You could use a symbolic link. This method starts by moving “drawings” to /bigdrive as before:

mv /home/fred/drawings /bigdrive/drawings

And then:

ln -s /bigdrive/drawings /home/fred/drawings

For almost all uses, this is entirely transparent: no scripts need to be modified, no users need to be notified or retrained. Any action that accesses /home/fred/drawings will end up accessing /bigdrive/drawings instead. Well, almost any action – “ls -l” does work slightly differently after the move:

# before the link
$ ls -l drawings
total 53776
-rw-r--r-- 1 apl wheel 17086 Nov 12 09:46 01-04-06_0537.jpg
-rw-r--r-- 1 apl wheel 42590 Nov 12 09:46 12-28-05_0521.jpg
-rw-r--r-- 1 apl wheel 48270 Nov 12 09:46 12-28-05_1038.jpg
-rw-r--r-- 1 apl wheel 39134 Nov 12 09:46 12-28-05_1042.jpg
-rw-r--r-- 1 apl wheel 39138 Nov 12 09:46 12-28-05_1043.jpg
-rw-r--r-- 1 apl wheel 41874 Nov 12 09:46 12-28-05_1044.jpg
-rw-r--r-- 1 apl wheel 41578 Nov 12 09:45 12-28-05_1045.jpg
etc.
# after the link
$ ls -l drawings
lrwxr-xr-x 1 apl apl 9 Nov 22 08:10 tfoo -> /bigdrive/drawings

Also, “rm” won’t complain if you just “rm /home/fred/drawings”: it won’t warn about this being a directory – but it doesn’t remove /bigdrive/drawings, just the symbolic link in /home/fred. But watch out: some Unixes have slightly different treatments – see Different symbolic link behaviour.

But there’s nothing else that would confuse a script or program ordinarily. A program can determine that it has crossed a symbolic link if it wants to, but there’s seldom any reason to, so you are unlikely to have any issues.

Symbolic links can be confusing if you are running around with “cd”. The issue is this: if you were sitting in /home/fred and then did “cd drawings”, and you then “cd ..”, where are you? Will “pwd” show /home/fred or /bigdrive?

Well, it depends on your shell and perhaps some environment variables. For example, with Bash, you could “set -P” to make “cd” follow the physical path for symbolic links. You can also just say “cd -P” or “cd -L” to have specific control if necessary.

Mac OS X has symbolic links, but it also has something similar called aliases that have some features symbolic links do not.

Way back when, Microsoft MSDOS had something that was a little bit like a Unix symbolic link. Actually, it was a way to “mount” drives, but they called it “JOIN”. For example, you could “JOIN D: C:\NEWDRIVE”, and then if you “CD \NEWDRIVE” you’d be sitting on D:. Hardly anybody ever used this, but it has been there just the same (there’s a Unix “join” also, but that’s a database like command that merges files based on a common key- see “man join”).

*Originally published at APLawrence.com

Add to Del.icio.us | Digg | Reddit | Furl

Bookmark WebProNews:

A.P. Lawrence provides SCO Unix and Linux consulting services http://www.pcunix.com

Second, and this was much earlier and most likely will be denied but I swear it was true – he said that there would come a time where disk was unnecessary because memory would keep getting bigger and cheaper. His first argument is coming to fruition, albeit slowwwwwly. His second didn’t take into account the absurd rate of data growth and therefore the economic impact associated with that.

Dell made this announcement which is using removable disk technologies from ProStor as a direct low-end/mid-market replacement for tape based media and systems. These cartridges have the same basic form factor as tape and even higher levels of tolerance (for dropping them, for example) and longer media life – AND they are disk – good old random access, readable in twenty years, disk. The possibilities are endless.

If there is a big business right now replacing tape and tape function with disk based subsystems, what if the data de-duped frenzy going on now could also have a removability element for deep bunker archive?

Comments

Tag:

Add to Del.icio.us | Digg | Yahoo! My Web | Furl

Bookmark WebProNews:

Steve Duplessie is the author of the “Steve’s IT Rants” blog, and the founder and Sr. Analyst of the Enterprise Strategy Group.

Here’s what I’ve discovered, followed by my requirements.

The List

I’ve evaluated exactly zero of these so far. That’s next.

s3sync.rb is written in Ruby as a sort of rsync clone to replace the perl script s3sync which is now abandonware. Given that I already use rsync for much of my backup system, this is highly appealing.

Backup Manager appears to now have S3 support as of version 0.7.3. It’s a command-line tool for Linux (and likely other Unix-like systems).

s3DAV isn’t exactly a backup tool. It’s provides a WebDAV front-end (or “virtual filesystem”) to S3 storage, so you could use many other backup tools with S3. Recent versions of Windows and Mac OS have WebDAV support built-in. Java is required for s3DAV.

S3 Backup is an Open Source tool for backing up to S3. It’s currently available only for Windows. Mac and Linux versions appear to be planned. The UI is built on wxWidgets.

duplicity is a free Unix tool that uses S3 and the librsync library. It is written in Python but not considered suitable for backing up important data quite yet.

S3 Solutions is a list of other S3 related tools on the Amazon Developer Connection.

Brackup is a backup tool written by Brad Fitzpatrick (of LiveJournal, SixApart, memcached, perlbal, etc…). It’s written in Perl, fairly new, and doesn’t have a lot in the way of documentation yet.

Jungle Disk provides clients for Mac, Windows, and Linux. It also offers a local WebDAV server.

For those keeping track, non-S3 options suggested in the comment on my previous post are Carbonite, rsync.net, and a DreamHost account.

Are there other S3 tools that I’m missing?

Also, I’ve found that Amazon’s S3 forum is quite helpful. The discussion there is generally of good quality and the software does the job nicely. Perhaps we should do something similar for YDN instead of using Yahoo! Groups?

My Requirements

Most of what I need to backup lives on Linux servers in a few collocation facilities around the country (Bowling Green, Ohio; San Jose, California; San Francisco, CA). My laptop and desktop windows boxes have USB backup and get automatically synced to a Unix box on a regular basis already using the excellent SyncBack SE, so I don’t need to re-solve that problem.

I don’t really need a fancy GUI. I’m really looking for a stand alone tool that’s designed to work with S3 and keep bandwidth usage to a minimum. Alternatively, something that works at a lower level (such as a filesystem driver) to provide a “virtual drive” type of interface might work as well.

Bookmark WebProNews:

Jeremy Zawodny is the author of the popular Jeremy Zawodny’s blog. Jeremy is part of the Yahoo search team and frequently posts in the Yahoo! Search blog as well.

Visit Jeremy’s blog: Jeremy Zawodny’s blog.

Not content to simply attack the lower-end markets, they tacked on a 1 million hour MTBF (which means not only will the drive never break, it will be spinning post-Apocalypse) and a 5 year warranty.

Sounds great right? It sure does. Now, did the company use its marketing muscle to come up with a crafty high-end “enterprise” caliber name for the line? Something like DS500 (death star) or Infinity500 or DC500 (data center)??? Nope. It uses Deskstar. Yep. Deskstar. Why not Lunch box 500? or CameInTheCerealBox 500? I can see the ad campaign now, “Mr. VP of IT, if you have a desk, now you are a star”. Ugh.

How many desks have critical enterprise data on them? Maybe the age of consolidation has gone so far as to shrink data centers down into furniture looking things. HP used to make an MPE machine that looked like a desk. I sat on it in the Boston, MA Putnam Investments data center once, having no idea it was not furniture. That might have been OK, except I was the EMC sales rep/installer that was going to plug a new memory board into that desk, which probably did not instill a very high degree of confidence in the customer. I’m pretty sure I left the board there for one of the guys techies and took him to a bar to wait it out.

I digress. My point is that if you wanna make people feel comfortable about applying your product or technology into their worlds, you should first try to understand what those requirements mean, and second, you might want to consider a naming convention that connotes confidence. Who would buy an Ab Blaster if it were called a “You’ll never use it anyway, and end up putting it in the basement with clothes hanging from it that don’t fit you anyway, you fat slob”?

Tag:

Add to Del.icio.us | Digg | Yahoo! My Web | Furl

Bookmark WebProNews:

Steve Duplessie is the author of the “Steve’s IT Rants” blog, and the founder and Sr. Analyst of the Enterprise Strategy Group.

Both of these are based on work developed at the University of Maryland called Tree Maps. There are references at that link to many other implementations and similar projects on Windows and Linux.

When you want to see disk usage, a graphical view is intuitive and easy to understand. For example, here’s how GrandPerspective sees my home folder:

GrandPerspective is very simple: the larger the rectangle, the larger the file it is representing. You click on the rectangle to see the path and name of the file. The only options available are to change the colors to have them selected by depth, directory, name or extension. That’s it: this is a simple and very direct application.

Disk Inventory X is more ambitious. Like GrandPerspective, it displays rectangles based on the sizes of files. But it also shows much more information:

(Click to see image)

All of the gold rectangles in that image are Parallels Workstation documents (the same files are the large blue boxes in the GrandPerspective view). Notice to the right that documents of that type consume a total of 14.2GB of my hard drive. I can also see that I have 20.5MB of PDF files and so on. That’s helpful information.

I can’t really say that one of these is better than the other: it depends on what I’m looking for. The simplicity of GrandPerspective is attractive for a quick overview, but Disk Inventory X includes information GrandPerspective doesn’t have. I’ll keep them both.

*Originally published at APLawrence.com

Add to | DiggThis | Yahoo! My Web

A.P. Lawrence provides SCO Unix and Linux consulting services http://www.pcunix.com