Communication Breakdown: When Email Goes Down
At the end of December, some Hotmail users experienced problems with their email – it was gone. Messages and folders went completely missing from their accounts. Luckily, for those users, the emails came back.
Microsoft says it recovered 100% of email and folders for the accounts affected. Unfortunately, for those who didn’t sign into their accounts between the time of the incident and the time the account was restored, any messages sent to their accounts during that time would have bounced.
Microsoft has apologized for the incident, but it can’t have been very good for the service’s reputation with users, particularly considering there plenty of other options out there. Hotmail has hundreds of millions of users and competitors like Yahoo and Google will be happy to take as many of them as possible.
The whole thing makes you stop and consider how much users are relying on third-parties for essential communication. Who’s to say people didn’t miss extremely important messages during that period?
Microsoft’s Mike Schackwitz details exactly what happened on the company’s Inside Windows Live Blog:
In Hotmail, one way we monitor the health of the email service is through automated tests. We set up a number of accounts with different configurations, and then use automated tests to log into these accounts, simulate normal user activity and behavior, and report when errors are found. We use scripts to create and delete these test accounts in bulk. The way we delete a test account is to remove its record from a group of directory servers that route users and incoming mail to the correct mailbox.
On December 30th, we had an error in a script that inadvertently removed the directory records of a small number of real user accounts along with a set of test accounts. Please note that the email messages and folders of impacted users were not deleted; only their inbox location in the directory servers was removed. Therefore when they logged in, a new mailbox was automatically created for them on a new storage server that didn’t contain their old messages and folders. This is why the accounts received the “Welcome to Hotmail” message.
Read the post for further explanation.
It’s not like Microsoft is the first provider to experience downtime. Google has always bragged about its Gmail uptime (and has a dashboard where users can monitor it), but it’s gone down on occasion too. Facebook is trying to redefine email and electronic communication with its social inbox, but Facebook recently went down for a lot of users itself. Twitter is no replacement for email, but a lot of people communicate with it frequently, and that fail whale appears fairly frequently.
Microsoft says it’s updating its infrastructure, and changing its alert process, as well as its feedback process to take preventative action against future incidents. Unfortunately, and this goes for any company, it’s usually the issues you don’t think to prevent that end up costing people.