Save your bacon: Backup your stuff

Today was yet another one of those days that come far too often.

A day when someone tells me their computer crashed and they have no backups. For months.

This isn’t a computer at home that’s used for email, Facebook and maybe an occasional game. This computer is used to manage their customers’ technical data and no one has bothered to back it up. We’re talking several gigabytes of contact information, among other things.

The stumper for me is this: Despite the fact that a sizable portion of this company’s tens of millions in revenue depends on the data this software manages, they haven’t backed it up for months.

Computers can come and go – it’s the data that matters. Except for specialty units like servers and such, many of the computers that do what this one does could be replaced by new, much faster hardware for $300-$400. But none of that matters much if you don’t backup your business data.

Every time I think I should give clients a choice about backing up the data created in systems I create, one of these situations pops up to remind me that no choice is necessary.

I have to keep my clients’ best interest at heart *even when they don’t*.

It happens.

Know anyone whose dog chewed through a computer’s power cord? I do. Ever had the power fail while you were doing something important? Did it mess up your data? It will.

Ironically, I lost power a few hours after I wrote the outline for this piece.

Ever had a client call and tell you their computer was stolen and their backup media was sitting on top of the computer – and it’s gone too? I have. Don’t make me nag and wag my finger at you. Backup your stuff

Backup your stuff

Take your business data seriously. Yes, it’s one more thing to do, even though you can automate it – just be sure those automated backups really do work. Do it every week, if not every day for stuff that you truly cannot afford to lose. Don’t be that person who calls and says “…..and we haven’t backed up since…”

The second most important thing about backups is that you can restore them. Save a copy of the backup on a different device – not the same drive your data is on. If your backups are on the same hard drive as your data, you’re doing it wrong. If that drive dies, your backup dies with it.

At least once a month, try to restore on your backups to a different computer. If you can’t, you’re no better off than the businesses who don’t backup at all.

That electricity thing…

I’m a little NASA-ish when it comes to backup power systems. I have an APC SmartUPS uninterruptible power supply (UPS) with automatic voltage regulation (AVR) on every computer as well as the TV. Yes, I really do mean every computer.

While I rarely watch the tube, I don’t want to replace it if I don’t have to. The UPS units are why I have servers that still run after 10 years.

Computers like stable electricity. They, like your data, are an asset. Depending on what type of computer you use, you might be able to replace it for a couple of hundred dollars – but you can’t replace the data.

You can’t get the time back that you’ll waste replacing hardware, reinstalling software, reconfiguring your network and finally, re-keying your data – if you have it.

A $200-300 UPS will pay for itself with the first outage. Having even two minutes to close files and shut things down normally before firing off an email saying “losing power” etc is worth every penny vs. having it all shut down in a millisecond with no notice, damaging data as it goes down.

If and when electricity spikes or failures cream your machine or your data, there is rarely anything your computer person can do to make things right. Quite often, it’s time to replace the computer and start over.

Sound like fun? It isn’t. Save your bacon. Backup your data. Test your backups by restoring them to a different machine. Sleep better at night.

Worth saying twice: I have to keep my clients’ best interest at heart *even when they don’t*.

Put your mask on first

Fire Smoke IndyW 1428

Professional development mentors remind us that we must take care of ourselves first.

They advise that we improve ourselves mentally, physically and emotionally – in other words, attend first to our overall health – so that we’re better prepared to perform well in our roles at work, at home and in our community.

Personal finance mentors do the same when they remind us to pay ourselves first. If we don’t, something will always come up that consumes those funds, leaving us ill-prepared for our future.

Airline flight attendants ask us to put on our oxygen mask first, then help others sitting near us, because we can’t help our kids or significant others if we’re unable to breathe.

Here’s the technology version of putting your mask on first:

  • Backup your business data.
  • Test your backups regularly to be sure you can restore them.
  • Rotate your backup media off-site so that a theft or on-site fire or water damage don’t render your backups useless at the time you’ll need them most.
  • Document your backup and restore process so that you can restore and get systems running again even though your technology wizard is on a 16 hour flight to Australia.
  • Investigate, plan and implement real-time disaster recovery for your business data, particularly if your business model has little downtime tolerance.

This may seem like a hassle. It may seem like unnecessary overhead. Don’t be tempted by those thoughts.

Fact is, if you put your mask on first, you’ll be in a better position to help your customers solve their problems, grow their business and keep paying you. Why? Because your business will be more resilient.

Look back at the business impacts from an event like Hurricanes Sandy or Katrina. If you were impacted by those storms, how would you service customers who weren’t in the storm track? If you can’t, you know they’re likely to find someone else who can.

Your “Someday” is coming

These kinds of things that happen when your business can’t take a power outage, a hard drive crash or similar disruptions. The question is… when?

No one can point to a date and declare (in their Darth Vader voice) that “Your systems are going to fail on this day.”

What I can guarantee, even without considering Katrina, Sandy, Boardwalk fires, blizzards and ice storms, is that it’ll happen…Someday. These things happen to electronic, mechanical devices. You can either be prepared for them or not.

At least once a week, I hear from someone whose “Someday” has arrived. Three times last month I saw it happen to businesses who didn’t have backups. Like a TV show involving the Kardashians, it’s drama you don’t need.

You might think that hardware failures happen more often to businesses that don’t have backups. The reality is that businesses with good backups simply restore them and keep working, so we don’t hear much about their hardware problems. One result of this is that making backups is ignored until it’s too late.

This puts the security of your clients, your employees, your clients’ employees and the families of all these people at risk.

If your most important database disappeared right now, how would that impact your business? How would you recover? How long would it take to get back to where you are right now, productivity-wise? When did you last test your ability to restore your data from a backup?

If you don’t know the answers, ask your technology people. Don’t do it in an accusing fashion, just explain that you’re concerned about the possibility of hardware failure and natural disasters, so you’d like to know what the backup and recovery plan is and how long the recovery period will take for your business. These are things management should know.

Remember, it’s an asset

While there is no good time for this to happen, history suggests that failures are likely during your busy season, or during financial month / quarter / year end.

The good news is that if you have your backup and restore act together, you might lose some time and productivity when your Someday comes, but you’re far less likely to lose your job or your business.

Backup your data. Test your backups to make sure the restores will work. Schedule these tasks.

Care for your data like an irreplaceable asset.

Insulating yourself from disaster

Late last week, a friend of mine’s website exploded.

More accurately, his webhost’s electrical room exploded and caught fire and took down NINE THOUSAND servers with it.

Because my friend knows these things happen, he was well prepared for it.

He had backups in place. He had an alternate server in place and in more or less the time it took to switch a “Go here” sign from server A (the exploding place) to server B, his site was back up for the most part.

Meanwhile, all 9000 servers are still not back in service after 4 days.

Some of this is their fault, and some is not.

The fire department wouldn’t let them run their backup generators, which is ultimately what forced 9000 servers (and no doubt hundreds of thousands of web sites) offline, but physical damage to the facility would have made that a moot point as they later found.

When the fire department had finished their work, the generators were started, but failed. More delays as a new generator had to be brought in and installed.

Now the generator is up but each of 9000 servers has to be started manually and then checked to make sure it is working. And some of those are going to be problematic because of destroyed cabling on a lower floor.

Imagine if you had your e-commerce online store on those servers. Maybe you’re Brad Fallon and your $750k a month wedding favors online store has been down for a week. A week to Brad is almost $200 grand, and remember, he has a pile of employees and warehouse space to pay for. Can you afford that kind of hit?

Redundancy is an expense AND an investment

The real issue here is a lack of redundancy on the part of The Planet. My understanding is that they have 5 different data centers, yet it appears that they are simply 5 standalone data centers that do not replicate each other.

Ironically, their blog talked about redundancy and recovery from catastrophic events just a few short (pardon the pun) weeks ago. Ironic?

While the expense of redundancy at that level (estimates are that they have 50,000 servers) is substantial, exactly how much do you think it will cost them because they didn’t have that redundancy in place?

It should be noted that they do offer a backup server service, where you get the main server and a backup server (presumably in another location). Wonder how many of those will get sold this month?

So how many customers will this cost them? 9000? 900?

I doubt we’ll ever know, but let’s put on the speculation hat and take a look at the math.

If it cost them 10% of their customer base at that location (900 lowest price dedicated server accounts), at the lowest service level, that’s a loss of $80,100 PER MONTH.

What about future customers who will find out about the explosion and the lack of redundancy and decide to go elsewhere? Pretty hard to measure that. After only 4 days (probably far sooner), there are Google AdWords ads on the net for the keywords “Houston webhost” (and probably others) that suggest “moving off of The Planet”. Their competition isn’t going to let this go away.

And what does this have to do with you?

Are you as well prepared as my friend was?

  • What happens if your web host provider explodes tonight?
  • Do you have backups?
  • Do you have a plan in place – or at least the knowledge – to move your site elsewhere and restore backups to that location in a timely manner?
  • Do you know what kind of redundancy your web host offers? Some have duplicate systems in other locations, some do not. Some do backups for you. Some do not. All serious web hosts have power protection in the form of battery backup systems and generators.

What doesn’t matter to you: The Planet’s problems. There are plenty of other good web hosts out there if you need one.

What does matter to you: How does an event like this affect your business when those problems occur? How many customers do you lose because your systems are unavailable? How many new customers give up after you’ve spent X dollars to get them to your site in a mindset that is ready to buy?

With that in mind, spend some time thinking about what makes sense regarding an investment in redundancy. Then take action.

What do I do for my sites and the ones I manage?

Weekly – I have automated server-level backups taken. They are downloaded to a server here in my office in Montana (the web server is in Michigan). They are then copied to a high end RAID-5 network addressable storage (NAS) drive in my office. All of this happens automatically.

Daily – SQL databases on my web sites (and my clients’ sites) are backed up to a different web server, and in the wee hours, they are downloaded to a server here in my Montana office. They are then copied to the NAS drive mentioned earlier. All of this happens automatically. In addition, all web programming and images here on my main development machines is copied to the NAS drive on a nightly basis. Automatically. Finally, those same items are copied to a laptop (yes, nightly and automatically) so that if my office and the server in Michigan decide to explode, I have the laptop as a last resort.

What’s next? I am in the process of establishing a second web hosting center account so that these automated backups can be pushed to that server so that I can quickly switch to the new site in the event of a disaster, without having hours of restore-from-backups time. Even though the backups are automated and kept current, the time to restore them can be critical for some accounts.

These are the kinds of processes and situations you need to be thinking about.

Don’t depend on someone else to protect your business assets. Let them, but make sure you are covered by things within your control.

These events would likely bury a lesser-prepared competitor. Handled properly, they will make you shine even brighter as the expert in your market – even if your market isn’t tech-related.

Related info: Lessons learned: Think like a fire marshal.