Is Your Facility Prepared for a Disaster?

BY GLEN PRIDGEN

This is a question I ask myself every morning: “Am I ready to completely rebuild my network within 24 hours if necessary?” A disaster recovery plan should target two very different types of situations; an isolated disaster and a community disaster. Hurricane Katrina was both.

An isolated disaster usually affects a single company and its customers. The area affected is small and includes employee turnover, office incidents, computer crashes, server crashes, Internet connection failure, and phone system failure. This is the most common situation and is where we dedicate the majority of our resources. An isolated disaster recovery plan (IDRP) should address the following:

• A daily back-up. Back up all data on all servers and personal computers daily to tape and store that tape offsite; to do so, have your administrator take the password-protected, encrypted tape home every night. Two weeks’ worth of back-up tapes is a must. Off-site back-up via the Internet is reasonably priced and should be utilized if possible. Archive one tape a month and save it for a year. It is crucial to verify that the back-up tape actually has data on it and that the restore works. There are countless nightmares of information technology (IT) managers trying to restore important data from tapes that are either corrupt or simply contain no data at all. You may be surprised to learn that one in every 10 back-ups incurs some type of failure. If your back-up plan is executed correctly, the most data you should ever lose is one day’s worth.
• Hardware. Your network should include a real server with Raid 5 hard drives. The consequence of turning a PC into a server to save a few bucks could be catastrophic.
• Internet connection failure. Occasionally, connection to the Internet will fail. Most service-level agreements allow for one down day every 100 days. Have one computer with access to a trusted dial-up account. Most Internet service providers will include access to their dial-up network as part of their standard offering. Also, have a Blackberry onsite that will at least give you access to one e-mail account.
• Phone system failure. Common sense tells us that signing a maintenance agreement for anything is a waste of money, and for the most part, that is true. But when it comes to the main means of communicating with the outside world, I feel it is a worthy investment. The fact is that if you don’t have a maintenance agreement on your phone system, then most companies will make it a point to respond as slow as possible to your outage. Bite the bullet, sign the contract, and know that if there is trouble, you will be taken care of in a timely manner.
• Employee sabotage. In most cases, it sounds more sinister that it really is. Most employees delete many e-mails and files before turning in their notice or quitting with no notice. In their haste to remove “personal” data, many business files become innocent casualties. The more sinister acts involve employees who delete files with the intent to inflict damage. With a minimum two-week supply of back-up tapes, both scenarios can be easily overcome. Your network should be protected by a firewall, and virus protection should be installed on every computer and server.

A regional disaster usually affects more than one company, its customers, and its neighbors. The area affected can include the loss of a building or the damage and destruction of your community. I have experienced two types of these disasters (theft/building loss, and community destruction). These experiences have taught many lessons and brought with them some unexpected challenges. When Bill Belichick, coach of the New England Patriots, was asked how he prepares for a season, he stated, “We focus most of our energies on the little things, because the big things will take care of themselves.” I will focus on the little things and assume that we all know if a building catches on fire to take the stairs and when officials tell us to run from a hurricane, then we should run.

When it comes to office theft and building loss, both of these incidents have the same challenges. Replace equipment and restore data as fast as possible. The building loss is more complicated. The following contingency precautions will assure an efficient and effective transition. For scheduling patients, you should use a fax number from a company such as maxemail.com or efax.com. This way, all faxes are e-mailed to your scheduler and a copy is backed up by the hosting company. You can have multiple e-mail destinations as well as the ability to change where the faxes are delivered immediately. Also, your phone company should provide a service that allows you to forward your phones from a remote location. The service is not a standard offering but is inexpensive and absolutely crucial in a crisis. Have an executive office space in mind as a contingency plan; they provide all necessities, including instant phone service and connection to the Internet while only requiring a short-term lease. If the building is gone or damaged, then you won’t be able to do cases, but with good planning you will continue collecting for cases already performed and will have a “war room” to plan for your grand reopening.

The biggest surprise for me was the complete collapse of our ability to communicate within the affected areas. Not one of our high-tech toys worked in these areas after the storm. Cell towers were either blown over or without power, thereby cutting off most service. There was no access to the Internet, and most land-line phones were out of service. While cell phones were unable to make voice calls, most were able to send and receive text messages intermittently.

Immediately after the storm hit, the only updates I could get directly from our Biloxi facility were via the AARL amateur radio network, a group of volunteer ham radio operators that immediately started helping rescue teams and families communicate. That’s right, we spend billions of dollars on Blackberries, cell phones, and laptops, and the only way we could communicate was with 100-year-old technology. I am not saying that we should all become ham radio operators; I am just saying that we should all at least get to know one.

Now, we have executed our IDRP and Hurricane Katrina has just destroyed our area. As we have seen, our biggest failure is an inability to communicate. To overcome this, we must designate a single point of contact (SPOC) for all communication immediately after the storm. This person should be located outside of the affected area and utilize a phone and phone number from that location. Even though you are in Atlanta, you are safe to assume that your Biloxi cell phone will not work. Instruct all employees to check in with the SPOC and let them know their status. In our situation, the biggest stress for all employees was accounting for the safety and wellbeing of co-workers.

Gulf Coast Outpatient Surgery Center (GCOSC) is a center Alliance Surgery manages and is located in Biloxi, Miss. Some of the casinos just a few miles away were completely destroyed and several hospitals in the vicinity were severely damaged. Although GCOSC sustained some flooding, Katrina spared us debilitating damage. While assessing the damage we predicted we would be doing cases in six to eight weeks. With incredible courage and dedication from the GCOSC team, our facility was performing cases within two weeks of the storm, well ahead of the initial forecast.

There is no disaster recovery plan that can execute itself. Without a dedicated and highly skilled team to execute the plan, it isn’t worth the paper it’s printed on. The GCOSC team was determined to overcome Katrina and provide an invaluable service to their community. To execute a plan, we must first have a plan.

Share this article: Email, Slashdot, Digg, Del.icio.us, Yahoo!MyWeb, Windows Live Favorites, Furl
Add this article feed to: RSS, My Yahoo, Newsgator, Bloglines