Make a Disaster Recovery Plan

it disaster recovery

Last month I wrote about cybersecurity in schools and what steps we can take to prevent attacks from happening. I wrote in the blog that there are many things we can do to harden our security and to make our institutions less of a soft target. Yet, as I pointed out in that piece, chances are that one way or another we will all get hit by some type of attack. This month, I am writing about preparing in advance for what to do after an attack happens. In IT lingo, have a disaster recovery plan.

So, let’s assume you have done everything possible to harden your systems, and yet, you still got hit by an attack. What have you done in advance to lessen the overall impact of this attack, and to lessen the downtime? Most of the following suggestions can be watered down to backup, practice, backup more and practice again.

When we think of the AV industry, in particular, the first step is to think about what needs to be backed up. I recommend thinking about the following:

  • Design
  • Plans
  • Programming (including touch panels, control panels, DSPs, etc),
  • Wiring designs
  • IP addresses
  • MAC addresses
  • Anything else that you would need in order to get a system back to functioning status.

Essentially you want to back up anything that you keep any records or information about.

An essential piece of a strong disaster recovery plan is to test that plan. My suggestion here will actually sound pretty crazy, but I think it is a worthwhile exercise because it will help you strengthen your plan. Perform a test where you assume that a classroom has been compromised. The attackers have bricked every single device in your system. What would you do? To start this exercise, get an inventory of every piece of equipment in your room (and I mean literally every piece). Then figure out which devices are on the network, regardless of whether you think those pieces are at risk or not. Now, from that list, determine anything that takes any type of setup, configuration or programming. This could be as simple as a dial on the front of an amp, or as complicated as a programmed DSP or control system.

Why look at the dials on an amp, if the attack is coming over the network? I suggest this for two reasons. The first is that while it is more likely an attack comes from the outside over the network, you can not rule out a physical in-person type of interruption. While this may or may not be considered a “cyberattack,” it could still have the same effect, classrooms out of order for some extended period of time. After you have completed those steps, have another member of your team take each piece of that equipment and reset it to factory settings. I suggest another team member so that you don’t get re-familiarized with the system. Now, go ahead and put it back in working order. Before you get started though, do all of this work with a brand new computer, don’t use your regular computer (you have to assume that was lost in the attack). After you have your computer set back up with the necessary software, re-loading configurations should not take you more than 15-30 minutes. Likely, the first time you perform this exercise it will take you longer than that, so I don’t suggest you do this in a space that is going to be used imminently.

What you will likely find out is that you forgot certain aspects, like the calibration of interactive touch screens, whether some devices had IP addresses hard-coded as opposed to dynamic, etc. Depending on how you configured some systems you may have security locks on them that don’t get erased by a factory reset. Do you remember those codes? Performing this type of disaster recovery exercise will give you a clear understanding of everything you did not think of when you thought you had documented and backed up properly. It will also give you information on how long this will take. During a real recovery, that is going to be a question you will need to answer — how long before you can have rooms up and running?

Now let’s think about where and how you store these backups. You have to make a few assumptions as you back up your data. The first assumption is that the place you put the backup data will be compromised, and the second assumption is that the backup data was compromised before you put it in place. These two issues feed into each other. First, you want to back up all your data in an organized manner, including any instructions or reminders of the restore process. Then you want to consider backing up to multiple locations. Google Drive, Dropbox, etc. are all good sources of backup. These companies are very good about backing up and protecting their data. Important to remember, however, is that you should treat these backups as cold storage. That is, once you have stored a set of backups on any system, you disconnect from that system and only reconnect for recovery. This is so that you don’t accidentally corrupt your backups. The second assumption is that your data is at risk when you backed it up. Many attacks are successful because the data was already carrying the attack when it was backed up. Therefore, when an organization wipes the attacked system and restores it from a backup, the problem is not resolved because it simply replicates all over again. This is why you should consider a system where you put something in “cold” storage and don’t keep copying more files into it, so you don’t introduce the attack into your backups.

Another idea for doing this is to talk with your IT department. See if they have any old laptops that they are going to retire. The laptops should have a solid-state drive. Configure the laptop with everything you need to do a full restore, including software, programming and configurations. Then, set this in a fireproof location. Do this every six months and keep the laptops for two years. This is an inexpensive but solid method of backing up and protecting this data.

The end of summer is an extremely busy time in higher education. But — this blog is timely because it is also a time when you have done lots of work over the summer and may not have it all properly backed up. Giving consideration to the suggestions in this blog will help you be the hero if (when) your school ever gets hit by an attack. As always, I love feedback on these blogs. What tricks and tips do you have for others to develop a disaster recovery plan? Have you developed one that differs? Catch up with me on Twitter or LinkedIn and let me know your thoughts.