When Everything Goes Wrong

hope-wrong-0916It was my husband’s birthday, and the only thing standing between me and our birthday date was a quick service call. Update some network settings, reboot the equipment so they’d take effect, check to make sure everything was online. Easy peasy, right?

Yeah, I was like the AV equivalent of the old cop sitting in a diner during the opening scene of a buddy cop movie. Hey guys, I’m two days away from retirement, who wants to see pictures of my new boat?! The door opens, the gang we’ve been hunting down walks in, everything goes wrong.

My equipment didn’t come back online. In fact, it didn’t come back on at all. I couldn’t ping it, I couldn’t connect to it over USB. The status lights were all dark and sad. It was deader than that cop’s retirement plans. And it was mission critical. I had just over three hours until our movie reservations. What was I going to do?

First Things First, Engage Tech Support

As soon as I realized that something was wrong, I opened up a chat session with the manufacturer. I also reached out to one of my contacts to see if I could get escalated to a higher tier. I had someone on the line pretty quickly, with an offer to get me escalated if necessary. Tech support wasn’t able to get my equipment recovered, but having someone on the other line who knew exactly what to try gave me the confidence to pull the rip cord when I knew we were never going to get it back up and running.

When In Doubt, Phone a Friend

I pride myself in my ability to think on my feet. When things get tough, I take a deep breath and dive right in. But this was an extreme situation. I didn’t have the luxury of figuring things out in my own time. I needed to get everything back up and running, and I needed to do it fast. So I called my boss. He told me to keep working on the recovery process, but he was going to have someone drive over with a spare piece of equipment. It wasn’t exactly the same part, so he said to use that time to prep everything for a change-over. With a plan in place, I was able to get to work. Having a timeline gave me a chance to take those deep breaths and focus on making those changes.

You Never Known When You Might Need Those Tools

I’m a programmer. I don’t pull cables. I don’t install equipment. I don’t touch our racks, unless it’s to plug in a network or USB cable. I still carry a full tool kit in my car. You never know when you’re going to need that screwdriver and those snips. I was able to get my broken equipment out of the rack and the new part installed very quickly. I even installed a new pair of rack ears. All those years working service sure paid off. Even if your job doesn’t technically involve touching the equipment, everyone who might need to work in a rack should know how to install rack ears, rack a heavy box and move cabling around.

Understand How the Entire System Works

I didn’t have six hours to do a full commission of the large system (and let’s be honest, the client probably wanted me out of their house). But I knew how all of the different components worked together. I knew what my failure points would be. In short, I knew exactly what to test to make sure everything was back up and running. Without pressing every button, I could say with confidence that our replacement was working. I also knew my client, which means I knew exactly what to test first.

In the end, I left my client with a working system, an RMA in place for their equipment and the knowledge that my boss and I were able to work quickly and decisively to make sure they had no interruption of service. And I even made it home before the babysitter showed up!

It’s scary and stressful when things go wrong. But it’s always a good feeling to know that you have the tools and resources to make things right.