The most often ignored or forgotten problem in AV/IT systems can be summed up in this quote from Benjamin Franklin: “By failing to prepare you are preparing to fail.”

Technically speaking, a failure is declared when the system does not meet its desired objectives. Therefore we can consider any system that cannot meet minimum performance or availability requirements to be “failed.”

As AV systems blend with ever more IT technology, many of the IT acronyms have started to be used to describe various kinds of “failure” or similar problems. To clarify this, let’s examine what those common acronyms really mean.

FAILURE METRICS (derived from Wikipedia online public domain content)

• Mean Time to Failure or MTTF is the average length of time that a system is online between outages or failures. This metric showcases the fact that it is important to think of reliability in statistical terms.
• Mean Time to Repair or MTTR is the is the averaged repair metric and represents a point on the curve of the amount of time required to repair/re-boot a system and bring it back online.
• Mean Time Between Failures or MTBF is the most common failure related metric. It is also the one most commonly used incorrectly. “Mean time between failures” or “MTBF” refers to the average amount of time that elapses between one failure and the next. Mathematically, this should be equivalent to the sum of MTTF and MTTR and represent the total time required for a device to fail and that failure to be repaired. (Please note that manufacturers don’t usually specify or discuss MTTR/MTTF and instead use MTBF as a shorthand for average failure rate over time, albeit incorrectly.)

While the screen above is not quite the infamous “blue screen of death,” it is indicative of the kind of ‘unknown cause’ failure that can occur in today’s complex integrated and interlocked AV/IT systems. If it were a medical condition it would be termed idiopathic or “without apparent cause.”

For the end user, this is THE most frustrating and panic-inducing kind of problem any system can encounter because the cause is not obvious and the solution is not immediately evident. If the system could talk it would say:

The point is — systems will fail, and how that is resolved and or mitigated is the key to keep the end-user both comfortable and sane.

Tiny, Tiny, Tiny

Let’s look at a recent “failure” and how technology both enabled it in the first place and common sense fixed it.

The story begins, according to a colleague, when the tech team at a large house of worship recommended to the worship team that they consider replacing the 10-year-old hand-held wireless system with a new UHL system using head-worn mics.

The rationale behind the recommendation was two-fold:

1. The current system used RF frequencies that were going to become unavailable within a year due to the FCC re-allocation program.
2. The existing system was suffering from both age- and quality-related issues because the technology employed was essentially obsolete.

A further issue of a lack of available replacement parts to keep the old system running was becoming ever more serious.

So after some considerable debate, funding discussions and several round-robin sessions with various new system options it was decided to go with a product that incorporated extremely small lightweight head worn boom mics and current multi-frequency UHF technology allowing for both expansion and flexibility in deployment.

Small Is Good and Bad

Here is where the failure process began. The new mics are so small that they are almost invisible. So the worship team members were having difficulty in learning how to position them and keep them in position — there was no obvious microphone element to position and the monochromatic color scheme of the headgear made it very hard for the users to know where their mic actually was. This led to many instances of poor pickup and off-axis positioning. As you might suspect the complaints came rapidly and loudly.

For all intents and purposes the new costly UHF systems that were supposed to solve all the existing poor quality and reliability problems of the old system had failed. In fact, they had created a whole new set of problems and in the opinion of the worship team, the whole exercise was viewed as both a failure and a bad reflection upon the judgment of the tech team.

Mitigating Failure

This was clearly not a good situation and it was one that required both diplomacy with and careful re-education of the worship team members. Finding a way to get them to accept and adapt to the new hardware and learn how to properly use it and work with it meant spending time planning out some rehearsal schedules, working with each worship team member on how to use their new mics and most importantly — finding a way to remove the sense of failure from the mindset of the whole worship team and garner acceptance for the new system.

First and foremost was the need to train each team member on how to deploy the headgear and position the microphone element for proper pickup and minimum noise. Since these mics were hard to visually identify, a clever solution was found: Placing a tiny red dot of nail polish on the inside (invisible to the audience) of the 3/8 inch long mic element at the tip of the headgear boom to aid the users in positioning them appropriately for that specific user’s delivery style, voice and physique. So a $3 bottle of bright red nail polish provided the biggest single element of the solution to proper use and positioning.

The Proof is in the Performance

The whole tech team was hanging by its proverbial fingernails on the first Sunday after all the rehearsing and training to see if their solution would provide what the worship team wanted and expected.

To everyone’s great relief, the system functioned relatively flawlessly — not perfectly, but a far better performance than it had just a week earlier.

After services concluded, a post-event meeting with the worship team’s members provided useful additional fine-tuning feedback and allowed the tech team to work out the remaining minor bugs and kinks with the system so that within a week or two everyone had forgotten the failure and was comfortable with the new gear.

From imminent failure to functional usability, accomplished by thinking not about the technology or assigning blame or fault but by looking for a cause and finding a simple, logical and immediate solution that worked.

Failure does not always have to be an option, but it will always be a problem. However, if you plan for the possibility and prepare to solve the issue(s), it can be turned into a success.