Suggested by a conversation with John Mayberry – Emmaco

Engineering history is replete with decisions that, upon any kind of reasonable reflection, would prove at best questionable or at worst disastrous — for example, the rapid and urgent deployment of the legendary Boeing designed, engineered and constructed B-29 Superfortress in May of 1944 during WW II. Like many things that are rushed into use or deployed before they are thoroughly tested, the B-29’s engines caused massive and deadly failures until all the problems that should have been solved before use were finally corrected (which did not occur until after WW2 was over and the fully tested and revised engines were used in the now iconic Douglas DC-7).

The story goes like this:

It’s May 1944 and the U.S. Army Air Force (USAAF) is battling with the U.S.Navy over how to bring the war to a close and trying to convince the President to choose their option. The USAAF plan was based on a controversial strategy devised by now legendary cigar-chomping Major (later General and creator of SAC) Curtis LeMay. LeMay had designed and eventually implemented a controversial systematic carpet bombing campaign of Japan. His weapon of choice — the single most expensive weapons project undertaken by the United States in World War II, exceeding the cost of the Manhattan Project by between 1 and 1.7 billion 1940s dollars, was the B-29 Stratofortress, a plane that could finally reach targets in Japan from recently captured Pacific island bases.

The problem was that the plane was never tested in the high temperatures of the tropical airfields where the B-29s were to be based. Added to that, the early B-29 tactics of maximum weights, when combined with operating conditions produced overheating problems in the brand new Wright Duplex Cyclone’s R3350 piston engines
that were never completely solved until replaced with a competitor’s engine. Sadly, the R-3350 engines had an unfortunate tendency to swallow their own valves, catch fire and demolish the engine (often on take -off when they were under maximum stress). Because of a high magnesium content in the potentially combustible crankcase alloy, the resulting engine fires — sometimes burning with a core temperature approaching 5,600 °F (3,100 °C) fed by the burning crankcase alloys — were often so intense the main spar could burn through in seconds, resulting in catastrophic wing failure. What eventually occurred is shown in this photo from sometime in 1945, wherein hundreds of failed engines are just lying in scrapheaps to be salvaged for whatever useful parts they might have.

Doomed to fail!

This cautionary tale relates to our industry because we too often deploy products and install designs which are shall we say politely — unproven. Now to be fair, that is not always the integrator, designer or engineering folks’ fault. Manufacturers bear blame as well for pushing products out well before validating and verifying their real world performance in field operation and functionality — essentially leaving the beta testing to the users — (any large northwestern U.S. software company come to mind?). This would be the collateral damage caused by making an assumption of truthfulness or validation which simply was not viable.

There are numerous stories not all that dissimilar to the B29 debacle in the audio/video industry. They continue to arise far too regularly. I’m sure you can name companies who had and may still have a ‘standard’ practice’ of sending product out the door they knew didn’t work as described or at all.

This practice is often blamed (there is always a scapegoat somewhere) on some ill-defined ‘management’ decision that arrived in the form of a dictum from on high to meet promised production and billing milestones. Thus, product was shipped under the assumption it would be fixed in the field (more collateral damage?). This is chiefly found when the relationship between manufacturer and integrator(s) was less than professional. I don’t know about you but in the last 30+ years I certainly can recall more than one project where large numbers of amplifiers, or digital audio processors, or control hardware/software left the factory untested and were simply doomed to failure but not in the company’s hands.

BUT… that’s not the whole story folks!

Now let’s add in the other half of the equation. The often-ignored truth is that the quality of some of our work is awful. Inescapably it usually boils down to poor education and bad management and the aforementioned questionable products. Think about it — do you need more than 10 fingers to count the number of times have you walked into a room with a dead monitor, unintelligible speech or push buttons that don’t work? I certainly do, and I bet you do as well.

I think it’s well past time for the industry’s integrators, designers, consultants and specifiers to rise up and take a firm stand! We must insist — nay, demand — that every product (hardware or software) is fully tested and documented prior to leaving the factory. I don’t recall seeing that statement /requirement on very many if any purchase orders, design specifications or dealer/reseller agreements but it should be automatic boilerplate, just like payment terms. I don’t think that anyone should ever again sign any document that “holds harmless” any supplier or manufacturer, and puts us squarely in the crosshairs. Start shredding your old paperwork right now!

If we don’t fix this, the recent market analysis (I will not mention who is responsible to protect the guilty) that purported to show that 54 percent of their respondents assert that a lack of systems integration was a big challenge for marketers this year might well prove to be real data.

When shielded (no names used in the reporting) interviews show, repeatedly, that well over 50 percent (closer to 65 percent) of our own people selling our work to a client think their own firm is a major impediment to their own success we have a crisis to handle, and right now. You many argue the point, but ultimately it’s really hard to use real world feedback from actual end-users to dispute this claim (AV — the “stuff that doesn’t work” mentality).

OK, so Houston, we have a problem — what needs to be done?

The AV/IT/IP industry needs to mature — now! Let’s start with something simple and doable with off-the-shelf solutions. All of our components should automatically report their failures and real time performance back to industry standard ERP (Enterprise Resource Planning) compatible management products like SAP and Microsoft Dynamics.  Via their computer dashboards, our end customers should be able to encapsulate the overall system performance of our hardware and software in seconds. It’s not fantasy land, aircraft, telecom systems in buildings, and tons of other systems already do this every day, 24/7/365.

Don’t think it will work? Let me point out that our security brethren have provided many of these capabilities for years. They have no problem remotely dialing up a CCTV camera in Poughkeepsie and watching from their desk in Bismarck, Berlin or Baku.

Unfortunately, the same is not true for a large amount of the audio/video world. A school principal is not able to tell from his desk that the system is ready for the annual holiday pageant. Exceptionally few systems even report failures back to the integrator, let alone total system performance to the end client in real-time.

How many of you provide ERP data to your clients? My guess is a few percent at best, but I truly hope I am wrong and someone has already blazed the path — if you have, please comment on this column and let me know. Virtually every principal financial, healthcare infrastructure and security company uses ERP dashboards now to manage their other disciplines or to manage their cash flow, measure and monitor their development, and maximize the efficiencies of their resources. Imagine if our systems by design reported what messages were sent and the quality of them as reproduced in the actual space!

An AV dashboard indicating system up-time and failures (particularly across an enterprise network) tied to standard management tools would prove informative, useful and not the least hugely enlightening.

Our systems need to account for imperfect processes, people, technologies, designs or specifications do not actually exist. Remember, Murphy was an optimist and it is without doubt that any mission critical imperfections will show up, at the most inconvenient point in time.

We have to accept that our buyers, customers and their employees are self-governing; will act under the influence of internal and external forces we can’t possibly imagine; have at best a massively limited knowledge of our systems and no concept of their impact on our carefully implemented designs; and often do the unanticipated or unexpected. It’s the 21st century update to the old adage about idiot-proofing: “The problem with idiot proofing anything is that they keep making smarter idiots.”

To quote Mr. Mayberry, “It’s time to put on the big boy and girl pants” and make our systems work properly for years to come.