When Resiliency and Contingency is Critical to the Business

Commentary by
David Finn, Health IT Officer at Symantec
LinkedIn Profile

A lot of you know that I’ve been hanging around Health IT for over 20 years.  Some of you know that I’ve been involved in IT for nearly 30 years.  A few of you may know that well before I discovered programming and IT, I worked in the professional theatre.

What none of you know is that all I learned about resilient systems and business continuity and contingency planning I learned in the theatre.  Not in IT.  I say none of you knew that (until now) because I didn’t know it until last night. Well, you’ve either quit reading by now or are ready to have me explain. So, here goes . . .

Last night I went to the theatre.  It was 2 legendary Texas singer/songwriters – – you’d know them if I named names but I won’t.  Bare stage, acoustic guitars, minimal lighting, and 2 mics.  It was an historic theatrical venue, probably under 1,000 seats on 3 levels and any decent singer or actor could fill the place with no mic but I’m already looking around for extra lights, backup sound . . . My wife, whom I met when we worked in the theatre, reminds me that they really wouldn’t need that for this kind of show.  She had already done the risk analysis.

I was thinking back to other theatrical events.  I worked my way through undergraduate studies (in theatre) as a card-carrying stagehand in a roadhouse that presented touring shows, touring groups, regional and local rentals.  In show business, just like healthcare, it is all about the audience member.  Let’s call them the patient.  In healthcare, they are looking for care for a pain, injury or ideally just to protect their health.  In show business they want to be entertained – – another way being taken care of.

So, here was my introduction to business continuity:  A national Broadway tour of “Fiddler on the Roof” was coming to our roadhouse.  It was February in North Dakota and on this particular day that meant high winds and snow.  The cast had arrived on time but the trucks carrying the sets and costumes arrived at the theatre about the time the curtain was supposed to be going up.    The local presenter called the producer in NY who said  “Hell, yes, there will be a show.”  It took about 10 minutes to decide to do this:  Bring down the front curtain, have the principal (star) do his nightclub act on the stage in front of the curtain while we did the set up behind the curtain; we cut some of the scenic elements to get the set up down to about 2 hours.  The audience got a nightclub act; the actor got a “free” tryout. The show went on.  I’m sure much union overtime was paid but 3,000 “cultural patients” left the “hospital” healthy that night.  The company was so high on adrenaline that it was a stunning performance!  While no one locally thought of it, the NY producer knew exactly what to do and doing nothing was not an option.  Contingency planning!

Resiliency and high availability was a different story.  It was one of the first shows in that roadhouse and I was working the show as an electrician.  The show was The Four Seasons.  Yes, as in Frankie Valli and The . . .  This was before Jersey Boys . . . mid-70’s.  No sets but lots of costumes and lots and lots of sound equipment.  We unloaded what looked like a complete sound set up and I thought we were ready to go to sound checks when the road manager for The Four Seasons called us all over and said “Let’s finish unloading.”  We all went back to the loading dock and there was a complete, identical, fully redundant sound system (these mixing boards and set up probably cost ~$200,000 – – back then).  We unloaded it and set it up right next to the one we had just set up.  I had to ask: “Why?”  The answer is one I’ve not ever forgotten:  “We make our money making this sound; if we can’t make this sound we don’t make money.  If it’s a different sound, people won’t pay for it.  We can’t afford to miss one note, one word.”  Let me just say the answer made perfect sense to me then . . . and it still made sense 30 years later when I was asked to lead the selection and implementation of an EMR at the IDS where I was CIO.

Doing a nightclub act while the show was being set up?  A completely redundant sound system for The Four Seasons?  These were business decisions . . . the stage hands didn’t decide, the actors didn’t decide although they were all part of the decision and had to participate.  It took more money, it took more time and effort, but all the “patients” were assured of getting the show they paid for.  They got what they wanted, needed and expected.  No one said “the computer is down” or ”we can’t do that right now”.

Healthcare has to be able to do availability and continuity at least as well as a theatrical road house.  I can’t even think of what it means if we can’t.  Clinicians depend on the data to treat and care for patients and that does not happen on the planned schedules of IT or the CFO or the doctor.  It needs to happen no matter what – – “. . . we can’t afford to miss one note, one word.”

Remind me to tell you about giving Ray Charles’ his 40th birthday cake backstage in the Green Room . . .