The most popular rendition of Murphy's Law is, "What can go wrong will, and at the worst possible time…" In today's automation world, we are building ever more complicated automation and management systems, designed to eek the last bit of quality and production performance our of our processes. We are creating some fertile ground for the production of future Murphy's Law crops. Minimizing these risks, from all perspectives – Automation Vendor, System Integrator, and End User, is essential to create solutions that will degrade gracefully to minimize downtime.
Why am I writing about this now? Two Words – Hard Drive. As Baz Luhrmann once said in a commencement speech turned into "Sun Tan Song," "The real troubles in your life are apt to be things that never crossed your worried mind; the kind that blindside you at 4pm on some idle Tuesday." OK, in my case it was 3:15 on a Monday and I had to do a hard boot. That was it – "Drive not recognized." New Drive in hand, some software upgrades at the same time, backups that were out of date and a day later of loading, copying, recovering and I was 90% whole again, (not what the plant manager would want to hear, right?). A few shortcuts to get back on line quickly – Antivirus software can wait, documentation of the new system setup can wait, I won't forget to update that temporary license I got from my software vendor, to get back up and running… By now you're likely shaking your head saying Yup, I've been there… Oh, but it gets better. The next morning I wake up to a message saying there is a problem with the Operating System, and the system can't boot. Recover with a Boot Disk, scan the drive and there are bad clusters. The Bathtub Curve still exists…
The bright side – Some aspects of my technical life continued to work. Webmail gave me access to email. I could also go to another machine that had parallel email access thanks to IMAP rather than POP technology. I happen to use a combination of Web Based "Hosted or Cloud Based" solutions. They keep working. This reminded me of the term "Graceful Degradation," when systems can encounter failures yet will continue to deliver value.
How do you design for Graceful Degradation? And, how do you design for a quick recovery?
Let's explore these issues from the three perspectives outlined earlier – Automation Product Vendors, System Integrators and the End User perspective.
Product Vendors should design products with Graceful Degradation in mind. Products should be modular, to enable flexibility to meet the task, but if a Module fails, the entire product shouldn't stop delivering value. For example, if security authentication is not available as a service, can you still authenticate locally? What's the backup? Licensing of technology. Will licensing survive Disk Mirroring? If not, are your procedures for update documented and readily available? Does it require a product expert? Are your application files neatly stored in one place? Do you rely on Registry settings or is the entire configuration stored within a set of application files? Are the resources of your product well documented in terms of Ports or Operating System Services that are needed? Focus on delivering solutions and not toolsets. In today's world, we need to deliver fish, not deliver tools and teach people to fish. There is no time for learning and becoming product experts today, and the variability in the resulting applications extends development and troubleshooting times, and complicates documentation.
Consider following an "Appliance Paradigm." There is a range of new "Purpose Built," commercial of the shelf (COTS) products that are applied rather than engineered for an application. Examples include Enterprise Transaction Modules, Panel HMIs, and the Universal Protocol Gateway. Appliances are quickly replaced with an off the shelf spare and their program is updated through Flash Memory cards or some simple configuration settings.
System Integrators need to resist their tendency to engineer everything. In the past, when vendors provided toolkits, there was a heavy reliance on system integrators that have the ability to amortize a development tool learning curve over many projects. Integrators would then develop end user applications efficiently. That was a time when end users had engineers on staff that would be the process expert, guiding the developments of their integration partner. Today, with rightsizing in all industries, the process expertise lays in the heads of both integration partners and a fewer number of process engineers. The System Integrator should really be changing their focus from engineering custom solutions, to configuring purpose built products to solve problems. By engineering, I mean software development and scripting – using development tools rather than purpose built solutions. Development Tools require high levels of QA and Documentation, items that can quickly eat into the profits of a job. The IEEE Computer Society statistics show that development costs make up 40% of overall software costs. 60% is taken up with QA, bug fixing and enhancement. QA really needs to be separate from development in order objectively test an application and it takes as long to QA a development as it does to develop in the first place. World class companies have ratios of 1:1 or higher, QA to Developers. Going forward, System Integrators will be delivering more value to customers by delivering process expertise, delivering value in product selection, chosing Purpose Built products to solve a customer problem. It is the Process Expertise that is the new value of an Integrator, not the expertise in a specific vendor tool. Vendors creating purpose built "Appliance" solutions, which benefit from high deployment volumes, and unmatched levels of QA, situational testing, and both user and application documentation.