Perhaps the most advanced automation is found in aircraft, both military and commercial. People in that field talk about situational awareness and opaque automation when discussing controlled flight into terrain. Bad things happen when pilots either are unaware of what the automation is doing or are aware and can't understand what is going on, distracting them from the real situation.
Developing Machine Intelligence
The faults of human operators could be overcome by computers that can learn from teachers and create novel solutions from known or deduced properties of the process and its equipment. While anyone who has operated a computer would find this highly unlikely, there are some amazing developments in neuromorphic processors that could make that level of computing a reality in a decade or so.
The prospects for truly intelligent computers that can learn from each other raise the possibility of machines that quickly become far more intelligent than we are. This could go badly if we have given them control of our manufacturing and utility processes, not to mention weapons. See Our Final Invention by James Barrat.
Machines Need People
Machines require maintenance performed by people carrying tool boxes and spare parts. Automating the maintenance function will require dexterous mobile machines with true intelligence. The computers that replace operators can be as large as necessary, but maintenance robots need something like Asimov's fictional positronic brain. The nearest thing we have to that is quantum computing, where things are still going slowly. I/O functions are proving to be particularly difficult.
It appears that we can't design factories that don't need humans for the foreseeable future. We need to design automation to work with people for the time being. Discrete process machines, PID controllers and the like work quite well with people. The main problem is the programming of procedures.
Normal operation is straightforward to write as a procedure. Process procedures may have 10 times as many lines of code to deal with the possible exceptions than are needed for normal operation. Accidents happen when there was no plan for what to do when the first incident occurs in a chain that becomes a disaster.
The Future: Human Language Programming
One way to allow an operator to know what the automation is doing is to write procedures that an operator can read. Combined with some indication of the part of the procedure that is being executed now, an operator can tell if the program is actually happening and what might happen next.
This will require an interpreter in the computer that can turn the written procedure into commands to equipment. This is not difficult—BASIC is an interpreter. However, BASIC is very computer-centric and mathematical compared to what is needed to express a readable procedure. The users need to be able to define the vocabulary that the interpreter will use. See my book Automating Manufacturing Operations for a complete discussion of this subject.
Success depends on a structure described in ISA 88 where product procedures command equipment modules. The modules are designed to provide process functions that are independent of the product being made. Examples are CNC milling machines, pick-and-place robots, reactor jacket temperature control and distillation columns.
The work to create the interpreter will be simpler if a group of vendors and users can agree on a user requirements specification. ISA standards committees have done this kind of work in the past, but sometimes industry organizations are faster. Perhaps the work could be completed before opaque automation causes a serious accident, as management continues to reduce the number of operators for a process.