Figure 1: Augmented reality refers to physical reality extended with virtual aspects and is a subset of "mixed reality," which covers any combination of virtual reality (complete simulation) and physical reality (no simulation).
Wherever software supports people at work, the question of the best human-machine interface (HMI) arises. In other words, how and where can the data the person needs be presented, so it can be understood, is easy to find and can be used in practice? This also applies in the daily work of automation engineering, and especially in the process industry, whether in the operation of the plant or in servicing and maintenance work. Can solutions from the field of mixed reality simplify users’ work in the future?
Since many people are not necessarily wholly familiar with the terms virtual, augmented and mixed reality, here are a few definitions. Virtual reality is understood as the complete simulation of a virtual reality. Augmented reality, on the other hand, extends physical reality by adding virtual aspects. Finally, the term “mixed reality” is used at present to describe all media that lie between the poles of physical and virtual reality; thus it's used as a general or umbrella term (Figure 1). The mixed reality spectrum is thus delimited on one side by physical, and on the other by virtual reality.
Typical application cases
Before discussing whether it's already technically possible to use mixed reality in the process industries today—and if so, how—the question arises of what the potential applications might be? Are there applications that might benefit from an interface of this kind, either now or in the near future? The answer is yes. There is a great potential in plant maintenance.
In the extensive plants of the process industries, it can prove difficult to even find the components that need to be maintained. In this situation, it could be envisaged that a mixed-reality solution would, so to speak, “navigate” an employee through the plant. Directly on the spot, it could then display the information on the components requiring maintenance, and provide access to the necessary documents. And while all this was going on, the technician would have both hands free to concentrate on the actual task.
There is also a potentially useful application at the control level. Instead of displaying the information about the distributed control system on big screens at a central point, it could be displayed on head-mounted devices (HMDs). Thanks to the associated freedom of mobility, it would then be possible to compare the apparent status of the plant with the measured values from individual devices onsite.
Figure 2: Plant Assist Manager software supports the user in performing and documenting process steps and workflows.
Another potential area of application is repetitive processes—monotonous but important tasks. Here, too, there is a use for mixed-reality solutions. When performing such activities, people tend to tire and make mistakes more easily. Appropriate mixed-reality concepts could help employees work more efficiently, carrying out the unvarying processes in an automated form and documenting all the steps.
There are certainly many other potential areas of application for mixed-reality solutions apart from the examples mentioned here; thus it's definitely justifiable to examine the question of technical feasibility.
Mixed reality: a use case
Rösberg Engineering GmbH in Karlsruhe, Baden, Germany, has taken up this challenge. The enterprise has specialized in the process industry and offers services in the modernization and building of process plants, as well as software solutions that support plant engineers and operators in their work throughout the plant lifecycle. As automation experts, we always focus on the question of what trends will influence the sector in the future, and address the aspect of how far these can realistically be implemented. For this reason, the company entrusted Stefan Stegmüller, a student from the Information Technology dual study course, with the task of investigating the present state of technology in this area for his bachelor thesis. Stegmüller also examined the practical use of a mixed-reality solution by designing an experiment that reproduced a real situation in a plant, with the aim of assessing the time and resources necessary to realize the solution and identifying potential difficulties.
The use case chosen for the research project was based on the Plant Assist Manager (PAM) developed by Rösberg's automation experts (Figure 2). This software supports users in performing and documenting workflows such as filling of tank trucks. The workflows are optimized, automatically documented and then displayed, for instance, on mobile devices. With the help of a specific checklist, the user is guided through work processes and has all relevant information available at a glance. For this application, an HMD would be ideal as the HMI. The bachelor thesis focused on testing the feasibility and usefulness of this approach.
Figure 3: Graduate student Stfan Stegmüller uses a tank filling system to investigate the practical use of the Microsoft HoloLens for process industries.
To do this, Stegmüller worked with an experimental construction that was a scaled-down version of a real tank filling plant (Figure 3). When filling tanks with chemicals, it's essential to avoid all mistakes, as they may be dangerous. Here, support is provided by the relevant checklists. The aim was to navigate the user through the filling process using a Microsoft HoloLens.
Practical benefits and challenges
The advantages of this solution are obvious: employees no longer have to switch to and fro between the task to be performed and the tablet with the required instructions. They have all the information directly “in view” while working. They can confirm completed actions by voice or gesture command, and have both hands free to work. Moreover, operating in this way is much more intuitive if the necessary work instructions are directly linked to the relevant plant component or the valve to be opened.
Major challenges were posed by the choice of the right tracking method, i.e. the question of how best to anchor the virtual content in the real environment. This is essential for the displayed information to be shown at the right place and time. Basically, the following methods are available: magnetic or infrared tracking; tracking with visible light; or a system using inertia (inertial measurement unit [IMU]). In tracking with visible light, a differentiation is additionally made between tracking with natural features, referenced tracking or model-based tracking.
Figure 4: Problems with the non-specificity of QR codes for indexing real and virtual information were overcome by tracking artificial features marked with triangles of random size, sequence and rotation.
Stegmüller reports: “In the case I am concerned with, I mainly considered the possibilities of tracking with natural features, meaning prominent features of perceived natural images, and tracking with reference marks such as QR codes. The final choice fell at first on QR codes, and this was subsequently modified to markings in the form of images with triangles of random size, sequence and rotation (Figure 4). This variant proved to be advantageous as these perceived images can be uniquely identified by the algorithm used. When using QR codes as anchor points for virtual content, mistaken identities can occur during evaluation, due to their basic structural similarity.”
Seamless interaction
Another big challenge in implementing the system was the interplay between individual components, such as PLCs, databases, software and the HoloLens. What components are compatible, and where do interfaces still need to be developed? The concepts initially developed for the system had to be rethought and adapted several times. During the research project, it became clear that the use of a mixed reality solution in industrial practice is definitely realistic. Of course, the input in terms of resources has to be weighed against the benefits in each case. Stegmüller sums up: “In the next few years, a lot will be happening in this field, and I can certainly envisage mixed-reality headsets like the HoloLens opening up practical applications in the process industry in the foreseeable future. Practical tests have shown that this HMI and tools such as Plant Assist Manager ideally complement one other. We look forward to the first applications in industrial practice.”
Evelyn Landgraf, marketing, Rösberg Engineering GmbH, can be reached at [email protected].
