article_522_fig2
article_522_fig2
article_522_fig2
article_522_fig2
article_522_fig2

Capturing and implementing undocumented knowledge

June 10, 2002
Electric Power Research Institute (EPRI) takes on the problems presented by the loss of experienced workers.
By Madeleine Gross, Lewis Hanes, and Thomas Ayres

SOCIETAL, demographic, and market forces have brought recent and ongoing changes to the workforce in the U.S. energy industry. The loss of highly knowledgeable managers and workers represents a growing problem in the energy enterprise, a situation also found in other industries and many government organizations.

Workforces are aging, expert knowledge and skills are being lost, and qualified replacements are increasingly difficult to find and retain. The loss or departure of undocumented knowledge associated with this situation can be particularly problematic with complex systems that have been in place for many years. Younger incoming staff are unlikely to have had education, training, or experience that is directly applicable to older technical systems. The unavailability of valuable knowledge, and, more specifically, tacit (or undocumented) knowledge (See Table I below), can have negative operational, environmental, safety, and economic consequences, since such knowledge is unique, known to one or a very few individuals or teams, and not available to others through procedures or normal training.

The Electric Power Research Institute (EPRI) Strategic Human Performance Program undertook during 1999-2001 a multi-faceted research project, “Capturing Undocumented Worker-Job-Knowledge.” The goals were to assess the problems related to this potential loss of tacit knowledge, to determine and assess possible approaches to deal with this loss, and to develop practical guidelines for use in energy industry settings. The main phases of the project work are described below, along with some conclusions and recommendations.

Current Practices
Initial interviews with energy industry personnel, as well as with persons from government, military, and other industry settings, confirmed the criticality of undocumented knowledge problems. They also demonstrated how knowledge loss is interwoven with broader issues of knowledge management and the aging of the workforce. Although many electric utilities have efforts in place to capture and store valuable knowledge before key employees leave, typically these activities involve capturing information in procedures or training programs and through job rotation. Mentoring, also a common industry approach, has been reduced by the pressure to minimize the number of employees.

The ultimate objective of this EPRI project, therefore, was defined as the delivery of a practical guidance document, including methods, techniques, and types of tools to consider for each step of the process (a process that would take advantage of site programs already in place), with the following aims:

  1. Identify managers and workers who possess potentially valuable undocumented knowledge.
  2. Evaluate the knowledge and determine if it is worth capturing.
  3. Elicit and store the valuable knowledge.
  4. Retrieve and present this knowledge to other personnel when needed.

A survey was conducted of management and supervisory staff from diverse utility settings. One finding was that nearly all respondents (92%) believed that loss of unique, valuable expertise would pose a problem within the next five years. But only 30% indicated a planning effort was in place to retain knowledge from experienced personnel and make it accessible and usable by new or replacement workers. Rather, methods frequently mentioned for dealing with departing personnel included: hiring outside contractors or consultants; searching for suitable replacements; using raises, promotions, or other incentives to induce departing workers to remain; and hiring employees back after they retire.

The EPRI project also involved a literature and background review, encompassing the areas of knowledge management, cognitive psychology, applied psychology, and artificial intelligence/expert systems. Based on the survey and literature review, a prototype process, suitable for field testing and for step-by-step elaboration based on such testing, was developed for capturing valuable undocumented knowledge.

The process provided guidance for identifying managers and workers who possess valuable undocumented knowledge and evaluating if such knowledge is worth capturing; and selecting and applying methods, techniques, and tools for eliciting, storing, retrieving, and presenting the valuable knowledge to other personnel when needed.

Testing the Prototype Process
Several approaches were selected for utility site testing. A number of tools and methods were considered for field evaluation. Examples include:

  • Applied cognitive task analysis, including a specific knowledge audit approach--expertise is elicited, analyzed, and represented by trained elicitors.
  • Critical incident and critical decision methods analysis.
  • Lessons-learned documentation.
  • Observation and encouragement of think-aloud protocols during actual work, simulation, or reconstructed scenarios (with collection of results by, for example, digital video recording).

Also explored were possible approaches for knowledge storage, retrieval, and/or presentation, including:

  • Knowledge repositories.
  • Concept maps--Perceived regularities in events or objects are defined as concepts and labeled. Maps (See Figure 1 below) represent their relationships with words describing the nature of the relationships. Concepts may have resources associated with them such as images, sounds, web pages, or other concept maps. Concept maps also can be used for knowledge elicitation.
  • Communities of practice--A group of people whose work centers on a particular subject, who have similar job activities, and/or who work for a common goal, with agreement on a knowledge domain, willingness to share, and mechanisms such as a web portal (See Figure 2 below) for sharing.
FIGURE 1: CONCEPT MAP
Useful for knowledge elicitation as well as storage, concept maps show the relationships among perceived regularities in events or objects (concepts). This high-level map is for decontamination in a nuclear power generation facility.


OVER A three-year period we performed background work, development of a prototype process, and site testing to support further refinement and elaboration of the guidelines. The guidelines consisted of both a process to follow and the knowledge elicitation methods. During the final stages of this project, the process and selected methods were implemented and tested at four utility sites with 20 workers/teams representing a range of organizations and work types.

The final guidance report was developed incorporating enhancements of the prototype process and selected methods based on these field test results. Some of the many important lessons that were learned during the development of the final guidelines are discussed below.

FIGURE 2: PUTTING IT INTO PRACTICE
A Community of Practice is a group of people whose work centers on a particular subject, who have similar job activities, and/or who work for a common goal, with agreement on a knowledge domain, willingness to share, and mechanisms for sharing among themselves and outside the community. Source: Price & Mynett, IHE Delft


Identify the Experts
An initial activity is to identify key employees who may be leaving their current jobs for whatever reasons, or may have knowledge so valuable that it should be available to others when they are absent due to travel, vacation, illness, etc. Methods to identify these key employees may range from simply asking managers to identify key employees to periodic corporate-wide workforce surveys. 
Some considerations include:

  • Individuals who are recognized by his/her or other managers and peers as being the only expert about something of high importance, or one of only a few local site experts.
  • Individuals with expertise in handling rare or infrequent events (e.g., repair of a unit that fails on average once every 10 years, or handling extensive repairs necessitated by a hurricane in areas not normally experiencing hurricanes).
  • Individuals with expertise for systems, etc., that are going to be replaced with different technology involving different skills (e.g., "old" computer system being replaced about the same time the expert on that system retires)--these skills need not be maintained.

Following identification of the experts, it is important to determine if these workers are willing to permit their valuable tacit knowledge to be elicited and made available to others.

Many workers are willing--and in some cases, eager--to share their knowledge, and for a variety of reasons. A worker may view it as an honor to be recognized as an expert. Others may feel an obligation to share their valuable knowledge with others because of the benefits received during their careers, or because it is the right thing to do. Others may participate because their manager has asked them and made time available. It is simply part of the job.  It has been found, however, that some workers (about 10%) are not willing to share their expertise for a variety of reasons. Some of the reasons include:

  • Knowledge is viewed as an individual's intellectual property, and may be used by that person as a basis for consulting work or another job.
  • Fear of layoff because of the perception that the unique knowledge provides job protection, and making it available to others may increase vulnerability.
  • Alienation against the company for some real or imagined reason (e.g., lower than expected salary increase or being passed over for promotion).
  • Belief that he/she does not possess any valuable knowledge, even though the person has been selected as an expert.
  • Expectation that elicited knowledge will “go into a file cabinet and never be seen again,” thus wasting the time of the expert (may be based on previous experience at company).
  • Current work assignments leave no time available to participate in knowledge elicitation.
  • Fear of loss of status because he/she no longer will be recognized as the expert in the organization.

Develop a Plan
A plan should be prepared to elicit, store, and retrieve valuable undocumented knowledge from each person selected. The plan should identify the specific knowledge elicitation methods selected for each expert or group of experts with similar skills, define the methods for storage, and describe how the stored knowledge will be retrieved.

Development of this plan will require consideration of a number of factors, such as types of knowledge, availability of the departing expert, and capabilities and resources of the personnel responsible for knowledge elicitation.

Most utilities already have programs to capture and disseminate expert-worker knowledge. For example, most companies have training groups and programs, procedure groups, human resources organizations, etc., that routinely identify, collect, and disseminate important information. In addition, some companies have effective mentoring, apprentice, job rotation, and cross-training programs.

To the extent feasible, existing resources and infrastructure should be used to collect and disseminate valuable undocumented knowledge. Thus, time and costs to initiate a new program may be minimized. In fact, in many organizations, a very important step will be to assign an existing department, group, or individual with the responsibility for any expanded undocumented knowledge capture efforts.

The knowledge elicitor should be at least somewhat familiar with the domain. The elicitor(s) may have the required familiarity through previous experience, or he/she may be given time to be bootstrapped into the domain prior to knowledge elicitation. There are several reasons why such domain knowledge is necessary. Most importantly, it permits the knowledge elicitor to understand specialized domain terminology, be able to ask intelligent questions, and have some recognition of the specific areas to probe further to obtain the needed knowledge.

Most knowledge elicitors need guidance regarding the valuable domain knowledge to be captured. This may not be necessary if the elicitor is extremely familiar with the domain and the knowledge that needs to be captured. Without this depth of knowledge, however, he/she may need to rely on someone else for direction.
 
In many cases, the expert has a wide range of expertise, some of which is unique and some that is also known by others. It should be noted, however, that some workers who are identified as experts may say, “I don't have any valuable knowledge; other people know what I know.” It is not uncommon for an expert not to realize he/she has valuable knowledge not known by anyone else.

Experts are usually extremely busy because they are the ones assigned the most demanding and difficult tasks, and may be consulted by others needing access to their unique knowledge. It may be difficult for the knowledge elicitor to have much time with the expert. Therefore, the time must be used wisely to capture the knowledge that is most valuable and not available to others.

In any event, the expert's manager or other people familiar with the situation should be queried regarding the specific knowledge to elicit. They will have an understanding of the knowledge areas that are important. They will be able to identify the most valuable and needed information that should be collected and subsequently made available to others.

Select Elicitation Methods
Knowledge elicitation efforts usually take place in stages, and the nature of the knowledge is a major consideration in selecting the appropriate elicitation methods. The first stage is for the elicitor to develop an understanding of the general knowledge of importance available to the expert. Methods are available to develop a high-level description or overview of the expert's valuable knowledge, e.g., the concept mapping method. The description created by applying the method can be reviewed with the expert and his/her manager to select areas to drill down to the levels at which the most valuable undocumented knowledge is held. 

Following selection of the specific areas of importance, the elicitor may drill down to a deeper level of expertise applying the same methods used to create the high-level overview of the expert's knowledge. Alternatively, another method may be selected that is more appropriate for the nature of the knowledge. For example, if the knowledge is based in large part on significant events occurring in the past, then an interview approach focusing on critical incidents or critical decisions may be appropriate.

Other approaches may be more suited to knowledge that relates to operations and maintenance processes and equipment. Such knowledge may be elicited with the help of simulations and scenarios using mock-ups or actual equipment. The simulations and constructed scenarios method and the think-aloud problem-solving method encourage the expert to describe what he/she is doing and thinking about as he/she performs the simulated or actual tasks. Video or audio recordings and photographs may be taken at appropriate times during the elicitation sessions, edited, indexed, and made available to others when access to the expertise would prove beneficial.

It may be desirable to drill down to a more detailed level of knowledge at certain points during the elicitation process. For example, the expert may report that he/she senses almost unconsciously that something is in alignment, and that one “thing” can be inserted into another. If this capability to perform the action more quickly and better than anyone else has high value, then an unstructured interview approach might be applied to ferret out the important cues that are present. The elicitor may ask about visual, auditory, and tactual cues that are being used, possibly at almost an unconscious level.

Consider Storage, Presentation, and Use
Previous researchers working in the field of expert systems and knowledge management have observed the existence of a knowledge acquisition bottleneck. The knowledge elicitation methods, applied appropriately in the context and situation, can alleviate serious knowledge acquisition bottleneck problems.
 
Despite such reduction of knowledge acquisition bottlenecks, however, care must be exercised to facilitate the subsequent steps of knowledge storage, presentation, and use. The very methods that can alleviate bottlenecks in knowledge acquisition can create time and effort barriers for subsequent stages of the process.

For example, methods such as structured and unstructured interviews that rely on audio recording of elicitation sessions can create a transcription bottleneck. Transcription, editing, and reviewing audio records of interview sessions are time-consuming activities. Techniques to minimize the editing required to format knowledge for use by others include careful and selective audio recording and, for certain kinds of knowledge capture, use of video recording.

Computer speech recognition might be considered as an approach for avoiding the transcription bottleneck. At this time, the technology is not yet advanced enough to make this approach feasible. Both the elicitor and expert would need to train the speech recognition system in their respective voice patterns, and technical terms not in the speech recognition lexicon would need to be entered prior to the elicitation session. Speech recognition technology is moving ahead rapidly, however, and it may help reduce the transcription problem in the future.

The elicited knowledge should be formatted and packaged in a knowledge module. A knowledge module is explicit knowledge related to a specific task, activity, job, etc., that is retrievable when needed after having been elicited from an expert; evaluated, edited, and formatted to be in a form usable by others; and stored in electronic and/or hard-copy form.

There are at least two issues to consider when preparing knowledge modules. One issue relates to the use of the expert knowledge: Is it going to be incorporated with other material used by those receiving the information, or is it going to be used in standalone fashion? For example, the expert knowledge could be incorporated into a training class together with other training material. Alternatively, the expert knowledge could be linked to a step in a procedure, automatically appearing when it is time to perform that step. An example of standalone use involves a person in the field who inserts a CD-ROM in a laptop computer to receive guidance on how to perform a task, either just before or during task performance.

A second issue relates to the characteristics of the person using the knowledge module. If that person is not expected to be familiar with some of the technical terminology used or with the location of parts or tools discussed by the expert, then additional information may be required.

The knowledge modules must be stored appropriately and in accessible locations. Their existence must somehow be made obvious to potential users at the critical time that the knowledge should be accessed, and they must be presented in a timely fashion when needed.

It is essential that the knowledge modules be updated and corrected as appropriate. Changes will occur in equipment, processes, procedures, practices, regulations, and responsibilities over time. For a knowledge module to be useful over an extended period, it must be updated. Also, with use, some of the knowledge may be found to be incorrect. It is essential that the errors be eliminated and correct information provided. Knowledge modules that no longer have value should be eliminated.

No Right or Wrong Methods
The process of capturing valuable undocumented knowledge hinges on the development of an effective plan. It is important to determine whether potentially valuable undocumented knowledge will be lost with unavailability of experienced personnel; evaluate whether this knowledge is worth capturing; select appropriate method(s) to use in eliciting knowledge; and store, retrieve, and present this knowledge when needed.

The importance of each of these steps does not, however, imply that there is a right or wrong knowledge elicitation method or set of methods. The choice depends on a range of considerations, some of which may not come into play until knowledge elicitation is under way.

For example, the knowledge elicitor may find that an elicitation method not considered or selected during planning may be more appropriate for the type of knowledge used by the expert. In such instances, it may prove desirable to revise the plan as the knowledge elicitation moves forward. Thus, understanding and access to a range of methods, and the flexibility to alter methods being used or planned, will result in greater benefit from the knowledge capture endeavor.

Conclusion
The findings of this project have important implications for the way in which organizations manage data and information. Just as some valuable knowledge is stored only in the minds of experts, some valuable information and data may not be collected, adequately stored, easily accessible, or usable when accessed.

Strategic work in the rapidly developing fields of data, information, and knowledge management is exploring these challenges and helping identify solutions. By providing approaches, tools, and capabilities for optimizing management of human and facility performance, the Undocumented Knowledge Project and the EPRI Strategic Human Performance Program focus on helping energy companies maximize the long-term value of the data and information that they generate and the human capital that they develop and manage.

Acknowledgements
This work was performed as part of the EPRI Strategic Human Performance Program. Papers were presented at IEEE Seventh Conference on Human Factors and Power Plants, September 15-19.

  About the Authors
Madeleine Grossis manager, strategic human performance program, andLewis Hanesis project manager at theElectric Power Research Institute(EPRI).Thomas Ayres, human factors consultant, works with EPRI on several projects.