International Journal of Intelligent Information Systems
Volume 4, Issue 4, August 2015, Pages: 71-78

Development of Operating Instructional System Using AR Technology in Chemical Plants

Atsuko Nakai1, Shun Motoyoshi2, Fuminori Oomori2, Kazuhiko Suzuki1, 2

1Centerfor Safe and Disaster-Resistant Society, Okayama University, Okayama, Japan

2Graduate School of Natural Science & Technology, Okayama University, Okayama, Japan

Email address:

(A. Nakai)
(S. Motoyoshi)
(F. Oomori)
(K. Suzuki)

To cite this article:

Atsuko Nakai, Shun Motoyoshi, Fuminori Oomori, Kazuhiko Suzuki. Development of Operating Instructional System Using AR Technology in Chemical Plants. International Journal of Intelligent Information Systems. Vol. 4, No. 4, 2015, pp. 71-78. doi: 10.11648/j.ijiis.20150404.11

Abstract: In recent years, Japan's industrial accident rate has shown an increasing trend. This is especially remarkable due to chemical industrial complexes. As is well-known, many kinds of hazardous materials are being controlled in chemical facilities. If a serious accident occurs, there is the potential for severe damage to employees and the residents of local communities. A primary factor in these accidents is the lack of safety awareness, safety knowledge, safety management system deficiencies, and insufficiency of safety ethics. In addition, industrial technology is highly diversified and complicated. As a result, operators cannot grasp the whole situation of the abnormalities and potential crises present. In other words, operators are unable to take the appropriate safety measures to prevent accidents. In some cases, equipment failure shave developed into serious accidents due to incorrect operation by the operator. This paper presents systems that provide information to operators by using augmented reality (AR) technology in chemical plants. AR can enhance real-world environments using virtual objects such as computer graphics. This system can help plant operators to confirm procedures in order to ensure proper operation. Furthermore, the operator can recognize the equipment to be operated properly using a tablet PC with a built-in camera. The proposed system can provide the plant information based on the dynamic simulator (DS). In an emergency, chemical plant operators are required to make quick decisions to prevent the escalation of an accident. To convey accurate indication information of the work, it is useful to recognize target equipment using AR marker in addition to the output information by individual voice from control room. Our developed systems can support chemical plant operators to make quick decisions and to follow correct operating procedures.

Keywords: Instructional System, AR, Operation Support, Human Error

1. Introduction

Improving the safety and reliability of large-scale industrial facilities is extremely important for the safety of workers and also for the local residents. Consequently, a number of activities, countermeasures, and assessment techniques have been proposed. There is a need for detecting potential risks before an accident or disaster occurs, taking preventative/reactionary measures, and organizing a management system that is compatible with the risks. In addition, a safe process design is necessary for preventing human error. Control problems are increasingly significant in both software and hardware as the economic situation changes over time [1].

In a chemical plant, operators must be responsible for their own areas in order to ensure the safe operation of all plant components. There are two types of operators: field operators and board operators. Field operators are responsible for operating equipment in the field. Board operators’ control refineries from a control center or control room. In order to achieve smooth operation in a chemical plant, cooperative work is necessary [1]. There were human factors at play in the recent accident, including misjudgment and miscommunication.[2]Accidents also often occur in non-steady-state operation.[3][4]When an accident or disaster, including during non-steady-state operation, emergency response operation is needed. Operators must make accurate and quick decisions in spite of being in high pressure situations.[5][6]At present, operational guidelines and procedures for emergencies have not been fully developed. Accurate judgment, and the corresponding operational steps, are very important in order to prevent the escalation of the accident.[6][7]Howeverwith the retirement of skilled workers, opportunities for technology inheritance, training and education have decreased. We are concerned that the flexible response capability of workers has also reduced. In this study, we propose a system to provide instructional information for field operators by using augmented reality (AR) technology in chemical plants. AR can enhance real-world environments by using virtual objects such as computer graphics. [8]By using this system, operators can confirm the required operating procedures in the field. Furthermore, the operator can recognize the proper operating equipment by using a camera-equipped tablet PC. Our system has been designed to aid chemical plant operators in making quick decisions and in taking the correct operating steps. The concept of the proposed system is shown in Fig. 1.

Fig. 1. Concept of the Proposed System.

2. Purpose and Approach

In chemical plants, a large amount of dangerous chemical materials of a very wide variety are controlled. Moreover, they are managed in high-temperature, high-pressure environments. The various chemical fluids that flow through into the pipe and the equipment (such as tanks or reactors) are arranged in close proximity. Based on the information from the distributed control system (DCS), board operators manage the plant process until the product from raw materials. Board operators instruct the field operators on the necessary operations by monitoring the conditions in plant. Field operators work according to instructions from the central control room. When an alarm is issued, field operators perform an emergency response in the facility. Operating instructions on the site is a qualitative indication, not a quantitative data, "Turn the valve to the right", such as radio contact (through the voice of the board operators).Also, due to the installation of similar aggregating devices, such as pipes and valves, there is a risk of serious accidents occurring due to operators operating the wrong device.

In recent years, workload shave increased due to labor-saving measures and cost reduction. Young, immature plant operators are being forced to handle complicated facilities with fewer people due to the aging workforce situation. As a result, the potential for human error to occur has been increasing.

Human errors caused by field operators can be classified into the following types:[3]

1. Misjudgment: This involves a lack of understanding of the implications of the current situation and work

2. Incorrect operation: This includes mistakes made in operating procedures and operation details

3. Cognition and verification mistakes: This involves mistaking the operation target and omitting to give confirmation after completing an operation.

This study shows an information system that presents the relevant operating procedures and accurate equipment information to field operators. Using AR, the proposed system is created with additional information on the real image. The operators captured the equipment through camera; our system displays the name of equipment and operating information. The advantages of using AR include:

1. Virtual information is displayed simultaneously with real visual information

2. A real-time and interactive interface

3. Access to 3D representation

With advances in computer technology, AR technology has been implemented in various scenarios.AR can function in ways that enhance one’s current perception of reality.[9]It does so by incorporating computer vision and object recognition, so that the information about the real world of the user becomes interactive and digitally alterable. Virtual information about the environment and its objects can be overlaid on the real world.[9][10][11]AR systems are effectively used for emergency management.[12] Also previous research for chemical industries operators are presented by companies.[13][14]As previous research for the safe operation of large-scale facilities, there is the wearable supervision system that can provide to support the on-site operator. Using the system, the on-site operator have the ability to do the same things as if he or she were in the control room.[15] In nuclear power plant, Augmented Reality is a promising technology that will improve the efficiency and safety of maintenance work. [16]

By enhancing the information available on the real-world environment, the proposed system is able to provide the correct procedures and identify the target equipment for the operators. This system is linked to the plant dynamic simulator, instead of a real chemical plant, to provide plant information. Actual chemical plant facilities are controlled by the DCS. The number of equipment operated by the field operator is managed by the board operator in the control room. We used a dynamic simulator (DS) as a virtual plant in this system. DS is a software tool for building a virtual plant on a computer by modeling an existing plant or a plant that is yet to be built.[17] With the DS, operators can be educated in various situations. This system can be implemented with the actual control and safety systems using plant data. [18]Field operators operate the plant on the basis of the procedures outlined in the plant information.

By using the AR markers, the proposed system makes it possible to accurately recognize the target device. Field operators can determine whether the correct operation has been performed by using the camera-equipped tablet PC. The AR marker presents an image of a set pattern, which is indicator for specifying the installation site and displaying additional information, the name of equipment and process value etc. The operator identifies an operation object with the camera, and can confirm the correct operating procedure on the tablet PC. First, the work information is displayed to the operator; after the task has been completed, the next work procedure is presented by updating the plant information. Human errors often occur when a field operator works alone or in a non-steady-state operation. [3][5]The purpose of this study is to support the work of young, immature operators and to reduce human error.

3. Operation Instructional System

3.1. Outline of Operation Instructional System

The proposed system consists of two subsystems: the operating procedure system, and the device recognition system. Plant information for the operating procedure is stored in the database and presented as required. Operators can confirm the procedures and recognize the target device using the system through the camera-equipped tablet PC. Fig. 2 shows an outline of the proposed system.

An approximate workflow utilizing the system is shown in Fig. 3.

This entails the following steps:

1. Check the displayed procedure on the tablet screen.

2. Operate the equipment based on the instructions of the system.

3. Recognize the operation target device.

4. Operate the target device.

5. Check the operation and proceed to the next task.

3.2. Operating Procedure System

Traditionally, the operating procedure manual is created on printed paper. Usually, operating procedures are displayed in the form of a list, with no detailed information provided on the timing and the operation. Operating manual slack is descriptions of operational targets and their location through photos or figures. The instructional information that is presented to workers is, therefore, not considered to be sufficient. Conventional operating procedures are hard to understand intuitively. In order for the operator to perform alone without a trainer, considerable money and time must be spentFurthermore, the retirement of skillful workers is advancing in Japan, which is posing labor-related challenges.

These experts’ knowledge and skills are not being sufficiently transferred to the younger generation. Moreover, skilled workers may not remember the procedures of non-steady-state work.

Operating procedure systems compensate for the gaps in knowledge of safe operation. First, work procedure is indicated by the system. The field operator operates the equipment with reference to the work procedure outlined by the system. The virtual plant changes the process value by reflecting on the result of the work. After this, the next procedure can be displayed. The field operator then refers to the tablet PC and advances to the next task. In this study, we propose ways to present operators with the correct operating procedures that can be used in the field of chemical plants using AR.

Fig. 2. Outline of the Operation Instructional System.

Fig. 3. Workflow Utilizing the System.

3.2.1. Divide Operating Procedures

Existing operating procedures are presented in a list. In order to systematize the operating procedures on paper, the contents of the procedure should be divided into steps. The standard operating procedures contain too much information. Therefore, there is a need to clearly describe the role of the workers. Accordingly, we sorted the information by using the 6w2h method. 6w refers to "what", "who", "when", "where", "whom" and "why"; 2h refers to "how" and how many. The procedure of each operation is further divided and described in length it can be understood at once in the form of cards. The cards can be freely positioned on the display and be overlaid on real image using AR technology. Fig. 4 shows the image of the operating display.

3.2.2. XML Description of Operation Procedure

Operation procedures using this system are conveyed based on the progress of the work. Procedural information is written using Extensible Markup Language (XML). XML defines a set of rules for encoding documents and allows users to specify their own tags. It is possible for users to create their own meaning, structure and hierarchy in a unified manner. XML is also useful for creating files to store data [16].Fig. 5 shows the operating procedure written in XML.

The operating procedure to be used in this system, as shown in Fig. 5 is divided into steps<Step>, the ID information of each device <Target>,name <Name> and the associated markers <Marker>.Information such as the absolute coordinates of the markers are entered in the target space <X> · <Y> · <Z>. By using XML, documentation can be easily created, changed and appended.

3.3. Operation Support Database

Fig. 4. Image of the Operating Display.

The created procedures, plant conditions, and information of the equipment are stored in the database. Operating data of the dynamic simulator is stored in the Excel file in PC. The sensor data of the plant equipment from the DS is stored in Excel files on the database in this system. Exchange data with dynamic simulator by using the Excel file may be performed smoothly. Field operators can request procedures and equipment information from the database by operating the tablet PC. The information in this database is updated by performing the operation of the plant in the simulator; this can be presented to the worker. The next procedure is updated for display in the system after the completion of the first procedure to prevent a slip in operation. The information in the database is thus capable of being modified when necessary. And it may be further expanded.

3.4. Device Recognition System

3.4.1. Device Recognition Using AR marker

The device recognition system identifies equipment using the camera on the tablet PC. AR markers are used to recognize the plant equipment; the system then presents the target device to the field operator. The AR marker, in an image recognition AR system, is an indicator for specifying a position to display additional information and the image of a set pattern. When the AR system recognizes the marker, it can display information that has been specified so as to overlap with the real image. Simple black-and-white graphics or QR codes are often used as AR markers. There are many similar equipment, such valves and pipes, in a chemical plant. When installing a marker on the operation target device, the system is able to identify and determine the correct operational target and the operating procedure is presented.

Fig. 5. Operation Procedure Written in XML.

The field operator can perform the task by referring to the displayed procedure.

When the procedure is completed, the result of work is reflected on the plant information. The database, which stores the process value of the virtual plant, is also updated. This system links the DS as a virtual plant. According to the updated values, the next operating procedure is displayed.

3.4.2. Identify the Target Device

Workflow utilizing the system is performed based on the operating procedure to be accessed from the database. The operator checks the procedure in the field and identifies the AR marker of the target device with the help of the tablet PC camera. When the camera captures the correct target device, the device recognition system displays an orange object. If field operator tries to operate the wrong equipment, a blue object on the tablet PC indicates the direction of the target device. After the camera detects the target device, information on the device is displayed, followed by the operating procedure. The field operator can then perform along with the procedure displayed. If the operating instructions cannot be understood, the operator can refer to additional information in the system. In addition, it is possible to access device information from the plant, such as temperature and pressure.

3.5. Verification Experiment of the Device Recognition System

To check the operation of the system, experiments were carried out to identify the operation target device from a plurality of devices. Operating procedures were created using XML as follows:

Fig. 6. Verification Experiment of the Proposed System.

1. Close blue valve

2. Open red valve

3. Check the increase in pressure

4. Open the water supply valve

When the camera recognizes the operation target device by the AR marker, the name of the target device is presented on an orange object. The system shows the correct device direction with a blue object, if the camera recognized a wrong device. When performing the operation of opening the water supply valve after the blue valve is closed, the system displays the screen shown in Fig. 6.

3.6. Conjunction with the Sensor Data

This system is linked with the dynamic simulator (DS). When the equipment is in operation, the system indicates the operating procedure. When field workers operate the target device, the fluctuant sensor data are updated in real time. Therefore, changes in the process values are reflected in the virtual plant. The device recognition system compares the threshold, processes the value change of the DS, and displays the operating procedure. [18]When the process values meet the threshold, the operating procedure progresses and the system updates to display the next procedure. Fig. 7 shows the flow of the operation as it progresses in the system.

Fig. 7. Flow of the Operation in Progress.

4. Operating Experiment of the HDS Plant Procedure

4.1. Application of the Hydrodesulfurization (HDS) Process

This system is supposed to simulate the operating procedure of the HDS process. This process involves the reaction that comes with hydrogen separation and the removal of sulfur content from petroleum using the metal catalysts (such as nickel, cobalt, molybdenum, etc.) under high temperatures and high pressures; it also involves producing products with low kerosene and sulfur content and light oils. In this experiment, our system was applied to the field work. The Piping and Instrumentation Diagram (P&ID) of operating range is shown in Fig. 8.As the dynamic simulator is virtual; this experiment was carried out in our laboratory.

4.2. Making the Operation Procedures

In this paper, we referred to an operating procedure that had experienced Vacuum Gas Oil(VGO)charge pump failure trouble. The prerequisites for carrying out the HDS simulator training operations were as follows:

1.  The time to activate a spare pump starts when the main pump is stopped in order to close the flow time of the real device. This takes about 3 or 5 minutes.

2.  Aspects such as communication and reporting should reflect the training of the operators, and should protect what has been done within the department or company.

The outline of the corresponding operations is as follows:

1.  The field operators need to switch from the main pump to the spare pump very quickly.

2.  It is necessary to pay attention to the rising temperature of the furnace, due to the rise in temperature of the reactor.

3.  A board operator should operate the DCS using caution, monitoring any change in the temperature of the inflow and outflow of the furnace.

4.  A field operator should confirm the level of tanksD-402 and D-406 to prevent a rapid decrease liquid level.

In order to present the procedure on the system screen, the normal printed operating procedure should be divided into cards that clearly present individual steps, as follows:

Step1. Start a spare charge pump.

Step2. Adjust the temperature of the heating furnace.

Step3. Adjust the liquid level in the tank.

Step4. Open the main valve to 30% of the maximum value.

Step5.Resume the raw material supply to the heating furnace.

Fig. 8. P&ID of the Operating Range.

This procedure involves the cooperated work both field operator and board operator. The board operator control work flow from step1 to step5. Step 3, a required field operation, was applied to the operating experiment. It could be divided more finely. For storing information on the operating procedures in the system, we defined the parameters for this step as follows.

Number of the procedure

Procedure information

Start information

Operation information

Detailed information

Confirm information

Step 3 was divided into the following procedural cards:

Step 3: Adjust the liquid level in the tank

Adjust the level of the LGO (light gas oil).

Adjust the level of the tank.

Adjust the level of LGO.

Fig. 9. Model of Operation Procedures.

Refer to LC432.

Check and adjust the level of LC432.

Adjust the level of LC432.

Confirm the level of D-402.

The operating procedure model for Step 3 is shown in Fig.9. Similarly, operating procedures were created for each step.

4.3. Operating Experiment

Using the operating procedure, the operating experiment of the proposed system was performed. Fig. 10 shows Procedure 1, and Fig. 11 shows Procedure 2. As a result, we confirmed that the developed system can perform as intended.

Fig. 10. Result of Operating Experiment Procedure1.

Fig. 11. Result of Operating Experiment Procedure2.

5. Results and Discussion

In this paper, we proposed a system to provide information for field operators by using AR technology in chemical plants. AR can enhance real-world environments through the use of virtual objects. By using AR markers, this system has made it possible to accurately recognize target devices. The operator can recognize the AR marker set for the target device with the help of a camera-equipped tablet PC. Field operators can detect the correct equipment to perform procedures on and more easily differentiate between similar aggregating devices, such as pipes and valves. Thus, the system can reduce the risk of serious accidents caused due to operators mistaking the device to be operated. In addition to identifying the correct equipment, the operator can confirm the correct operating procedure through the tablet PC. Operating procedures are displayed for the field operator, and after the procedure is completed, the next procedure is displayed by updating plant information. In order to systematize the operating procedure, we divided some steps to offer more clarity to operators. The system can then instruct operators on each operating procedure through cards on the tablet PC.

We used the dynamic simulator (DS) as a virtual plant in this system; actual chemical plant facilities are controlled by the DCS. The number of equipment operated by the field operator is managed by the board operator in the control room. In addition to presenting detailed plant information, the functionality of this system should expand to include presenting instructions to board operators as well. In future, this system can be implemented with the actual controls and systems, and by using real plant data. Now in the chemical plant, the field operator cannot confirm the real-time process value in the field.

6. Conclusion

In this study, we developed an operating instructional system using AR technology for chemical plants. This information system can indicate real-time operating procedures and accurate equipment information for field operators. In chemical plants in Japan, there have been rapid generational changes in the workforce. Technology and knowledge of the skilled workers is being gradually lost. Inherited technology and past accomplishments are essential factors in enhancing competitive production. Human errors often occur when a field operator works alone or performs non-steady-state operations. The proposed system tackles the issue of human error by recognizing correct equipment, presenting relevant information, and having a confirmation function to prevent erroneous operation. Moreover, when an accident occurs, the correct operation must be performed without any human error in order to prevent an escalation of the situation. The proposed system can help operators achieve this level of accuracy. Furthermore, the system can also be used for safety education and training. We especially hope that the proposed system can aid the next generation of operators in the field and also help clarify non-steady-state operations.


  1. A. Nakai, Y. Kaihata, K. Suzuki, 2014. "The Experience-Based Safety Training System Using Vr Technology for Chemical Plant", International Journal of Advanced Computer Science and Applications (IJACSA), Vol.5, No.11, pp.63-67.
  2. K. Suzuki, Y. Munesawa, A. Nakai, 2013 "Recent Accidents and Safety Activities in JAPAN", AIChE Spring Meeting & 9th Global Congress on Process Safety, April.
  3. A. Komatsubara, 2008."Human Error (second edition)", Maruzen Publishing, in Japanese.
  4. Ministry of Health Labor and Welfare, 2003"Interim Report on Industrial Accident Survey", in Japanese.
  5. S. Jürgen, 2013. "Process and Plant Safety – Research & Education Strategy to Keep Long Term Competences", Chemical Engineering Transactions, vol.31.
  6. Norton, C., Cameron, I., Crosthwaite, C., Balliu, N., Tade, M., Shallcross, D., Hoadley, A., Barton, G., Kavanagh, J. 2008. "Development and deployment of an immersive learning environment for enhancing process systems engineering concepts", Education for Chemical Engineers, vol. 3, 2,December pp.75-83.
  7. T Nakata, 2007. "Wisdom for Preventing Human Errors: Can all mistakes beeliminated ?",Kagaku-Dojin Publishing, 2007 in Japanese
  8. Chen, Brian X, 2009. "IfYou’re Not Seeing Data, You’re Not Seeing", Wired, 25 August.
  9. Nikkei Communications editorial office, 2009"AllAbout AR: Technology to innovate mobile phones and the Internet", Nikkei Business Publishing, in Japanese.
  10. K. Oafish, Y. Sakurai, Y. Ishii, 2013. "An Adaptive AR Photographing Parameter Tuning for Industry Workspaces", SICE Annual Conference 2013, September.
  11. T. Fabio,2013. "The HMI of the future will look very familiar," Control Engineering, Vol. 60, No. 7, pp. 26-29.
  12. G. Baron, 2009. "Augmented Reality--Emerging Technology for Emergency Management" [90]
  13. Y. Ishii, K. Ooishi, Y. Sakurai, 2013."Industrial Augmented Reality -Innovative operator assistance in collaboration with Augmented Reality-", Yokogawa Technical Report English Edition Vol.56 No.2.
  14. H. Hara, H. Kuwabara, 2015. " Innovation in Field Operations using Smart Devices with Augmented Reality Technology" FUJITUU,66,1 pp.11-17 (01 2015) in Japanese.
  15. O.Naef, P. Crausaz, 2006. " 6th Sense System Augmented Reality Chemical Plant Supervision System"International Journal of Online Engineering (iJOE), Vol.2, No.4.
  16. H. Ishii, Z. Bian, H. Fujino, T. Sekiyama, T. Nakai,A. Okamoto, H. Shimoda,M. Izumi, Y. Kanehira, and Y. Morishita, 2007. "Augmented Reality Applications forNuclear Power Plant Maintenance Work", International Symposium on Symbiotic Nuclear Power Systems for 21st Century (ISSNP).
  17. G. Fukano, K. Yokoyama, Y. Yahata, 2013. "MIRROR PLANT On-line Plant Simulator and its Applications", Yokogawa Technical Report English Edition, vol.56 No.1, pp. 11- 14.
  18. K. Yamamoto, A. Nakai, K. Suzuki, 2013. "Development of Experienced-based Training System combined with Process Dynamic Simulation" Asia Pacific Symposium on Safety 2013.

Article Tools
Follow on us
Science Publishing Group
NEW YORK, NY 10018
Tel: (001)347-688-8931