Kodai TSUSHIMA*
Kazuya HIRATA*
*
Technologies, R and D Division xR Application Development Section
In recent years, xR technology (general term for VR, AR, MR technology, etc.) has begun to be actively used not only in the entertainment industry but also in the industrial field. EBARA Group is also promoting the use of xR technology in order to increase the added value of its products and services and to improve conventional operations. This paper explains cases using various elemental technologies of VR, such as contact interference in VR space, visualization of flow analysis results, and participation in VR space from remote locations (remote coordination) for industrial applications using VR devices (HMD) and VR equipment (CAVE). In addition, we introduce our efforts to utilize 360-degree images obtained by 360-degree cameras and point cloud models obtained by 3D scanners, which are peripheral technologies of xR.
Keywords: xReality, Industrial application, Contact interference, Flow analysis, Visualization, Remote coordination, Virtual Reality,
Augmented Reality, Mixed Reality, 3D-CAD, Digital Transformation
Virtual reality (VR) technology, which has been developed mainly for the entertainment industry, is an immersive technology that mainly uses computer graphics (CG) to allow users to perceive a virtual world as if it were reality. With acceleration in improvement of the hardware and software and the advancement of 3D-CAD technology in recent years, VR technology has begun to be enthusiastically applied in industrial fields. As in the case of the entertainment industry, the most common way to use VR in industrial fields is to immerse users in the virtual space with a head mounted display (HMD) and stereoscopic views of 3D-CAD models (hereinafter 3D models). Some designs using 3D models can be visualized only on a PC screen or in printed matter. Using this technology, however, the size, shape, and structure of such designs can be checked in a virtual space as if there were a real prototype. Accordingly, the designs can be modified at the design stage based on the check results.
In recent years, technologies such as VR, argumented reality (AR), and mixed reality (MR) have been collectively referred to as xReality (xR), which is referred to as xR technology in this paper.
The EBARA Group, whose main products are industrial machinery such as pumps and compressors, public infrastructure such as pumping and drainage stations, waste treatment plants, and semiconductor manufacturing equipment, is also promoting the application of xR technology to improve sales, design, after-sales services, and other operations, as well as to increase customer value1). This paper will explain Ebara’s xR equipment, and introduce contact interference, visualization of flow analysis results, and remote coordination as application examples of xR technology. Finally, 360-degree images taken by a 360-degree camera and their usage will be introduced.
As Ebara’s xR equipment, in addition to a general HMD (operational since 2017), the Cave Automatic Virtual Environment system (hereinafter referred to as the CAVE system) (Figure 1) was established and started operation in 2018. The CAVE system is a large system with multiple screens on the front, sides, and bottom. Each screen has an individual projector that projects images for the right and left eyes alternately. The user is immersed, not with an HMD, but with LCD shutter glasses linked to the projector. In addition, since the images from each projector change according to the position and the line of sight of the immersed person, the user can view the 3D model stereoscopically without feeling any discomfort regarding the position of the 3D model. An HMD blocks the entire field of vision of the wearer. Once immersed, wearers cannot perceive their surroundings or even their own limbs. The CAVE system, on the other hand, allows multiple people to recognize and share the same 3D model because its LCD shutter glasses do not block their view. Therefore, it is possible for multiple people to discuss the 3D model while pointing at it to indicate specific locations in the 3D model. In addition, since LCD shutter glasses are lighter than HMDs, they are more comfortable to wear during immersion.
Fig. 1 The CAVE system operated by Ebara
When using an HMD, we also use additional equipment: a PC equipped with a high-performance graphics board (we use a notebook mobile workstation), a controller for operation in the virtual space, a tracker to identify the immersed person's position in the virtual space, and VR gloves to display the section from elbow to fingertips of the immersed person in the virtual space during contact interference. All of these devices are portable, unlike the CAVE system, which is a fixed facility. Thus, VR technology also provides the advantage of usability in various locations, both inside and outside the company.
“Contact Interference” is a technology to determine whether a 3D model is in contact with the hand model of the immersed person or another 3D model in a virtual space where 3D models are viewed stereoscopically. This technology is also used in 3D-CAD, as well as in virtual space of VR. First, the user holds the controller in both hands, or in one hand and wears the VR glove on the other hand and the trackers on the upper and lower arms, as shown in Figure 2.
Fig. 2 VR immersed person wearing equipment.
The movements of the controller and the VR glove of the immersed user are displayed as the hand and arm movements in the virtual space, without inconsistency with their actual positions. The immersed user can then touch, grasp, and move some parts of the 3D model with the virtual hand and arm that are displayed in the virtual space. Figure 3 shows an Ebara high-pressure multistage pump. As shown in the figure, the pump body is surrounded by many small pipes and auxiliary equipment such as valves.
Fig. 3 Ebara high-pressure multistage pump.
Figure 4 shows an example of an immersed person wearing the VR glove and trackers performing an interference check on a 3D model of a high-pressure multistage pump. When the hand of the immersed person wearing the VR glove touches pipes and auxiliary equipment, which are usually displayed in the color specified when the 3D model was created, the color changes to blue, and the glove vibrates to notify the person of contact.
Fig. 4 Example of notifying an immersed person of contact
In addition, the VR glove responds to the movements of the immersed person's five fingers. When the person holds a part, the piping and auxiliary equipment turn red to let the person know that the part has been held and can be moved. Figure 5 shows an immersed person actually holding the piping of a 3D model of a high-pressure multistage pump to confirm the piping route.
Fig. 5 Example of confirmation of the piping layout
When the held pipes collide with each other, they stop moving without passing through each other, and the arrows indicate where they are interfering. This allows the immersed person to experience the piping as if the person were handling real piping. This technology resolves one disadvantage of HMD: once wearing it, the user is unable to recognize their own limbs, making it impossible to check the distance from the model and reach of the user during immersion. Video 1
shows actual contact interference.
The Ebara Group is considering the application of contact interference mainly for design review at the design stage, and for disassembly and assembly training. For example, in the case of a large high-pressure multistage pump, as shown in Figure 3, it is difficult to manufacture a prototype with the actual dimensions, because it has many complicated pipes. This technology, however, makes it possible to check the arrangement of the piping around the pump body and the auxiliary equipment such as valves, as well as the piping route. In addition, as shown in Figure 6, by projecting a human model in a virtual space, it is possible to assess workability for assembly and maintenance.
Fig. 6 Example of a combination of a 3D model and an avatar
The combination of these technologies is used to confirm the layout of pumping and drainage stations and plant facilities during planning, check the workability of footholds and confined spaces during work in a high place, make customers feel the scale of each product on plant tours and at various exhibitions, and compare the shapes of old and new products.
Ebara uses “EasyVR” of FiatLux for the CAVE system and “IC.IDO” of ESI Japan for the HMD as software for stereoscopic viewing of 3D models and contact interference technology.
In recent years, flow analysis technology has been widely used, not only for research but also as a tool in the field of manufacturing such as design and development. Almost all of the present flow analyses are three-dimensional; accordingly, the results provide three-dimensional flow field data. The results are displayed on the screen or printed on paper as velocity vectors and pressure contours for evaluation. In conventional evaluation methods for these data, mainly the part of the flow field to be observed is cut and displayed as a cross section. For unknown flow phenomena, an arbitrary cross section is observed. Therefore, the information in the direction perpendicular to the cross section is lost in the flow field with high three-dimensionality and complexity, or in the flow field in a complex geometry such as the internal flow path of a pump. For this reason, it is difficult to evaluate the flow field in detail. A bird's-eye view or animated view is also used in addition to a cross-sectional view, but it is difficult to grasp a complex three-dimensional flow field.
Therefore, we tried to evaluate the flow field in detail by immersing ourselves in the flow field made in the VR space using the CAVE system or an HMD. In this study, a large vertical shaft pump (Figure 7) was used to analyze the flow around the suction tank, which leads water to the pump, and the suction bell mouth, which is the inlet of the pump.
Fig. 7 Ebara large vertical shaft pumpｖ
First, as an example of the display of the analysis results, the particle paths around the bell mouth from the upstream of the suction tank are shown in Figure 8.
Fig. 8 Particle paths around the bell mouth
We released fish-shaped markers for visualization from the upper right of the figure on the upstream side of the suction tank, and then observed how water flows into the bell mouth shown at the lower left of the figure. As shown in Figure 8, it can be confirmed that the particle paths are disturbed, especially on the downstream side of the bell mouth, and the fish-shaped markers indicate complex flow. However, due to the high three-dimensionality of the flow field, it is difficult to accurately detect the generation position and direction of underwater vortexes and the air-sucking vortexes that would exist in the area.
Next, video 2 and Figure 9 show an example of evaluation in the VR space with an HMD and the CAVE system based on the same results. The software used in this study was “STAR-CCM+VR” of SIEMENS for the HMD and “EasyVR” of FiatLux for the CAVE system. In video 2, the local flow is visualized by emitting particles from the controller held in the hand in the immersed VR space and making them flow according to the flow analysis results. The immersed person can examine complex three-dimensional flow by changing their line of sight and observation position, as if those were real. This is extremely difficult to achieve with conventional two-dimensional evaluation using cross-sectional cutting, but the use of xR technology has made it possible to easily realize highly accurate and detailed evaluation.
Fig. 9 Example of flow analysis results using the CAVE system
In the CAVE system, multiple people can recognize the same virtual space in common. In addition, they are immersed in the space while recognizing each other's presence and appearance. Thus, they can have discussions using intuitive words like “this,” “around here,” and “this kind of feeling,” while pointing to the stereoscopic analysis results. In addition, HMD equipment is portable, it allows evaluation using VR technology at various locations. In both methods, it is easy to understand locally complicated flow patterns by moving inside the flow field and changing the observation direction as desired. In this analysis, VR technology was particularly effective in understanding the positional relationships between the bell mouth and underwater vortexes and air sucking vortexes, as well as complex three-dimensional flow patterns caused by vortexes.
“Remote coordination” is a technology that allows users to immerse themselves in the same virtual space and view the same stereoscopic 3D model from different remote locations connected via the Internet. Each immersed person can mutually recognize each other's standing position, line of sight, and even pointing position in the virtual space because they are displayed as avatars whose face and hands are stereoscopically viewed there. Also, as shown in Figure 10, changes made by one avatar, such as placing markers or adding dimensions to the 3D model, will be immediately reflected in the model viewed by all immersed persons.
Fig. 10 Example of placing a marker to a 3D model during remote coordination
As an example of the utility of remote coordination, a local worker and a head office technician could be immersed in the same virtual space to handle a pump failure at a local pumping and drainage station.” In the virtual space, the worker could point to the 3D model and explain the pump failure, while the technician could use the same model to provide a solution for the problem. Remote coordination is possible between HMDs or between an HMD and the CAVE systems. If only HMDs are distributed to and set up at local offices, there is no need for technicians to travel around the country, and a wide range of troubleshooting can be offered quickly. This technology will be especially effective in situations where long-distance travel is restricted due to measures to prevent the spread of infectious diseases, as has been the case recently. In addition, when the research, development, design, and production departments of a company are dispersed in remote areas, it is possible to conduct design reviews with stereoscopic views, as if a prototype were in before the immersed person’s eyes, instead of having discussions using 2-dimensional information, such as drawings displayed on a conventional video conference screen. The understanding of the structure and shape of a product during planning can be shared more accurately and in greater detail by the responsible person of each relevant department, Thus, it is expected to improve product quality and reduce development lead time and rework time during installation at the site.
The following is an application example of Ebara’s remote coordination technology. Fig. 11 shows an example of remote coordination implemented with VR systems (HMDs), which were distributed to Ebara Headquarters (Tokyo) and Kumamoto Plant (Kumamoto Prefecture), connecting to the CAVE system in Fujisawa Plant (Kanagawa Prefecture). The CAVE screen in Figure 11 shows the heads and hands of immersed persons from different locations as light blue avatars. In the same virtual space, we confirmed the layout inside the factory building and the workability during assembly of the product, while commonly viewing the 3D model. Also, we are planning to establish connections to our overseas bases.
Fig. 11 Example of remote coordination between Ebara Headquarters, Kumamoto Plant, and Fujisawa Plant
This paper next describes the use of 360-degree images and videos, which are a peripheral xR technology. A 360-degree image is a panoramic image taken with a special camera called a 360-degree camera (Figure 12), which includes all directions (up, down, left, and right). 360-degree images are sometimes referred to as 720-degree images (a pair of 360-degree images in the horizontal and vertical directions), but in this paper, “360-degree images” refers to both still images and videos. An example of a 360-degree image is shown in Figure 13, which shows a 360-degree image of the lobby of Ebara Headquarters taken by a 360-degree camera.
Fig. 12 Example of a 360-degree camera
Fig. 13 360-degree image of the lobby of Ebara Headquarters
The black pump shown on the left side of Figure 13 is a huge pump with a height close to the ceiling, and it is usually difficult to take an overview image of it.
In a 360-degree image, the top, bottom, left, and right are compressed into one image. Thus, a 360-degree image looks like a compressed panoramic image when viewed in a plane. If a special application on a smartphone is used to view 360-degree images, left and right images with parallax can be obtained. In addition, since the image changes according to the movement of the user’s line of sight, the user can have a live-action VR experience as if standing in the scene. What is noteworthy about the use of 360-degree images is that the equipment can be easily introduced. At the time of writing, a 360-degree camera with sufficient performance can be purchased for less than 50,000 yen. Dedicated applications for operating the 360-degree camera and viewing the captured 360-degree images are also available free of charge and are very easy to use. VR goggles that hold a smartphone can be purchased for as little as 300 yen, making it relatively easy to experience and use xR technology.
There are a wide variety of possible uses for 360-degree images. VR can be widely used, not only in manufacturing divisions, but also in administrative divisions, for creating training manuals for hazard prediction training in factories, creating company profile materials for public relations activities, recruitment activities, and sales activities; and remotely conducting witnessed tests in combination with remote coordination technology.
The challenge in expanding the use of VR technology in the future is that there are not many 3D models available for the product line of the company's business. In the case studies in the previous chapters, VR immersion was introduced mainly for relatively small stand-alone products. On the other hand, the EBARA Group also provides large facilities, such as pumping and drainage stations and waste treatment plants. Contact interference and remote coordination will be effective in reviewing plans of pumping and drainage stations and waste treatment plants, checking delivery routes, installation procedures, and piping layouts of various equipment, and verifying workability during operation and maintenance after the commencement of operation. For new projects, the use of 3D-CAD has been progressing since the planning stage. However, there are almost no 3D models for the existing facilities. Therefore, when xR technology is used for these facilities, it has been necessary to create new 3D models.
In order to obtain a 3D model of an existing facility, point cloud data acquired from the facility with a 3D laser scanner (hereafter referred to as 3D scanner) is used. By using a 3D scanner, point cloud data of an existing facility can be obtained in about half a day, which offers a suitable 3D model faster than creating a new one2). By combining point cloud data and 3D models of individual products such as pumps in a virtual space, it is possible to assess the ease of disassembly and assembly of pumps, check the delivery routes of new pumps, and verify the piping layouts in virtual space.
Figure 14 shows an example of the utilization of point cloud data. Figure 14 shows an immersion with an HMD into a virtual space with point cloud data acquired by a 3D scanner and a 3D model created by 3D-CAD. For the VR immersion in Figure 14, “XVL Studio VR option” of Lattice Technology was used.
Fig. 14 Example of the utilization of point cloud data.
In the lower center of the image, the presence of the HMD controller is visible in the hands of the immersed person. The immersed person can touch 3D models in the virtual space by using this controller as a pseudo hand and pick up and move parts. It is also possible to check for contact interference between parts to be picked up and moved and the point cloud data in the space.
The utilization of point group data in xR technology is expected to be applied to more various operations. For that reason, we are planning to collect information on this technology and promote the further introduction of the technology.
In addition to what has been presented in this paper, xR technology is expected to rapidly become widespread in various fields and further technologically to develop in the future. The EBARA Group will not only continue to research and develop xR technology, but will also promote the use of xR technology at work sites to improve the efficiency and precision of various operations. In addition, the group is promoting the use of this technology in its sales and public relations activities as a tool for improving customer satisfaction and corporate value, aiming to contribute to ESG management of the group.
1) Kazuya Hirata, “Industrial Application of VR Technology at Ebara Corporation,” 86th CG/Visualization Workshop
2) Kodai Tsushima, “DX Using VR Technology: The forefront of VR Use in the Manufacturing Industry,” Journal of sewerage, monthly, July 2020