1 s2.0 S0278612522001650 Main

Journal of Manufacturing Systems 65 (2022) 260–278
Contents lists available at ScienceDirect
Journal of Manufacturing Systems

journal homepage: www.elsevier.com/locate/jmansys
Review
Challenges and opportunities on AR/VR technologies for manufacturing

systems in the context of industry 4.0: A state of the art review
M Eswaran, M V A Raju Bahubalendruni *
Industrial Robotics and Manufacturing Automation Laboratory, Department of Mechanical Engineering, National Institute of Technology Puducherry, Puducherry
609609, India
A R T I C L E I N F O A B S T R A C T
Keywords: Manufacturing industries are currently experiencing the fourth revolution with penetration of new immersive
Augmented reality technologies extremely focused on the flexible manufacturing and human machine interaction. The immersive
Virtual reality technologies such as Augmented Reality (AR) and Virtual Reality (VR) have received much attention among the
Industry 4.0
manufacturing industries to achieve mass production and mass customization for different product variants. This
AR/VR for manufacturing
paper demonstrates the research progress and developments in the AR/VR technologies for product design and
Object identification
evaluation, Repair & Maintenance, Assembly, Warehouse management, Quality control, Plant layout and CNC
simulation. The critical challenges in AR/VR technology implementation at hardware and software level and
opportunities for the further developments are discussed.
1. Introduction processes prior to their implementation in a production environment.

This will result in cost savings in reworks and revisions to manufacturing
The current era of Industry 4.0 has led to significant changes in activities such as product design, process planning, machining, and so
manufacturing domain, with the goal of producing innovative, high- on [5,6].
quality products with shorter time-to-market by increasing the flexi The immersive technologies such as AR and VR were first used by
bility of real-time instruction or knowledge exchanges between the retailers to promote their products and services to the customers.
various streams in a product development life cycle [1,2]. The customer However, the advancements in software algorithms and lower costs in
looking for different product variants to satisfy their need and economic AR/VR devices made the immersive technologies deployed into a wide
caliber demanding the manufacturing industries to be more flexible at range of applications in recent years. The implementation of AR/VR
design and manufacturing [3]. Moreover, the current manufacturing technology in industry has centred on guiding and training semi-skilled
tycoons of the world are aiming to adopt technologies for meeting workers to do challenging tasks efficiently. In augmented reality (AR),
aesthetic, functional and quality requirements of 21st century mankind. viewers might experience a dominant scene of the real world in which
Technology adoption at the manufacturing stage can eliminate the the augmented scenes of computer-generated digital information or
bottle necks on achieving high product variance with uncompromised virtual things are overlaid on real-world objects. Furthermore, AR
quality at minimum time. The growing trend of globalized technology allows the user to experience human sensory modalities such
manufacturing environments demand the real-time information ex as vision, haptic, and hearing, among others [7–9]. However, the special
changes and seamless task coordination among the many nodes in a equipment is necessary to visualize the virtual scene in the real envi
product development life cycle, such as design, setup planning, sched ronment. There are three AR visualization devices, such as a Head
uling, machining, assembly and so on. Therefore, the manufacturers Mounted Display (HMD), a see-through hand-held display, or spatial
have been reconsidered their process of product manufacturing to projected devices [10,11]. These devices have been used to superimpose
maintain pace with the emerging technology at par with industry 4.0 the augmented virtual scene over the real surroundings and displays it in
standards [4]. Immersive technologies such as AR and VR are being used the user’s sight. The virtual scene information layered on the actual
to assist workers in overcoming these obstacles and challenges by world is additive or masking, depending on whether the virtual scene is
providing support, simulation, and assistance in enhancing industrial in the form of symbolic or pictorial data [1,12,13]. Moreover, this
* Corresponding author.
E-mail address: bahubalindruni@gmail.com (M.V.A.R. Bahubalendruni).
https://doi.org/10.1016/j.jmsy.2022.09.016
Received 10 February 2022; Received in revised form 24 April 2022; Accepted 19 September 2022
Available online 30 September 2022
0278-6125/© 2022 The Society of Manufacturing Engineers. Published by Elsevier Ltd. All rights reserved.
M. Eswaran and M.V.A.R. Bahubalendruni Journal of Manufacturing Systems 65 (2022) 260–278
AR-based interactive experience seamlessly integrates with the actual designed to accurately simulate the manufacturing process, they
surroundings, creating an immersive sense of the real world. nevertheless have limitations. To begin with, physical interactions be
VR is one of the immersive technologies in current era where the user tween users and the virtual environment are removed, resulting in a loss
is completely shifted from real environment experience to the fully of the users’ actual experience and diminishing the VR manufacturing
immersive virtual environment. The term VR is defined as "an immersive process’ fidelity and trustworthiness. Furthermore, it is difficult to fully
technology in which users can experience all of their human senses in a and accurately replicate the actual working conditions that are impor
prominent virtual scene in a virtual world [14]." The VR concept was first tant to the manufacturing processes, the VR experience may not be very
proposed in the 1950s and the Sensorama simulator is a first VR machine convincing. Even though there are improved techniques to speed up the
which is regarded as the first fully functional VR device capable of computation process, real-time performance remains a challenging one
providing users with an interactive, realistic immersion experience [15]. [36]. Virtual manufacturing system (VMS) is kind of monitoring soft
VR technology may create a virtual world experience for the user by ware in production system and using for product design analysis, quality
simulating all human senses such as vision, hearing, touch, taste, and inspection and performance analysis of assembly system. It requires
smell. Users might bear experience of visualization, manipulation, and various virtual models of product to accomplish the task of inspection
interaction with computers in a realistic immersive virtual environment. and monitoring of production systems[37].
Furthermore, VR gives users with interactive realistic graphics of The current paper organised with following sections. Section 2
non-visual modalities like as tactile, audio, and feedback, allowing the covers the main elements of AR/VR technologies associated with
user to sense the presence of a real-world environment [16–18]. Several manufacturing applications. Section 3 provides the detailed architecture
researchers focused their research interests on this technology, resulting of AR assisted manufacturing system. Section 4 discusses the method
in development of various concepts and prototypes [19,20]. ologies of AR/VR in manufacturing activity. Section 5 provides the
In addition to VR/AR, the term Mixed reality (MR) has been used in detailed review on application of AR/VR technologies in manufacturing
the article to represent the combination of both real and virtual envi activities. Section 6 addresses the challenges and research needs in AR/
ronment scene. Milgram statement on “virtuality continuum” and it is VR assisted manufacturing system. Finally, the last section ended with
defined as a range between the Augmented reality (AR) and Augmented discussion of conclusion.
virtuality (AV), where the Augmented reality has been covered more
prominent scene of real environment in user’s view compared to over 2. Main elements of AR/VR technologies associated with
laying of digital computer-generated virtual 3D scene information or manufacturing system
context [21,22]. However, Augmented virtuality has been covered more
prominent scene of virtual environment compared to the real environ Developing successful AR/VR based applications for manufacturing
ment scene as shown in Fig. 1. Azuma et al. performed the detailed domain is a challenging task. Despite significant breakthroughs in AR/
survey on AR emerging technology, in which the role of VR technology over the last two decades, the majority of AR/VR systems
Reality-Virtuality Continuum has discussed[23]. built to date are lab-based implementations. In this AR/VR technology,
Extended reality (XR) is another new emerging technology in in the synchronisation between the real and virtual worlds must be
dustry 5.0 and is being employed in various field of applications such as established in the shortest feasible time, real-time tracking and
assembly line monitoring, remote assistance, maintenance, health edu computing are essential. Globally, extensive research has been con
cation/training and so on [24–27]. Reddy et al. stated that the enhanced ducted and addressed few elements to solve some of the most pressing
communication technologies, edge computing, high precision compu concerns in AR/VR technology. AR/VR systems comprises of numerous
tation capabilities and highly capable devices can improve the appli hardware as well as software elements which enables the users to
cability of XR in industry 5.0 [28]. execute and complete the task successfully in manufacturing domain.
Over the last two decades, AR technology has improved virtual ob Moreover, special equipment is necessary, such as display devices and
ject positioning approaches from marker-based to marker-less tech precise trackers. There is a large range of commercially accessible
niques, as well as mobile context-aware technologies that can hardware and software tools in the current market. The main elements
incorporate AR into mobile and manufacturing workshop environments of AR for assistive manufacturing are indicated in the Fig. 2. Hardware
[29–31]. Thanks to the increasing number of AR efforts from multiple and software are the two major elements of AR system.
prominent corporations (Google, Sony, HP, IBM, and others), this
unique technology has been successfully used in a variety of areas, 2.1. AR hardware elements
including medicine, education, maintenance and repair, and cultural
heritage [5,14,32,33]. Google Glass, a compact and light optical An AR hardware system consist of display devices, sensing devices,
see-through monocular display [34], in particular, provides two key haptic and force feedback rendering devices, and tracking devices.
augmented reality features: encumbrance-free and quick access to in HMD devices are extensively preferring in AR applications at which
formation by stick on it in the visual environment. the eye-level display allows for real world experience of the unified AR
Many VR systems, such as CAVE, VADE, Vshop, IVY, SHARP, and scene. Consequently, wearing of these HMD devices is inconvenient and
HIDRA [35], have been successfully proposed to use for handling critical can cause neck pains, nausea, and tiredness, with its use for after pro
manufacturing tasks. Despite the fact that virtual reality systems are longed use [38,39]. Recently, other display technologies, such as pro
jectors and handheld devices, have been investigated by researchers.
Most of the new HMD types evolved in 2020 are lighter than previous
ones and have the feature of larger field of view (FOV), and
high-resolution displays like SXGA (1280 × 1024) or SVGA (800 × 600)
[40]. Some HMDs comprises of built-in cameras and sensors, that allow
for image capture and display the virtual scene at proper position when
the users in dynamic motion. Furthermore, Significant research has been
carried out and they suggested few customisation services in optical
system, such as optical see through (OST) and video see-through (VST)
HMDs, which are compatible with various video input configurations
(DVI, VGA, composite, etc.).
Cheng et al. [41] examined the use of freeform prisms and lenses to
Fig. 1. Milgram Virtuality Continuum. achieve a large FOV for HMDs, and Zheng et al. [42] designed an off-axis
261
Fig. 2. Hardware and Software Elements of AR Associated Manufacturing System.
optical system employing relay lenses with a polynomial surface. The images, and the resulting texture can be projected onto a physical model
development of visualisation and interaction methods in a design review for evaluating the design. Kitagawa and Yamamoto [49] were used a 3D
system have been focused by Caruso and Re [43]. Moreover, two LCD projector for the applications of 3D puzzle assembly instructions on
motor-driven video cameras were used to create a video see-through a desktop. Portable projector with AR technology has been used for the
HMD that can automatically adjust camera convergence. This medical applications of liver surgery in the hospital [50]. Schoning et al.
self-adjusting method tackles the problem of distortion in a virtual [51] have developed the projector-based phones design which com
prototype (VP) image which usually occur when it is too close to the prises of mini portable projectors with mobile phones, where the con
designer in a standard VST HMD. struction of camera optical axis and projector axis are aligned inside the
The focus of AR research is on mobility can be achieved through the mobile phone. Its influence on the field of projection (FoP) and FOV
use of handheld devices (HHD) and which are available business or were addressed. Willis [52] addressed the usage of similar device in the
customized model for the specific applications [44,45]. These mobile gaming industry. The Sixth Sense project created a user interface in
devices contain high processing capability and hardware components, which the user wears a mini projector on their head. RePro3D was
such as a gyroscope, high-resolution camera, and an interactive screen. developed by Yoshida et al. [53], who used retro-reflective projection
Off-the-shelf mobile phones can also be used for AR applications. technologies to create motion parallax, allowing users to view 3D im
However, these have limited processing capacity and internal storage. In ages without any special equipment.
order to achieve real-time performance, several researchers have used a In AR applications, haptic and force feedback is studied to improve
client-server architecture. PC-based tablets are preferred and capable to the user’s immersive and interactive experience. Wearable data gloves
generate the 3D Design of rendering scene than the mobile phones. have been used by researchers for applications on mobile and desktop
The "napkin" metaphor concept is used to describe a 3D design space, processes like as design, assembly etc. Valentini concentrated on the
which enables the display of virtual plane on a collection of ARTag interaction method in an AR environment while employing a data glove
markers. A mobile gadget called MARTI was developed as a combination for a virtual assembly [54]. Based on the identification of three common
of UMPC and camera [46]. The two-handed gripping mechanism of manipulation motions, the research focuses on grasping and manipu
MARTI device can make it is more comfortable to use. In this HHD de lating virtual assembly components. Path planning for a virtual robot
vice, the real time performance could not be achieved due to limitations has also been done using haptic devices [55].
at storage and processing capacity. Therefore, the experts suggested
client-server architecture system to achieve robustness in HHD device.
The system of client–server architecture includes mobile phone and PC, 2.2. Software systems
that enables the rendering of complicated CAD models or generated
virtual scene on the object of real environment. Hakkarainen et al. Merging of Real and virtual objects, 3D registration, and real-time
examined the mobile phone for the applications of assembly guidance in interaction are all essential elements of AR [23,56]. Various ap
manufacturing industry. Similarly, Ha et al. [47] used an Android mo proaches and systems have been developed in recent years to overcome
bile phone to implement a client-server approach for real-time tracking tracking and registration concerns in AR. Several well-known AR tech
applications in which the image capture and virtual information nology platforms have been created based on these algorithms to aid in
rendering are handled by the smartphone, while image analysis and the development of diverse AR applications. The fundamental enabling
object recognition are handled by the server. technology of AR is tracking and registration. Various sensing devices
The manufacturing industries are using the spatial projectors for the have been used to track the users’ heads in order to calculate their
AR assisted manufacturing activities which led to eliminate the users to perspectives, which is necessary for static and dynamic registration to
wear HMDs to visualize the augmented virtual scene on the real envi work. In this section, several sensors were introduced. CV-based
ronment. Hence, the researchers termed, this projection - based AR tracking and registration will be discussed in this part since these
arrangement as spatial augmented reality (SAR) in the industrial envi tracking methods analyse images captured by a depth camera to esti
ronment. The applicability of projectors in AR can be divided into two mate the camera position[57]. Natural feature-based tracking,
categories based on the projector’s installation, namely permanent Marker-based tracking, and model-based tracking are the three types of
installation and portable installation. During the early design stage, tracking and registration algorithms currently used in AR systems.
Saakes and Stappers [48] used AR technology with a fixed SAR config In AR applications, fiducial markers are widely utilized for object
uration, where the real-world texture can be combined with digital identification and tracking. These markers include geometric aspects or
unique patterns that make them simple to identify in a video stream. For
262
a known environment, marker-based tracking is a reliable and stable items that can be tracked in this package. Texture images with good
solution. Different markers can be recognised with an appropriate contrast are used as image targets. It is possible to track up to five picture
camera posing using feature detection and pattern matching approach. targets at the same time. Objects with several picture targets and fixed
The prominent tracking tool in this sector is AR Toolkit, which can spatial relationships are referred to as multi targets. If one of the targets
create square markers with asymmetric patterns [58,59]. Once the is discovered, other image targets can be detected based on the rela
marker position is identified the virtual object is superimposed on it tionship. A frame marker is a fiducial marker that has a unique ID
seamlessly. AR Toolkit-based marker generation has been used in encoded in a binary pattern along the marker image’s border. Fig. 3
several manufacturing applications including design [5,60], assembly represents the application development platform tools for AR assisted
[61–65], robot programming [8,66–69], teaching [70], CNC machining manufacturing activities.
[71–73], etc. osgART is another marker generating tool for object
tracking [74,75]. Moreover, libraries based on ARToolKit have been 2.3. VR elements associated with manufacturing system
created for a variety of programming platforms.
ARTag is an open-source platform that was built following AR Fig. 4 represents the hardware and software elements of VR system,
Toolkit [76]. ARTag employs more advanced image and digital symbol where the input devices, output devices and VR engine or computer are
processing techniques. In terms of stability and tolerance to illumination considered as hardware elements and software elements can be software
variations, ARTag leads ARToolKit. ARTag uses 2D barcode markers, application and database. The input devices are essential elements,
which eliminates the requirement to train or load pattern files. FLAR which enables the interaction between the user input and virtual world.
ToolKit is a toolkit for creating web-based augmented reality apps. VR engine or PC is a main heart of VR configuration system, which
ARMES is a commercially available marker tracker and which is based receives input through user input devices and after processing the
on SDK circular markers. BazAR [77] is a computer vision-based library suitable responses can be delivered to the user in real time via the output
for detecting and matching feature points. A key point-based technique devices. The input devices, such as voice recognition, 3D mouse and
has been created in this library. To greatly reduce on-line processing gloves are used to transmit user input to the VR Engine. In addition, the
time, the authors structured the wide-baseline matching of key points tracking sensors includes electromagnetic, optical, mechanical, ultra
between the camera images and those in the model images as a classi sonic and data gloves are used to track the position of the user. Virtual
fication issue. A simple and fast key point detector was used to overcome engines or PC were utilized to generate 3D models, object rendering,
concerns such as huge perspective and scale fluctuations in this simulation and visualize in real time. 3D sound, visualization devices
technique. and haptic devices are output devices, which makes the VR based
Tracking natural features can improve tracking stability and range. application more realistic.
AR applications for the unprepared environment can be built using
natural feature tracking. The robust point matching methodology is used 3. Architecture of AR assisted manufacturing system
in most of the current natural feature tracking techniques. To aid feature
detection and matching, a variety of feature descriptors have been AR technology enhances the interaction between the users and sys
investigated, including ferns features, Binary Robust Independent tems by enabling the user to move around the AR environment and
Elementary Features (BRIEF), Speeded Up Robust Features (SURF), visualize the virtual models naturally. Therefore, AR system has
Scale Invariant Feature Transform (SIFT) features, and so on. emerged as a promising technology to facilitate the mechanical
Parallel Tracking and Mapping (PTAM) [58] is a well-known method manufacturing activities. The architecture of AR assisted Manufacturing
for predicting camera posture in a new scene. Detail maps of the un system is illustrated in Fig. 5. The system comprises of six modules such
known area can be recreated with many landmarks using methods of as CAD model of parts/virtual scene generation, Generation of
processing the two tasks, namely tracking and mapping, in parallel manufacturing applications instruction, simulate complete
threads using keyframe-based mapping. Virtual items can be registered manufacturing operations in the CAD environment and unity tool, Real
onto the real world using the comprehensive maps. Only static and environment Capture, Tracking and rendering virtual scene/Real time
constrained environments can be tracked using this platform. As an interaction.
extension of PTAM, Parallel Tracking and Multiple Mapping (PTAMM)
[78] was developed. Model-based tracking, rather than employing 4. Methodologies for manufacturing activities in AR/VR
fiducial markers, matches observed features with natural attributes from
pre-created objects, such as shape, texture, motion, and so on. The Open This section reviews and addresses the methodologies that enable
Tracking Toolkit (OpenTL) [79] is a model-based tracking library with AR/VR manufacturing applications in industrial environment. Fig. 5
user-friendly APIs and support for tracking various objects. To achieve represents the methodologies of AR/VR assisted manufacturing activ
real-time performance, this library makes use of multithreading and ities. The creation of digital models is one of the primary tasks for
GPU-based processing. It was created for overall model-based object developing the AR/VR assisted manufacturing application. A 3D model
tracking, not exclusively for AR applications. can be generated by using CAD software or 3D reconstruction technique
AR applications based on mobile devices have gained significant involves collection of geometric data in digital form from an existing
attention in recent years due to their high processing capabilities and physical part. Unlike gaming VR/AR, the generated 3D model has to be
omnipresent features. AndAR (Android ARToolkit) is a Java-based accurate in geometric dimension and functionalities behaves like real
software framework that enables the creation of augmented reality object in the virtual environment. Following that, the AR/VR in
(AR) applications on the Android platform. ARToolkit for iOS is a structions have been generated for the various manufacturing activities
version that supports the construction of augmented reality apps for such as design, assembly, maintenance/repair, etc., Then the simulation
Apple’s iOS platform. NyARToolkit is a streamlined version of the of manufacturing activities will be carried out in both CAD environment
ARToolKit toolkit that works with a variety of programming languages and unity engine platform. The next step, object identification and
and platforms. It may also be used to create AR apps for Android phones tracking which plays a crucial role in AR technology assisted
and tablets. Layar is a smartphone platform that shows digital infor manufacturing activities simulation, where the identification of a
mation about the environment in the user’s immediate vicinity. Layar is particular object, positioning and orientation of virtual information over
available in many versions for Android, Symbian, and iOS. On Android the physical object are the main challenges in AR environment [80–83].
2.1, 2.2, and 2.3 devices, the Qualcomm Augmented Reality (QCAR) These methods rely on computer vision technology for identification and
SDK [122] facilitates the building of AR applications. tracking of an object. Point cloud tracking is an appropriate method for
Image targets, multi targets, and frame markers are the three types of manufacturing applications, where 3D point clouds are used to track the
263
Fig. 3. Application Development Platform Tools for AR assisted Manufacturing Activities.
estimation and overlaying. Fig. 6 represents the object identification of

physical components from the point cloud techniques.
• Reference object point clouds management: A CAD model is gener

ated for all physical objects in the manufacturing environment and it
is converted into point clouds (Ri) [88–91]. This point clouds are
stored in the data base and it is used for the process of matching with
environment object points. All reference objects must be generated
in advance and stored in tracking database.
• Extraction of cloud points from depth image: The role of depth sensor
is to generate the point cloud of the components, tools present in the
assembly environment under the focus of depth sensor. When the
physical object is positioned in the focus of the depth sensor, depth
can able to identified by the visible surface of the object at different
Fig. 4. VR Elements Associated with Manufacturing system. depths and stored as a point cloud [92,93]. Fig. 7 represents the
depth of object surface from depth sensor camera.
object in the real environment [20,84–87].
Point cloud tracking is organized into six steps: Reference object • Initial overlapping: The generation of environment point clouds is a
point clouds management, Extraction of cloud points from depth image, process, where the input depth image from depth sensor camera is
Initial overlapping, Matching, minimization of spatial distance, pose
Fig. 5. Architecture/Methodologies of AR Assisted Manufacturing System.
264
• Minimization of spatial distance: it is a process at where the spatial

distance between cloud points is minimized. The spatial distance of
cloud points is represented by a translation and rotation vector form,
which is used to measure the camera pose for AR.
• Pose estimation and overlaying: The goal of camera pose estimation
is to render the virtual objects and proper positioning or registering
of rendered virtual scene over the physical objects in the real
environment.
5. AR/VR for manufacturing applications
Manufacturing applications needs skilled engineers to do the oper

ation effectively with minimum errors. On job guidance and visual un
derstanding of the tasks further reduce the training time and equip the
engineer with appropriate knowledge to do the specific operation. AR/
Fig. 6. Object Identification Based on Cloud Points Method.
VR technologies have used in several manufacturing applications as
indicated in. Fig. 9.
The list of AR/VR devices and Visualization methods used for
different manufacturing applications from the cited literature is given in
Table1. Majority of manufacturing applications are focused on
improving productivity, reduce training cost and time, error rate
reduction, quality assurance, minimize the maintenance time and cost,
evaluation of product design, real time guidance, optimized inventory
management.
Fig. 7. Depth Camera Scanning.
subjected to map generation process (i.e., vertex map, normal map)

and this map generation form the environment object cloud points
(Xi) [74,94,95]. In matching process, the point clouds Ri and Xi are
overlapped initially by the process of translation and orientation of
the cloud points.
• Matching: Once the object is initially overlapped, it can be tracked by
the process of continuous matching of cloud points datasets [83,96].
Fig. 8 represents the reference object and the 3D point clouds for the
selected reference object respectively. Fig. 9. AR/VR Assisted Manufacturing Applications.
Fig. 8. Cloud Points Based Matching Algorithms.
265
Table1
List of AR/VR devices and Visualization methods used for different manufacturing applications.
Mechanical Activities Research Group Focus Tracking Method Visualization Method of
device Visualization
HMD HHD SD Optical Video
Maintenance/ Repair / M.M.L. Chang et al.[112] Enhance efficiency and productivity Natural feature-based X – – X –
Disassembly C.Y. Siew et al.[113] Enhance efficiency and safety Marker based X – – X –
A. Z. Abdul Halim[114] Shorten production time and improve Marker less tracking X – X X X
efficiency
Crystal Young and Rahul Rai Reducing manual maintenance task Marker and Model X – – X X
[115] time based
Mario Lorenz et al.[116] Support system for Maintenance worker Projector based AR X – X X X
system
A. Sanna et al.[117] Shorten production time and improve Marker based X – – X X
efficiency
Assembly Mika Hakkarainen et al.[118] Step by step guidance Marker based X – – X X
Yan Pang et al.[82] Assembly feature design Marker less X – – X X
Gabriel Evans et al.[119] Step by step guidance Marker based X – – X X
Alves et al.[120] Assistance to assembly worker Marker based X – X X X
Stefan Werrlich et al.[121] Assembly Training Marker based X – – X X
Patrick Renner and Thies Step by step guidance Marker based X – – X X
Pfeiffer[122]
Design and Evaluation Dimitris et al.[104] Product Design and Evaluation Marker based X – – – X
Marco de Sá and Elizabeth F.C Effective product design Marker based – X – – X
[123]
Min Ki Park et al.[124] Product appearance design evaluation Marker based – – X – X
Marco Schumann et al.[110] Enhance Efficiency and flexibility of Marker and Marker X – – X –
Design less tracking
Thitirat Siriborvornratanakul Increase productivity and decrease Marker based – X – – X
[125] unnecessary workloads
Shane R et al.[126] Interactive rapid prototyping Marker based – – X – X
Mechanical Activities Research Group Concentrated on Tracking Method Visualization Method of
device Visualization
HMD HHD SD Optical Video
Warehouse Management Stoltz et al.[127] Time management and reduce cost Marker less tracking X – – X –
Arnis Cirulisa and Egils Reduce error rate and decision-making Marker based X – – X –
Gintersa[128] time
Dimitris Mourtzis et al.[129] Warehouse design Marker based – X – – X
Ananthi et al.[130] Reduce error rate Marker based – – X – X
Marie et al.[131] Improve efficiency of Warehouse Marker based – X – – X
Wei Wang et al.[132] Object recognition Marker based X – – X –
Quality control Amelessodji K. E. et al.[133] Improve Quality Sensor based X – – X –
Petra Tschirner, Axel Gräser Improve Quality Sensor based X – – X –
[134]
Daniel Segovia et al.[135] Quality monitoring Sensor based X – – X –
Uros et al.[136] Quality Inspection Marker-based X – – X –
João Alves et al.[137] Improve Quality Marker-based X – – X –
Moritz et al.[138] Quality assurance Marker-based X – – X –
Plant layout Dominik et al.[139] Enhance production line planning Marker-based X – – – X
Andreas K, George-Christopher Layout design Marker-based X – – X –
V[73]
A.Y.C. Nee et al.[14] Enhance layout design Marker-based X – – – X
Katharina et al.[140] Factory design and planning Marker-based X – – X –
Jonghwan Lee et al.[141] Layout design Marker-based X – – X –
S. Jiang et al.[142] Enhance layout design Marker-based X – – X –
CNC Simulation Zhang et al. [71] Machining simulation Marker-based X – – X –
Alex Olwal et al.[143] Machining simulation Marker-based – – X X –
J. Zhang et al. [56] Machining simulation Marker-based X – – X –
J. Zhang et al. [144] Machining simulation Marker-based X – – X –
Yun Suen Pai et al.[145] Machining simulation Marker-based X – – X –
Zhang et al. [72] Validate NC codes Marker-based X – – X –
5.1. Product design and development position analysis, challenges in current techniques and evolution of
available resources. Hence, it is requiring the group of people from the
To keep the pace with advanced digital technology revolution in the different background in order to achieve the desired result. Different
industry 4.0 era, many experts have been reconsidering their product methods have been reported in literature to systemize the process of
design and development (PDD) in the manufacturing industry. More product design [97,98]. Moreover, the product design has been associ
flexible and efficient PDD methods and strategies have to be imple ated with the manufacturing of final product, thus considering how the
mented to acquire the dynamic customer’s requirements and shortened essential parts have to be manufactured and assembled together to
the production cost and time of a product. In such a context, the current obtain a complete product. Hence it is important to observe that the
product design and evaluation must be upgraded in order to maintain experts have to be maintained high level knowledge and experience
the competitiveness in the global market. which may result the increase in effectiveness of manufacturing a
Product design is one of the significant phases in manufacturing, product [99,100]. Thus, the newly joined engineers have to be trained or
which is a complex process that involves creativity, drafting, market guided in proper way, in order to complete the product design phase
266
without any flaw. Towards that goal, the manufacturing industry need 5.2. Maintenance/repair
to bear perfect learning platform, that will enable the new design en
gineers to have required knowledge and technical experience in the Maintenance system is one of the essential and challenging activity
industrial environment. The above issues could be rectified through in the manufacturing industry, where the maintenance workers have to
digitalization of design information with novel visualization. ensure the objective of maximum reliability and availability of ma
technologies which allows the design engineer to review the com chines. Nowadays, the manufacturing industries have been focusing
plete product design, available resources and identify an optimum their research activity on the significant development of optimum
design from the list of sources in a realistic scale environment maintenance system in order to achieve the maximum productivity. AR
[101–103]. is one of the promising technologies in industry 4.0 which enables many
Moreover, these visualization technologies assist the engineers from approaches in the maintenance and repair system, guiding workers by
the different groups or companies to evaluate the resulting of complete providing the auditory and visual instruction [115,152,153]. In con
product and detect the errors earlier in order to avoid the increase in ventional maintenance system, the workers mostly review the hand
production cost [104]. Having perception of the designed product is a books for instructions that consumes more time to accomplish the
most challenging one, the experts from the group of companies or complicated tasks.
different stream are experiencing more difficulties in designing a com The researchers are proposed an innovative machine service system
plex product. The current AR and VR visualization technology offer new that enables maintenance service through immersive AR/VR technology
competences in projecting the designed products in more realistic way in [154–157]. Fig. 11 represents the architecture of proposed maintenance
AR/VR environment [105–107]. Fig. 10 illustrates the AR/VR assisted system and exchange of data between the onsite technician and
product design and evaluation in manufacturing environment. head-mechanic of the industry. The proposed system is composed of
Currently, the collaboration between stakeholders in product design three steps: (i) Failure report preparation, (ii) Diagnosis and generation
and development have been increased by AR/VR immersive technology of maintenance instruction by AR and, (iii) Maintenance and evaluation.
[108–110]. Since the stake holders can visualize the final product in The first step in the system of architecture is registering the failure
more realistic way in an immersive environment, these technologies report. In manufacturing industry, the service report has to be prepared
enable the stake holders to provide feedback and suggestions quickly prior to the action of scheduled maintenance or unexpected failure.
which may lead to decrease in rework cost, and increase in productivity In the conventional maintenance system, the onsite technicians
[111]. prepare the report manually, which composes of potential problems and
Moreover, Ford industry has developed a Ford immersive vehicle action required details in the form of text. However, the proposed sys
environment (FIVE) system for PDD [146]. These FIVE systems which tem that enables the technician to use the lead of communication
converts CAD models design into a virtual vehicle model and that en technologies to prepare a digitalized data report [113,158,159]. The
ables the engineers to inspect the product in both interior and exterior malfunction of machines could be recorded in the form of text, image,
design in the absence of prototype model and also enables the design audio and video format by the onsite technician and these failure report
engineer to operate the design model virtually in immersive environ can be conveyed to the head mechanic or expert through cloud platform.
ment. In the FORD FIVE system, the VR technology has been used to Where the cloud system facilitates the communication between the ex
visualize the final product in virtual environment through VR headset perts and onsite technician by enabling the exchange of failure report to
[147,148]. Park demonstrated an AR assisted re-formable model system the experts and AR instruction to the onsite technician [160–162]. After
for evaluation of design, which allows for interactive modification of viewing the failure report, the experts will initiate the main part of
shapes, textures, colours, and user interfaces in tangible design evalua service; Identification of failure cause and generation of AR instructions.
tion. Many VR approaches have been developed to create design model The solution for the failure (i.e., AR instruction) could be conveyed to
of product in manufacturing [149–151]. Stark et al. and Perkunder et al. the onsite technician through the cloud system [163]. Fig. 12 illustrates
have been developed an immersive environment for modelling the the AR assisted Remote maintenance/Repair system.
product by merging desktop CAD with VR technologies. For example, the special machine manufacturing industries are
selling their product in all over the world for the technology trans
mission. Some industries, especially in emerging economies, less skilled
personal are working in the field of maintenance and repair. Therefore,
Fig. 10. AR/VR Assisted Product Design and Evaluation.
267
Fig. 11. Architecture of Proposed AR Assisted Maintenance System.
be rectified in order to obtain the goal of increased productivity and

reduction in maintenance cost. In a competitive environment, the
manufacturing industry have to be run in an efficient manner without
arise of down time of machines. This goal could be achieved by adopting
the AR technology for remote assistance in the field of industry repair
and maintenance [167–169].
Currently, the AR technology enables the remote experts to convey
the step-by-step instruction to the local technicians in real time through
the AR visualization device of HMD and HHD. These reduces 10% of
remote assistance time, when compared to the phone assistance method
[170–173]. Fig. 13 represents, the role of AR in all the categories of
maintenance system and also bear out the effect of AR in maintenance
system. More recently, the manufacturing industries like BMW and
Bosch are implemented this AR technology in their plant to support the
complex maintenance task [174–176]. Moreover, this technology en
Fig. 12. AR Assisted Remote Maintenance/Repair System. ables the efficient communication between the experts and on-site
technician in terms of exchange in synchronous and asynchronous in
the experts are required to fly over there to solve any severe failures in formation [177].
machines. This leads to long idle times of machines in the production
lines, resulting the lack of production and increase in repair cost [164]. 5.3. Assembly guidance
In this situation, the manufacturers could contact the experts instantly
through mobile phone for receive the instruction to solve the issues in Assembly system can be considered as a flow process, where the parts
the machines. However, these method of receiving instruction reduces can be joined systematically to obtain the required final product. As
the need of experts in physical appearance for repair operation and the sembly of any product consumes more cost time in the overall
same time, it is not inadequate for repairing the complicated mainte manufacturing, an assembly worker should have complete idea on
nance task in the machine [165,166]. In such a context, this issue must sequence of assembly operations, tools to be used and collision free
Fig. 13. Role of AR in Maintenance System.
268
assembly paths. There exist numerous literatures on virtual assembly 5.4. Warehouse management
planning but the solution cannot be accessed at the shopfloor for im
mediate reference [178–180]. Thanks to AR, now all the assembly at Warehouse management (WHM) plays an important role to achieve
tributes and task information can be visualised for a better task the goal of productivity, which comprises of activities like inventory
execution. Initially, the researchers have introduced the model of AR management, order allocation, order picking and material processing.
based assembly system, where the process of head motion sensing and Towards the goal of industry 4.0 era [196], the manufacturers have been
real-world positioning or registration of virtual content with HMD are focusing their objective to shift from conventional WHM to smart WHM
playing important role in AR system, such that a computer-generated with the help of upgraded technologies in industry 4.0. More flexible
virtual scene, containing related information, can be overlaid and sta and efficient methods and technologies have been developed in order to
bilized on a precise position on a real object in the physical environment. meet the goal of smart WHM [197,198].
Moreover, Assembly system enables the assembly line workers to Inventory management is one of the significant and challenging ac
perform complex assembly tasks and which eliminates the possibility of tivity in WHM system where the inventory checking is one of the critical
replacing manpower with automated machine to complete complex and brain-numbing activity among other activities [199]. In conven
assembly tasks. However, Motion sensing and registration could be tional inventory control, the workers have to do manual stack checking
critical and challenging one for the users in this AR based assembly by moving physically around the warehouse of the industry and then
system. From then on, many researchers have focused and reported the they explored the details of stack, status of pair parts (i.e., it is ready to
works on AR based assembly activities. Molineros and Sharma [181] go or it is in the good condition) for the inventory control management.
suggested an effective visualization scheme using assembly graphs to These stacking activities are carried out by paper-based methods which
control the augmented virtual information. They proposed a series of AR resulting more time consumption and possibilities of human error in
based assembly system that specifically concentrated on the enactment stacking [200]. Moreover, the workers have to be provided with
of an AR assembly system to a real configuration setting in industry. This cost-intensive training to obtain the desired outcomes. Realizing that,
could be achieved by integrating design software tools these stacking systems have to be streamlined in order to obtain the
(CAD/PLM/PDM) for assembly process in assembly environment. L Hou effective inventory control management. Therefore, the manufacturers
et al. [182] addressed the usage of AR aid educational tool for engi are thriving to reconsider their conventional system into smart in
neering assembly tasks. Researchers proposed a simplified model of AR ventory control system with emerging technologies that enables to
based assembly system, such as assembly guidance, assembly training obtain the outcomes of industry [201]. Currently, the manufacturing
and assembly simulation, planning and design [63,173,183]. Makris industries are utilizing the digital technologies like barcode scanning
et al. [184] suggested an assembly system with AR technologies and this system and AR based technology for inventory control management
system provides the visual instruction to the assembly workers in the system [130]. The arrival of bar code scanning system has built a new
real environment, where they used CAD environment for generation of innovation and revolution in inventory control system. Barcode scanner
rendering models or instructions. Sukan et al. [185] proposed an is a device, which provides the detailed information related to the in
interactive system referred as ParaFrustum, that enables the users to ventory checking and stacking control management and these digital
visualize the target object from certain viewpoints in the current envi details have been obtained by scanning the bar code in the inventory
ronment and it also eliminates occlusions. Wang et al. suggested a [202,203]. However, despite the positive effects, the barcode scanners
real-time occlusion managing solution in assembly system, that elimi are too expansive when the demand for hardware increases and also not
nates the occlusions in the assembly environment [186]. Michela and pertinent to read more than 15 ft around the scanner.
Gino proposed an AR prototype system to assist the operator during AR technology adoption at the inventory control management can
panel fitting operations in car body assembly process, by providing in eliminate the bottle necks on achieving the effective stacking, ordering
structions to rectify the problems in the alignment such as gap and and moving the parts from inventory to the respective shopfloor of the
flushness[187]. Lai et al. presented an approach combining AR and industry [204]. Fig. 14 represents, the major challenges in inventory
R-CNN techniques to aid assembly workers in tool identification and control that are being improved by AR technology and it is allows the
task execution in the assembly station[188]. Zhang et al. [189] pre manufacturing industries to keep pace with the digital technology rev
sented an approach to execute the RFID technology in assembly task olution in industry 4.0. For example, the AR technologies allows the
operations in an AR environment, aiming to enable the workers for worker to identify and reach the target object in less time by means of
visualization of just-in-time instruction rendering and navigation of virtual digital content overlaying on the targeted object. Moreover, the
intuitive instruction. Boeing has carried out the investigation on the AR AR technology provides the foremost effects like the reduction in human
aid assembly operation and reported the effectiveness of the system error, palletization error removal and minimization of damage risk in a
[190]. Chu and Ko proposed new AR functionalities to aid manual as WHM [205]. These reduction in error rates are significantly improved by
sembly in an occluded situation. The emphasis is on proving the effec increasing the visibility of objects in a warehouse.
tiveness of various assistance information in AR, such as the assembly In order-picking, the time management is one of the most labour-
interface, and assembly worker hand movements [191]. Andersen et al. intensive functions because the time required to handle the items and
[192] developed the system of POC (proof -of-concept) to estimate the the mobility time between the work centres are the main challenges in
human pose by matching the raw captured image features, where the picking [206–208]. Order picking is referred as the most crucial stage in
features include corners and edges. Many research has been conducted the WHM, which can influence 50–60% of total operating cost in
on AR aid assembly activities and analysed with printed manual as warehouse. However, the AR technology provides the direction to
sembly approach [193–195]. The objective of this research is focused to identify the object that leads to reduction in searching time and picking
examine how effectively the AR visualization approach can be used to of appropriate item [209]. Some researchers are developed AR based
guide the workers in performing the assembly activities manually. The pick-by-vision system which work usually with visualization devices like
outcomes of the research shows that the AR aid method were more HMD, HHD and projector-based devices. Therefore, an integration of AR
effective than the paper-based guidance in assembly task. Moreover, AR technology with order picking can supports the reduction of strain in
aid approach has to be more suitable to complete difficult task as workers which leads to increase in rate of overall production.
compared to conventional paper-based instructions, whereas for simple For the transportation, the optimized planning of loading, unloading
tasks, the paper-based method is not differed significantly from the AR and vehicle utilization have to be formulated significantly [210], in
aid instructions. order to obtain the feasible transportation with in the industry. These
can be possible by accumulating the more digital information in the
separate planning software. However, the AR technology has been
269
Fig. 14. AR Aid Inventory Control.
utilized for minimizing the package identification time and routing assurance. These reports will allow the manufacturers to know the
strategies for transporting the items from the WHM of industry. number of defective products and to implement significant decisions in
real time.
5.5. Quality control

5.6. CNC simulation
Quality control is one of the essential elements when it is come to any
service or product in manufacturing industries. In current marketing Many efficient techniques and algorithms were developed by the
scenario, the manufacturing industries are focusing to produce the high- experts for both physical and geometrical simulation of CNC system, and
quality product with increased productivity. In other words, the quality many 3D graphics-based CNC machining simulation were commercially
of products has to fulfil the customer needs, in order to withstand the available in the system. In CNC simulation, the workers must have the
competition in the current market. In manufacturing industry, the ability to analyse the machining information and spontaneous decision
quality of product has been monitored usually by visual inspection implementation are very crucial. These have been achieved by the sys
approach. Despite the advance technologies adoption in the automation tem of high-speed computers equipped with features of 3D graphics
system, the current system still allows the worker to carry out the quality simulation and rapid accessibility of database.
checking process in the manufacturing industries. The final stage of Currently, the AR technology provides real world augmented with
product completion lifecycle, quality checking tasks are not completely machining information for the users to carry out the critical CNC
carried out by the machines, partial work has been carried out by the simulation process by projecting the virtual content machining infor
workers. However, the workers in the quality checking could not ensure mation over the real environment object in the manufacturing shop floor
the quality of products completely by visual inspection approach. [219]. Moreover, the user’s knowledge in CNC simulation has been
Therefore, the manufacturing industries adopted the advance technol improved by projecting the augmented scene of 2D or 3D models of parts
ogies to modernize quality assurance system [211–214], that will enable and tools, machining information, CNC programs and cutting parame
the optimum results of quality. The role of AR in the current quality ters [220,221]. The benefits of using AR in these applications increase
system is more significant, which provides the helpful information of the ability of the user to higher knowledge while compared to
defective product components in less time to the workers [135,215]. computer-based and VR-based simulation systems. Many researchers
With AR technology, the product quality checking can be carried out have undertaken on the application of AR technologies for
by the workers, where the workers are moving around the outlet of information-intensive and time-consuming manufacturing processes.
machines by wearing the AR glasses [136]. Besides, the IoT sensors are AR technology has been applied in CNC machining in a relatively small
embedded on the outlet products, that provides the details of product in number of applications. This is most likely owing to the fact that the
terms of quality constraints assurance and this useful information has processing operations are machine-centric, and there are fewer human
been displayed over the AR screen of the workers. As a result, the quality elements engaged at this point [222]. Olwal et al., proposed a
of particular product has been ensured by the workers in less time and projection-based AR display techniques, that allows the workers to
also, it eliminates unintentional mistakes by the workers[216]. visualize the machining information data which is displayed over the
Moreover, the AR technology provides significant training for new real machining scene [143,223]. The AR technology, which collects
workers to carry out the quality inspection in quality control system machining process data and cutting forces, then displays them on a
[133,217,218]. Therefore, the new workers can be trained by displaying holographic optical element window and which is mounted on the
the guidelines of quality inspection over the AR screen and these AR lathe’s sliding door. Weinert et al. [224] have proposed AR based CNC
displayed information will guide the new workers at every step of in simulation system, where the machining simulation has been carried out
spection process. As a result, the overlooking of product components can for the 5-axis CNC machines. In the proposed system, ARToolKit-based
be eliminated and also, the product has been checked accurately tracking was used to track the cutter’s movement in relation to the
matching with the quality features. In addition, AR assisted quality machine table, and dexel boards were used to model the workpiece in
assurance has been carried out at every step-in quality inspection, that the system. This simulation system has been used to forecast the colli
resulting, the assured quality products only allowed to move further sions between the head stock and the workpiece and also used to esti
subsequent process in quality control. mate the cutting forces in the machining process of the system.
In quality report generation, the displayed information’s on the AR Zhang J et al. [144] have developed the AR-assisted in situ CNC
screen are gathered and this will be used to create the report of quality Machining Simulation system, where the proposed method has been
270
applied on 3 axis CNC machine system to carry out the machining op environment through VR devices.
erations in the system. This AR based CNC simulation system comprises Currently, the manufacturing industries are utilizing the AR
of three main elements, such as vertical CNC machine (3-axis), a Visu immersive technology to design the PLP system in the manufacturing
alization device and human-machine interfaces, which incorporates the shop floor [228]. This AR technology based PLP system provides the real
CCD camera for the capturing the image and high-end PC for processing environment of shop floor, where the designed for factory layout plan
the raw image and generated the virtual scene model to superimpose ning (FLP) models are superimposed over the real environment of floor
over the real environment of the machining system. The display device factory [139,229,230]. This AR technology allows the design engineer
in the in-situ system might be either a head-mounted display or to lay out the virtual PLP 3D models in the real environment of the
desktop-based monitor system. In situ CNC simulation system, the manufacturing shop floor, during which the design engineer will com
hybrid tracking method has been used to register the position of the plete the evaluation of resulting layout arrangement. Finally, the ability
cutter in the coordinate measuring system and also used to enable the to simulate work flow and material movement at a human scale in the
extraction of constraints from the given NC codes. shop floor and in conjunction with a virtual modified production layout
This proposed system is intended for inexperienced machinists who allows for unprecedented verification. This might help user to prevent a
wants to change NC codes and see how the CNC machine responds number of blunders before making modifications on the shop floor.
without risking tool damage or machine failure. Where the alarms can
be presented in the augmented display to warn the users of threats and 6. Challenges and research needs in AR/VR associated
faults, according to simulation results. The machining simulation has manufacturing system
been carried out in-between a real cutter and a virtual model of work
piece on 3-axis CNC system, and then the results are exhibited to the Despite the increased interest in AR and the significant amount of
users via a video see-through rendering approach. In this system, the research and developments, there are still a number of bottlenecks and
material removal simulation processes has been displayed to the user to issues that need to be addressed. Many researchers have conducted a
aid in the inspection and assessment of machining operations prior to comprehensive assessment of AR-based manufacturing applications
executing the real machining, which further decreases the material through relevant articles, highlighting the numerous challenges that
waste and power usage. Furthermore, the users are enabled to investi must be addressed before the technology is fully adopted by the in
gate the physical features of the machining processes based on dustry. In this section, various aspects are used to categorise the Chal
machining conditions, which are augmented to the real environment of lenges and limitations of AR, that define the current state of the art in AR
the machining system. The suggested system’s use of video see-through using the following criteria: Technological based, Organization based
technology allows various users to focus on different machining infor and Environmental based. The key challenges of AR are illustrated in
mation and process, which can be advantageous during training with Fig. 15.
multiple learners. Tracking and rendering (i.e., challenges in sensors and sensing
technologies for human position estimation and environment percep
5.7. Layout planning tion), processing speed, and Ergonomics are the critical technical
challenges.
Nowadays, the rapid changes in customer demand, both in terms of
quantity and variety, the necessity to design and construct a new factory
plan or reconfigure an existing one has been increased. Plant layout 6.1. Tracking
planning (PLP) is a critical activity in the manufacturing industry, as
design engineers must ensure that the physical arrangement of facilities Tracking and registration are the key enabling technologies of AR.
in the industrial plant is efficient. The main objective of layout process The virtual and real models of objects cannot be effectively blended
planning is to position the facilities by considering space, foundation, without proper tracking and registration Various sensing devices have
and surrounding areas, as well as laws and regulations relating to er been used to track the user’s posture in order to calculate their per
gonomics and safety, such a way that it facilitates efficient material flow spectives, which is necessary for static and dynamic registration to work.
for the desired product families and quantities. Whereas the industrial Methods for predicting human pose and estimation of environment
facilities are referred as machines, processing equipment, perception demands advanced sensors and sensing technologies.
manufacturing cell, department and warehouse in the shop floor. The One of the most important tasks in AR based Manufacturing is to
Industrial facility arrangement requires high cognitive effort to design create a real-time tracking system that is suitable for industrial sce
and simulate the layout planning in the manufacturing shop floor [225, narios, where the characteristics (e.g., poorly textured product with
226]. many smooth surfaces, strong light variation, very small object sizes,
The researchers proposed an innovative technique in PLP that en etc.) present a challenge to the existing techniques. To this day, marker-
ables the result of optimum layout planning in manufacturing shop floor
through VR and AR technology [141,227]. Initially, the VR immersive
technology have been introduced to carry out the PLP tasks in the
manufacturing industry. Since, VR technology is one the predominant
tool for PLP tasks and which allows the design engineer to ensure the
optimum PLP design by simulating the PLP task in the virtual environ
ment of the factory floor [20]. These VR technologies are quite similar to
the conventional on-line layout planning platform for the design engi
neer. However, these PLP VR system are particularly designed for
simulation of layout planning, that enables the design engineer to refine
the theoretical results before implement to the actual shop floor.
Moreover, the entire factory environment has been simulated virtually
in the virtual environment, as a result, the identification of deviation
from the real factory environment and this enables the design engineer
to modify the current design of PLP. However, the VR technology in
cludes time consuming process for creation of detailed 3D models of
entire factory floor space to simulate the PLP tasks in the virtual Fig. 15. Key challenges of AR in Manufacturing system.
271
based tracking is the most extensively used and popular tracking method 235] proposed a closed-loop registration strategy in which the intended
in AR based Manufacturing. Because of its precision, versatility, and synthetic imagery (actual and virtual) was implemented directly as the
convenience of use, several researchers adopt a marker-based tracking objective. Yang et al. [236] developed a single-shot method for
technique in AR assisted manufacturing systems. In this tracking, A obtaining extrinsic calibration of a generically constructed RGB and
camera captures a predetermined marker, which is then identified and depth camera rig using partially known metric information from the
tracked on each frame. The marker’s location and orientation in relation observed scene. The key way to addressing the accuracy issue at this
to the camera are then calculated. Finally, the calculated pose is used to time is to implement improved tracking systems, specific registration
render a virtual model over the real-world objects. Wang et al. [194] tactics, and effective calibration approaches.
investigated tracking tool models using non-square visual markers and a The latency problem refers to alignment errors of the virtual models
colour-based marker such that operators may manage scanned articu caused by the time gap between when an observer moves and when the
lated real objects naturally. Marker-based approaches for camera image that corresponds to their new position is displayed. Moreover, the
posture estimation are extremely accurate since markers can be con head rotations and user operations become more rapid, this problem has
structed with optimal tracking patterns. These methods, however, are become more serious.
inconvenient in some circumstances, particularly for small or irregular
items on which markers cannot be mounted. Furthermore, when the 6.3. Processing speed
marker is obstructed, the tracking can be lost.
In recent years, marker-less tracking has been regarded as a desirable According to Stoltz et al. [127] and Porcelli et al. [237], AR systems,
successor to marker-based tracking. SLAM (Simultaneous localization particularly HMDs, lack appropriate processing power, resulting in poor
and mapping) technology is a well-known technique for creating a map user experience and performance. AR devices require a lot of computing
in an unprepared location while maintaining track of its present location power to run real-time scenarios with very high-quality digital infor
using extracted scene elements [195]. The extracted information from mation and tracking algorithms. Although Moore’s Law states that the
the real environment scene element, such as floors and intersections of density of transistors are doubled for every two years on integrated
walls and edges of an object, rather than identifying a specific marker, circuits [238], this means that the processing speed of AR devices rises
and then uses these features to estimate the object’s location and with time, but the amount of compute power demands increases when
orientation. SLAM technology is a self-navigation technique that uses AR is applied in more industrial applications. Cloud computing, as
sensing data (e.g., GPS, Lidar, visual and odometer) to build or update a proposed by [239], appears to be an acceptable option because it allows
map of an unknown area while keeping track of an agent’s location for better resource and content use. To provide smooth content distri
inside that map. Moreover, the marker-less tracking offers non-invasive bution, it requires a high-quality network connection and security.
alternatives to marker-based technologies. This category includes Fernández et al., [11] proposed a cloud-based AR system. According to
Microsoft Kinect [231], which is widely used. It has a depth sensor that (Bravo et al., 2021) [240], Cloud-based AR systems require strong
detects image depth and uses that information to generate network capabilities to translate data from the cloud to the visualisation
skeleton-based human position estimate. Intel RealSense and Structure device. The current 5 G internet network provides a sufficiently
Sensor are two commercially available depth cameras [93]. Without the low-latency and high-bandwidth cellular connection for AR to operate
use of depth information, it is now possible to effectively estimate the effectively.
human position directly from a digital image thanks to recent break
throughs in artificial intelligence (AI), particularly deep learning[4, 6.4. Ergonomics challenges
232]. However, SLAM methods do not consider the impact of updated
3D structures corresponding to moving object, the target working scene The ergonomic design of a product has an impact on user acceptance.
must be kept static, or the recovered 3D models will quickly become Not only its hardware ergonomics, such as weight and FOV [39,241],
obsolete, and the system would have to start again for each frame of the but also its software ergonomics, such as camera angles and light
obtained video. The first real-time multi-person method used to deter sensitivity of AR algorithms. Long-term usage of AR devices (particu
mine the human posture from a single photograph was Open Pose [93], larly HMDs) leads to diplopic vision, depth perception problems, fatigue
which was first released in 2017. It can detect crucial spots on the and visual discomfort [242]. Moreover, AR minimises eye and neck
human body, hands, and face. Because it does not require dedicated movement during user working condition.
cameras, Open Pose is a promising option for human pose estimation.
However, it is computationally costly and requires high-end GPUs to 6.5. Organization challenges
achieve real-time person tracking, which is one of its constraints.
Another reliable tracking method is sensor-based tracking, which has The user acceptance is also linked to privacy and security [127]. The
been frequently used in AR assembly systems even before marker-based monitoring of AR users and their gestures, speech, and actions observed
tracking was introduced. The tracking principle uses many types of by the AR system is required for training assessment and task/error
sensors, such as magnetic, inertial, optical, acoustic, and/or mechanical tracking. Researchers have evaluated AR in the workplace and identified
sensors, whereas vision-based tracking uses computer vision techniques. several issues and difficulties [243]. According to several field studies,
In general, sensor-based trackers used in AR applications must have high AR has a negative influence on performance due to software restrictions,
precision, low latency and jitter, and be able to operate reliably in a HMD comfort, privacy concerns, and high cognitive load. In addition,
variety of environments [233]. Optical sensors, for example, provide the level of work complexity is linked to AR usage [244]. AR systems,
excellent accuracy, versatility, and low latency, but they are expensive, like any other network, are susceptible to cyber. As a result, companies
bulky, and require extensive calibration. may face cybersecurity issues in protecting their own intellectual
property (IP) as well as the integrity of their gear and processes.
6.2. Registration
6.6. Environmental challenges
The fundamental goal of tracking in AR is to make it easier to register
virtual models and/or manual user instructions with the right position in The environment of an organization employing AR, according to
the augmented environment. In this domain, there are two major con [245], may influence how the technology is employed. Third-party data
cerns: accuracy and latency. The term "accuracy" refers to the error integration, industry-wide standardisation of AR solutions, the
caused by sensor misalignment, inaccuracy of sensory device, and/or employment regulatory environment, or the need for external help, to
inaccurate tracking results [234]. To eliminate errors, Zheng et al. [231, name a few [127]. It is also necessary to examine the evolving behaviour
272
of employing AR in a system in order to gain a better understanding of its products to satisfy the customer needs in the market. The manufacturing
environmental influence at a higher level. industries are facing more complexities for sharing their product infor
mation to the various streams of multiple workers in the industrial
environment. The approach of cloud-based collaboration has to be
6.7. Research needs developed to allow the multiple workers to share their informative
knowledge and experience among the various stream of industries from
Research and development efforts must be carried out in order to different locations. The following are a few research possibilities in the
achieve improved VR/AR technology for the manufacturing applica domain of collaborative and sharing:
tions in industrial environment. The figure represents the research needs
for AR/VR immersive technologies in manufacturing applications, such • Improve the flexibility of multiple users in the VR/AR environment,
as improving the virtual interface for realism, recognising worker be that leads to multiple users precisely register to the same coordinate
haviours, collaboration and sharing among various streams in industrial system and keep their shared information synchronized on a regular
environment. More research and development activities are required to basis.
improve VR/AR technology to increase the realism of virtual interface in • Approaches for enabling the users to have natural, effective distance
a VR/AR environment, such as upgradation of optical system for high collaborations in the VR/AR Environment.
fidelity and low tendency of visual interface, adoption of advanced 3d • Develop the methodologies for simultaneous interface and concur
auditory feedback system, improvement of multichannel haptic devices, rent interactions among collaborators in VR/AR Environment.
upgradation of locomotion interface (realistic running/walking experi
ence), improvement of marker-less tracking method, optimum algo 7. Conclusion
rithms for object identification and recognition, Methods of human pose
estimation and suitable approach for effective human – environment This paper presents a detailed review of AR/VR research and de
interaction. velopments in various applications of manufacturing activities,
In manufacturing industries, the worker activity recognition has including few challenges and research opportunities of AR/VR immer
been considered for evaluating the performance of workers. Fig. 16 il sive technologies associated with current manufacturing systems. ter
lustrates the various research opportunities of AR/VR technologies in minologies and concepts in AR/VR technologies for manufacturing
the current manufacturing system. To study the worker behavior, activities have been discussed, and challenges in tracking and registra
research has to be carried out in the following aspects. tion techniques for environmental perception and human pose estima
tion were reviewed with detailed discussion. Moreover, the outlines of
• Techniques for real-time worker activity recognition- has to be AR/VR elements, such as hardware and software elements utilized for
researched to address the current state of workers activity in the various manufacturing task were demonstrated comprehensively.
manufacturing process that enables to calculate the operation time Further, this review article includes the methodologies of AR/VR
for evaluation purpose. manufacturing system with detailed discussion on object identification
• Methods for worker facial recognition, techniques have to be and tracking methods. According to the detailed analysis of compre
developed for identification of the worker’s emotional state during hensive review conducted, it is likely to conclude that the adoption of
the manufacturing process. immersive technologies of AR/VR system in manufacturing industries is
• Virtual guidance techniques have to be developed with consideration persistently increased. Moreover, the potential application of this tech
of worker’s current state for improve the interaction efficiency of nology facilitates the manufactures to adopt other recent advancement
guidance to the workers. technologies of industry 4.0, such as IoT, cloud computing and block
chain for their manufacturing operations in the industrial environment.
Currently, the industries are at demand of fabricating new variant of
Fig. 16. Research opportunities of AR/VR technologies in the current manufacturing system.
273
It is foreseeable that the AR/VR technologies will play significant role in [24] Doolani S, Wessels C, Kanal V, Sevastopoulos C, Jaiswal A, Nambiappan H, et al.
A review of extended reality (XR) technologies for manufacturing training.
the next decade, and this technology wave to enhance the human ma
Technologies 2020;8:77. https://doi.org/10.3390/technologies8040077.
chine interaction through real time instruction exchanges in a product [25] Gong L, Fast-berglund ÅSA, Johansson B. A Framew Ext Real Syst Dev Manuf
development life cycle. 2021:9. https://doi.org/10.1109/ACCESS.2021.3056752.
[26] Almaguer CAG, Acosta ACA, Murueta POP. Gamification and simulation: distance
education for industrial engineering students. 2021 Mach Learn Digit Technol
Funding Educ Innov Work 2021:1–7.
[27] Pappas G. Extended Reality (XR) \& Gamification in the Context of the Internet of
Things (IoT) and Artificial Intelligence (AI). Michigan State University, 2021.
The research/review presented in the article is partially supported by
[28] Maddikunta PKR, Pham QV, Deepa BP, Dev N, Gadekallu K. TR, et al. Industry
DST-ARTPARK with grant No. UG-11. 5.0: A survey on enabling technologies and potential applications. J Ind Inf Integr
2022:26. https://doi.org/10.1016/j.jii.2021.100257.
[29] He F., Ong S.K. A Mobile Solution for Augmenting a Manufacturing Environment
Declaration of Competing Interest with User-Generated Annotations 2019. https://doi.org/10.3390/info10020060.
[30] Chauhan VP, Kayasth MM. Object detection and pose tracking in augmented
reality. Int J Sci Res 2014;3:2623–9.
The authors declare that they have no known competing financial [31] Katiyar A., Kalra K., Garg C. Marker Based Augmented Reality 2015;2:441–445.
interests or personal relationships that could have appeared to influence [32] Bernd S, De Blandine L. An augmented reality system for training and assistance
to maintenance in the industrial context. J Winter Sch. Comput Graph 2003;11:
the work reported in this paper.
101–10.
[33] Kaviyaraj R, Uma M. A survey on future of augmented reality with AI in
References education. Proc - Int Conf Artif Intell Smart Syst ICAIS 2021 2021:47–52. https://
doi.org/10.1109/ICAIS50930.2021.9395838.
[34] Kulli‾ya‾t al-Taqni‾yah al-ʻUlya‾ (United Arab Emirates), Institute of Electrical
[1] Wang X, Ong SK, Nee AYC. A comprehensive survey of ubiquitous manufacturing
and Electronics Engineers. 2019 Sixth HCT Information Technology Trends (ITT).
research. Int J Prod Res 2017;7543:1–25. https://doi.org/10.1080/
2019:305–8.
00207543.2017.1413259.
[35] Cruz-Neira C, Sandin DJ, DeFanti TA. Surround-screen projection-based virtual
[2] Phuyal S, Bista D, Bista R. Challenges, opportunities and future directions of
reality: the design and implementation of the CAVE. Proc ACM SIGGRAPH 93
smart manufacturing: a state of art review. Sustain Futur 2020;2:100023. https://
Conf. Comput Graph 1993:135–42.
doi.org/10.1016/j.sftr.2020.100023.
[36] Brown C, Hicks J, Rinaudo CH, Burch R. The use of augmented reality and virtual
[3] Ong SK, Yuan ML, Nee AYC. Augmented reality applications in manufacturing: a
reality in ergonomic applications for education, aviation, and maintenance. Erg
survey. Int J Prod Res 2008;46:2707–42. https://doi.org/10.1080/
Des 2021:1–9. https://doi.org/10.1177/10648046211003469.
00207540601064773.
[37] Wang Q. Virtual Manuf Syst 2005;vol. 16. https://doi.org/10.2495/978-1-85312-
[4] Karomati D, Chu C, Wang L. Systematic literature review on augmented reality in
998-8/09.
smart manufacturing: collaboration between human and computational
[38] Itoh Y, Langlotz T, Sutton J, Plopski A. Towards indistinguishable augmented
intelligence. J Manuf Syst 2020. https://doi.org/10.1016/j.jmsy.2020.10.017.
reality: a survey on optical see-through head-mounted displays. ACM Comput
[5] Nee A.Y.C., Ong S.K. Virtual and augmented reality applications in
Surv 2021:54.
manufacturing. vol. 46. IFAC; 2013. https://doi.org/10.3182/20130619–3-RU-
[39] Palmarini R, Erkoyuncu JA, Roy R, Torabmostaedi H. A systematic review of
3018.00637.
augmented reality applications in maintenance. Robot Comput Integr Manuf
[6] Ong SK, Wang ZB. Augmented assembly technologies based on 3D bare-hand
2018;49:215–28. https://doi.org/10.1016/j.rcim.2017.06.002.
interaction. CIRP Ann - Manuf Technol 2011;60:1–4. https://doi.org/10.1016/j.
[40] Santos P, Fleisch T, Graf H, Stork A. 3D interactive augmented reality in early
cirp.2011.03.001.
stages of product design. 10th Int Conf Hum-Comput Inter 2003:1203–7.
[7] Hassan S, Yoon J. Haptic assisted aircraft optimal assembly path planning scheme
[41] Cheng D, Wang Y, Hua H, Talha MM. Design of an optical see-through head-
based on swarming and artificial potential field approach. Adv Eng Softw 2014;
mounted display with a low f-number and large field of view using a freeform
69:18–25. https://doi.org/10.1016/j.advengsoft.2013.12.003.
prism. Appl Opt 2009;48:2655–68.
[8] Yew DNAWW, Nee SKOAYC. Haptic and visual augmented reality interface for
[42] Zheng Z, Liu X, Li H, Xu L. Design and fabrication of an off-axis see-through head-
programming welding robots 2017. https://doi.org/10.1007/s40436–017-
mounted display with an x–y polynomial surface. Appl Opt 2010;49:3661–8.
0184–7.
[43] Caruso G, Re GM. Interactive augmented reality system for product design
[9] González C, Solanes JE, Munoz A, Gracia L, Girbés-Juan V, Tornero J. Advanced
review. Eng Real Virtual Real 2010 2010;vol. 7525:75250H.
teleoperation and control system for industrial robots based on augmented
[44] Stutzman B, Nilsen D, Broderick T, Neubert J. MARTI: mobile augmented reality
virtuality and haptic feedback. J Manuf Syst 2021;59:283–98.
tool for industry. 2009 WRI World Congr Comput Sci Inf Eng 2009;vol. 5:425–9.
[10] Blaga A., Tamas L. Augmented Reality for Digital Manufacturing. MED 2018 -
[45] Xin M, Sharlin E, Sousa MC. Napkin sketch: handheld mixed reality 3D sketching.
26th Mediterr Conf Control Autom 2018:173–8. https://doi.org/10.1109/
Proc 2008 ACM Symp Virtual Real Softw Technol 2008:223–6.
MED.2018.8443028.
[46] Lotsaris K, Fousekis N, Koukas S, Aivaliotis S, Kousi N, Michalos G, et al.
[11] Fraga-Lamas P, Fernández-Caramés TM, Blanco-Novoa Ó, Vilar-Montesinos MA.
Augmented Reality (AR) based framework for supporting human workers in
A review on industrial augmented reality systems for the industry 4.0 shipyard.
flexible manufacturing. Procedia CIRP 2020;96:301–6. https://doi.org/10.1016/
IEEE Access 2018;6:13358–75. https://doi.org/10.1109/ACCESS.2018.2808326.
j.procir.2021.01.091.
[12] Fernando L, Cardoso DS, Cristina F, Queiroz M, Zorzal ER. A survey of industrial
[47] Ha J, Cho K, Rojas FA, Yang HS. Real-time scalable recognition and tracking
augmented reality. Comput Ind Eng 2019:106159. https://doi.org/10.1016/j.
based on the server-client model for mobile augmented reality. 2011 IEEE Int
cie.2019.106159.
Symp VR Innov 2011:267–72.
[13] Billinghurst M, Clark A, Lee G. A survey of augmented reality. Found Trends
[48] Saakes D, Stappers PJ. A tangible design tool for sketching materials in products.
Hum-Comput Inter 2014;8:73–272. https://doi.org/10.1561/1100000049.
AI EDAM 2009;23:275–87.
[14] Nee AYC, Ong SK, Chryssolouris G, Mourtzis D. Augmented reality applications in
[49] Kitagawa M, Yamamoto T. 3D puzzle guidance in augmented reality environment
design and manufacturing. CIRP Ann - Manuf Technol 2012;61:657–79. https://
using a 3D desk surface projection. 2011 IEEE Symp 3D Use Interfaces 2011:
doi.org/10.1016/j.cirp.2012.05.010.
133–4.
[15] Heilig M.L.U.S. Patent No. 3,050,870. Google Patents 1962;870.
[50] Gavaghan KA, Peterhans M, Oliveira-Santos T, Weber S. A portable image overlay
[16] Aukstakalnis S., Blatner D. Silicon mirage: The art and science of virtual reality
projection device for computer-aided open liver surgery. IEEE Trans Biomed Eng
1992.
2011;58:1855–64.
[17] Bamodu O., Ye X.M. Virtual Reality and Virtual Reality System Components. Adv.
[51] Schöning J, Löchtefeld M, Rohs M, Krüger A. Projector phones: a new class of
Inf. Comput. Technol. Eng. Manuf. Environ. Eng., vol. 765, Trans Tech
interfaces for augmented reality. Int J Mob Hum Comput Inter 2010;2:1–14.
Publications Ltd; 2013, p. 1169–72. https://doi.org/10.4028/www.scientific.
[52] Willis KDD. Character interaction with handheld projectors. Proc fifth Int Conf
net/AMR.765–767.1169.
Tangible, Embed embodied Interact 2010:445–6.
[18] Jayaram S, Vance J, Gadh R, Jayaram U, Srinivasan H. Assessment of VR
[53] Yoshida T, Shimizu K, Kurogi T, Kamuro S, Minamizawa K, Nii H, et al. RePro3D:
technology and its applications to engineering problems. J Comput Inf Sci Eng
full-parallax 3D display with haptic feedback using retro-reflective projection
2001;1:72–83.
technology. 2011 IEEE Int Symp VR Innov 2011:49–54.
[19] Berg LP, Vance JM. Industry use of virtual reality in product design and
[54] Valentini PP. Interactive virtual assembling in augmented reality. Int J Inter Des
manufacturing: a survey. Virtual Real 2017;21:1–17. https://doi.org/10.1007/
Manuf 2009;3:109–19.
s10055-016-0293-9.
[55] Ong CJCSK, Zhou AYCNYQ. Haptic-based interactive path planning for a virtual
[20] Gong L, Berglund J, Fast-Berglund Å, Johansson B, Wang Z, Börjesson T.
robot arm 2010:113–123. https://doi.org/10.1007/s12008–010-0088–2.
Development of virtual reality support to factory layout planning. Int J Inter Des
[56] Zhang J, Ong SK, Nee AYC. AR-assisted in situ machining simulation: architecture
Manuf 2019;13:935–45. https://doi.org/10.1007/s12008-019-00538-x.
and implementation. Proc 7th ACM SIGGRAPH Int Conf Virtual-Real Contin Its
[21] Milgram P. A taxonomy of mixed reality visual displays. Ind Eng 2011:1–14.
Appl Ind VRCAI 2008. https://doi.org/10.1145/1477862.1477897.
[22] Milgram P, Kishino F. A taxonomy of mixed reality visual displays. IEICE Trans
[57] Yang Y-I, Yang C-K, Chu C-H. A virtual try-on system in augmented reality using
Inf Syst 1994;77:1321–9.
RGB-D cameras for footwear personalization. J Manuf Syst 2014;33:690–8.
[23] Azuma RT. A survey of augmented reality. Presence Teleoperators \ Virtual
Environ 1997;6:355–85.
274
[58] Klein G, Murray D. Parallel tracking and mapping for small AR workspaces. 2007 [89] Birdal T, Ilic S. Point pair features based object detection and pose estimation
6th IEEE ACM Int Symp Mix Augment Real 2007:225–34. revisited. Proc - 2015 Int Conf 3D Vis, 3DV 2015 2015:527–35. https://doi.org/
[59] Park JS. AR-Room: A rapid prototyping framework for augmented reality 10.1109/3DV.2015.65.
applications. Multimed Tools Appl 2011;55:725–46. https://doi.org/10.1007/ [90] Drost B, Ulrich M, Navab N, Ilic S. Model globally, match locally: efficient and
s11042-010-0592-1. robust 3D object recognition. Proc IEEE Comput Soc Conf Comput Vis Pattern
[60] Yang SS, Nasr N, Ong SK, Nee AYC. Designing automotive products for Recognit 2010:998–1005. https://doi.org/10.1109/CVPR.2010.5540108.
remanufacturing from material selection perspective. J Clean Prod 2017;153: [91] Herrmann S. Object detection with Microsoft A comparison between image and
570–9. https://doi.org/10.1016/j.jclepro.2015.08.121. point cloud 2021.
[61] Wang ZB, Ong SK, Nee AYC. Augmented reality aided interactive manual [92] Newcombe R a, Molyneaux D, Kim D, Davison AJ, Shotton J, Hodges S, et al.
assembly design. Int J Adv Manuf Technol 2013;69:1311–21. https://doi.org/ KinectFusion slides. IEEE Int Symp Mix Augment Real 2011:127–36.
10.1007/s00170-013-5091-x. [93] et al. RA. Real-Time dense surface mapping and tracking with kinect. Ismar 2011:
[62] Ong SK, Pang Y, Nee AYC. Augmented reality aided assembly design and 127–36.
planning. CIRP Ann - Manuf Technol 2007;56:49–52. https://doi.org/10.1016/j. [94] Magistri F, Chebrolu N, Stachniss C. Segmentation-based 4D registration of plants
cirp.2007.05.014. point clouds for phenotyping. IEEE Int Conf Intell Robot Syst 2020:2433–9.
[63] Vasilis S., Lydia A., Kosmas A. Applying the Teaching Factory paradigm and https://doi.org/10.1109/IROS45743.2020.9340918.
Augmented Reality technology for operator training in assembly operations n.d. [95] Zhang J, Ong SK, Nee AYC. RFID-assisted assembly guidance system in an
[64] Sääski J, Salonen T, Hakkarainen M, Siltanen S, Woodward C, Lempiäinen J. augmented reality environment. Int J Prod Res 2011;49:3919–38. https://doi.
Integration of design and assembly using augmented reality. IFIP Int Fed Inf org/10.1080/00207543.2010.492802.
Process 2008;260:395–404. https://doi.org/10.1007/978-0-387-77405-3_39. [96] Bogoslavskyi I, Stachniss C. Analyzing the quality of matched 3D point clouds of
[65] Webel S, Bockholt U, Engelke T, Gavish N, Olbrich M, Preusche C. An augmented objects. IEEE Int Conf Intell Robot Syst 2017;(2017-Septe):6685–90. https://doi.
reality training platform for assembly and maintenance skills. Rob Auton Syst org/10.1109/IROS.2017.8206584.
2013;61:398–403. https://doi.org/10.1016/j.robot.2012.09.013. [97] Gupta S, Dangayach GS, Singh AK. Key determinants of sustainable product
[66] Makhataeva Z, Varol HA. Augmented Reality for. Robot: A Rev 2020. https://doi. design and manufacturing. Procedia CIRP 2015;26:99–102. https://doi.org/
org/10.3390/robotics9020021. 10.1016/j.procir.2014.07.166.
[67] Masehian E, Mohamadnejad N. Path planning of nonholonomic flying robots [98] Kaebernick H, Kara S, Sun M. Sustainable product development and
using a new virtual obstacle method. Int Conf Robot Mechatron, ICROM 2015 manufacturing by considering environmental requirements. Robot Comput Integr
2015:612–7. https://doi.org/10.1109/ICRoM.2015.7367853. Manuf 2003;19:461–8. https://doi.org/10.1016/S0736-5845(03)00056-5.
[68] Blankemeyer S, Wiemann R, Posniak L, Pregizer C, Raatz A, Stief P, et al. [99] Ji Y, Jiang Z, Li X, Huang Y, Wang F. A multitask context-aware approach for
ScienceDirect A new methodology to analyze the functional and physical design lesson-learned knowledge recommendation in collaborative product
architecture of Robot Programming Using Augmented Reality Nantes, France design. J Intell Manuf 2022:1–23.
Reality Intuitive Robot Programming Using Augmented existing products for an [100] Sharma D., Kumar P., Singh R.K. Quantifiable Contribution of Sustainable
assembly oriented product family. Procedia CIRP 2018;76:155–60. https://doi. Manufacturing Enablers in Indian SMEs. Recent Adv. Ind. Prod., Springer; 2022,
org/10.1016/j.procir.2018.02.028. p. 123–35.
[69] Ni D, Yew AWW, Ong SK, Nee AYC. Haptic and visual augmented reality interface [101] Qiu C, Zhou S, Liu Z, Gao Q, Tan J. Digital assembly technology based on
for programming welding robots. Adv Manuf 2017;5:191–8. https://doi.org/ augmented reality and digital twins: a review. Virtual Real Intell Hardw 2019;1:
10.1007/s40436-017-0184-7. 597–610. https://doi.org/10.1016/j.vrih.2019.10.002.
[70] Nee A.Y.C. An Application of Augmented Reality ( AR) in the Teaching of an Arc [102] Masehian E, Ghandi S. ASPPR: a new assembly sequence and path planner/
Welding Robot n.d. replanner for monotone and nonmonotone assembly planning. CAD Comput
[71] Zhang J, Ong SK, Nee AYC. A volumetric model-based CNC simulation and Aided Des 2020;123:102828. https://doi.org/10.1016/j.cad.2020.102828.
monitoring system in augmented environments. 2006 Int Conf Cyber, CW’06 [103] Shen Y, Ong SK, Nee AYC. Augmented reality for collaborative product design
2006:33–40. https://doi.org/10.1109/CW.2006.11. and development. Des Stud 2010;31:118–45. https://doi.org/10.1016/j.
[72] Zhang J, Ong SK, Nee AYC. Design and development of an in situ machining destud.2009.11.001.
simulation system using augmented reality technology. Procedia CIRP 2012;3: [104] Mourtzis D, Siatras V, Angelopoulos J, Panopoulos N. An augmented reality
185–90. https://doi.org/10.1016/j.procir.2012.07.033. collaborative product design cloud-based platform in the context of learning
[73] Kokkas A, Vosniakos GC. An augmented reality approach to factory layout design factory. Procedia Manuf 2020;45:546–51. https://doi.org/10.1016/j.
embedding operation simulation. Int J Inter Des Manuf 2019;13:1061–71. promfg.2020.04.076.
https://doi.org/10.1007/s12008-019-00567-6. [105] Purdy TG, Choi YM. Enhancing augmented reality for use in product design. Conf
[74] Yu L., Ong S.K., Nee A.Y.C. A tracking solution for mobile augmented reality Hum Factors Comput Syst - Proc 2014:1303–8. https://doi.org/10.1145/
based on sensor-aided marker-less tracking and panoramic mapping 2016: 2559206.2581251.
3199–3220. https://doi.org/10.1007/s11042–014-2430–3. [106] Ng LX, Wang ZB, Ong SK, Nee AYC. Integrated product design and assembly
[75] Adrianto D., Hidajat M., Yesmaya V. Augmented reality using Vuforia for planning in an augmented reality environment. Assem Autom 2013;33:345–59.
marketing residence. 2016 1st Int Conf Game, Game Art, Gamification, ICGGAG https://doi.org/10.1108/AA-10-2012-058.
2016 2017. https://doi.org/10.1109/ICGGAG.2016.8052642. [107] Shen Y, Ong SK, Nee AYC. A framework for multiple-view product representation
[76] Cawood S., Fiala M. Augmented reality: a practical guide. 2008. using augmented reality. 2006 Int Conf Cyber, CW’06 2006:157–64. https://doi.
[77] Lepetit V, Fua P. Keypoint recognition using randomized trees. IEEE Trans Pattern org/10.1109/CW.2006.6.
Anal Mach Intell 2006;28:1465–79. [108] Li T, Lockett H, Lawson C. Using requirement-functional-logical-physical models
[78] Castle R, Klein G, Murray DW. Video-rate localization in multiple maps for to support early assembly process planning for complex aircraft systems
wearable augmented reality. 2008 12th IEEE Int Symp Wearable Comput 2008: integration. J Manuf Syst 2020;54:242–57. https://doi.org/10.1016/j.
15–22. jmsy.2020.01.001.
[79] Panin G, Lenz C, Wojtczyk M, Nair S, Roth E, Friedlhuber T, et al. A unifying [109] Slater M, Wilbur S. A framework for immersive virtual environments (FIVE):
software architecture for model-based visual tracking. Image Process. Mach Vis speculations on the role of presence in virtual environments presence governs
Appl 2008;vol. 6813:681303. aspects of autonomie responses and higher-level behaviors of a participant in a
[80] Radkowski R. Object tracking with a range camera for augmented reality VE. Presence Teleoperators Virtual Environ 1997;6:603–16.
assembly assistance. J Comput Inf Sci Eng 2016;16:1–8. https://doi.org/10.1115/ [110] Schumann M, Fuchs C, Kollatsch C, Klimant P. Evaluation of augmented reality
1.4031981. supported approaches for product design and production processes. Procedia
[81] Yuan ML, Ong SK, Nee AYC. A generalized registration method for augmented CIRP 2020;97:160–5. https://doi.org/10.1016/j.procir.2020.05.219.
reality systems 2005;29:980–97. https://doi.org/10.1016/j.cag.2005.09.014. [111] Mourtzis D, Zogopoulos V, Vlachou E. Augmented reality supported product
[82] Pang Y, Nee AYC, Ong SK, Yuan M, Youcef-Toumi K. Assembly feature design in design towards industry 4.0: a teaching factory paradigm. Procedia Manuf 2018;
an augmented reality environment. Assem Autom 2006;26:34–43. https://doi. 23:207–12. https://doi.org/10.1016/j.promfg.2018.04.018.
org/10.1108/01445150610645648. [112] Chang MML, Nee AYC, Ong SK. Interactive AR-assisted product disassembly
[83] Stancelova P., Sikudova E., Cernekova Z. 3D feature detector-descriptor pair sequence planning (ARDIS. Int J Prod Res 2020;58:4916–31. https://doi.org/
evaluation on point clouds. Eur Signal Process Conf 2021;2021-Janua:590–4. 10.1080/00207543.2020.1730462.
https://doi.org/10.23919/Eusipco47968.2020.9287339. [113] Siew CY, Ong SK, Nee AYC. A practical augmented reality-assisted maintenance
[84] Kim M, Choi SH, Park KB, Lee JY. A hybrid approach to industrial augmented system framework for adaptive user support. Robot Comput Integr Manuf 2019;
reality using deep learning-based facility segmentation and depth prediction. 59:115–29. https://doi.org/10.1016/j.rcim.2019.03.010.
Sens (Switz) 2021;21:1–21. https://doi.org/10.3390/s21010307. [114] Halim A.Z.A. Special Issue MAINTENANCE PROCESS IN AUTOMOTIVE
[85] Yan Y, Poirson E, Bennis F. An interactive motion planning framework that can INDUSTRY 2018.
learn from experience. CAD Comput Aided Des 2015;59:23–38. https://doi.org/ [115] Young C, Rai R. Automated procedure reconfiguration framework for augmented
10.1016/j.cad.2014.07.007. reality-guided maintenance applications. J Comput Inf Sci Eng 2021:21. https://
[86] Della Corte B, Bogoslavskyi I, Stachniss C, Grisetti G. A general framework for doi.org/10.1115/1.4051054.
flexible multi-cue photometric point cloud registration. Proc - IEEE Int Conf [116] Lorenz M, Knopp S, Klimant P. Industrial augmented reality: requirements for an
Robot Autom 2018:4969–76. https://doi.org/10.1109/ICRA.2018.8461049. augmented reality maintenance worker support system. Adjun Proc - 2018 IEEE
[87] Decker N, Wang Y, Huang Q. Efficiently registering scan point clouds of 3D Int Symp Mix Augment Real, ISMAR-Adjun 2018 2018:151–3. https://doi.org/
printed parts for shape accuracy assessment and modeling. J Manuf Syst 2020;56: 10.1109/ISMAR-Adjunct.2018.00055.
587–97. https://doi.org/10.1016/j.jmsy.2020.04.001. [117] Sanna A, Manuri F, Lamberti F, Paravati G, Pezzolla P. Using handheld devices to
[88] Wang C. Depth Sens 2021. sup port augmented reality-based maintenance and assembly tasks. 2015 IEEE Int
275
Conf Consum Electron ICCE 2015 2015:178–9. https://doi.org/10.1109/ [144] Zhang J, Ong SK, Nee AYC. A multi-regional computation scheme in an AR-
ICCE.2015.7066370. assisted in situ CNC simulation environment. CAD Comput Aided Des 2010;42:
[118] Billinghurst M, Hakkarainen M, Woodward C. Augmented assembly using a 1167–77. https://doi.org/10.1016/j.cad.2010.06.007.
mobile phone. MUM’08 - Proc 7th Int Conf Mob Ubiquitous Multimed 2008:84–7. [145] Pai YS, Yap HJ, Md Dawal SZ, Ramesh S, Phoon SY. Virtual planning, control, and
https://doi.org/10.1145/1543137.1543153. machining for a modular-based automated factory operation in an augmented
[119] Evans G, Miller J, Iglesias Pena M, MacAllister A, Winer E. Evaluating the reality environment. Sci Rep 2016;6:31678. https://doi.org/10.1038/srep27380.
Microsoft HoloLens through an augmented reality assembly application. Degrad [146] Monfared M, Shukla VK, Dutta S, Chaubey A. Reshaping education through
Environ Sensing. Process Disp 2017 2017;10197:101970V. https://doi.org/ augmented reality and virtual reality. Cyber Intell Inf. Retr. Springer; 2022.
10.1117/12.2262626. p. 619–29.
[120] Alves J., Marques B., Oliveira M., Araújo T., Dias P., Santos B.S. Comparing [147] Doerner R, Tesch A, Hildebrand A, Leenders S, Tropper T, Wilke W, et al. VR/AR
Spatial and Mobile Augmented Reality for Guiding Assembling Procedures with Case Studies. Virtual Augment. Real: Springer; 2022. p. 331–69.
Task Validation. 19th IEEE Int Conf Auton Robot Syst Compet ICARSC 2019 2019. [148] Checa D, Urbikain G, Beranoagirre A, Bustillo A, de Lacalle LN. Using machine-
https://doi.org/10.1109/ICARSC.2019.8733642. learning techniques and virtual reality to design cutting tools for energy
[121] Werrlich S., Nitsche K., Notni G. Demand analysis for an augmented reality based optimization in milling operations. Int J Comput Integr Manuf 2022:1–21.
assembly training. ACM Int Conf Proceeding Ser 2017;Part F1285:416–22. [149] da Silva AG, Mendes Gomes MV, Winkler I. Virtual reality and digital human
https://doi.org/10.1145/3056540.3076190. modeling for ergonomic assessment in industrial product development: a patent
[122] Renner P, Pfeiffer T. Evaluation of attention guiding techniques for augmented and literature review. Appl Sci 2022;12:1084.
reality-based assistance in picking and assembly tasks. Int Conf Intell Use [150] Bernardo N., Duarte E. Immersive Virtual Reality in an Industrial Design
Interfaces, Proc IUI 2017:89–92. https://doi.org/10.1145/3030024.3040987. Education Context: What the Future Looks Like According to its Educators 2022.
[123] De Sá M, Churchill EF. Human factors in augmented reality environments. Hum [151] Stark R. Major Technology 7: Virtual Reality—VR. Virtual Prod. Creat. Ind.,
Factors Augment Real Environ 2013:139–64. https://doi.org/10.1007/978-1- Springer; 2022, p. 305–26.
4614-4205-9. [152] Mourtzis D, Zogopoulos V, Vlachou E. Augmented reality application to support
[124] Park MK, Lim KJ, Seo MK, Jung SJ, Lee KH. Spatial augmented reality for product remote maintenance as a service in the robotics industry. Procedia CIRP 2017;63:
appearance design evaluation. J Comput Des Eng 2015;2:38–46. https://doi.org/ 46–51. https://doi.org/10.1016/j.procir.2017.03.154.
10.1016/j.jcde.2014.11.004. [153] 2021-Investigating the Potential of Smart Manufacturing Technologies.pdf n.d.
[125] Siriborvornratanakul T. Enhancing user experiences of mobile-based augmented [154] Sun L, Osman HAl, Lang J. An augmented reality online assistance platform for
reality via spatial augmented reality: Designs and architectures of projector- repair tasks. ACM Trans Multimed Comput Commun Appl 2021:17. https://doi.
camera devices. Adv Multimed 2018;2018. https://doi.org/10.1155/2018/ org/10.1145/3429285.
8194726. [155] Siew CY, Chang MML, Ong SK, Nee AYC. Human-oriented maintenance and
[126] Porter SR, Marner MR, Smith RT, Zucco JE, Thomas BH. Validating spatial disassembly in sustainable manufacturing. Comput Ind Eng 2020;150:106903.
augmented reality for interactive rapid prototyping. 9th IEEE Int Symp Mix https://doi.org/10.1016/j.cie.2020.106903.
Augment Real 2010 Sci Technol ISMAR 2010 - Proc 2010:265–6. https://doi.org/ [156] Siew CY, Nee AYC, Ong SK. Improving maintenance efficiency with an adaptive
10.1109/ISMAR.2010.5643599. AR-assisted maintenance system. ACM Int Conf Proc Ser 2019:74–8. https://doi.
[127] Stoltz M-H, Giannikas V, McFarlane D, Strachan J, Um J, Srinivasan R. org/10.1145/3351180.3351203.
Augmented reality in warehouse operations: opportunities and barriers. IFAC-Pap [157] Guo Z, Zhou D, Zhou Q, Zhang X, Geng J, Zeng S. Applications of virtual reality in
2017;50:12979–84. maintenance during the industrial product lifecycle: A systematic review. J Manuf
[128] Cirulis A, Ginters E. Augmented reality in logistics. Procedia Comput Sci 2013;26: Syst 2020;56:525–38. https://doi.org/10.1016/j.jmsy.2020.07.007.
14–20. https://doi.org/10.1016/j.procs.2013.12.003. [158] Palmarini R, Erkoyuncu JA, Roy R, Torabmostaedi H. A systematic review of
[129] Mourtzis D, Samothrakis V, Zogopoulos V, Vlachou E. Warehouse design and augmented reality applications in maintenance. Robot Comput Integr Manuf
operation using augmented reality technology: a papermaking industry case 2018;49:215–28. https://doi.org/10.1016/j.rcim.2017.06.002.
study. Procedia CIRP 2019;79:574–9. https://doi.org/10.1016/j. [159] Chang MML, Ong SK, Nee AYC. AR-guided product disassembly for maintenance
procir.2019.02.097. and remanufacturing. Procedia CIRP 2017;61:299–304. https://doi.org/10.1016/
[130] Ananthi K, Rajavel R, Sabarikannan S, Srisaran A, Sridhar C. Design and j.procir.2016.11.194.
fabrication of IoT based inventory control system. 2021 7th Int Conf Adv Comput [160] Aschenbrenner D., Latoschik M.E., Schillingz K. Industrial maintenance with
Commun Syst ICACCS 2021 2021:1101–4. https://doi.org/10.1109/ augmented reality: Two case studies. Proc ACM Symp Virtual Real Softw Technol
ICACCS51430.2021.9441701. VRST 2016;02–04-Nove:341–2. https://doi.org/10.1145/2993369.2996305.
[131] Stoltz MH, Giannikas V, McFarlane D, Strachan J, Um J, Srinivasan R. Augmented [161] Lamberti F, Manuri F, Sanna A, Paravati G, Pezzolla P, Montuschi P. Challenges,
reality in warehouse operations: opportunities and barriers. IFAC-Pap 2017;50: opportunities, and future trends of emerging techniques for augmented reality-
12979–84. https://doi.org/10.1016/j.ifacol.2017.08.1807. based maintenance. IEEE Trans Emerg Top Comput 2014;2:411–21. https://doi.
[132] Wang W, Wang F, Song W, Su S. Application of augmented reality (AR) org/10.1109/TETC.2014.2368833.
technologies in inhouse logistics. E3S Web Conf 2020:145. https://doi.org/ [162] Zhu J, Ong SK, Nee AYC. A context-aware augmented reality system to assist the
10.1051/e3sconf/202014502018. maintenance operators. Int J Inter Des Manuf 2014;8:293–304. https://doi.org/
[133] Etonam AK, Di Gravio G, Kuloba PW, Njiri JG. Augmented reality (AR) 10.1007/s12008-013-0199-7.
application in manufacturing encompassing quality control and maintenance. Int [163] Zhu J, Ong SK, Nee AYC. An authorable context-aware augmented reality system
J Eng Adv Technol 2019;9:197–204. https://doi.org/10.35940/ijeat. to assist the maintenance technicians. Int J Adv Manuf Technol 2013;66:
A1120.109119. 1699–714. https://doi.org/10.1007/s00170-012-4451-2.
[134] Tschirner P, Gräser A. Virtual Augment Real Qual Improv Man welds 2002;vol. [164] Ariansyah D., Erkoyuncu J.A., Eimontaite I., Johnson T., Oostveen A.-M., Fletcher
15. IFAC. https://doi.org/10.3182/20020721-6-es-1901.00906. S., et al. A head mounted augmented reality design practice for maintenance
[135] Segovia D, Mendoza M, Mendoza E, González E. Augmented reality as a tool for assembly: Toward meeting perceptual and cognitive needs of AR users. Appl
production and quality monitoring. Procedia Comput Sci 2015;75:291–300. Ergon 2022;98:103597.
https://doi.org/10.1016/j.procs.2015.12.250. [165] Schoitsch E., Michel B., Kremer S., Krapp M. Special theme Augmented Reality
[136] Urbas U, Vrabič R, Vukašinović N. Displaying product manufacturing information 2015.
in augmented reality for inspection. Procedia CIRP 2019;81:832–7. https://doi. [166] Of F, Engineering I. FACULTY OF INDUSTRIAL ENGINEERING AND ROBOTICS
org/10.1016/j.procir.2019.03.208. MAINTENANCE PLATFORM IN INDUSTRY 4. 0 - Abstract - n.d.:1–41.
[137] Alves JB, Marques B, Dias P, Santos BS. Using augmented reality for industrial [167] Eder M, Hulla M, Mast F, Ramsauer C. On the application of augmented reality in
quality assurance: a shop floor user study. Int J Adv Manuf Technol 2021;115: a learning factory working environment. Procedia Manuf 2020;45:7–12. https://
105–16. https://doi.org/10.1007/s00170-021-07049-8. doi.org/10.1016/j.promfg.2020.04.030.
[138] Quandt M, Knoke B, Gorldt C, Freitag M, Thoben KD. General requirements for [168] Yew AWW, Ong SK, Nee AYC. Immersive augmented reality environment for the
industrial augmented reality applications. Procedia CIRP 2018;72:1130–5. teleoperation of maintenance robots. Procedia CIRP 2017;61:305–10. https://doi.
https://doi.org/10.1016/j.procir.2018.03.061. org/10.1016/j.procir.2016.11.183.
[139] Herr D, Reinhardt J, Reina G, Krüger R, Ferrari RV, Ertl T. Immersive modular [169] Bernd S de. lAvAl Blandine, An Augmented Reality System for Training and
factory layout planning using augmented reality. Procedia CIRP 2018;72:1112–7. Assistance to Maintenance in the Industrial Context. J WSCG 2003;11:3–7.
https://doi.org/10.1016/j.procir.2018.03.200. [170] Safi M, Chung J, Pradhan P. Review of augmented reality in aerospace industry.
[140] Pentenrieder K, Bade C, Doil F, Meier P. Augmented reality-based factory Aircr Eng Aerosp Technol 2019;91:1187–94. https://doi.org/10.1108/AEAT-09-
planning - an application tailored to industrial needs. 2007 6th IEEE ACM Int 2018-0241.
Symp Mix Augment Real, ISMAR 2007:31–42. https://doi.org/10.1109/ [171] Sahu CK, Young C, Rai R. Artificial intelligence (AI) in augmented reality (AR)-
ISMAR.2007.4538822. assisted manufacturing applications: a review. Int J Prod Res 2021;59:4903–59.
[141] Lee J, Han S, Yang J. Construction of a computer-simulated mixed reality https://doi.org/10.1080/00207543.2020.1859636.
environment for virtual factory layout planning. Comput Ind 2011;62:86–98. [172] Lundgren C, Bokrantz J, Skoogh A. A strategy development process for Smart
https://doi.org/10.1016/j.compind.2010.07.001. Maintenance implementation. J Manuf Technol Manag 2021;32:142–66. https://
[142] Jiang S, Ong SK, Nee AYC. An AR-based hybrid approach for facility layout doi.org/10.1108/JMTM-06-2020-0222.
planning and evaluation for existing shop floors. Int J Adv Manuf Technol 2014; [173] Gattullo M, Evangelista A, Uva AE, Fiorentino M, Gabbard JL. What, How, and
72:457–73. https://doi.org/10.1007/s00170-014-5653-6. why are visual assets used in industrial augmented reality? a systematic review
[143] Olwal A, Gustafsson J, Lindfors C. Spatial augmented reality on industrial CNC- and classification in maintenance, assembly, and training (From 1997 to 2019.
machines. Eng Real Virtual Real 2008 2008;6804:680409. https://doi.org/ IEEE Trans Vis Comput Graph 2022;28:1443–56. https://doi.org/10.1109/
10.1117/12.760960. TVCG.2020.3014614.
276
[174] Leone D, Pietronudo MC, Dezi L. Improving business models through augmented [204] Rejeb A., Keogh J.G., Wamba S.F., Treiblmaier H. The potentials of augmented
reality applications: evidence from history, theory, and practice. Int J Qual 2022; reality in supply chain management: a state-of-the-art review. vol. 71. Springer
6:28–42. International Publishing; 2021. https://doi.org/10.1007/s11301–020-00201-w.
[175] Iftene A. Introduction to mixed realities 2022. [205] Bruni R., Piccarozzi M. Industry 4.0 enablers in retailing: a literature review. Int J
[176] Marques B, Silva S, Alves J, Rocha A, Dias P, Santos BS. Remote collaboration in Retail \& Distrib Manag 2022.
maintenance contexts using augmented reality: insights from a participatory [206] Weber A, Werth J, Epro G, Friemert D, Hartmann U, Lambrianides Y, et al. Head-
process. Int J Inter Des Manuf 2022:1–20. Mounted Hand-Held Disp Diminish Eff Fall-Resist Skills 2022:1–14.
[177] Stark R. Major Technology 8: Augmented Reality—AR. Virtual Prod. Creat. Ind., [207] Nahavandi D, Alizadehsani R, Khosravi A, Acharya UR. Application of artificial
Springer; 2022, p. 327–51. intelligence in wearable devices: opportunities and challenges. Comput Methods
[178] Bahubalendruni MVAR, Biswal BB. A review on assembly sequence generation Prog Biomed 2022;213:106541.
and its automation. Proc Inst Mech Eng Part C J Mech Eng Sci 2016;230:824–38. [208] Saurabh Jha P, Khan MS. An experimental study on the automotive production
https://doi.org/10.1177/0954406215584633. line using assembly line balancing techniques. Int J Mech Eng Technol 2017;8:
[179] Bahubalendruni MVAR, Biswal BB. An intelligent approach towards optimal 22–33.
assembly sequence generation. Proc Inst Mech Eng Part C J Mech Eng Sci 2018; [209] Lehman S, Elezovikj S, Ling H, Tan C. ARCHIE++: a cloud-enabled framework
232:531–41. https://doi.org/10.1177/0954406216684159. FOR Conducting AR system testing in the wild. X:1–1 IEEE Trans Vis Comput
[180] Bahubalendruni MVAR, Gulivindala AK, Varupala SSVP, Palavalasa DK. Optimal Graph 2022. https://doi.org/10.1109/tvcg.2022.3141029.
Assembly Sequence generation through computational approach. Sadhana - Acad [210] Savla M., Pandhare A., Gulunjkar S., Pandit P., Dhawale P. Automation in
Proc Eng Sci 2019;44:1–9. https://doi.org/10.1007/s12046-019-1157-2. Inventory Management in MSME (Micro, Small, Medium Enterprises) Warehouse
[181] Raghavan V, Molineros J, Sharma R. Interactive evaluation of assembly sequences by Use of Robots. Adv. Mater. Manuf. Energy Eng. Vol. I, Springer; 2022, p.
using augmented reality. IEEE Trans Robot Autom 1999;15:435–49. https://doi. 263–75.
org/10.1109/70.768177. [211] Republic C. Sample pages to be followed exactly. Power 2005;1:363–8.
[182] Hou L, Wang X, Bernold L, Love PED. Using Animated Augmented Reality to [212] Navab N. Developing killer apps for industrial augmented reality. IEEE Comput
Cognitively Guide Assembly. J Comput Civ Eng 2013;27:439–51. https://doi.org/ Graph Appl 2004;24:16–20. https://doi.org/10.1109/MCG.2004.1297006.
10.1061/(asce)cp.1943-5487.0000184. [213] Zheng L, Liu X, An Z, Li S, Zhang R. A smart assistance system for cable assembly
[183] Kolla SSVK Sanchez A, Plapper P. Comparing Effectiveness of Paper Based and by combining wearable augmented reality with portable visual inspection. Virtual
Augmented Reality Instructions for Manual Assembly and Training Tasks. SSRN Real Intell Hardw 2020;2:12–27. https://doi.org/10.1016/j.vrih.2019.12.002.
Electron J 2021:6–8. https://doi.org/10.2139/ssrn.3859970. [214] Muñoz A, Mahiques X, Solanes JE, Martí A, Gracia L, Tornero J. Mixed reality-
[184] Makris S, Pintzos G, Rentzos L, Chryssolouris G. Assembly support using AR based user interface for quality control inspection of car body surfaces. J Manuf
technology based on automatic sequence generation. CIRP Ann - Manuf Technol Syst 2019;53:75–92. https://doi.org/10.1016/j.jmsy.2019.08.004.
2013;62:9–12. https://doi.org/10.1016/j.cirp.2013.03.095. [215] Wu C, Yuan Y, Tang Y, Tian B. Application of terrestrial laser scanning (TLS) in
[185] Sukan M, Elvezio C, Oda O, Feiner S, Tversky B. ParaFrustum: Visualization the architecture, engineering and construction (AEC) industry. Sensors 2022;22:
techniques for guiding a user to a constrained set of viewing positions and 265.
orientations. UIST 2014 - Proc 27th Annu ACM Symp Use Interface Softw Technol [216] De Pace F, Manuri F, Sanna A. Augmented Reality in Industry 4.0. Am J Comput
2014:331–40. https://doi.org/10.1145/2642918.2647417. Sci Inf Technol 2018:06. https://doi.org/10.21767/2349-3917.100017.
[186] Li W, Wang J, Liu M, Zhao S. Real-time occlusion handling for augmented reality [217] Sommer M, Stjepandić J. Machine learning in manufacturing in the era of
assistance assembly systems with monocular images. J Manuf Syst 2022;62: industry 4.0. Digit. An Approach Prod. Process Optim. a. Built Environ. Springer,;
561–74. 2022. p. 79–102.
[187] Dalle Mura M, Dini G. An augmented reality approach for supporting panel [218] Runji J, Lee Y-J, Chu C-H. User-requirements analysis on augmented reality-based
alignment in car body assembly. J Manuf Syst 2021;59:251–60. https://doi.org/ maintenance in manufacturing. J Comput Inf Sci Eng 2022:1–52. https://doi.org/
10.1016/j.jmsy.2021.03.004. 10.1115/1.4053410.
[188] Lai Z, Tao W, Leu MC, Yin Z. Smart augmented reality instructional system for [219] Stepanek V, Brazina J, Holub M, Vetiska J, Kovar J, Kroupa J, et al.
mechanical assembly towards worker-centered intelligent manufacturing. Implementation of Industry 4.0 Elements in Industrial Metrology–Case Study.
J Manuf Syst 2020;55:69–81. https://doi.org/10.1016/j.jmsy.2020.02.010. Digit. Prod. Syst. Springer; 2022. p. 296–308.
[189] Huang GQ, Zhang YF, Jiang PY. RFID-based wireless manufacturing for walking- [220] Mao Y., Peng Z., Hu X. Mixed reality-based digital twin implementation approach
worker assembly islands with fixed-position layouts. Robot Comput Integr Manuf for flexible manufacturing system design 2022:1–21.
2007;23:469–77. https://doi.org/10.1016/j.rcim.2006.05.006. [221] Liu YK, Ong SK, Nee AYC. State-of-the-art survey on digital twin implementations.
[190] Barfield W, Caudell T. Basic concepts in wearable computers and augmented Adv Manuf 2022. https://doi.org/10.1007/s40436-021-00375-w.
reality. Fundam. wearable Comput. Augment. Real: CRC Press,; 2001. p. 19–42. [222] Adam A., Yusof Y., Latif K., Kadir A.Z.A., Sam T.-H., Nazrein S., et al. The
[191] Chu C, Ko C. An experimental study on augmented reality assisted manual Implementation of a Novel Augmented Reality (AR) Based Mobile Application on
assembly with occluded components. J Manuf Syst 2021. https://doi.org/ Open Platform Controller for CNC Machining Operation. Recent Trends
10.1016/j.jmsy.2021.04.003. Mechatronics Towar. Ind. 4.0, Springer; 2022, p. 227–34.
[192] Thøgersen M, Andoh J, Milde C, Graven-Nielsen T, Flor H, Petrini L. [223] Olwal A, Lindfors C, Gustafsson J, Kjellberg T, Mattsson L. ASTOR: An
Individualized augmented reality training reduces phantom pain and cortical autostereoscopic optical see-through augmented reality system. Proc - Fourth
reorganization in amputees: a proof of concept study. J Pain 2020;21:1257–69. IEEE ACM Int Symp Symp Mix Augment Reality. ISMAR 2005 2005 2005:24–7.
[193] Wiedenmaier S, Oehme O, Schmidt L, Luczak H. Augmented reality (AR) for https://doi.org/10.1109/ISMAR.2005.15.
assembly processes design and experimental evaluation. Int J Hum Comput Inter [224] Weinert K, Zabel A, Ungemach E, Odendahl S. Improved NC path validation and
2003;16:497–514. manipulation with augmented reality methods. Prod Eng 2008;2:371–6. https://
[194] Wang X, Ong SK, Nee AYC. Multi-modal augmented-reality assembly guidance doi.org/10.1007/s11740-008-0115-3.
based on bare-hand interface. Adv Eng Inform 2016;30:406–21. https://doi.org/ [225] Shariatzadeh N, Sivard G, Chen D. Software evaluation criteria for rapid factory
10.1016/j.aei.2016.05.004. layout planning, design and simulation. Procedia CIRP 2012;3:299–304. https://
[195] Wang X, Ong SK, Nee AYC. Real-virtual interaction in AR assembly simulation doi.org/10.1016/j.procir.2012.07.052.
based on component contact handling strategy. Assem Autom 2015;35:376–94. [226] Inkulu A.K., Bahubalendruni M.V.A.R., Dara A., SankaranarayanaSamy K.
https://doi.org/10.1108/AA-02-2015-012. Challenges and opportunities in human robot collaboration context of Industry
[196] Ammar M., Haleem A., Javaid M., Bahl S., Nandan D. Improving the Performance 4.0-a state of the art review. Ind Robot Int J Robot Res Appl 2021.
of Supply Chain Through Industry 4.0 Technologies. Adv. Mater. Manuf. Energy [227] Ciuffini AF, Di Cecca C, Ferrise F, Mapelli C, Barella S. Application of virtual/
Eng. Vol. II, Springer; 2022, p. 197–209. augmented reality in steelmaking plants layout planning and logistics. Met Ital
[197] Zhen L, Li H. A literature review of smart warehouse operations management. 2016;2016:5–10.
Front Eng Manag 2022:1–25. [228] Tan CH, Yap HJ, Musa SN, Chang SW, Sivadas CS, Hisaburi AS, et al. Augmented
[198] Mocan A, Gaureanu A, Szabó G, Mrugalska B. Arguments for Emerging reality assisted facility layout digitization and planning. J Mech Sci Technol 2021;
Technologies Applications to Improve Manufacturing Warehouse Ergonomics. 35:4115–23. https://doi.org/10.1007/s12206-021-0823-6.
Sustain. Innov. Manuf. Enterp. Springer,; 2022. p. 115–64. [229] Pai YS, Yap HJ, Md Dawal SZ, Ramesh S, Phoon SY. Virtual planning, control, and
[199] Yan Z, Shan Y, Li Y, Yin K, Li X. Gender differences of cognitive loads in machining for a modular-based automated factory operation in an augmented
augmented reality-based warehouse. Proc - 2021 IEEE Conf Virtual Real 3D Use reality environment. Sci Rep 2016;6:1–19. https://doi.org/10.1038/srep27380.
Interfaces Abstr Work VRW 2021 2021:500–1. https://doi.org/10.1109/ [230] Jiang S, Nee AYC. A novel facility layout planning and optimization methodology.
VRW52623.2021.00132. CIRP Ann - Manuf Technol 2013;62:483–6. https://doi.org/10.1016/j.
[200] Damise MA, Sirahbizu B, Ph Y. Minimizing Total Inventory Cost Mech Spare Parts cirp.2013.03.133.
( Case Messobo Cem Ind 2021;9:627–47. https://doi.org/10.13140/ [231] Zhang Z. Microsoft kinect sensor and its effect. IEEE Multimed 2012;19:4–10.
RG.2.2.26322.68809. https://doi.org/10.1109/MMUL.2012.24.
[201] Özda\ugo\uglu A, Bahar S. Logistics 4.0 and Smart Supply Chain Management. [232] Chu C-H, Wang L, Liu S, Zhang Y, Menozzi M. Augmented reality in smart
Ind. 4.0 Glob. Businesses, Emerald Publishing Limited; 2022. manufacturing: enabling collaboration between humans and artificial
[202] Transport U. Implementation of Barcode Technology to Logistics Processes of a intelligence. J Manuf Syst 2021;61:658–9.
Company 2022. [233] Pang Y, Yuan ML, Nee AYC, Ong SK, Youcef-toumi K. A Marker Regist Method
[203] Daulay JT, Suhendrik S. SISTEM PARKIR SEKOLAH BERBASIS JAVA Augment Real Based Affin Prop 2006:50.
MENGGUNAKAN BARCODE SCANNER STUDI KASUS SMKN 3 TANAH PUTIH. [234] Ong SK, Yuan ML, Alliance S. Track Points Using Proj Reconstr Augment Real
Lentera Dumai 2022:13. 1999:421–4.
277
[235] Wang Z, Bai X, Zhang S, He W, Zhang X, Yan Y, et al. Information-level real-time [241] Maly I, Sedlacek D, Leitao P. Augmented reality experiments with industrial robot
AR instruction: a novel dynamic assembly guidance information representation in industry 4.0 environment. IEEE Int Conf Ind Inform 2016;0:176–81. https://
assisting human cognition. Int J Adv Manuf Technol 2020;107:1463–81. https:// doi.org/10.1109/INDIN.2016.7819154.
doi.org/10.1007/s00170-020-05034-1. [242] Murauer C.S. Full shift usage of smart glasses in order picking processes
[236] Yang Q, Wu DL, Zhu HM, Bao JS, Wei ZH. Assembly operation process planning considering a methodical approach of continuous user involvement. 2019.
by mapping a virtual assembly simulation to real operation. Comput Ind 2013;64: [243] Funk M., Bachler A., Bachler L., Kosch T., Heidenreich T., Schmidt A. Working
869–79. https://doi.org/10.1016/j.compind.2013.06.001. with augmented reality? A long-term analysis of in-situ instructions at the
[237] Porcelli I, Rapaccini M, Espíndola DB, Pereira CE. Technical and organizational assembly workplace. ACM Int Conf Proceeding Ser 2017;Part F1285:222–9.
issues about the introduction of augmented reality in maintenance and technical https://doi.org/10.1145/3056540.3056548.
assistance services. IFAC Proc 2013;Vol(46):257–62. https://doi.org/10.3182/ [244] Blattgerste J., Strenge B., Renner P., Pfeiffer T., Essig K. Comparing conventional
20130522-3-BR-4036.00024. and augmented reality instructions for manual assembly tasks. ACM Int Conf
[238] Moore A. A tangible augmented reality interface to tiled street maps and its Proceeding Ser 2017;Part F1285:75–82. https://doi.org/10.1145/
usability testing. Prog. Spat. Data Handl., Springer; 2006, p. 511–28. 3056540.3056547.
[239] Hsiao K-F, Chen N-S, Huang S-Y. Learning while exercising for science education [245] Egger J, Masood T. Augmented reality in support of intelligent manufacturing – a
in augmented reality among adolescents. Inter Learn Environ 2012;20:331–49. systematic literature review. Comput Ind Eng 2019:106195. https://doi.org/
[240] Bravo A, Maier AM, Cash PJ. Watch that seam! designing hybrid presentations 10.1016/j.cie.2019.106195.
with data visualisation in augmented reality. Int J Des 2021;15:1–28.
278

1 s2.0 S0278612522001650 Main

Uploaded by

Copyright:

Available Formats

1 s2.0 S0278612522001650 Main

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

1 s2.0 S0278612522001650 Main

Uploaded by

Copyright:

Available Formats

Journal of Manufacturing Systems 65 (2022) 260–278

Contents lists available at ScienceDirect

Journal of Manufacturing Systems

Challenges and opportunities on AR/VR technologies for manufacturing

1. Introduction processes prior to their implementation in a production environment.

Fig. 2. Hardware and Software Elements of AR Associated Manufacturing System.

Fig. 3. Application Development Platform Tools for AR assisted Manufacturing Activities.

estimation and overlaying. Fig. 6 represents the object identification of

• Reference object point clouds management: A CAD model is gener­

Fig. 5. Architecture/Methodologies of AR Assisted Manufacturing System.

• Minimization of spatial distance: it is a process at where the spatial

5. AR/VR for manufacturing applications

Manufacturing applications needs skilled engineers to do the oper­

Fig. 7. Depth Camera Scanning.

subjected to map generation process (i.e., vertex map, normal map)

Fig. 8. Cloud Points Based Matching Algorithms.

HMD HHD SD Optical Video

Fig. 10. AR/VR Assisted Product Design and Evaluation.

Fig. 11. Architecture of Proposed AR Assisted Maintenance System.

be rectified in order to obtain the goal of increased productivity and

Fig. 13. Role of AR in Maintenance System.

Fig. 14. AR Aid Inventory Control.

5.5. Quality control

You might also like

• Reference object point clouds management: A CAD model is gener

Manufacturing applications needs skilled engineers to do the oper