Contained in:
Book Chapter

The Value of Extended Reality Techniques to Improve Remote Collaborative Maintenance Operations: A User Study

  • Corentin Coupry
  • Paul Richard
  • David Bigaud
  • Sylvain Noblecourt
  • David Baudry

In the Architecture, Engineering and Construction (AEC) sector, data extracted from building information modelling (BIM) can be used to create a digital twin (DT). The algorithms of a BIM-based DT can facilitate the retrieval of information, which can then be used to improve building operation and maintenance procedures. However, with the increased complexity and automation of the building, maintenance operations are likely to become more complex and may require expert intervention. Collaboration and interaction between the operator and the expert may be limited as the latter may not be on site or within the company. Recently, extended reality (XR) technologies have proven to be effective in improving collaboration during maintenance operations, through data display and shared interactions. This paper presents a new collaborative solution using these technologies to enhance collaboration during remote maintenance operations. The proposed approach consists of a mixed reality (MR) set-up for the operator, a virtual reality (VR) set-up for the remote expert and a shared Digital Model of a heat exchanger. The MR set-up is used for tracking and displaying specific information, provided by the VR module. A user study was carried out to compare the efficiency of our solution with a standard audio-video collaboration. Our approach demonstrated substantial enhancements in collaborative inspection, resulting in a significative reduction in both the overall completion time of the inspection and the frequency of errors committed by the operators

  • Keywords:
  • Virtual Reality; Mixed Reality; Operation & Maintenance; Collaboration; Digital Twin,
+ Show More

Corentin Coupry

LARIS, France - ORCID: 0000-0003-0037-5606

Paul Richard

LARIS, France

David Bigaud

LARIS, France - ORCID: 0000-0001-8474-2885

Sylvain Noblecourt

LINEACT, France - ORCID: 0000-0001-6075-382X

David Baudry

LINEACT, France - ORCID: 0000-0002-4386-4496

  1. Anton, D., Kurillo, G., & Bajcsy, R. (2018). User experience and interaction performance in 2D/3D telecollaboration. Future Generation Computer Systems, 82, 77–88. DOI: 10.1016/j.future.2017.12.055
  2. Bai, H., Sasikumar, P., Yang, J., & Billinghurst, M. (2020). A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–13. DOI: 10.1145/3313831.3376550
  3. Chenechal, M. L., Duval, T., Gouranton, V., Royan, J., & Arnaldi, B. (2016). Vishnu: Virtual immersive support for HelpiNg users an interaction paradigm for collaborative remote guiding in mixed reality. 2016 IEEE Third VR International Workshop on Collaborative Virtual Environments (3DCVE), 9–12. DOI: 10.1109/3DCVE.2016.7563559
  4. Coupry, C., Noblecourt, S., Richard, P., Baudry, D., & Bigaud, D. (2021). BIM-Based Digital Twin and XR Devices to Improve Maintenance Procedures in Smart Buildings: A Literature Review. Applied Sciences, 11(15), 6810. DOI: 10.3390/app11156810
  5. Fakourfar, O., Ta, K., Tang, R., Bateman, S., & Tang, A. (2016). Stabilized Annotations for Mobile Remote Assistance. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 1548–1560. DOI: 10.1145/2858036.2858171
  6. Gao, L., Bai, H., Lee, G., & Billinghurst, M. (2016). An oriented point-cloud view for MR remote collaboration. SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications, 1–4. DOI: 10.1145/2999508.2999531
  7. Gartner Top 10 Strategic Technology Trends for 2023. (2023). Gartner. https://www.gartner.com/en/articles/gartner-top-10-strategic-technology-trends-for-2023
  8. Grandi, J. G., Debarba, H. G., & Maciel, A. (2019). Characterizing Asymmetric Collaborative Interactions in Virtual and Augmented Realities. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 127–135. DOI: 10.1109/VR.2019.8798080
  9. Jamwal, A., Agrawal, R., Sharma, M., & Giallanza, A. (2021). Industry 4.0 Technologies for Manufacturing Sustainability: A Systematic Review and Future Research Directions. Applied Sciences, 11(12), 5725. DOI: 10.3390/app11125725
  10. Karl-Bridge-Microsoft. (2023, June 2). Windows Device Portal overview—UWP applications. https://learn.microsoft.com/en-us/windows/uwp/debug-test-perf/device-portal
  11. Kolkmeier, J., Harmsen, E., Giesselink, S., Reidsma, D., Theune, M., & Heylen, D. (2018). With a little help from a holographic friend: The OpenIMPRESS mixed reality telepresence toolkit for remote collaboration systems. Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology, 1–11. DOI: 10.1145/3281505.3281542
  12. Ladwig, P. (2019). Remote guidance for machine maintenance supported by physical LEDs and virtual reality. ACM International Conference Proceeding Series, Query date: 2022-06-03 15:39:34, 255–262. DOI: 10.1145/3340764.3340780
  13. Microsoft. (n.d.-a). Microsoft HoloLens. Retrieved April 22, 2021, from https://www.microsoft.com/en-us/hololens
  14. Microsoft. (n.d.-b). Microsoft Teams. Retrieved June 28, 2023, from https://www.microsoft.com/en-us/microsoft-teams/group-chat-software
  15. Microsoft. (2022, October 4). QR code tracking overview—Mixed Reality. https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/qr-code-tracking-overview
  16. Oda, O., Elvezio, C., Sukan, M., Feiner, S., & Tversky, B. (2015). Virtual Replicas for Remote Assistance in Virtual and Augmented Reality. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, 405–415. DOI: 10.1145/2807442.2807497
  17. OpenXR. (2016, December 6). The Khronos Group. https://www.khronos.org/openxr/
  18. Photon Unity Networking. (n.d.). Retrieved June 22, 2023, from https://www.photonengine.com/pun/
  19. Pierce, J. S., Stearns, B. C., & Pausch, R. (1999). Voodoo dolls: Seamless interaction at multiple scales in virtual environments. Proceedings of the 1999 Symposium on Interactive 3D Graphics - SI3D ’99, 141–145. DOI: 10.1145/300523.300540
  20. Piumsomboon, T., Day, A., Ens, B., Lee, Y., Lee, G., & Billinghurst, M. (2017). Exploring enhancements for remote mixed reality collaboration. SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications on - SA ’17, 1–5. DOI: 10.1145/3132787.3139200
  21. Piumsomboon, T., Dey, A., Ens, B., Lee, G., & Billinghurst, M. (2019). The Effects of Sharing Awareness Cues in Collaborative Mixed Reality. Frontiers in Robotics and AI, 6, 5. DOI: 10.3389/frobt.2019.00005
  22. Piumsomboon, T., Lee, G. A., Irlitti, A., Ens, B., Thomas, B. H., & Billinghurst, M. (2019). On the Shoulder of the Giant: A Multi-Scale Mixed Reality Collaboration with 360 Video Sharing and Tangible Interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–17. DOI: 10.1145/3290605.3300458
  23. Serubugo, S. (2018). Self-overlapping maze and map design for asymmetric collaboration in room-scale virtual reality for public spaces. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST, 229(Query date: 2022-06-02 15:15:28), 194–203. DOI: 10.1007/978-3-319-76908-0_19
  24. Teo, T., Lawrence, L., Lee, G. A., Billinghurst, M., & Adcock, M. (2019). Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–14. DOI: 10.1145/3290605.3300431
  25. Wang, P., Wang, Y., Billinghurst, M., Yang, H., Xu, P., & Li, Y. (2023). BeHere: A VR/SAR remote collaboration system based on virtual replicas sharing gesture and avatar in a procedural task. Virtual Reality, 27(2), 1409–1430. DOI: 10.1007/s10055-023-00748-5
  26. Wang, P., Zhang, S., Bai, X., Billinghurst, M., He, W., Sun, M., Chen, Y., Lv, H., & Ji, H. (2019). 2.5DHANDS: A gesture-based MR remote collaborative platform. The International Journal of Advanced Manufacturing Technology, 102(5–8), 1339–1353. DOI: 10.1007/s00170-018-03237-1
  27. WebRTC. (n.d.). WebRTC. Retrieved May 10, 2022, from https://webrtc.org/?hl=fr
PDF
  • Publication Year: 2023
  • Pages: 23-33

XML
  • Publication Year: 2023

Chapter Information

Chapter Title

The Value of Extended Reality Techniques to Improve Remote Collaborative Maintenance Operations: A User Study

Authors

Corentin Coupry, Paul Richard, David Bigaud, Sylvain Noblecourt, David Baudry

DOI

10.36253/979-12-215-0289-3.03

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Bibliographic Information

Book Title

CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality

Book Subtitle

Managing the Digital Transformation of Construction Industry

Editors

Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Publisher Name

Firenze University Press

DOI

10.36253/979-12-215-0289-3

eISBN (pdf)

979-12-215-0289-3

eISBN (xml)

979-12-215-0257-2

Series Title

Proceedings e report

Series ISSN

2704-601X

Series E-ISSN

2704-5846

69

Fulltext
downloads

63

Views

Export Citation

1,307

Open Access Books

in the Catalogue

1,949

Book Chapters

3,290,448

Fulltext
downloads

4,134

Authors

from 860 Research Institutions

of 63 Nations

63

scientific boards

from 339 Research Institutions

of 43 Nations

1,150

Referees

from 345 Research Institutions

of 37 Nations