Contained in:
Book Chapter

Seamless Indoor/Outdoor Marker-Less Augmented Reality Registration Supporting Facility Management Operations

  • Leonardo Messi
  • Francesco Spegni
  • Massimo Vaccarini
  • Alessandra Corneli
  • Leonardo Binni

Augmented reality (AR) still struggles to be widely used in real processes in the construction industry despite its great potential. This is partly due to the difficulties that exist in aligning holograms and maintaining their stability, especially for outdoor applications. In addition, being indoor-outdoor interactions crucial for built environment management, it would be important that AR apps can work seamlessly. Alignment in indoor environments cannot make use of methods such as GNSS, nor can all environments be assumed to have been previously initialized with AR tools. Thus, marker-less AR registration is crucial for indoor applications. This paper presents an approach for marker-less AR registration seamlessly in both outdoor and indoor environments. Real-time kinematic positioning (RTK) and Inertial Measurement Units (IMU) technologies have been chosen for outdoor registration, while image comparison based on convolutional neural networks (CNN) for indoor registration. In this research, the application of these two technologies and their integration have been studied and tested on site on a real Facility Management use case related to a university campus. The proposed approach has shown very promising results in displaying BIM elements of the electrical system seamlessly superimposed through AR to their physical counterparts in mixed indoor-outdoor environments

  • Keywords:
  • Augmented Reality,
  • Seamless Registration,
  • Feature Matching,
  • Pose Estimation,
  • Real-Time Kinematic,
  • Facility Management,
+ Show More

Leonardo Messi

Università Politecnica delle Marche, Italy - ORCID: 0000-0003-4714-7758

Francesco Spegni

Università Politecnica delle Marche, Italy - ORCID: 0000-0003-3632-3533

Massimo Vaccarini

Università Politecnica delle Marche, Italy - ORCID: 0000-0001-9305-5956

Alessandra Corneli

Università Politecnica delle Marche, Italy - ORCID: 0000-0002-0380-9353

Leonardo Binni

Università Politecnica delle Marche, Italy

  1. Ahmad, A., Claudio, P., Alizadeh Naeini, A., & Sohn, G. (2020). Wi-fi rss fingerprinting for indoor localization using augmented reality. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 5(4), 57–64. DOI: 10.5194/isprs-Annals-V-4-2020-57-2020
  2. Albahbah, M., Kıvrak, S., & Arslan, G. (2021). Application areas of augmented reality and virtual reality in construction project management: A scoping review. Journal of Construction Engineering, 4, 151–172. DOI: 10.31462/jcemi.2021.03151172
  3. Ashour, Z., Shaghaghian, Z., & Yan, W. (2022). BIMxAR: BIM-Empowered Augmented Reality for Learning Architectural Representations. DOI: 10.48550/arXiv.2204.03207
  4. Baek, F., Ha, I., & Kim, H. (2019). Augmented reality system for facility management using image-based indoor localization. Automation in Construction, 99, 18–26. DOI: 10.1016/j.autcon.2018.11.034
  5. Cao, B., Araujo, A., & Sim, J. (2020). Unifying Deep Local and Global Features for Image Search. http://arxiv.org/abs/2001.05027
  6. Carbonari, A., Franco, C., Naticchia, B., Spegni, F., & Vaccarini, M. (2022). A Mixed Reality Application for the On-Site Assessment of Building Renovation: Development and Testing. Sustainability (Switzerland), 14(20). DOI: 10.3390/su142013239
  7. Chen, K., Chen, W., Fellow, P., Li, C. T., Student, M., & Cheng, J. C. P. (2019). A BIM-based location aware AR collaborative framework for facility maintenance management. In Journal of Information Technology in Construction (ITcon) (Vol. 24). http://www.itcon.org/2019/19
  8. Cheng, J. C. P., Chen, K., & Chen, W. (2020). State-of-the-Art Review on Mixed Reality Applications in the AECO Industry. Journal of Construction Engineering and Management, 146(2). DOI: 10.1061/(asce)co.1943-7862.0001749
  9. Costanza, E., Kunz, A., & Fjeld, M. (2009). Mixed Reality: A Survey. Lecture Notes in Computer Science, 5440. DOI: 10.1007/978-3-642-00437-7_3
  10. Cyrus, J., Krcmarik, D., Moezzi, R., Koci, J., & Petru, M. (2019). Hololens used for precise position tracking of the third party devices - Autonomous vehicles. Communications - Scientific Letters of the University of Žilina, 21(2), 18–23. DOI: 10.26552/com.c.2019.2.18-23
  11. DeTone, D., Malisiewicz, T., & Rabinovich, A. (2017). SuperPoint: Self-Supervised Interest Point Detection and Description. http://arxiv.org/abs/1712.07629
  12. El Barhoumi, N., Hajji, R., Bouali, Z., Ben Brahim, Y., & Kharroubi, A. (2022). Assessment of 3D Models Placement Methods in Augmented Reality. Applied Sciences (Switzerland), 12(20). DOI: 10.3390/app122010620
  13. Ethan Rublee, Vincent Rabaud, Kurt Konolige, & Gary Bradski. (2011). ORB: An efficient alternative to SIFT or SURF. 2011 International Conference on Computer Vision. DOI: 10.1109/ICCV.2011.6126544
  14. Google LLC. (2023). ARCore. https://developers.google.com/ar?hl=it
  15. Guarese, R. L. M., & Maciel, A. (2019). Development and Usability Analysis of a Mixed Reality GPS Navigation Application for the Microsoft HoloLens. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11542 LNCS, 431–437. DOI: 10.1007/978-3-030-22514-8_41
  16. Hansen, L. H., Fleck, P., Stranner, M., Schmalstieg, D., & Arth, C. (2021). Augmented Reality for Subsurface Utility Engineering, Revisited. IEEE Transactions on Visualization and Computer Graphics, 27(11), 4119–4128. DOI: 10.1109/TVCG.2021.3106479
  17. He, Z., Xia, Z., Chang, Y., Chen, W., Hu, J., & Wei, X. (2006). Research on underground pipeline augmented reality system based on ARToolKit. In H. Wu & Q. Zhu (Eds.), Proc. SPIE 6421, Geoinformatics 2006: Geospatial Information Technology (p. 642112). DOI: 10.1117/12.713123
  18. Humenberger, M., Cabon, Y., Guerin, N., Morat, J., Leroy, V., Revaud, J., Rerole, P., Pion, N., de Souza, C., & Csurka, G. (2020). Robust Image Retrieval-based Visual Localization using Kapture. http://arxiv.org/abs/2007.13867
  19. Jurado, D., Jurado, J. M., Ortega, L., & Feito, F. R. (2021). Geuinf: Real-time visualization of indoor facilities using mixed reality. Sensors (Switzerland), 21(4), 1–21. DOI: 10.3390/s21041123
  20. Kaizu, Y., & Choi, J. (2012). Development of a tractor navigation system using augmented reality. Engineering in Agriculture, Environment and Food, 5(3), 96–101. DOI: 10.1016/S1881-8366(12)80021-8
  21. Kim, C., Park, T., Lim, H., & Kim, H. (2013). On-site construction management using mobile computing technology. Automation in Construction, 35, 415–423. DOI: 10.1016/j.autcon.2013.05.027
  22. Kneip, L., Scaramuzza, D., & Siegwart, R. (2011). A Novel Parametrization of the Perspective-Three-Point Problem for a Direct Computation of Absolute Camera Position and Orientation. DOI: 10.1109/CVPR.2011.5995464
  23. Koch, C., Neges, M., König, M., & Abramovici, M. (2014). Natural markers for augmented reality-based indoor navigation and facility maintenance. Automation in Construction, 48, 18–30. DOI: 10.1016/j.autcon.2014.08.009
  24. Kuo, C., Jeng, T., & Yang, I. (2013). An invisible head marker tracking system for indoor mobile augmented reality. Automation in Construction, 33, 104–115. DOI: 10.1016/j.autcon.2012.09.011
  25. Lee, S., & Akin, Ö. (2011). Augmented reality-based computational fieldwork support for equipment operations and maintenance. Automation in Construction, 20(4), 338–352. DOI: 10.1016/j.autcon.2010.11.004
  26. Li, Y., Snavely, N., Huttenlocher, D., & Fua, P. (2012). Worldwide Pose Estimation using 3D Point Clouds. DOI: 10.1007/978-3-642-33718-5_2
  27. Lindenberger, P., Sarlin, P.-E., & Pollefeys, M. (2023). LightGlue: Local Feature Matching at Light Speed. http://arxiv.org/abs/2306.13643
  28. Ling, F. F., Elvezio, C., Bullock, J., Henderson, S., & Feiner, S. (2019). A Hybrid RTK GNSS and SLAM Outdoor Augmented Reality System. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 1044–1045. DOI: 10.1109/VR.2019.8798315
  29. Marchand, E., Uchiyama, H., & Spindler, F. (2016). Pose estimation for augmented reality: a hands-on survey. DOI: 10.1109/TVCG.2015.2513408
  30. Microsoft. (2023). World Locking Tools documentation. https://learn.microsoft.com/en-us/mixed-reality/world-locking-tools/
  31. Naticchia, B., Vaccarini, M., Corneli, A., Messi, L., & Carbonari, A. (2021). Leveraging Extended Reality technologies with RFID to enhance on field maintenance of buildings. http://itc.scix.net/paper/w78-2021-paper-038
  32. Park, C. S., Lee, D. Y., Kwon, O. S., & Wang, X. (2013). A framework for proactive construction defect management using BIM, augmented reality and ontology-based data collection template. Automation in Construction, 33, 61–71. DOI: 10.1016/j.autcon.2012.09.010
  33. PTC Products. (2023). Vuforia Enterprise AR Software. https://www.ptc.com/en/products/vuforia
  34. Roberts, G. W., Evans, A., Dodson, A. H., Denby, B., Cooper, S., & Hollands, R. (2002). The Use of Augmented Reality, GPS and INS for Subsurface Data Visualisation. https://doi.org/https://api.semanticscholar.org/CorpusID:112931894
  35. Salman, A., & Ahmad, W. (2023). Implementation of augmented reality and mixed reality applications for smart facilities management: a systematic review. Smart and Sustainable Built Environment. DOI: 10.1108/SASBE-11-2022-0254
  36. Sarlin, P.-E., Cadena, C., Siegwart, R., & Dymczyk, M. (2018). From Coarse to Fine: Robust Hierarchical Localization at Large Scale. http://arxiv.org/abs/1812.03506
  37. Sarlin, P.-E., Debraine, F., Dymczyk, M., Siegwart, R., & Cadena, C. (2018). Leveraging Deep Visual Descriptors for Hierarchical Efficient Localization. http://arxiv.org/abs/1809.01019
  38. Schönberger, J. L., & Frahm, J.-M. (2016). Structure-from-Motion Revisited. https://github.com/colmap/colmap.
  39. Teruggi, S., & Fassi, F. (2022). Mixed Reality Content Alignment in Monumental Environments. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, 43(B2-2022), 901–908. DOI: 10.5194/isprs-archives-XLIII-B2-2022-901-2022
  40. Trimble Inc. (2023). Site Vision. https://sitevision.trimble.com/
  41. Vaccarini, M., Carbonari, A., Spegni, F., & Giretti, A. (2022). Enhancing BIM through Mixed Reality for Facility Management. In Building Information Modeling - A Sustainable Approach and Emerging Technologies [Working Title]. IntechOpen. DOI: 10.5772/intechopen.106186
  42. vGIS Inc. (2023). Engineering-grade AR for AEC. https://www.vgis.io/augmented-reality-bim-gis-ar-aec-civil-construction-engineering-bentley-autodesk-esri/
  43. Zhao, S., Chen, Y., & Farrell, J. A. (2016). High-Precision Vehicle Navigation in Urban Environments Using an MEM’s IMU and Single-Frequency GPS Receiver. IEEE Transactions on Intelligent Transportation Systems, 17(10), 2854–2867. DOI: 10.1109/TITS.2016.2529000
PDF
  • Publication Year: 2023
  • Pages: 109-120

XML
  • Publication Year: 2023

Chapter Information

Chapter Title

Seamless Indoor/Outdoor Marker-Less Augmented Reality Registration Supporting Facility Management Operations

Authors

Leonardo Messi, Francesco Spegni, Massimo Vaccarini, Alessandra Corneli, Leonardo Binni

DOI

10.36253/979-12-215-0289-3.11

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Bibliographic Information

Book Title

CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality

Book Subtitle

Managing the Digital Transformation of Construction Industry

Editors

Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Publisher Name

Firenze University Press

DOI

10.36253/979-12-215-0289-3

eISBN (pdf)

979-12-215-0289-3

eISBN (xml)

979-12-215-0257-2

Series Title

Proceedings e report

Series ISSN

2704-601X

Series E-ISSN

2704-5846

133

Fulltext
downloads

140

Views

Export Citation

1,361

Open Access Books

in the Catalogue

2,368

Book Chapters

3,870,371

Fulltext
downloads

4,536

Authors

from 942 Research Institutions

of 66 Nations

67

scientific boards

from 357 Research Institutions

of 43 Nations

1,249

Referees

from 381 Research Institutions

of 38 Nations