RNS
Browse Projects > Detailed View

Virtual and Augmented Reality for Integration and Effective Use of Heterogeneous Data from Visual Inspection, Nondestructive Testing/Evaluation, and Structural Health Monitoring

Description:

To ensure safety and longevity of infrastructure, on-site Visual Inspection (VI), Nondestructive Evaluation (NDE), and Structural Health Monitoring (SHM) are used to assess performance and condition and quantify the deterioration. Nevertheless, these actions generate large amounts of data (e.g., condition data of various components, crack locations and crack size, section loss information, internal force, vibration, load ratings, FEM results, etc.) and metadata (e.g., plans with locations of sensors, type / specification of sensors, etc.), which is challenging to access and visualize. These data often are hard to manage and relate with each other in a seamless fashion in a quick timeframe. This may lead to underutilization of the available information to make more informed decisions. This, in turn, may also lead to less than optimal decision making, and in some cases wrong or costly decisions. Main challenges with how the data and metadata records are managed are the following:

(1) The heterogeneous nature of the data and metadata (condition data of various components, crack locations and crack size, section loss information, internal force, vibration, load rating data, FEM results, etc.) makes it difficult to access and visualize in a combined yet meaningful manner.

(2) The size and geometry of infrastructure components (e.g., bridges, pipelines, etc.) are large and complex, which presents a challenge to directly correlate the data with metadata (e.g., sensor readings with their position, results of data analysis with the location of the damage, etc.), which in turn is crucial to understand the data, and decide and perform subsequent actions.

(3) A diverse audience consults of the infrastructural data and metadata; the inspector, evaluation engineer, planners, maintenance engineer, designer, and decision maker are frequently not the same person, and they may have different backgrounds and needs in terms of access and visualization.

Therefore, there is a need for a novel and robust method of managing data and metadata that can offer a new strategy for successful structural assessment where data is not separated from their environment (infrastructure) but becomes part of it. Combining virtual tours (VT), informational modeling (IM), and augmented reality (AR) may be suitable way to address the above challenges. Development of such a platform also has a great potential to integrate data from new technologies with visual inspection data in a seamless fashion, leading to stakeholders adopting new and innovative technologies.

Objective:

The desired research outcome of this work is a novel methodology for documenting, organizing, and visualizing (i.e., managing) data and metadata for visual inspection (VI), nondestructive evaluation (NDE), and structural health monitoring (SHM). The main two approaches will combine both virtual reality (VR) and augmented reality (AR) to implement human-centered assessment of decaying infrastructure.

The final product of this work will be a software application which applies this methodology. Hence, the methodology and software to be developed must encompass the best cognitive practices. This will guarantee human-centered features of the final product. This research will be applied to the needs of industry. Through the duration of this research AR will be benchmarked both in laboratory settings, and on-site with traditional VI, NDE, and SHM tools chosen by users. It is expected that the regular interaction of users with both AR and VR will inform them of the potential and direction towards the use of this technology in field decisions and applications. Time, cost, and safety indexes will be measured to compare the ability of the new system to improve current inspection practices, management operations for infrastructure, and ultimately in providing safer and more effective results.

The main application of this technology will be transportation infrastructure, including bridges and tunnels. This tool would also be applicable to includes aviation, dams, energy, inland waterways, ports, rail, schools, solid waste, drinking water, hazardous waste, levees, public parks, roads, transit, and waste water. To facilitate the implementation of the research products across these many types of infrastructure, researchers should disseminate the results to practitioners through presentations and publications at the venues participated by owners, AASHTO bridge and maintenance meetings. They should also organize workshops and short courses to instruct others on how to use the new technology.

Benefits:

The desired research outcome of this work is a novel methodology for documenting, organizing, and visualizing (i.e., managing) data and metadata for visual inspection (VI), nondestructive evaluation (NDE), and structural health monitoring (SHM). The main two approaches will combine both virtual reality (VR) and augmented reality (AR) to implement human-centered assessment of decaying infrastructure.

The final product of this work will be a software application which applies this methodology. Hence, the methodology and software to be developed must encompass the best cognitive practices. This will guarantee human-centered features of the final product. This research will be applied to the needs of industry. Through the duration of this research AR will be benchmarked both in laboratory settings, and on-site with traditional VI, NDE, and SHM tools chosen by users. It is expected that the regular interaction of users with both AR and VR will inform them of the potential and direction towards the use of this technology in field decisions and applications. Time, cost, and safety indexes will be measured to compare the ability of the new system to improve current inspection practices, management operations for infrastructure, and ultimately in providing safer and more effective results.

The main application of this technology will be transportation infrastructure, including bridges and tunnels. This tool would also be applicable to includes aviation, dams, energy, inland waterways, ports, rail, schools, solid waste, drinking water, hazardous waste, levees, public parks, roads, transit, and waste water. To facilitate the implementation of the research products across these many types of infrastructure, researchers should disseminate the results to practitioners through presentations and publications at the venues participated by owners, AASHTO bridge and maintenance meetings. They should also organize workshops and short courses to instruct others on how to use the new technology.

Related Research:

In general, there are two main typologies of data visualization for VI, NDE, and SHM: two-dimensional (2D) and three-dimensional (3D). 2D documentation include Computer-Aided Design (CAD) programs.

Additionally, many NDE and SHM systems have their own proprietary 2D visualization software. While 2D methods can be valuable for a project since they are less costly in terms of time and money than their 3D counterparts, these methods do not always capture the full details and do not guarantee a comprehensive or intuitive understanding of a structure, sensing system, or data and results of data analysis. Future development and implementation of advanced sensors will make it even more complicated to use. 3D methods could be more advantageous for depicting complicated topologies of structures and networks.

3D methods for documenting structure include technologies such as LiDAR, Bridge/Building Information Modeling (BRIM/BIM), and Model View Definition Summary (MVDS). A currently funded project on TRID is “Implementation of Aerial LiDAR Technology to Update Highway Feature Inventory”. There are currently nine different projects on BIM ranging in structural applications from bridges to airports. Virtual Reality (VR) BIM models have also been used by many major companies including LERA and AECOM to facilitate the understanding of a complex space. Yet, while these methods are well-suited for depicting 3D concepts about a structure, these 3D methods can be costly for a project in terms of time, money, and management.

Hence, there is a need for a method of data access and visualization that enables 3D understanding yet does not necessitate building a 3D digital model. Virtual tours (VT) have been implemented for tourism and educational purposes to communicate 3D space to a broad audience in a simple and intuitive manner. Previously, scholars have suggested that a combination of Virtual Reality (VR) and Informational Modeling (IM) could document a building. A VT/IM based method for SHM was preliminary developed to fill the gap in methods for SHM data access and visualization. Augmented Reality (AR) refers to an enriched real world with a complimenting virtual world. In the case of Virtual Reality (VR), the real world is replaced by virtual objects and systems. In contrast, AR enhances the real world by anchoring virtual information into it. Recent achievements have shown promise for collecting measurements. While there are many studies of the applications of AR, there is very little on the interaction between human cognition, information overlays, and motion controllers. Hence in addition to points (1)-(3) identified above, there is a need to (4) build a theory of AR cognition (ARC), using the built-environments as the foundation for studying how to complement human cognition with visual representations and motor actions from wearable AR devices.

Tasks:

Specific tasks include generating architectures and capabilities, accessing and adding information off site and on-site, workshop and reporting, and conference presentation of the results. A final report is expected to summarize the results obtained as well as the benchmarking of this technology with current techniques and methods, as well as recommendations for future developments.

Implementation:

The target audience of this project is wide in scope and includes:

(i) Inspectors, technicians, engineers, planners, decision makers, and other professionals dealing with VI, NDE, and SHM;

(ii) Construction and infrastructure managers, owners, and other stakeholders in the following fields: bridges, buildings, aviation, dams, energy, inland waterways, ports, rails, schools, solid waste, drinking water, hazardous waste, levees, public parks, roads, transit, and waste water;

(iii) Government officials, state and county engineers, military, and other entities in charge of the safety and protection of our nation’s infrastructure and the wellbeing and prosperity of society; (iv) Academics – researchers, students, and teachers whose field of work is related to maintenance and preservation of infrastructure.

Key decision makers who can approve, influence, or champion implementation of the research products:

AASHTO (see the next subsection), but also Federal Highway Administration (FHWA) could approve, influence, and champion the implementation (FHWA’s new website of regarding NDE technologies is developed, and study for similar website regarding SHM is being carried out).

AASHTO committee with likely responsibility for adoption of the results:

AASHTO committee T-18 Bridge Management, Evaluation, and Rehabilitation and/or AASHTO committee T-9 Bridge Preservation would likely take responsibility for adoption of the results.

Early adopters:

Early adopters would be state transportation agencies such as DOTs and/or fragments of it, local owners, bridge authorities, counties, cities, small and medium railroads, transportation districts, etc.

Institutional or political barriers to implementation:

We do not expect to encounter any institutional or political barrier. On the contrary, the political discussions on infrastructure are expected to support innovation of new approaches to quantify and better assess the condition of our existing infrastructure.

Sponsoring Committee:AFF40, Field Testing and Nondestructive Evaluation (NDE) of Transportation Structures
Research Period:12 - 24 months
Research Priority:Medium
RNS Developer:Branko Glisic, Ph.D. Associate Professor, Princeton University, Tel: 609-258-8278, E-mail: bglisic@princeton.edu; Fernando Moreu, Ph.D. Assistant Professor, The University of New Mexico, Tel: 505-277-1784, E-mail: fmoreu@unm.edu; Sreenivas Alampalli, Ph.D., P.E., MBA Director, Structure Management Bureau, New York State Department of Transportation, Tel: 518-457-4544, E-mail: sreenivas.alampalli@dot.ny.gov
Date Posted:04/20/2018
Date Modified:04/25/2018
Index Terms:Virtual reality, Augmented reality, Nondestructive tests, Structural health monitoring, Inspection, Metadata, Data collection,
Cosponsoring Committees: 
Subjects    
Construction
Maintenance and Preservation
Research
Bridges and other structures
Transportation (General)

Please click here if you wish to share information or are aware of any research underway that addresses issues in this research needs statement. The information may be helpful to the sponsoring committee in keeping the statement up-to-date.