WiP abstract: System-level integration of mobile multi-modal multi-sensor systems

Heterogeneous roaming sensor systems have gained significant importance in many domains of civil infrastructure performance inspection as they accelerate data collection and analysis. However, designing such systems is challenging due to the immense complexity in the heterogeneity and processing demands of the involved sensors. Unifying frameworks are needed to simplify development, deployment and operation of roaming sensors and computing units. To address the sensing needs, we propose SIROM³, a Scalable Intelligent Roaming Multi-Modal Multi-Sensor framework. SIROM³ incorporates a CPS approach for infrastructure performance monitoring to address the following challenges: 1. Scalability and expandability. It offers a scalable and expandable solution enabling diversity in sensing and the growth in processing platforms from sensors to control centers. 2. Fusion foundations. SIROM³ enables fusion of data collected by logically and geo-spatially distributed sensors. 3. Big data handling. Automatic collection, categorization, storage and manipulation of heterogeneous large volume of data streams. 4. Automation. SIROM³ minimizes human interaction through full automation from data acquisition to visualization of the fused results. Illustrated in Fig. 1, SIROM³ realizes scalability and expandability in a system-level design approach encapsulating common functionality across hierarchical components in a Run-Time Environment (RTE). The RTE deploys a layered design paradigm defining services in both software and hardware architectures. Equipped with multiple RTE-enabled Multi-Sensor Aggregators (MSA), an array of Roaming Sensor Systems (RSS) operate as mobile agents attached to vehicles to provide distributed computing services regulated by Fleet Control and Management (FCM) center via communication network. A series of foundational services including the Precise Timing Protocol (PTP), GPS timing systems, Distance Measure- ent Instruments (DMI) through middleware services (CORBA) embedded in the RTE build the fusion foundations for data correlation and analysis. A Heterogeneous Stream File-system Overlay (HSFO) alleviates the big data challenge. It facilitates storing, processing, categorizing and fusing large heterogeneous data stream collected by versatile sensors. A GIS visualization module is integrated for visual analysis and monitoring. SIROM³ enables coordination and collaboration across sensors, MSAs and RSSes, which produce high volume of heterogeneous data stored in HSFO. To fuse the data efficiently, SIROM³ contains an expandable plugin system (part of RTE) for rapid algorithm prototyping using data streams in the architectural hierarchy (i.e. from MSAs to FCM) via the HSFO API. This makes an ideal test-bed to develop new algorithms and methodologies expanding CPS principles to civil infrastructure performance monitoring. In result, SIROM³ simplifies the development, construction and operation of roaming multi-modal multi-sensor systems. We demonstrate the efficiency of SIROM³ by automating the assessment of road surface conditions at the city scale. We realized an RSS with 6 MSAs and 30 heterogeneous sensors, including radars, microphones, GPS and cameras, all deployed onto a van sponsored by the VOTERS (Versatile Onboard Traffic Embedded Roaming Sensors) project. Over 20 terabytes of data have been collected, aggregated, fused, analyzed and geo-spatially visualized using SIROM³ for studying the pavement conditions of the city of Brockton, MA covering 300 miles. The expandability of SIROM³ is shown by adding a millimeter-wave radar needing less than 50 lines of C++ code for system integration. SIROM³ offers a unified solution for comprehensive roadway assessment and evaluation. The integrated management of big data (from collection to automated processing) is an ideal research platform for automa