Real-Time Data Streaming Architecture and Intelligent Workflow Management for Processing Massive Ecological Videos

We present data collection and storage utilities and a workflow management system for handling the processing of large volumes of videos collected from an ecological source over several years and still growing. They lie in the heart of an integrated system that brings together expertise from various disciplines, including marine science, image processing, high performance computing and user interface. A real-time data streaming architecture was developed for efficient collection and storage of videos. In the analysis part, a workflow management system with two main components was deployed, i) a workflow engine and ii) a workflow monitor. The workflow engine deals with on-demand user queries and batch queries, selection of suitable computing platform and invocation of optimal software modules, while the workflow monitor handles the seamless execution and intelligent error handling of workflow jobs on a heterogeneous computing platform. We discuss the challenges that lie ahead for the workflow system such as the demand for more sophisticated scheduling and monitoring.