The Technology of Processing Intensive Structured Dataflow on a Supercomputer

Modern experimental setups generate prolonged and intense data streams. For example, non-contact measurement techniques PIV (Particle Image Velocimetry), based on continuous image processing, are widely used in the experimental aerodynamics and hydrodynamics. These experimental stereo or volumetric velocimetry 3D PIV setups are able to generate long-lasting and intensive flows of images by using two or more cameras. High computational complexity of the image processing algorithms is the main limiting factor in the conditions of the low computational performance of PIV setups itself. Removal of these limitations by moving image processing tasks on remote supercomputer by using our proposed technology "Distributed PIV", which will allow users to apply their new high-precision parallel algorithms in a real-time and implement feedback to the experimental setup. This paper describes innovative technology for high-performance processing of the intensive flows of the structured data generated by an experimental setup and delivered through a high-speed DWDM backbone directly into computing nodes of a remote supercomputer. A high BDP (Bandwidth-Delay Product) problem case in the design of protocols such as TCP in respect of performance tuning to achieve maximum network throughput is solved by designed middleware. The effectiveness of the proposed technology has been proven as a result of the natural distributed experiment, which combines the experimental setup and supercomputer through the 10 Gbps and 900 km length channel. This experiment illustrates possibility of using the technology for the unique physical experiments.