Embryonic stream processing using morphogens

Stream processing has been shown to be a computing paradigm that is well suited to the distributed processing of massive amounts of continuous real-time data. In stream processing, tasks are distributed across processing nodes and the information flow passes from task to task. The allocation of tasks to nodes has typically been carried out by a centralized task manager. Whilst this allocation approach has allowed optimization of the task allocation for constrained domains, it is likely to suffer problems as the complexity of the processing tasks and the scale of the network rise and the network becomes more dynamic. In this paper we explore the potential for a distributed autonomous task allocation based on an embryonic approach combined with a node differentiation that uses reaction-diffusion techniques. In this approach each processing node contains a full description of the processing tasks, and determines its own optimal role based on interactions with its neighbors. We describe the approach and provide preliminary results that indicate that it is likely to provide elegant scalability and therefore warrants further consideration.