ResearchOps: The case for DevOps in scientific applications

DevOps (a portmanteau of “development” and “operations”) is a software development method that extends the agile philosophy to rapidly produce software products and services and to improve operations performance and quality assurance. It was born to accelerate the delivery of Web-based systems and quickly bring new value to users. Many Web-based systems evolve according to usage trends without a clear long-term goal. Before the widespread use of Web services, most software with a clear goal were delivered as packages that users installed on their own system. New versions were delivered with a much lower frequency, with periods in between versions ranging from months to years. Development cycles were divided into large design, coding and testing phases culminating in the release of a new stable version. In software development in the context of applied science, even when the goal is clear, the process to attain it is not. Hence, working releases that capture the current software state must be released frequently in order to reduce the risks for all stakeholders and to make it possible to assess the current state of a project and steer it in the right direction. This paper explores the usefulness of DevOps concepts to improve the development of software that supports scientific projects. We establish the similarities and differences between scientific projects and Web applications development, and discuss where the related methodologies need to be extended. Unique challenges are discussed herewith developed solutions, and still open questions. Lessons learned are highlighted as best practices to be followed in research projects. This discussion is rooted in our experience in real-life projects at the IBM Research Brazil Lab, which just as well apply to other research institutions.