Claims and supporting evidence for self-adaptive systems: A literature study

Despite the vast body of work on self-adaption, no systematic study has been performed on the claims associated with self-adaptation and the evidence that exists for these claims. As such an insight is crucial for researchers and engineers, we performed a literature study of the research results from SEAMS since 2006 and the associated Dagstuhl seminar in 2008. The study shows that the primary claims of self-adaptation are improved flexibility, reliability, and performance of the system. On the other hand, the tradeoffs implied by self-adaptation have not received much attention. Evidence is obtained from basic examples, or simply lacking. Few systematic empirical studies have been performed, and no industrial evidence is reported. From the study, we offer the following recommendations to move the field forward: to improve evaluation, researchers should make their assessment methods, tools and data publicly available; to deal with poor discussion of limitations, conferences/workshops should require an explicit section on limitations in engineering papers; to improve poor treatment of tradeoffs, this aspect should be an explicit subject of reviews; and finally, to enhance industrial validation, the best academy-industry efforts could be formally recognized by the community.