Specifying a framework for evaluating requirements Engineering Technology: Challenges and lessons learned

Evaluating requirements engineering technology is a challenging activity. It becomes even more difficult when having to evaluate the technology and thus to show its suitability in real settings, as access to industrial resources might be limited and the target domain might be complex or very sensitive. This paper reports on our experience in specifying an evaluation framework for requirements engineering technology. The technology aims to improve safety assurance and certification practices, and is being developed in the scope of a large-scale European research project. We focus on presenting the challenges encountered and the lessons learned while specifying the framework. These lessons summarise how we addressed, plan to address, or propose to address the challenges. This information can be useful for other researchers and practitioners that have to evaluate requirements engineering technology in general, and with industry and for safety assurance and certification in particular.