Is a Multi-Criteria Evaluation Tool Reserved for Experts?

The objective of this investigation was to determine whether the analytical hierarchy process algorithm is suitable for the evaluation of software by evaluators with little Information Technology experience. The scope of the research was the evaluation of two free and open source e-learning systems at the Open University of Tanzania using 33 stakeholders with diverse levels of Information Technology experience. Both quantitative and qualitative research methods were used. The qualitative methods comprised participative observation and interviews. Questionnaires and the analytical hierarchy process, a multiple-criteria decision¬making algorithm, represented the quantitative methods. The results showed that of the two e-learning systems evaluated, Moodle was preferred over ATutor. Furthermore it was found that the analytical hierarchy process algorithm is appropriate for the evaluation of software in a situation where Information Technology experience is limited. It is anticipated that the paper contributes to the theory and practise of decision making in developing countries such as Tanzania.