Evaluating Recommender Algorithms for Learning Using Crowdsourcing

Keeping focused on a certain goal or topic when learning with resources found on the Web is a challenge. Creating a hierarchical learning goal structure with activities and sub-activities can help the learner to keep on track. Moreover, providing useful recommendations to such activities can further support the learner. However, recommendations need to be relevant to the specific goal or activity the learner is currently working on, as well as being novel and diverse to the learner. Such user-centric metrics like novelty and diversity are best measured by asking the users themselves. Nonetheless, conducting user experiments are notoriously time-consuming and access to an adequate amount of users is often very limited. Crowd sourcing offers a means to evaluate TEL recommender algorithms by reaching out to sufficient participants in a shorter time-frame and with less effort. In this paper, a concept for evaluating TEL recommender algorithms using crowd sourcing is presented as well as a repeated proof-of-concept evaluation experiment of a TEL graph-based recommender algorithm AScore that exploits hierarchical activity structures. Results from both experiments support the postulated hypotheses, thereby showing that crowd sourcing can be successfully applied to evaluate TEL recommender algorithms.