Relationship between quality and payment in crowdsourced design

In recent years, the “power of the crowd” has been repeatedly demonstrated and various Internet platforms have been used to applied collaborative intelligence to areas that range from open innovation to image analysis. However, crowdsourcing applications in the fields of design research and creative innovation have been much slower to emerge. So, although there have been reports of systems and researchers using Internet crowdsourcing to carry out generative design, there are still many gaps in knowledge about the capability and limitations of the technology. For example on crowdsourcing platforms, like Amazon´s Mechanical Turk, the relationship between remuneration and the final quality of designs has not been established, so it is unclear how much payment should be offered to ensure a particular standard of result. To investigate the relationship between the crowd´s remuneration and the quality of their innovation, this paper reports how payment for a 2D interior design task (living room layout) was systematically varied and the quality of the output assessed by a ranking process that was also crowdsourced. Information about individual Mturk workers who participated in the study was also collected. The results suggest that while average design quality only slowly increases, the quality of the “best” design generated by the crowd improved dramatically with payment levels. In other words, increasing monetary rewards does not improve the average creativity of design but rather it increases the chance of an excellent solution being generated by an individual in the crowd.