Raynor Vliegendhart, Larson, Martha, Kofler, Christoph, Eickhoff, Carsten, Pouwelse, Johan. Investigating Factors Influencing Crowdsourcing Tasks with High Imaginative Load. In WSDM'11 Workshop on Crowdsourcing for Search and Data Mining. February 2011.
We report useful observations made during the design and test of a crowdsourcing task with a high “imaginative load”, a term we introduce to designate a task that requires workers to answer questions from a hypothetical point of view that is beyond their daily experiences. We find that workers are able to deliver high quality responses to such {HITs}, but that it is important that the HIT title allows workers to formulate accurate expectations of the task. Also important is the inclusion of free-text justification questions that target specific items in a pattern that is not obviously predictable. These findings were supported by a small-scale experiment run on several crowdsourcing platforms.