Crowdsourcing is important

• Example: NASA Clickworkers
• 100K+ volunteers classifying Mars craters from photography

• Results indistinguishable from expert geologists

Experts
Crowds

The problem with crowdsourcing

• Simple independent judgments are not enough for complex work
• Generating articles
• Producing software
• Scientific collaboration
• Need to coordinate the work of many individuals

Designing for collaboration

• Look around you...
• Suppose that needed to write an article together
• Each contribute one minute of effort
• This is close to the situation on MTurk

Doesn't work well enough

• Overall article quality low
• Huge variance in output from each individual
• Adding uncoordinated workers often decreases output quality

Prior work

• Castingwords: speech transcription with high quality results
1. Split audio, distribute each chunk to multiple workers
2. Other workers rate the work of the first wave
3. Automatically stitch highest rated text back together
• TurKit: iterative approach for improved quality*
• Generalization of the above approach
• Iteratively do tasks to get better results
Little, G., Chilton, L. B., Goldman, M., and Miller, R. C. Turkit: human computation algorithms on mechanical turk.

Our research contributions

1. Framework for accomplishing complex and interdependent tasks from many small contributions in crowdsourcing markets
2. Implementation in a software toolkit and using this toolkit to investigate the applicability, strengths, and weaknesses of the framework across a diverse range of tasks

MapReduce inspiration

• Common paradigm in distributed computing
• map(array) → array

• reduce(array) → value

CrowdForge

• Partition: workers break down large task into discrete subtasks
• Map: specified task is processed by one or more workers
• Reduce: multiple workers’ task results merged into single output

Notes

• Structure of the task is determined by the workers
• Many independent contributors can collaborate
• Big task can be split into turk-sized HITs

Example

Given a topic for an article

1. Partition topic into sub-topics
2. Collect facts on the topic
3. Combine these facts into a paragraph
4. Finally merge all of the paragraphs

Writing articles

Given a topic for an article

1. Partition topic into sub-topics
2. Collect facts on the topic
3. Combine these facts into a paragraph
4. Finally merge all of the paragraphs

Analysis

• Created 5 articles about New York City.
• Average: $3.26, 36 HITs, 5.3 headings/article, 658 words
• Comparison: 8 HITs which each requested one worker to write the full article, paying $3.26
• Rated all articles (plus Simple Wikipedia) using MTurk for quality based on criteria

Results

• Rated all articles using MTurk for quality based on criteria

• Group (μ=4.01) ≅ Simple WP (μ=3.95) > Individual (μ=3.75)

Approach to quality

• Highly variable output from workers
• A poor partition can ruin all subsequent results.
• Basic tool: get multiple results per task, vote on quality
• Partition quality approaches: merged > best

Making decisions

• Good results, but is the method applicable to other tasks?
• Double partition (x: make and model y: car buying criteria)

• Each cell mapped for facts, reduced for summary
• Tried and failed to get a single turker to produce, paying 2x total cost of CrowdForge

Writing scientific journals

• Grand challenge: Nature → New York Times
• Required 11 sub-flows, 31 sub-tasks, 262 workers, $100
• Difficult: scientific articles require time and expertise to understand.
• First sentence: identify key finding, succinctly summarize it, and also make it catchy and engaging

Evaluation

• 7/10 workers included key idea
• “Blockbusters, bestsellers, hit songs - the main variable that makes these things more popular than their lesser-known counterparts is not quality differences, according to a recent study. The answer lies in social influence - the more people know a certain movie is the one to watch, the more likely it will become popular.” (Best - 6.25)
• “The psychology of song preference. Song quality isn't the best predictor of song success.” (Worst - 1.25)

Technologies

• Python/Django server running on Apache.
• Boto.py for Amazon Web Services interfacing.

Main loop

• Poll AMT at regular periods
• Get new state of all relevant HITs, and update local database.
• Triggered events:
• resultRetrieved
• hitExpired
• hitComplete
• stageComplete
• More info: source released under Creative Commons

CrowdForge: arbitrary flows

• Flows respond to the triggered events
• Comes with pre-built flows
• Programatically define flows for tasks of any complexity

A better user interface

• Custom flows require a custom flow subclass
• CrowdForge comes with a very simple UI
• Create a powerful GUI to design complex flows without programming

More creative applications

• Writing software (partition: interface, map: method)
• Drawing pictures (map: sub-drawing, reduce: layout)
• Planning a trip
• May need more motivated/skilled workers (oDesk, TopCoder)

Thanks! Questions?

• Niki Kittur <nkittur@cs.cmu.edu>
• Boris Smus <boris@smus.com>
• Susheel Khamkar <susheelkhamkar2@gmail.com>
• Robert E. Kraut <robert.kraut@cs.cmu.edu>
• crowdforge.com