CrowdForge

Crowdsourcing Complex Tasks

Niki Kittur, Boris Smus, Susheel Khamkar, Robert E. Kraut

Crowdsourcing is important

The problem with crowdsourcing

Designing for collaboration

Doesn't work well enough

Prior work

Our research contributions

  1. Framework for accomplishing complex and interdependent tasks from many small contributions in crowdsourcing markets
  2. Implementation in a software toolkit and using this toolkit to investigate the applicability, strengths, and weaknesses of the framework across a diverse range of tasks

Task decomposition

MapReduce inspiration

CrowdForge

Notes

Example

Given a topic for an article

  1. Partition topic into sub-topics
  2. Collect facts on the topic
  3. Combine these facts into a paragraph
  4. Finally merge all of the paragraphs

Applications

Writing articles

Given a topic for an article

  1. Partition topic into sub-topics
  2. Collect facts on the topic
  3. Combine these facts into a paragraph
  4. Finally merge all of the paragraphs

Analysis

Results

Approach to quality

Making decisions

Writing scientific journals

Evaluation

Implementation

Technologies

Main loop

CrowdForge: arbitrary flows

A better user interface

More creative applications

Thanks! Questions?