CrowdTruth @DIR2015
•
0 likes•979 views
CrowdTruth is a framework for machine-human computation for harnessing disagreement in gathering annotated data. The slides come from our talk at DIR2015.
Report
Share
CrowdTruth @DIR2015
- 1. ★
- 4. • Annotator disagreement is signal, not noise. • It is indicative of the variation in human semantic interpretation of signs • It can indicate ambiguity, vagueness, similarity, over-generality, etc, as well as quality http://CrowdTruth.org
- 6. • Goals: ○ collecting a relation extraction gold standard ○ improve the performance of a relation extraction classifier http://CrowdTruth.org
- 7. TASK →
- 8. 1 1 1
- 9. 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 0 4 3 0 0 5 1 0
- 12. 1 0 1 1 0 0 4 3 0 0 5 1 0 Unit vector for relation R6 Sentence Vector Cosine = .55
- 15. • methodology • disagreement-aware crowdsourcing to collect gold standard data • metrics to capture disagreement • software • online platform for crowdsourcing task workflows and data analytics • ground truth collection • medical relation extraction • salience in news and tweets • sound annotation http://CrowdTruth.org