Using microtasks to crowdsource DBpedia entity classification: A study in workflow design

Tracking #: 1408-2620

Authors: 
Qiong Bu
Elena Simperl
Sergej Zerr
Yunjia Li

Responsible editor: 
Guest Editors Human Computation and Crowdsourcing

Submission type: 
Full Paper
Abstract: 
DBpedia is at the core of the Linked Open Data Cloud and widely used in research and applications. However, it is far from being perfect. Its content suffers from many flaws, as a result of factual errors inherited from Wikipedia or incomplete mappings from Wikipedia infobox to DBpedia ontology. In this work we focus on one class of such problems, un-typed entities. We propose a hierarchical tree-based approach to categorize DBpedia entities according to the DBpedia ontology using human computation and paid microtasks. We analyse the main dimensions of the crowdsourcing exercise in depth in order to come up with suggestions for workflow design and study three different workflows with automatic and hybrid prediction mechanisms to select possible candidates for the most specific category from the DBpedia ontology. To test our approach, we run experiments on CrowdFlower using a gold standard dataset of 120 previously unclassified entities. In our studies human-computation driven approaches generally achieved higher precision at lower cost when compared to workflows with automatic predictors. However, each of the tested workflows has its merit and none of them seems to perform exceptionally well on the entities that the DBpedia Extraction Framework fails to classify. We discuss these findings and their potential implications for the design of effective crowdsourced entity classification in DBpedia and beyond.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Accept

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Florian Daniel submitted on 05/Jul/2016
Suggestion:
Accept
Review Comment:

I think the authors solved the key points raised out in the last round of reviews, and the paper is ready for publication.

Review #2
By Marco Brambilla submitted on 27/Jul/2016
Suggestion:
Accept
Review Comment:

After this second round of revision, I think the paper is suited for publication, as the authors addressed with sufficient precision all the requests of the reviewers.