The uComp Protege Plugin: Crowdsourcing Enabled Ontology Engineering

Tracking #: 763-1973

Authors: 
Florian Hanika
Gerhard Wohlgenannt
Marta Sabou

Responsible editor: 
Guest Editors EKAW 2014 Schlobach Janowicz

Submission type: 
Conference Style
Abstract: 
Crowdsourcing techniques have been shown to provide effective means for solving a variety of ontology engineering problems. Yet, they are mainly being used as external means to ontology engineering, without being closely integrated into the work of ontology engineers. In this paper we investigate how to closely integrate crowdsourcing into ontology engineering practices. Firstly, we show that a set of basic crowdsourcing tasks are used recurrently to solve a range of ontology engineering problems. Secondly, we present the uComp Protege plugin that facilitates the integration of such typical crowdsourcing tasks into ontology engineering work from within the Protege ontology editing environment. An evaluation of the plugin in a typical ontology engineering scenario where ontologies are built from automatically learned semantic structures, shows that its use reduces the working times for the ontology engineers 11 times, lowers the overall task costs with 40% to 83% depending on the crowdsourcing settings used and leads to data quality comparable with that of tasks performed by ontology engineers.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
[EKAW] combined track accept

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 24/Aug/2014
Suggestion:
[EKAW] conference only accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

0 borderline paper

Reviewer's confidence
Select your choice from the options below and write its number below.

5 exprt

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

5 excellent

Novelty
Select your choice from the options below and write its number below.

3 fair

Technical quality

== 4 good

Evaluation

== 2 poor

Clarity and presentation

== 3 fair

Review

At the highest level, my issue with this paper is that the paper does not
get to the heart of difficult problems in ontology engineering. Ontology engineering
becomes expensive and hard as it gets bigger and the domain gets more complex.
This paper focuses on toy domains.

It is not clear why the task performed using the quiz cannot be completely
automated using information retrieval techniques. The relevance of a particular
term to a domain is a loosely defined relation. One ought to be able to use
frequency measures on wikipedia pages or domain documents to compute the relevance
quite reliably without using human in the loop solution.

The subclass task is not setup very well. The example considered in figure 3
is so vague, that it is not clear how one could get any reasonable response to it.
The interface does not seem to provide even textual definitions that could serve
as the basis of comparison.

The authors should make the ontologies created through this experiment open for
examination so that the reviewers could check what was acquired.

Shouldnt the crowds-sourcing be done through closed communities of people who specialize
in finance and climate change? What is the end application for the use of these ontologies?
To what extent are these ontologies specific to a particular application need?

The evaluation is weak because the only basis for correctness is inter-rater agreement the
values for which are not very strong. There needs to be at least one very high quality ontology as the baseline against which the crowd work could be measured. The low results on
the crowd agreement also indicate that we cannot yet claim that we have a successful
technique.

Review #2
Anonymous submitted on 26/Aug/2014
Suggestion:
[EKAW] combined track accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject
2

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)
4

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

5

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present

4

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Review

This paper describes a new protege plugin designed to support crowdsourcing as part of the ontology engineering design process. The well-thought out research questions in combination with the development of a real, usable plugin tool and comprehensive evaluation contribute to make this well-written paper a good candidate for inclusion in the combined track.

It would be good for the authors to discuss potential limitations of the crowdsourcing approach. In particular, in specialized domains such as scientific ontologies, a non-expert community will not be able to meaningfully contribute any of the typical crowdsourcing tasks as listed in Section 2.1. In that case the challenge becomes one of finding the appropriate community from which to crowdsource.

With this in mind, this work has relevance to the more general and difficult problem of facilitating bottom-up community engagement for ontology design. It would be interesting to see this work extended beyond traditional crowdsourcing tools like Crowdflower to investigate how tools similar to this one can facilitate ontology (and ontology pattern) design meetings where groups of specialists come together on specialized topics. ( See e.g., the VoCamp series: http://vocamp.org/wiki/Main_Page )

In section 3, steps 4 & 5 should just be collapsed into one step (i.e., step 4). If the collection and combination are significantly different steps then there should be more written on how both those steps are done (separately).

Minor comments:
Table 1 is outside of the margins.

Review #3
Anonymous submitted on 02/Sep/2014
Suggestion:
[EKAW] conference only accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
X 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
X 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
X 4 good
== 3 fair
== 2 poor
== 1 very poor

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
X 4 good
== 3 fair
== 2 poor
== 1 very poor

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
X 4 good
== 3 fair
== 2 poor
== 1 very poor

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
X 3 fair
== 2 poor
== 1 not present

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
X 4 good
== 3 fair
== 2 poor
== 1 very poor

Review:

The paper entitled 'The uComp Protege Plugin: Crowdsourcing Enabled Ontology Engineering' presents a plugin to the Protege ontology editor that enables the use of some common crowdsourcing strategies by directly embedding them (i.e., making them deployable) in the leading ontology engineering tool. The topic is interesting, important, timely, and a good match to the EKAW 2014 call. I would not rate this a classical research paper but rather a mix of a survey and a tools & systems paper. This is not so much an issue for the EKAW-only track but needs to be carefully considered for a potentially extended Semantic Web journal version.

The paper is well written and structured. It is easy to follow and the provided tables and figures are clear and contribute to the understanding of the material. One can argue whether figure 1 is really necessary but this is just a minor detail.

The paper is well motivated and provides a good overview of the related work. Section 2 is excellent and the table 1 gives a good overview of genres and addressed tasks. For a potential journal version, this part could be substantially extended to better work out the differences between some of the approaches. For section 2.1., I would like to see a comparison to tasks that cannot easily be crowdsourced. Again, this is more relevant for a potential future version. I will discuss some problems with the current presentation of 'ontology engineering' below.

Section 3 is the weakest part of the paper. While I admit that writing about the functionality and interface of a tool is difficult, this section should be reworked. A walk-through style with one consistent example may have been more appropriate. This part also begs the question whether a tool like uComp should be embedded into Protege at all. I would like to see a better argumentation here.

Section 4 provides the evaluation across the four dimensions time, cost, quality, and usability. While the evaluation is detailed and conceptually well worked out, it is also naive. Who are the 8 mentioned ontology engineers and what domain are they working on? The presented examples are toy ontologies at best, what do we learn from them about ontology engineering in-the-wild? Which test population rated the usability, the engineers?

My main concern with the paper and the evaluation, however, is the lack of a critical perspective. The crowdsourcing approach may work well for a 'default-meaning' such as 'Physics is a Science' or 'Car hasPart Engine' but it is very unlikely to work for more complex domains (and the required axioms). Even more, we would not need ontologies if we could simply agree on a canonical, simplistic, and realism-like definition of ontological constructs. [If so, we could simply hard-code them]. How can we expect the crowd to understand terms such as /Biodiversity/ (to stay with the author's climate change example) or /Deforestation/ if they differ within and between domains. The term /Forest/, for instance, has several hundred legally binding definitions in different countries and communities. Asking anonymous crowd-workers from all over the world will only lead to an semantically empty compromise. Ironically, the 'quality' evaluation in the paper points to exactly this problem. The inter-rater agreement between the experts is low, while it is high between the crowd-workers. This should have really raised red flags among the authors. In fact, this is a very common phenomenon among ontology engineers and even more so among domain experts. Likewise, it is ironic that the authors selected climate change as an example, which is among the most debated and misunderstood terms in science and society.

The question is how important this is for the overall quality of the paper. I would go as far as claiming that the examples provided in the paper could have been worked out using WordNet. On the other hand, this is a very nice tool and a solid paper on an emerging and important topic. Therefore, I would rate the paper as an accept for EKAW and propose to discuss with the authors how a potential SWJ version could look like (there is certainly potential for an extended version).

Review #4
Anonymous submitted on 04/Sep/2014
Suggestion:
[EKAW] combined track accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject

2

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

4

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

5

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

5

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

5

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Review
Please provide your textual review here.

The paper describes a Protege extension for crowd-sourcing ontology development tasks. It supports ontology developers by providing a user interface for submitting crowd-sourcing tasks, such as relevance verification and relation verification from within Protege. The approach is evaluated from different perspectives, such as time, cost, quality, and usability.

The paper does a good job of describing the problem as well as previous work in crowdsourcing for knowledge acquisition. Furthermore, the Protege extension seems to be technically sound and the evaluation is reasonable. However, there are some aspects of the paper that could be improved:

- There are several different versions/flavors of Protege, please state which versions the plug-in supports.

- Please provide more information about the background and level of experience the subjects (ontology engineers) in the evaluation.

- The cost calculation based on the wage of a research scientist is quite dubious. There are also benefits, overhead and vacation! That is, the actual cost per productive hour is much higher. It is better to just assume an hourly cost for the developer organization (which could be $26 per productive hour).

- There needs to be some discussion about what is actually evaluated here. For example, is it the Protege plug-in, the uComp framework, and/or crowdsourcing as a method.


Comments