Ontology Matching: Current trends among practitioners

Overall evaluation: -2
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject

Reviewer's confidence: 4
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

Interest to the Knowledge Engineering and Knowledge Management Community: 3
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

Novelty: 2
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

Technical quality: 3
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

Evaluation: 3
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present

Clarity and presentation: 4
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

Review

This paper aims at presenting the current trends of the research in ontology matching based on a questionnaire answered by participants in the OAEI contests. Eight open-ended questions were included in the questionnaire, being background questions or questions relative to the research field or to future challenges.

The objective is interesting and the study was properly conducted but for my point of view, it is a preliminary work to prepare for a more thorough survey based on a multiple choice questionnaire which could be established from the results of that first work. Such a deeper analysis will find its rightful place in a journal on Semantic Web.

The current conclusions are not sufficient to be specifically published either from a quantitative or a qualitative point of view. Open-ended questions are difficult to analyze and do not easily allow to draw clear conclusions. The study is too low to support generalizable conclusions. The number of respondents is not sufficient. 33 out of 288: is it a representative sample? They are all academic researchers. It could be good to contact also users in industry. Furthermore, some conclusions are obvious. Generally speaking, the reach of the current findings is not sufficient.

Overall evaluation
Select your choice from the options below and write its number below.

-3

Reviewer's confidence
Select your choice from the options below and write its number below.

5

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

4

Novelty
Select your choice from the options below and write its number below.

2

Technical quality
Select your choice from the options below and write its number below.

2

Evaluation
Select your choice from the options below and write its number below.

1

Clarity and presentation
Select your choice from the options below and write its number below.

3

Review
Please provide your textual review here.

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
=X= 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject

Reviewer's confidence
Select your choice from the options below and write its number below.

=X= 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

=X= 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
=X= 4 good
== 3 fair
== 2 poor
== 1 very poor

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
=X= 4 good
== 3 fair
== 2 poor
== 1 very poor

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
=X= 4 good
== 3 fair
== 2 poor
== 1 not present

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
=X= 4 good
== 3 fair
== 2 poor
== 1 very poor

Review
Please provide your textual review here.

This is a very interesting paper that looks at the current practices for
ontology mapping through the implementation of a survey completed by
practitioners.

The survey itself appears to be well-designed, with questions that
capture key aspects of the subfield of ontology mapping.
The paper contains many valuable insights along these lines,
and it should generate beneficial discussions at the conference.

Nevertheless, I am not certain that this approach offers insights
into the specific technical problems underlying ontology mapping.
What is needed are empirical and theoretical analyses.

Alternatively, another approach would be to take the challenges
identified by current researchers and specify these problems precisely
enough that they can serve as research problems that can shape future
directions (a la Hilbert's Problems in mathematics).
Of course, this is not the paper that was written, but it is the kind
of in-depth analysis and synthesis that would be more appropriate
for a journal article.

There are numerous grammatical errors (e.g. missing articles) throughout the paper.

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
X -2 reject
== -3 strong reject

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
X 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
X 4 good
== 3 fair
== 2 poor
== 1 very poor

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
X 4 good
== 3 fair
== 2 poor
== 1 very poor

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
X 2 poor
== 1 very poor

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
X 2 poor
== 1 not present

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
X 3 fair
== 2 poor
== 1 very poor

Review:

The paper entitled 'Ontology Matching: Current trends among practitioners' presents a survey among ontology matching researchers to uncover current and future research trends. This is an important and timely topic and suitable for EKAW 2014. The paper was submitted to the combined SWJ/EKAW full research paper track. This kind of (mostly anecdotal; see below) survey is not a good match to this track. I will argue below why it is also not suitable for the EKAW-only conference proceedings track and then propose to reject the paper and submit it to the 2015 edition of the ontology matching workshop at ISWC.

The paper's structure, language, and presentation need major revision but are still readable and easy to follow.

Unfortunately, the authors made two far reaching decisions that prevent me from recommending the paper. First, they only surveyed OAEI authors. Second, they asked for free text answers to overly generic questions. Additionally, the selected questions do not really reveal current trends (which is a pity given the paper's title). The authors should have included OAEI reviewers in their survey instead of only authors and should have reached out to communities that are interested in ontology matching and alignment but do not work on Semantic Web topics. These researchers are not well represented at OAEI but rather publish at FOIS and related events.

More importantly, however, are shortcomings related to the questions. Instead of preparing a pre-study and use the gained insights to ask more concrete and detailed questions, the authors decided to go with 8 rather generic questions. Four of those questions (1-4) are not really interesting and would typically only appear as meta-data about the participants. Unsurprisingly, what we are left with could equally well be taken out of the call for papers of this year's OM 2014 workshop. The very same argument can be made about the results 'uncovered' by the survey. The authors state that the participants raised concerns about the lack of real applications of ontology matching and pointed out the role of humans in the loop. For instance, in section 3.3 the authors list 'Human readable explanations for matches' while the OM 2014 call for papers lists 'Explanations in matching'. Similarity, the survey results point to the need of applications and the OM call asks for 'Matching for traditional applications (e.g., information integration)' and 'Matching for emerging applications (e.g., search, web-services).' On the one hand it is, of course, good that the survey confirms the CfP. On the other hand, however, it is not clear why it needs the survey? Finally, the only other important aspect brough up in the survey are more expressive matchings (called 'not 1:1' in the paper). Again, this is well known and has been discussed in the literature.

In addition to the problems outlined above, the survey results are presented in an anecdotal form which clearly reduces their value and hinders a more details, statistical evaluation.

Summing up, this is interesting and valuable work. However, the paper is rather suitable as a short paper at the OM workshop or a technical report. Maybe it could also be used as a pre-study for more mature work in the future.