Ontologies’ Mappings Validation and Annotation Enrichment Through Tagging

The paper describes a methodology for performing ontology alignment with the help of user annotation. The main contribution of the paper is the tagging system that allows users to input free-form tags for each alignment they are evaluating, while encouraging agreement through an auto-complete mechanism for tag names. The finding that collecting annotations without constraining the vocabulary is interesting and worth publishing.

However, the main message of the paper gets lost because of several issues with the writing. It is only after reading the related work section that it is clearly stated what the claims of the paper are. The main contribution of the paper should have been clearly stated in the introduction. Secondly, the paper needs a more thorough conclusion, reiterating the methodology and the main findings, as well as discussing future work in more detail. This should solidify the message of the paper, and make the contribution of the author more clear.

There are also some inconsistencies in the evaluation. Why was a different benchmark used in section 4.1 than in the other evaluations? Also, there is no evaluation for the choice of tag clustering method (spectral clustering) - I would be curious to see how this performs in comparison with clustering using similarity in a word embeddings space, for instance.

While the work appears to present a reasonable original contribution, the main message of the paper appears muddled, and needs to be made more clear in the introduction, discussion, and conclusion. Therefore, I recommend to accept the submission conditional upon major revisions.

Other issues:
* The related work on probabilistic techniques for selecting low quality mappings only describes a single method (random sampling) - are there no other probabilistic methods? If not, why did they need a separate subsection?

* The related work is also missing a discussion on the use of crowdsourcing in ontology alignment [1,2]. [3] have also explored how to encourage agreement between annotators for an alignment task.

* Section 2.1.2 cites paper [0] that is not in the reference list.

* In Section 3.2.2, I am not sure why it was necessary to add the variable names k and w, it just makes the formulas more difficult to follow.

* The use of the T notation was inconsistent throughout the paper - it referred to both a set, a tool, or a threshold.

* On page 8, there is a reference to Section 5.2, which does not exist.

* On page 15, there is a reference to a discussion in Section 5 about techniques that generate mappings that favor precision over recall. The discussion does not appear in Section 5.

* There are issues with rendering of the apostrophe character on pages 3 and 5.

* An article such as “the” or ”a” was almost always missing before using the phrase “similarity matrix”.

[1] Cheatham, Michelle, and Pascal Hitzler. "Conference v2. 0: An uncertain version of the oaei conference benchmark." International Semantic Web Conference. Springer, Cham, 2014.
[2] Sarasua, Cristina, Elena Simperl, and Natalya F. Noy. "Crowdmap: Crowdsourcing ontology alignment with microtasks." International Semantic Web Conference. Springer, Berlin, Heidelberg, 2012.
[3] Jiménez-Ruiz, Ernesto, et al. "Limiting Logical Violations in Ontology Alignment Through Negotiation." KR. 2016.

This paper presents an approach for tagging concepts from correspondences in order to validate alignments and enrich the ontologies involved in the alignment. It proposes to identify potential wrong correspondences based on the notion of unstable correspondences and analysis of similarity matrices. These correspondences are then presented to multiple users which tag the concepts involved in the correspondences. When pairs of concepts from a correspondence share tags, the correspondence is judged as correct. A tag suggestion strategy based on information gain and an initial corpus of concepts from entity annotations is also introduced. The similarity of tags is calculated using clustering techniques. The proposed approach has been evaluated on Conference and Anatomy tracks of OAEI.

This paper presents an interesting piece of work. However, the paper could be improved in several ways.

First, the notion of unstable alignment as it is presented in the paper is not really new. In fact, this refers to the conservativity principle in [1], which proposes that correspondences should not introduce new semantic relationships between concepts from one of input ontologies. The authors should refer to this work in the paper.

Second, the paper has some passages describing details that are not really necessary in the paper (section 3.5.1) while a deeper discussion on the experiments and results is missing (how many concepts have been tagged, the users have used some interface for providing the tags ? how it deals with the collaborative aspect of the tagging process ? could it be re-used ? is it publicly available ? how many clusters have been generated ? what are the thresholds for tag similarity used in the experiments ? what is the initial tag corpus in the case of ekaw ontology which does not have annotations ? etc).

Third, but still on evaluation, I was expecting to have the whole process (wrong correspondence selection, tag suggestion, user tagging and correspondence validation) applied on the same data set. However, these different steps have been evaluated mostly independently of each other. Although it is quite interesting to have the results of each step independently, it is hard to see the real impact of the approach as a whole. As stated above, a more quantitative analysis of the results could be introduced, better explaining the weaknesses of the proposed approach and how they could be addressed (for instance, what are the reasons for having 30% of novelty, how this could be improved ? how to improve the enrichment step what seems does not bring so much (0.03). With respect to the selected matching tool, it could be interesting to see how the proposed approach could improve the alignments provided by (OAEI) state-of-the-art tools.

Finally, as the authors state, "pay as you go technique is time-consuming". Anatomy is a relatively large ontology. How they deal with this aspect in the user tag processing ?

English has to be revised.

Minor remarks (not exhaustive) :

* Abstract
- ontology metadata => ontology annotations

* Introduction
- references to "pay as you go" definitions should be included in the introduction. It is not clear the specificities of this approach with respect to classical user validation and involvement.
- "They restrict users to only evaluate the accuracy" => precision and not accuracy

* Definition of terms
- self-sufficient => self-contained
- the authors refers to "mapping" what is introduced as "correspondence" in this section. This should be corrected in the paper. the same for the "Confidence" (page 4), introduced as "v" in the definition of correspondence.

* Related Work
- "implemented by the current literature" => references
- "techniques tools are employing to reach consensus"
- "instability in the final alignment" => what does it mean "instability" ? Authors should refer to the evaluation conducted on the OAEI conference track for a work on conservativity
- "free form nature" => please rephrase
- "Multiple user evaluation" => see [2]

* TagMatch interactive framework
- First paragraphs of Section 3.2 is a little confusing with respect to the content of this section
- hasExactSynomym, exact_synonym, other_synonym : please, indicate the prefix of this predicates

[1]
@Inbook{Solimando2014,
author="Solimando, Alessandro
and Jim{\'e}nez-Ruiz, Ernesto
and Guerrini, Giovanna",
editor="Mika, Peter
and Tudorache, Tania
and Bernstein, Abraham
and Welty, Chris
and Knoblock, Craig
and Vrande{\v{c}}i{\'{c}}, Denny
and Groth, Paul
and Noy, Natasha
and Janowicz, Krzysztof
and Goble, Carole",
title="Detecting and Correcting Conservativity Principle Violations in Ontology-to-Ontology Mappings",
bookTitle="The Semantic Web -- ISWC 2014: 13th International Semantic Web Conference, Riva del Garda, Italy, October 19-23, 2014. Proceedings, Part II",
year="2014",
publisher="Springer International Publishing",
address="Cham",
pages="1--16",
}

[2]
@inproceedings{Cheatham:2014:CVU:2717260.2717264,
author = {Cheatham, Michelle and Hitzler, Pascal},
title = {Conference V2.0: An Uncertain Version of the OAEI Conference Benchmark},
booktitle = {Proceedings of the 13th International Semantic Web Conference - Part II},
series = {ISWC '14},
year = {2014},
isbn = {978-3-319-11914-4},
pages = {33--48},
numpages = {16},
url = {http://dx.doi.org/10.1007/978-3-319-11915-1_3},
doi = {10.1007/978-3-319-11915-1_3},
acmid = {2717264},
publisher = {Springer-Verlag New York, Inc.},
address = {New York, NY, USA},
}

The paper describes a method to enhance the validation of mappings via tagging.

My main concern about the paper is its lack of focus and clear contribution. The paper includes some solutions for validation, alignment, partitioning, user tagging, and perhaps an user interface, but not a clear novel contribution. The paper fails at providing a clear position with respect to the literature and state of the art systems.

Detailed comments:

* The first section or introduction contains several claims that are not convincing.
- Page 1: The authors mention issues of automatic repair techniques. Although repair techniques may not chose the most suitable plans, they have shown to minimize the number of removed mappings (e.g. []), they also have a high rate of agreement. One could also use user interfaces like in [s1] and [s2] to manually select more suitable plans.
- As a follow-up of [10] the reference alignments of the OAEI largebio tracks were "tagged" as "?" (unknown) when they were identified by LogMap, AML or Alcomo as mapping leading to logical errors (http://www.cs.ox.ac.uk/isg/projects/SEALS/oaei/2017/oaei2017_umls_refere...). The work in [s3] also tried to propose an annotation mechanism for BioPortal mappings as alternative to a repair.
- Page 2: free tagging may lead to no agreement. I'm in favour in giving some degree of freedom but a (semi)controlled vocabulary for the mapping validation should be required.
- A claimed limitation of state-of-the art systems is that they are single-user validation. However some of the referenced papers use crowd-sourcing techniques which, afaik, indicate multiple user assessment.

* Section 2.1.3.
- The referenced papers mostly deal with Consistency (the title should be changed)
- Same concerns as above. Correctness in this case may be conflictive. It depends on the initial assumption (e.g. removing unsatisfiable classes).
- Consistency, conservativity and instability have not been introduced.
- I do not fully agree with points 2 and 3 (which somehow contradict the title of the section). Although
Alcomo, AML and LogMap focus on consistency issue they also implement techniques to reduce conservativity problems. Furthermore LogMap has a module/extension which specifically deals with this type of errors [s4].

* Section 2.3. KARMA [s7] and IncMap [s6] are other relevant systems involving the user and pay as you go techniques.

* Section 3.2:
- The section of low quality mappings based on "unstability" (defined later in page 6) is not novel.
- The concept of unstable alignments has been introduced in the literature before
(e.g. [s4, s5, s8])
- The paper presents a set of patterns to identify conservativity and consistency violations. Similar patterns have been proposed in the literature (e.g. [s8, s9]).
- D1 and D2 are not used in Ontology structure dependant techniques.

* Section 3.4:
- From section 0 (page 1) it seems the free tagging was with respect to the mappings since "corresponds to" and "similar to" are given as examples. However, the tagging given by the user is in the form of
new labels. This is something that is only explained in Page 9.
- Adding new labels (from a list or new) can be challenging if the context of the entities is not given. Are users given the context of the entities being aligned?
- Instead of reusing the labels from the ontologies (which may be limited as the author acknowledges) I would suggest to use background knowledge from WordNet, UMLS or BioPortal.
- Suggesting labels from parent classes may be conflicting if they represent a much broader concept.
- The techniques given after the addition of suitable tags from users can be seen as alignment techniques. Why not using these techniques from the very beginning, using both original and extended tags/labels?

* Section 3.5.1 is interesting but the paper loses the focus. Which are the benefits of the partitioning techniques? Were they completely necessary? Almost 3 pages are used to describe the partitioning algorithm. This is in principle fine, but the paper was not about partitioning. If partitioning is a key technique for the main contribution of the paper, this needs to be clearly motivated from th every beginning.

* The Evaluation is not very conclusive and leaves the reader more confused about the main contribution of the paper.
- For section 4.1 I would have expected the use of more than one ontology. The numbers does not seem to be very promising either.
- More systems should be used in Section 4.2. For example, systems with low precision in the OAEI.
- The numbers in Section 4.3, Table 4 are not fully comparable. Why not using AML and LogMap's user interfaces?
- The increase of AML is also smaller as it already produces very good results without user involvement.
- In Section 4.3, is SBOMT or AML used? Check Figure 3 and text.

* A longer Discussion/Conclusions section was expected.

Minor comments:
- Introduction ("section 0") does not have a title
- References to section are not correct in multiple cases (please thoroughly revise the references):
e.g. Page 2 (section 2) points to Section 3.1 instead to Section 2.1. This may be due to the fact that the "section 0" has not title.
- Typo in Algorithm 1.
- Page 3: column 2, paragraph 2. M\Mu -> Mf does not reflect an iterative process

Suggested literature:

[s1] Ontology Integration Using Mappings: Towards Getting the Right Logical Consequences. ESWC 2009
[s2] A Reasoning-Based Support Tool for Ontology Mapping Evaluation. ESWC 2009
[s3] Towards Annotating Potential Incoherences in BioPortal Mappings. ISWC 2014
[s4] Minimizing conservativity violations in ontology alignments: algorithms and evaluation. KAIS 2017
[s5] Ontology Matching. Springer, Heidelberg (2007)
[s6] IncMap: pay as you go matching of relational schemata to OWL ontologies. OM 2013
[s7] Semi-automatically mapping structured sources into the semantic web. ESWC 2012
[s8] Alignment incoherence in ontology matching. PhD report. Universität Mannheim
[s9] Ontology matching with semantic verification. J of Web Semantics 2009