VOAR : a configurable web-based environment for visualizing and manipulating multiple ontology alignments

Tracking #: 2154-3367

Authors: 
Bernardo Severo
Cassia Trojahn dos Santos
Renata Vieira

Responsible editor: 
Philippe Cudre-Mauroux

Submission type: 
Tool/System Report
Abstract: 
Ontology matching is an active research area aiming at providing solutions to the ontology heterogeneity problem. While most efforts in the field are dedicated to the development of matching approaches, less has been done in terms of alignment visualization. However, providing ways for visualizing alignments is required in many tasks involving users in the process (alignment evaluation, validation, comparison, repairing, etc.). This paper presents a Web-based environment for visualizing, editing and evaluating multiple ontology alignments. Users can configure their environment choosing different visualization modes and creating different profiles. They can apply a set of operations on the alignments (filtering, merge, etc.) and evaluate them against a reference one, using classical compliance metrics. Ontology and alignment libraries allow for users storing and searching them in the system.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Reject

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 16/Apr/2019
Suggestion:
Major Revision
Review Comment:

This paper presents a web based system for ontology alignment visualization with a strong focus on multiple alignments.
The user registers online and can upload ontologies as well as alignment files.
After preparing some profiles (choosing some visualization methods) the uploaded alignments are shown.
The main contributions, as stated in the paper, are the possibility to choose different methods for visualization (of possibly multiple alignments) and the user evaluation.

(1) Quality, importance, and impact of the described tool or system:

Visualization of ontologies and alignments is quite important for improving matching systems as well as alignments.
The tool is available as a web front end at voar.inf.pucrs.br but it can not be downloaded (source code is also not available).
With the restriction that the ontologies and alignments can not exceed 1MB, a lot of ontologies/alignments can not be visualized.

The impact of the tool mainly depends on the usability.
From my trials it is quite unintuitive to work with. When you get used to it, it might be okay, but not for new users.
Two ways to improve:
1) Introduce some documentation (which is currently missing) on how to use the tool
2) Revise the UI

Some problems I faced when using the tool:
- Can only upload small ontologies and alignments up to 1 MB (already oaei anatomy [1] is too large)
- Upload of oaei ontologies like ekaw ontology with url "http://oaei.ontologymatching.org/2017/conference/data/ekaw.owl" or with file upload fails with error message:
"Ontology already loaded in library. Please verify the URI." but the ontology is not in my library (this is counterintuitive).
- The lines/edges between the concepts in the intended trees visualization are drawn as an overlay,
any scrolling will destroy the visualization (tested with Chrome and Opera).
This happens easily with a horizontal split.
- Profiles can only be created at "My Library -> Profiles". This can not be done from the visualizations tab or any other page (a lot of clicking is required).
- In the graph and node-link view, it is difficult to see the actual correspondences (also depending on the color of the edges).
- The color of the ontologies can not be changed (especially in the graph visualizations).
- Online version is quite slow in comparison to the version in the video
- Not very intuitive: e.g. one has to create a profile first which is not easily comprehensible

My biggest doubts are about the usability study/user feedback.
The participants watch the video showing how the tool should be used and what functionality is available.
Afterwards the questions are of the form: "Can the tool deal with X?" which is already shown in the video that it actually does.
Thus the high values in figure 8 and 9 are explainable.
A better user study would be to ask the participants to use the tool and answer some questions about the alignment:
- What is the precision of Matcher X?
- What error do matcher Y make? etc
Afterwards a general user experience study can be used (like UEQ [2], NASA-TLX[3] or SUS - System Usability Scale[4]).
With such an approach the usability is much better covered. This can be also done with the master students mentioned in the paper.
People from the LinkedIn group can do this as well as some ontology matching system developers.
They usually have the need to analyze and visualize mappings.

The next contribution of the paper is to visualize multiple alignments.
From what I see in the figures this is done in the same visualization at the same time (like in figure 5).
Currently, I don't see any reason to do that. What information need can be satisfied with such a visualization?
If there are some, then they should be also introduced in the paper to better motivate the approach.

Some improvements regarding the visualization:
Another way of visualizing ontologies is VOWL[5] .
The authors can just use the web VOWL version and incorporate the alignment with "owl:equivalentProperty" or "owl:equivalentClass" assumptions.
Another way would be to just introduce some of the specification which is used to visualize OWL concepts (otherwise more complex OWL axioms are not shown).

The paper is well written and easy to follow.
It shows the capabilities and the limitations of the tool whereas for example figure 4 shows not more content that figure 5 and should be removed.

Some sidenotes for further improvement of the paper:
The link of the video should be referenced in the paper at the correspondent section 5 otherwise it is not easy to find.
The paper contains some formatting and spelling errors e.g.:
- p1: Visualizing tools -> Visualization tools
- p1: offer limited support to -> offer limited support for
- p3: Few Web-based user interfaces have been provided, as the for LogMap
- p6: rigth of Figure 6).
- p9: left column has a different line spacing

To sump up, each of the main contributions of the paper should be improved or at least motivated:
-Why is the simultaneous visualization necessary? What can be analyzed?
-Improvements of the usability of the tool
-Improvement of the user study to reflect the actual user feedback.

[1] http://oaei.ontologymatching.org/2018/anatomy/
[2] https://www.ueq-online.org
[3] https://humansystems.arc.nasa.gov/groups/TLX/
[4] Brooke, John. "SUS-A quick and dirty usability scale." Usability evaluation in industry 189.194 (1996): 4-7.
[5] http://vowl.visualdataweb.org

Review #2
Anonymous submitted on 13/May/2019
Suggestion:
Minor Revision
Review Comment:

The article on VOAR presents a tool for visually inspection of (automatic) ontology matchings and their manipulation.

The article defines the specific contribution of VOAR to enable the user to create profiles, GUI set-ups, for the task at hand. The article further introduces shortly the problem field of ontology matching and the related work it is based on followed by a on by one introduction of the different visualisation modalities. Finally a small user evaluation hints that the introduced profiles and therefore the overall concept are helping in the task of manually controlling the ontology matching.

The task of ontology matching is important and will become more important with the future independent introduction of always more specific ontologies in all fields of data management. This task is intrinsically hard to solve as it needs non explicit domain knowledge to validate that two ontologies (or parts of them) are matching. Though the oversight of domain specialists will be necessary for the foreseeable future, and therefor an efficient presentation of the ontology matches too.

The realisation that different sized ontologies call for different types of visual representation is most interesting. Unfortunately in the article the scale of ontologies is missing. This article would heavily profit of approximate size of ontologies (e.g. in total number of properties) which each type of visualisation is fit to handle. Most important but it is to define at least approximately what you call a “large ontology”.

Finally we like to investigate if there are any plans to open source the VOAR application.

Minor remarks:
- On page 9 left bottom paragraph has another line height then the rest of the article. Probably it is possible to refine some sentences to fill this paragraph.
- On page 9 orthographic: This should be better hightlight_
- On page 9 orthographic: we do not _made_ exhaustive tests

Review #3
Anonymous submitted on 08/Jul/2019
Suggestion:
Minor Revision
Review Comment:

The paper presents a web-based tool for visualizing and manipulating ontology alignments. The main idea on which the tool is based is that providing the user with several visualization methods and allowing simultaneous visualization of multiple alignments may facilitate the ontology alignment evaluation process. Thus, it offers visualization modes, such as trees and graphs, and the user can create several visualization profiles and switch between them depending on his needs. The tool is evaluated qualitatively by a limited number of users that were asked to answer a questionnaire regarding their experience using the tool.

The presented tool is a well-informed attempt to make a contribution to the ontology alignment visualization problem; it achieves that by supporting the several visualization modes and profiles. However, the paper focuses on describing the visualization modes, and discusses only very briefly other crucial functions that are available to support alignment visualization and editing. The paper mentions e.g. that trimming, filtering, etc. operations are available, but since it does not provide a more thorough presentation, the level of their support is unclear. In this respect, Section 4.4. should be extended to discuss in greater detail what exact type of operations are provided (i.e. what search, filtering, grouping, etc. options are available; how a "criterion" may be defined) and whether such operations are available in all visualization modes apart from the correspondences list and can be dynamically applied on any of them. Moreover, with respect to multiple visualization modes, although it is useful to provide two visualizations side-by-side, one could imagine a more dynamic interface where e.g. additional visualizations (more than two) might be added or removed as needed.

Apart from the above, as recognized by the authors themselves, the main drawback of the tool is that its usefulness is limited to rather small ontologies. The paper should state whether this is a scalability problem due to the fact the system cannot handle the computational load of the visualization of a very large number of alignments, or because the practical value of a complete graphical visualization of alignments is reduced when ontologies are very large. In this respect the authors could consider combining dynamic filtering, grouping, etc. operations discussed above as well as other approaches when visualizing alignments for large ontologies.

Finally, the practical use of the tool would be further supported if the authors provided some real use cases and challenges (apart from introductory courses on ontologies) where the tool was successfully used in practice.

Some other comments:

p.3 c.2 l.4-11: It should be defined what are simple/single, and complex/multiple alignments, so that e.g. the two sentences "However, they (i.e. [2,8,25]) are limited to visualization of **single** alignments and offer limited support ...", and "Differently from [4], VOAR is limited to the visualization of **simple** alignments" are clear and do not contradict each other.

p.5 c.1 l.7: "offer" -> offers

p.5 c.2 l.9-11: "Similar to the... 6)". sentence in bad English, please rephrase

p.5 c.2 l.23: "has an inefficient use of screen space displayed" -> phrase in bad English, please rephrase

p.6 c.1 last sentence: "Allowing ... preferences." the sentence seems to be incomplete, please check

p.7 c.2 l.6: "rigth" -> right

p.8 c.2 l.-10: "as" -> is

p.9 c.1 l.-3: "positives" -> positive