VOWL 2: User-Oriented Visualization of Ontologies

Tracking #: 750-1960

Authors: 
Steffen Lohmann
Stefan Negru
Florian Haag
Thomas Ertl

Responsible editor: 
Guest Editors EKAW 2014 Schlobach Janowicz

Submission type: 
Conference Style
Abstract: 
Ontologies become increasingly important as a means to structure and organize information. This requires methods and tools that enable not only ontology experts but also other user groups to work with ontologies and related structures. We have developed VOWL, a comprehensive and well-specified visual language for the user-oriented visualization of ontologies, and conducted a comparative study on an initial version of VOWL. Based upon results from that study, as well as an extensive review of other ontology visualizations, we have reworked many parts of VOWL. In this paper, we present the new version VOWL 2 and describe how the initial definitions were used to systematically redefine the visual notation. Beside the novelties of the language definition, which is based on a well-defined set of graphical primitives and an abstract color scheme, we briefly describe two implementations of VOWL 2. To gather some insight into the user experience with the new version of VOWL, we have conducted a small qualitative evaluation. We report on the study and its results, which confirmed that not only the general ideas of VOWL but also various of our enhancements for VOWL 2 can be understood by casual ontology users.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
[EKAW] combined track accept

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 19/Aug/2014
Suggestion:
[EKAW] combined track accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 2 accept

Reviewer's confidence
Select your choice from the options below and write its number below.

== 4 (high)

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent

Novelty
Select your choice from the options below and write its number below.

== 5 excellent

Technical quality
Select your choice from the options below and write its number below.

== 5 excellent

Evaluation
Select your choice from the options below and write its number below.

== 4 good

Clarity and presentation
Select your choice from the options below and write its number below.

== 5 excellent

Review
Please provide your textual review here.

This paper describe VOWL2, a visual language for visualizing
OWL ontologies aimed at "casual users" of ontologies. VOWL2
builds upon VOWL1, taking profit of a first user study to
improve its principles. Two implementation
of VOWL2 are presented, as well as a qualitative study
that confirms the direction that have been taken by the
authors.

All in all, this is a nice paper on OWL ontology visualization,
written by infoviz specialists. The proposal is interesting and
has the potential of becoming widely used.
The paper is densely yet carefully written, it has lots of contents,
a good related work section, and it obviously describes mature research.
Though only quantitative, and with few participants (6),
the evaluation has been well conducted and provides interesting
results and confirmations, as well as insightful discussion on
open issues.

I also enjoyed reading remarks that answered most of the questions
that did arise to my mind only some paragraphs after they did.

As a conclusion this paper should be accepted for presentation
at EKAW 2014. I am also in favor of its acceptance for
fast-track publication in the SWJ, provided that it
_changes its structure_ to focus more on VOWL2 by not insisting
too much on the historical developments from VOWL1 to VOWL2.
While describing incremental work seems ok for
a conference paper, the journal paper should focus more on
VOWL2 as a final proposal. Of course, and as stated in the call
for paper, the journal paper should also be largely extended:
an extended related work section with some visual examples,
a more thorough presentation of VOWL2 and a deeper
user study (with statistically significant results) would fit well.

Some more remarks along the paper:
- I would like to get more information on the way your proposal
can take into account or visualize the A-Box / instances from
an ontology. There is a bit of discussion on that topic p5, that
seems quite insufficient to me. You could consider this an
open issue, as instances cannot be dismissed easily, all the more
so when ontological modeling has to use instances to represent
"general knowledge".
- I would like to get the rationales
for the choices you made for graphical primitives or color schemes,
not just considerations on the differences between VOWL1 and VOWL2.
- please give short examples of some "guidelines on how the colors
should relate to each other" (p5) and on "how to combine several
of the additional text markers"
- I did not see "additional ontology-related information on the
selected elements" (the "sidebar" ?) (p8) on the online demo
- It is not that clear what are these "18 tasks, six per visualization,
and among those, three for each of the two ontologies" used for the
evaluation. Does your example task "Which classes appear to be crucial
for the ontology?" (4.1) count for one task or for three tasks? Maybe
you could clarify this description.
- I guess you could have had some more quantitative results, for
example related to accuracy and time used to complete the tasks. The fact
that you write about "within-subjects design" and "counterbalancing" (p10)
seems to indicate that the evaluation setting aims at statistically
significant results, but you may well spare for a journal
version of the paper.
- as stated by some participants of the study (p11) and discussed p12,
the double-ringed circles used for equivalent classes appear to raise
some questions, so I would have had a hint on how to improve your
proposal on that point.

Review #2
Anonymous submitted on 25/Aug/2014
Suggestion:
[EKAW] combined track accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject
2

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)
4

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4
Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4
Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present
4
Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4
Review
Please provide your textual review here.
The paper describe a system to visualize ontologies, its two implementations - one as a Protege plugin and one as a web application based on D3 - and a user evaluation study. The WebVOWL application is particularly interesting, given that the approach based on a json structure independent from the specific owl target ontology structure and parsers and the ability to display in a web browser makes it widely applicable. On the negative side, the design approach of providing users with a complete view instead than a stepwise approach, while it has its own merits and advantages seriously risks to impede effectiveness with real life ontologies of even moderately large size; this is also acknowledged by authors among the open issues to be further worked out. The other open issue reported, about the need to store location information from previous sessions so to mantain a fixed visualisation structure over time is a highly relevant one to make the system ore effecive in real usage.
The design choice about graphical primitives and color scheme looks reasonable, and the motivation from changes from previous version reasonable and well discussed. Evaluation, albeit limited in size by number of evaluators (6) has been designed well and seriously carried on.

Review #3
Anonymous submitted on 30/Aug/2014
Suggestion:
[EKAW] combined track accept
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject

1

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

4

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

3

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present

3

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor

4

Review
Please provide your textual review here.

The paper describes VOWL 2, developed based on a user study carried out with the first version. VOWL aims to support exploration of ontologies using a dedicated visual notation for graph visualisation. Its two implementations are targeted especially at casual users of ontologies. The authors carry out a good review of relevant ontology visualisation tools, and, by identifying gaps in requirements for support for the kinds of tasks users would carry out, describe how their implementation compares with existing tools.
The paper concludes with a discussion of areas in which the implementation could be improved.

The paper is, overall, well written and easy to follow. However, the discussion/conclusions in section 5 are a bit empty. In fact, what they really do is simply continue the previous section on results. I really can't see what new is said here. Maybe clearly stating in the introduction, or at the start of the evaluation section what key areas were to be addressed in this study, esp. wrt to EKAW, and then returning to those in the discussion might help? As it stands the evaluation seems to simply look at general usability. It could also be that I'm missing what was key because the evaluation tasks are not fully described (see comments also below).

**********

I disagree with the second half of this statement: "Many approaches visualize ontologies as graphs, which is a natural way to depict the concepts and relationships in a domain of knowledge." - this is natural ONLY if that actually maps to the structure of these relationships. This IS the case here, but the statement is general. And it appears to be the justification for the visualisation approach selected. (Later in the paper the authors make a similar statement but then specify that it is with reference to this particular type of data structure.)

I'm not completely clear, apart from implementation tools used and the latter being a plug-in, what the differences are between WebVOWL and ProtégéVOWL. If the latter had been available for evaluation would it have been worth comparing them against each other? OR would the results have been the same with either (everything else being equal)?

Also not clear if scaling IS available or not for the circles, or if this is saying that there are multiple options for the user to choose from.

I am conscious of space restrictions, otherwise would be useful to have a copy of at least one of the graphs in Fig. 2 in VOWL 1 to illustrate clearly where the changes are, especially as there are a few differences between what is shown in Fig 2a and of the MUTO ontology (Fig. 3) in the previous paper.

Evaluation

The impression is given that previous versions of SOVA do display datatype properties - if so, why not simply use the last version that did? Simply because this does represent a confounding factor. Unless, of course, there are improvements in the newest version with greater influence on the tasks evaluated?

The evaluation is described as within-subjects. However, this then follows: "In combination with the counterbalancing of the visualizations, this resulted in a setting where each task had to be solved for each of the visualizations by some of the participants. " - if "some" then must've been between, not within.

What were the actual tasks - the number and distribution is given, but not the tasks themselves, apart from the first set examples of TYPES of questions. A bit difficult to interpret the results otherwise. Were the same questions/tasks repeated for each ontology? Were they modified based on size or complexity?

"Overall, VOWL 2 was assessed to be well readable due to the comparably low number of edges and, in particular, edge crossings. This effect was achieved both by avoiding several edges present in other visualizations, such as between equivalent classes, and by applying the splitting rules for node multiplication."
Would this not have an effect on tasks that require a user to identify equivalent classes and/or their properties, for example? What alternatives exist to handle such cases?

"Two more participants asked for a clarification, as they wondered whether there would be multiple copies of the double-ringed equivalent class circles, one for each equivalent class, or only one in all." - do they or do they not - I can't tell either, not from the text.

"Looking at the multitude of approaches ..." - a fair number of approaches are discussed, but "multitude" is a lot more than would fit in a paper.

"were able to solve the vast majority of the tasks ..." - there were only 18 tasks, I wdn't call that vast...

"most of the graph would align itself automatically according to the users' input. " - align itself to what? What is the users' input here?

"Nonetheless, the force-directed layout led to highly connected nodes that could be easily identified, as intended; in this case, the significant, deeply integrated classes of an ontology would clearly stand out. " - don't really understand what this is saying - what exactly is a "deeply integrated class"?

*** OTHER POINTS ***

"Both implementations support interactive highlighting and display additional ontology-related information on the selected elements on demand to complement the visualization." - what exactly is this information? Maybe highlight it in one of the diagrams?

" ...even though both SOVA and GrOWL include some formal symbols." - would be useful to provide a few examples and compare to how VOWL represents these in a more intuitive manner.

*** Minor grammatical errors, e.g., (not all)

"various of our enhancements" either "various enhancements" or "various [elements/aspects] of our enhancements" or replace "various" with "some"

"A smaller number of works provide ..." -> "A smaller number of work provide[s]" - no 's' - work is not countable, and verb matches "a smaller number" (the subject)

"overview over " -> "overview OF"

"... relevant to the study comprising of classes, properties" - "comprising classes..." (no 'of')

"without getting in touch with their formal representations ..." - you get in touch with a person (or your inner self), not an inanimate object, should say something like "without making use of formal representations... "

"come in contact with ..." -> "come into contact with ... three visualizations ..."- but, as above, really used with people or, say, a disease, something simple like "had used/tried any of the ... three visualizations ..."

"the introduction into ontologies." -> introduction to

"of any doubts" -> "of any doubt" - not countable

"Gaph visualizations" -> gRaph

automatical -> automatic