Review Comment:
This manuscript was submitted as 'full paper' and should be reviewed along the usual dimensions for research contributions which include
(1) originality: the article touches interesting modelling problems related mainly to N-Ary relations and meta-classes
(2) significance of the results: results are not demonstrated to be particularly significant
(3) quality of writing: the article is clear and well written
The article is an overview of the authors' research on a language to represent the "background knowledge" of OWL ontologies. Distinctive features of PURO are the support for meta-classes and N-Ary relations. As such, it's objective is to be a tool for high-level modelling operations for drafting, interpreting, transforming, and align OWL ontologies. The article gives a detailed overview of the language, highlights its potential with relation to key use cases, reports on two user based studies and gives a broad discussion of the work in related areas. I find the concept of ontological background appealing and useful and sympathise with the authors' claim on the impact that low-level language features/limitations impact bad practices in ontology engineering. However, the article in the present form has some major drawbacks, mainly related to the unbalance between the broadness of the problems discussed (the potential of the approach) and the actual, measured efficacy of the contribution. Also, the article length and heterogeneity of content made it difficult to review.
1 - Introduction
Motivation for the work is the limits of OWL in representing n-ary relations and meta classes and this is illustrated with two applications of the Music Ontology (MO) and Good Relations. The proposed PURO language is meant to provide "1) better human interaction with semantic web ontologies, and 2) automatic checking of certain aspects of conceptualisation coherence." However, I struggled to relate the two user studies reported in the Evaluation section to these goals.
Objective of the article is to produce a coherent union of the core aspects of the language, considering past publications only focused on partial elements.
2 - Ontological background models
This section gives a definition of ontological background models (OBM). OBMs are conceived in relation to foreground models (OFM) as the latter are encoded in another language (here OWL) and have the objective of being operational, while the former has mainly the role of capturing the state of affairs of reality at a more intuitive level. The relation between OFM and OBM is named B-modeling. However, this definition seems to imply that foreground models generally pre-exist background models, which is not true even in many use cases presented in the article. Clearly it is not the case, but then why defining it from the FM? An easier way to think about it is to assume that any model has a background model that better captures reality while being less constrained (and this would apply to any PURO model as well, right?)
The definition: 1) BM is less biased than FM and closer to reality; 2) BM is partially aligned with FM; and 3) BM is in a different representational space.
Also, the biases are not meant to be deterministic (!), BM should not introduce additional biases (but they are not deterministic, so how to check what they could be...), and BM does not need to preserve all the information of FM. Also, we should assume that "closer to reality" implies that BM has some additional semantics that is somehow lost in FM. Also Definition 2 is not very useful, it just says that you can have an BM without a FM as soon as it is "possible" - clearly it always is.
I have the impression that the definition given is too vague and that any two models in different formalisms having some domain overlap could be OBM of each other! I guess "closer to reality" is the key element here, but I struggle to see this claim as something clear enough to be useful, especially in pragmatic contexts. I think that a clear characterisation of the concepts of "bias" and "closer to reality" is needed.
Also, there is no discussion on the ontological paradigm of OBMs and OFMs- open world, close world, not relevant, why?
Ultimately, Ontology engineering has developed a significant literature on methodologies but there is no discussion about the role of OBMs in that sense. For example, the formalisation of Competency Questions (CQ) [1] has a strong overlap with the ontology drafting use case discussed here.
3 - The language of PURO
This section presents PURO as a theory in first-order logic. The section is very clear and highlights the basic differences with RDF and OWL, primarily the capacity of representing N-Ary relations (with Roles) and meta-modelling features such as multi-level typing and what has been called "heterogenous types" - covering meta-concepts such as "Deprecated Type". However, I would like to see a discussion of these in the light of OWL2 Annotation properties. Considering that PURO is not proposed for its inferencing capabilities one could just develop a hierarchy of OWL2 AnnotationProperty to support multi-level typing and annotations, right?
An interesting element of the approach of PURO to N-Ary relations is the support for directionality in participating roles. For example, an N-Ary relation may include participants A,B,C, and D, in different roles, but some of them are origin and others target in the relationship. This feature support several combination of directional relationships. I am intrigued by this feature and expected more use cases and discussion about the impact in modelling pragmatic (user studies). A similar observation can be made for the MISO patterns for multi-object placeholder, referring to the interesting problem of referring to multiplicity ("Amazon sells books", many unspecified individuals not certainly the class Book). All this is really interesting.
4 - Gluing PURO with OWL: transformations and alignments
This section presents PURO as a language complementary to OWL and discusses how to transform and align both formalism. In Definition 5 the terms source and target are used, although these do not imply directionality. I would suggest to not use them and replace them with "PURO model" and "OWL model", to avoid confusion. Several operations are discussed for mapping and linking PURO terms with OWL structures, discussed as different processes such as "PURO model operationalisation in OWL", "PURO model reengineering" (from OWL to PURO), "PURO model coverage alignment" (only specifying a whole OWL ontology that cover the PURO model), and "OWL ontology annotation with PURO". I feel this section as redundant with Section 5. Since the theory presented here about these operations is purposely "weak" (meaning non prescriptive, flexible) the conclusions do not seem insightful.
5 - PURO use cases
This section presents a number of use cases partly derived from existing publications, and developed ad different stages of maturity. The objective of the section is to demonstrate the "versatility" of the language and not to "bring convincing evidence on the superiority of PURO to alternate approaches in handling each particular problem". I may be wrong but this sentence seems counterproductive. I am convinced that PURO could be better than OWL in its capacity to support users in conceptualising certain aspects of a domain (e.g. N-Ary relations). I may be wrong but it should be objective of this section to show that PURO is better than OWL at something, and of the Evaluation section to demonstrate that.
I appreciate your task is not an easy one. Generally, it is difficult for users to deal with one formal language (e.g. Warren [2]). How to obtain evidence that using two languages would benefit users?
I think the ontology drafting is a strong use case particularly on developing specific aspects. The ontology design pattern analysis and education use case is interesting, although difficult to judge (in comparison to what? I can imagine that using conceptual maps or UML diagrams but is this being evaluated?)
Overall, use cases seem to refer to different things related to PURO: the DL Formalism, its Graphic Language, and a set of tools (the ecosystem). Sometimes is not clear whether use cases refer to PURO as DL formalism, a graphical language or as a tool to support B-operations. All these are different things. Generally, the majority of presented use cases have not being properly evaluated, many are at a "Proof of concept" maturity level, and one is purely hypothetical! This is a bit underwhelming considering the article aims at providing a general overview of PURO. I would recommend to focus on few core properties of PURO, the ones that have been better studies/demonstrated, and leave the rest as for a future work section.
6 - PURO evaluations
This section extensively presents two user study, one conducted with the authors and another with students. Evaluating, for example, whether PURO adds anything to OWL or not is a good question (Q1). However, of the various questions posed here, none of them directly address the primary claim of the article, stated in the introduction (and reported above). Again, I think the problem is that authors should be clear on what it is that PURO should be better at doing and focus on evaluating that thing(s).
However, the conclusions are not particularly strong in any direction, nor statistically relevant, therefore, it is hard to judge whether the questions have been answered or not (probably not).
Experiment 1 has been conducted with the two authors of the paper. I think it should be conducted by at least three experts, not including authors/inventors of PURO, and results measured with Inter-Rated agreement, in order to be able to decide whether: 1) PURO is coherent; 2) it's expected mapping to OWL consistent; etc... you cannot do that by yourself! The second study is described in rich detail but its results not statistically relevant. Why even reporting it? Would it be possible to organise a study whose results would be then statistically relevant? This section is also very long compared to the actual content, and very difficult to review.
7 - Discussion
I disagree that there is a thing called 'ontoligistic' thinking and that some people may be better at doing it than others. I also thought that the objective of PURO was to support people with less expertise in OWL modelling, so justifying the results saying that some users are less good at the job does not seem to be fair. Also, it has been demonstrated that even simple operations like deciding between instance-of and subClassOf is not a trivial thing (see Warren [2]).
8 - Related research
This section is good except it misses a discussion of PURO use cases in the context of ontology engineering methodologies.
9 - Conclusions and future work
I like the definition of PURO as "a language allowing to graphically draft ontology skeletons" but the article tries to put together too many things, in my opinion. It is clear that there is a lot of work behind this submission but there is as much work to be done if the authors want to support all those claims about it's validity and utility. One option would be to narrow down the scope of the contribution and focus only on the use cases that have proven to be valid, such as the ontology drafting use case, eventually comparing with alternative approaches such as using protege to draft class and property hierarchies.
[1] Grüninger, Michael, and Mark S. Fox. "Methodology for the design and evaluation of ontologies." (1995).
[2] Warren, Paul, et al. "The usability of description logics." European Semantic Web Conference. Springer, Cham, 2014.
|
