Linked data schemata: fixing unsound foundations

The paper makes a number of helpful observations and a number of useful
conclusions related to combining third-party ontologies. There is some
reasonable work here, but also a couple of significant and possibly fatal
flaws in the work.

On the presentation side, the paper needs to be clearer about what is going
on. I have mentioned quite a number of these problems below.

On the work-performed side Dacura does not appear to be able to handle
ontologies of even moderate size, e.g., the DBpedia ontology. This is a
significant limitation as the DBpedia ontology is, in some sense, the
ontology of core of the linked data universe. As well, Dracura does not
handle OWL disjointness or equivalence. As these two aspects of OWL are
among its most-used parts in linked data, this is a severe limitation.

In a conference paper I don't think that the last two flaws would be a severe
problem, as one might expect a follow-on paper that improved the scope of Dacura.
In a journal paper, however, these two limitations are very significant, as
Dacura's analysis of just about every significant linked data ontology will
thus be severely partial. Just these two flaws are verging on being fatal
for the paper.

The paper needs some significant additions to be understandable to a wider
audience, particularly linked data practitioners. It mentions quite a few
points from the literature without adequately describing them. Researchers
with a deep background in logical modelling and a knowledge of the history
of the Semantic Web, RDF, and OWL will understand, but other readers will
have problems determining what is going on with, for example, predication
over classes and properties. It would be helpful for many readers if the
paper had examples for these points.

The placement of figures and tables in the paper is so far from ideal as to
make the paper harder to read.

Challenges

The paper needs to better define what a linked data schema needs to be. I
take it that the only requirement is that it is coherent (i.e., has no
necessarily unsatisfiable classes), consistent when combined with the data
at hand, and mentions each of the classes and properties (and datatypes?) in
the data at hand. This can be easily achieved with an empty schema, which
is probably not what is wanted. The paper also does not specify what
validation is.

The papeer claims to have an indentification and analysis of practical
and theoretical challenges. It thus needs to have a clear and complete
presentation of these challenges. The paper is not horrible in this area,
but has some significant flaws, which I outline below.

Challenge 1.

The paper should explicitly mention the mismatch between OWL and RDF(S),
which I take to be 1/ misuse of the (important) reserved vocabulary and 2/
using RDF lists (which is also a misuse of reserved vocabulary). (The other
mismatches are allowed in OWL 2 DL.) The paper should be careful to abide
by OWL 2 DL throughout, particularly as some of the works that it
prominently mentions predate OWL 2.

The paper appears to want to handle inputs that use old formalisms, e.g.,
DAML. No one should be using DAML at this point, as it has been superceeded
by OWL. The paper doesn't explicitly mention using RDF containers, but no
one should be using RDF containers either. [what to do if this happens?]

I don't think that the prohibition in OWL is on referring to classes of
classes. These are part of OWL 2 DL via punning. It is certainly the case
that OWL does not perform the full set of inferences on such constructs, but
that is by design. OWL 2 does not even do all the inferences that RDFS does
on these constructs.

Challenge 2.

Examples of how "follow your nose" goes wrong would be useful here. There
has been quite a bit of discussion on this problem.

Challenge 3.

Reuse is indeed a tough challenge. One of the problems is actually deciding
what kinds of alteration are bad. It is not useful to outlaw any alteration
because even adding a subclass to a class can be considered to alter the
existing class. The paper should provide some information on what sorts of
alterations are considered problematic.

The paper's description of the effects of modification is not correct.
It is not the case that an ontology's changes to another impact all users of
the other ontology, unless one believes that all ontologies are present in
all places. The paper needs to show where such changes actually become
problematic without postulating non-overridable use of all ontologies in the
web.

Challenge 4.

It is not that OWL is permissive per se. Instead OWL works differently from
object-oriented systems. This is indeed a potential problem.

It is not just OWL that works this way. RDFS has the same behaviour.
Singling out OWL as the problem is thus not appropriate there. In fact, the
example given fits within RDFS.

Related Work.

The description of SPARQL-based work should note that RDFUnit is very close
to being a union of SPIN and Stardog ICV (the successor to Pellet ICV).

There should be more mention of the work on integrity constraints in
Description Logics. One branch of this work underlies Stardog ICV and the
entire body of work can be easily found by looking at the papers that gave
rise to Stardog ICV and also from [Patel15].

Validation (description of Dacura)

As one of the claimed contributions of the paper is a description of Dacura,
the paper needs to describe Dacura in a clear and complete manner.

The basic notions underlying Dacura are fairly evident from the paper.
However, some of the details are not so clear. How is Dacura started? Do
you give it a URL and it goes from there? Does it follow its nose on all
IRIs it sees in its inputs, thus including the RDF, RDFS, and OWL definition
documents? Can Dacura work well starting from an RDF document that is not
an ontology?

It is strange that Dacura does not appear to support owl:disjointWith,
owl:equivalentClass, owl:differentFrom, and owl:sameAs. Disjointness is
perhaps the most important aspect of validity in an ontology that uses
open-world semantics. Equivalence is a very common tool to link ontologies
together in linked data (see the DBpedia ontology for many examples). Even
if Dacura uses a unique names assumption, it should handle owl:sameAs by
identifying the two individuals involved.

The particular solutions to the challenges above that are implemented in
Dacura seem largely reasonable. Some of these solutions are also
implemented in some OWL tools such as the OWL API. The paper should mention
these.

The general solution for properties that are not stated as object or domain
properties is to make them annotation properties, would this be possible in
Dacura?

It is not until this section that the version of "follow your nose" that
Dacura uses is given. This version is different from the standard one in
linked data. The paper should discuss these differences. An example would
be helpful as well. Having Dacura follow classes mentioned in disjointness
axioms even though its reasoner doesn't handle them is the correct decision.
However, it is not obvious that the listed terms exhaust all the ways that
OWL ontologies effectively use classes. The paper should be clear as to
whether Dacura actually follow all usese of classes in an OWL 2 ontology.

It appears that a triple of the form "object rdf:type class" does not induce
a structural dependence on the ontology that class comes from. On the other
hand a triple of the form "object property object" does induce a dependence
on the ontology that property comes from. Why is this so?

As ontology hijacking is not defined in the paper, it is hard to determine
whether Dacura does a reasonable job of detecting problematic axioms in
importing ontologies.

It is not really the closed/open world dichotomy that distinguishes between
what Dacura is doing from what is done in OWL. Orphan classes in OWL, or
indeed in RDFS, are orphan classes even in an open world setting. This is
not to say that detecting orphan classes is a bad idea, just that a
different discussion of the problem is needed.

My takeaway is that Dracut could be useful, but that it is hamstrung as an
analysis tool for linked data ontologies because it does not handle
disjointness and equivalence.

Evaluation

OpenCyc is quite large, partly because it contains a significant number of
individuals, and thus might be too expensive to process. However, it would
have been worthwhile to do a partial pass over it to see what other
ontologies it may drag in. (In my opinion this could easily be none.)

The DBpedia ontology, on the other hand, is not large at all and is very
simple, containing only a few OWL constructs like disjointness and
equivalence as far as I can recall (although more may have been added
recently). The inability of Dacura to process the DBpedia ontology is quite
surprising, and rather difficult to understand. The paper should try to
explain this significant limitation of Dracura. Even if the DBpedia
ontology cannot be handled by Dracura, it would have been useful to also see
what other ontologies it brings in.

DBpedia is an interesting case for Dacura. It includes equivalence
relationships into schema.org. Unfortunately Dacura does not handle this
part of OWL. DBpedia also includes disjointness which Dacura also does not
handle. As the DBpedia ontology can be considered to be the core ontology
of linked data, this is a decided problem for Dacura.

Schema.org is another interesting case for Dacura. It is uncertain whether
Dacura would be able to handle schema.org. This might be reasonble, as the
schema.org ontology doesn't quite fit correctly into linked data. However,
schema.org is becoming an important part of linked data, and thus an
important target for Dacura.

Results

The results listed are interesting, and indeed are rather depressing. Some
of the observations in this section can be argued against, for example that
external equivalences are erroneous, but the general thrust is reasonable.

There is one claim in this section that is completely incorrect. XML Schema
includes an int datatype (see http://www.w3.org/TR/xmlschema11-2/#int) and
has so done for quite some time. This error needs to be fixed.

The text also claims that three ontologies have tyops, but Table 6 lists
four ontologies, two of which are not problematic.

Quantification is not a notion generally associated with RDF. The paper
should describe what it means by quantification over classes and predicates
in RDF(S).

The problem with the dc ontology is misstated. It is dcterms:type that has
range of rdfs:Class.

Usage of the RDFS built-in vocabulary as domains, ranges, etc., is not
exactly higher-order logic. A better description of what the problematic
aspect of these uses are is needed.

In Table 7 some (mis)uses of OWL vocabulary are described as "overriding"
and others as higher order. Neither is quite true. It would be better to
describe these as misuse of reserved vocabulary.

Recommendations

The recommendations are, of course, the authors' opinion. One might argue
for different recommendations but none of the recommendations in the paper
are bad.

However, it is strange that there is one recommendation to avoid using
lists, but another recommendation to support misuse of the OWL reserved
vocabulary. Why not suggest a way of handling the use of RDF lists in
linked data ontologies?

I also would have liked to see some notion of how to overcome the problems
that exist. The OWL API can fix problems with missing declarations. Why
not advocate for tools that can fix other problems?

It would be interesting to see whether the misuse of the reserved vocabulary
outsdie of lists can be solved via a mechanism as simple as punning.

My view is that the work reported on is quite reasonable and useful but that
paper needs a full rewrite. The paper needs to start out with a firm
definition of what the problem is, one that is accessible to linked data
practitioners. It should then explore each of the parts of this problem,
providing definitions and examples so that the paper can be understood by
linked data practitioners. Then there should be a clear description of the
parts of Dacura that are relevant here. Finally the paper should actually
propose fixes to be in line with its title.

These changes are likely to add considerably to the length of the paper, but
I think that the result would be a much better paper.

This paper presents a discussion of implementation flaws in OWL DL ontologies and presents a web service for identifying these errors. The paper focuses on errors that are caused primarily by incompatibilities with reasoning in OWL DL over linked data models, which has some validity but is behind the state-of-the-art for linked data, where practitioners have largely rejected OWL and strict ontological reasoning. More recent techniques, notably SHACL, are mentioned only briefly and dismissively by the authors. It would be more interesting if this paper was more focused on errors that typically bother linked data practitioners (e.g., as their "typo" module does) than minor OWL issues, that seem to have little practical significance (as proven by the continued existence of such errors in major vocabularies).

The evaluation of this paper highlights some predictable errors in widely used ontologies, but does not really evaluate the methodology of this paper. As a reader, one has very little idea if this methodology captures all errors or captures more errors than a standard OWL-DL reasoner would. While the errors detected by the authors are of interest to the community, it does not provide a sufficient evaluation for a journal article.

This paper is not in the SWJ format.

The inability of the proposed system to handle circular subsumptions is concerning. Why did the authors chose a limited reasoning framework such as Prolog?

More care should be taken when citing ontologies. For example the WordNet ontology (the official one supported by Princeton and the Global WordNet Association) is available. I assume that some other one (I would guess the VU one) is being referred to. Similarly, models like SKOS and Dublin Core can be more clearly cited.

The typos section is insufficiently explained, is it simply a check for all entity names not declared in their defining ontology? If so how does it detect that XSD has a misspelled URL (this could be another ontology that is unavailable)? How does this compare to other contributions in the field of linked data quality.

The authors make a big deal about the use of rdf:List in schema definitions, claiming that it should be "stamped out in the wild". Actually this is quite acceptable in RDFS models and I wonder why they take such a hard line to this, especially as there is no standard OWL solution to this very basic problem (the authors do suggest some rather obscure solutions).

The authors do not reference the Luzzu system of Debattista et al., which is highly relevant here.

Minor
=====
p1. Semantic Web should be capitalized
p1. Linked data should not be capitalized (or at least be consistent)
p1. "in order to" is repeated
p2. Web of Data should be capitalized
p2. "the Dacura Quality Service" surely?
p2. Web of Data for logical (no comma)
p2. It is normal to capitalize Section X, Table X and Figure X
p3. oneOf has no namespace
Code examples should be formatted in a fixed width font
p4. metadata is a word (no hyphen)
p4. ManBearPig might not be considered a suitable example by some readers
p5. Please cite MG15 instead of saying "previous publication"
p5. Define ACID when first used (not on p8)
p9. form a model under a closed world assumption (as, in practice, ...)
p10. "takes pains" is colloquial
p15. "leads to a mess" could be stated more scientifically
p19. Footnote 5 doesn't work

The authors state that the contributions of the paper are (1) the identification and analysis of the challenges encountered when unifying the linked data schemata, (2) a description of the Dacura Quality service modelling unification, (3) an extensive evaluation of current linked data vocabularies, and (4) recommendation/best practises for constructing linked data vocabularies. Nevertheless, from my perspective, the contributions of the paper are mainly points (2), (3), as (1) is mainly summarization of the known issues and (4) is quite generic summarization of the approaches how to solve the challenges. So in terms of the originality, the main contributions seems to be (2), (3).

Regarding contribution (3) - the evaluation of LD vocabularies - the authors evaluated only certain portion of ontologies - ~60. But if we look at, e.g., Linked Open Vocabularies (http://lov.okfn.org/dataset/lov/), there are 500+ vocabularies worth considering. So the authors should evaluate more vocabularies, which will increase the significance of the results.

The section "recommendation/best practises for constructing linked data vocabularies" should better explain what is the propose way how to solve the challenge, e.g., using examples.

In terms of the quality of writing, there are couple of typos and formatting of the paper may be improved (e.g., by using different fonts for predicates/classes in the text).

This paper presents a method/tool for analysing issues with online ontologies (said to be Linked Data Schemata) that does not focus on local consistency, but on issues associated with the mixing of such ontologies through owl:import or URI referencing. Such issues include mixing of RDF and OWL (and sometimes DAML), re-definitions that are valid only locally, as well as undefined elements often (according to the authors) resulting from errors (e.g. typos) in the data.

The main claim contributions are the description and tools, as well as the analysis of existing ontologies with respect to the occurrence of such issues, and recommendation on the way to "fix" these issues.

This is a very interesting area with obvious relevance to the theme of the special issue. While debugging of ontologies have received some attention in the past, including from the point of view of avoiding pitfalls and the mixing of different, globally inconsistent models, the approach taken by the authors seem interesting. To summarise my view given in more details below however, I think the paper is not ready for publication, as the description of the method is too vague and the method itself has not been validated. Also, I do think that the presented use case (the open annotation ontology) is not sufficient either to demonstrate the value/validity of the method, or as an analysis of the current issues.

= Originality =

There is a large amount of literature on ontology debugging that seems to have been overlooked a bit. The related work section indeed talks about a number of works that are directly related to the current one, but a bit more of an overview of ontology debugging would give the reader a better idea of the way the proposed method places itself in the context of the state of the art. Also, there is a lot of discussion on the fact that the issue comes from unifying models. I'm surprised not to see more references to approaches (especially related to ontology alignments, distributed description logics or E-Connections) that have been taking a completely different approach to the issue of dependencies between inter-connected but possibly incompatible ontologies.

= Significance of the results =

The paper is made harder to assess by the fact that there are really 3 claimed contributions each with specific issues.

Regarding the method and tool, the overall description of what is being done can be followed from the point of view of the narrative, and the fact that the tool and sources are available certainly helps. Nevertheless, the description remains vague, and many of the terms employed are not defined. For this kind of work, I would have expected more formalisation of at least the basic notions being manipulated (e.g. what is an "Orphan Class"?). It is often the case in this sort of work that the biggest problems come from tiny details and there isn't much in the paper that helps the reader understand how things work.

The other issue on the methods is that it is not really evaluated. It is ran on an example which at least demonstrates that it can find issues, but there is no indication of how significant/real these issues are, whether it is complete, how well it performs, etc.

Regarding the analysis, if taken as a contribution in itself, it is much too weak. Despite the fact that it includes more than 60 ontologies, it remains that this is analysing (only partially) the dependency tree of only one ontology (which we have not reason to think is representative). I can't quite see what we can learn from this other than that the method works on this example (and even that is not clear; see above), and that this dependency tree shows a number of issues. I actually don't understand why the authors limited their study to this one example: if the tool and method work, why not doing it on all the ontologies/vocabularies of LOV then? Without something of this scale, many of the claims that are made in the paper about the current practices in ontology/schema building/publication are baseless and should not be included.

Finally, there are the recommendations. I am not quite sure what to say about this, in the sense that these should certainly not be evaluated for their scientific value (being based, as discussed above, mostly on anecdotal evidence, and rather representing, I think, the authors' opinion). These things can however be very useful if they propose solutions that can be implemented and that would lead to a clear improvement. While several of these recommendations are clear and sensible, others are just not realistic. For example "Ontologies in OWL need to immediately stop" using rdf:List - How do you suggest this can happen? Also, in the last one, the solution is "collaboration between ontology designers" - again, how would that be achieved? Is it even desirable that ontology designers will have to be coordinated? Isn't that against the whole semantic *web* idea?

In other words, while I welcome the recommendation section as a way to support better quality ontologies, that play well with others, the effectiveness of these recommendations is hampered by the fact that, for several of them, there is no concrete way to implement it.

= Quality of writing =

The paper is easy to read and follow. The narrative also seems to be flowing reasonably well. The main issues however, other than the paper not being in the right format (i.e. not applying the journal's template) are already mentioned above:
- The technical description of the method/tool is too vague
- Many claims are made, especially about the current ontology engineering practices, that are not substantiated (in the conclusion "The results of our survey give valuable information about the state of ontology development" - I don't think this can be said of a survey consisting in the debugging of the (partial) dependency tree of one ontology).

Also, this might just be me, but the abstract confused me a bit as I, for a minute, thought the authors were suggesting merging all online ontologies into 1 common, universal model.