Review Comment:
*The following text is written using basic Markdown syntax.*
# Summary
The paper presents an ontology for Information and Communication Technology (ICT) systems. The ontology is proposed for two separate purposes. First, to help detect anomalies within a network of ICT systems.
Second, to facilitate the analysis of root causes for such anomalies. To serve both purposes, the ontology is said to provide a formal model for a network's infrastructure and activity as well as associated incident situations and maintenance operations. The ontology's design is informed by a set of competency questions elicited from 16 domain experts. Said competency questions are used to derive requirements for the ontology that can be tested automatically using SPARQL queries. An example of the ontology's use is provided in terms of a fictitious use-case scenario involving a connection failure.
# Overall impression
The idea of an ontology that helps with the detection and analysis of network anomalies is interesting.
The collection of competency questions (CQ) from a reasonable number of domain experts provides a solid basis for the ontology's development. These CQs are also likely to be valuable for the wider community interested in ontologies for ICT systems. Furthermore, the goal of reusing existing ontologies can be seen as a contribution to a community-driven effort of solving non-trivial modelling problems for ICT systems.
Most of the computational artefacts and materials discussed in the paper are publicly available in an online repository on GitHub. The repository is well-organised and the available documentation is adequate for the purpose of reproducing what is presented in the paper (The available 'README.md' and 'makefile' files are clear to me. However, I haven't tried to reproduce the materials discussed in the paper).
Many design choices for the ontology as well as its practical use are only described on a high level of abstraction. As a consequence, most statements and claims about the ontology's use and design are vague and therefore hard to test and verify. In addition to the lack of technical detail, there are also more general shortcomings from a methodological point of view. In my more detailed comments below, I will outline a few examples of both issues.
# Comments on the conceptual description of the ontology
The ontology's intended use and design goals are only formulated in broad and general terms. In particular, there are no concrete statements that could be tested or verified. For example, it is said in various places of the paper that the ontology is designed for anomaly detection and root cause analysis in the context of ICT systems.
However, it remains unclear
- what kind of anomalies can be detected,
- what kind of analysis is supposed to be done in terms of root causes,
- and how exactly the ontology would help with either of these tasks.
The presentation of the fictitious use-case in Section 6 also doesn't explain how the ontology is supposed to be used w.r.t. the detection of anomalies or root cause analysis.
Adding to the confusion are formulations suggesting that the ontology does in fact *not* provide support for anomaly detection or root cause analysis. For example, in Section 5 it is written that:
> "[...] these questions require the implementation of more complex AI-based algorithms such as anomaly detection algorithms. For example, to answer CQ#11 (“What was the root cause of the incident?”), the explicit representation of alarms and logs associated with a given incident is not enough and needs to be enhanced with root cause analysis algorithms."
# Comments on the technical presentation of the ontology
The ontology is authored in OWL. Yet, there are no concrete statements about the ontology's design in terms of OWL axioms. This is particularly confusing because of the emphasis on reasoning in various places of the paper. In the introduction, it says the ontology is developed to provide
> "a consolidated semantic model for describing and reasoning on the combination of network infrastructure characteristics [...], network activity [...] and operations [...]."
In Section 3, the authors write
> "we consider incidents as a central notion towards i) computing and reasoning on 'anomaly signatures'".
However, the paper does not provide any details about what kind of inferences are made possible by the ontology or what kind of information these inferences would provide.
An inspection of the OWL file reveals many aspects of the ontology that are not mentioned in the paper. For example, the ontology features a very shallow class hierarchy (most classes have only one superclass/subclass in the inferred class hierarchy). This is not a problem in and of itself. However, there are many existing and non-existing subclass relationships that raise questions.
Here are only a few concrete examples that I don't understand:
- why a class called "CommunicationDevice" is not a subclass of "Device"?
- why is the class "StructuralProperty" not a subclass of the class "Property"? (and why is "StructuralObservable" a subclass of "Property"?)
- why is the class "Service" equivalent to the class "ServiceInstance"?
- why is the class "EventRecord" a subclass of "Event"?
The use of individuals is also not clear to me. For example, why are there five individuals from different namespaces with a local identifier "type" in addition to individuals "operationType" and "commentType"?
Note that these kinds of design choices are relevant for OWL reasoning. So, I would expect details like these to be discussed in the paper.
# Comments on the ontology's evaluation
The paper doesn't make a convincing case that the ontology meets the objectives of its designated purpose effectively and satisfactorily. The presented evaluation of the ontology is limited to test cases that have been derived as part of the formal specification for the ontology's design.
The derivation of test cases follows an ontology authoring approach based on competency questions (see reference 19: "Towards Competency Question-Driven Ontology Authoring" by Yuan Ren, Artemis Parvizi, Chris Mellish, Jeff Z. Pan, Kees van Deemter and Robert Stevens). So, passing the generated tests would only provide some evidence that the ontology has been constructed in accordance with design requirements as derived as part of the chosen ontology authoring approach. In particular, it is not warranted to equate a test-driven development approach with an evaluation of the ontology.
Also, it needs to be highlighted that passing tests are reported for only 16 out of 26 competency questions. Put differently, the ontology only meets about 60% of the formal design requirements derived from the formulated competency questions.
Overall, the provided evaluation is not sufficient to make a judgement about the ontology's quality, usability, or relevance.
# Recommendation
I do not see a clear pathway for addressing the issues outlined in this review.
|