Review Comment:
SUMMARY OF THE REVIEW:
This paper introduces a new ontology for the IoT domain, named as IoT-TTA, which aims to describe IoT technologies, tools and applications. In my opinion this paper requires a major revision to be accepted for publication. The main reasons for this assessment can be summarized as follows:
- The need for this ontology is unclear to me. The authors did not provide a convincing motivation that justify this work
- In the IoT domain, relevant industry standards, recommended practices and initiatives are critical. They define processes, terminologies, data models, etc. It is not clear to me which relevant standards, recommended practices and initiatives have been considered when building this ontology
- Before building a new ontology, a thorough analysis of relevant ontologies should be done to: (a) evaluate if the ontology is really needed, and (2) identify and import relevant terms that can be reused in the new ontology. This analysis has not been included in this paper
- Because the target audience of this paper does not only include experts on IoT domain, it would be recommended to provide a more detailed introduction of IoT concepts, technologies and tools. It would also improve readability if the authors split Section 3 in two sections. The first one would solely address the definitions of relevant terms in the IoT domain and the second section will provide a detailed description of the ontology including relevant modelling decisions
- Some modelling decisions behind the implementation of the ontology IoT-TTA are not clear to me. For instance, OWL 2 is a modelling language that provides a rich collection of constructors to build complex class definitions. However, the authors ignore this capability when defining the classes of the IoT-TTA ontology, where only disjoint restrictions are included. Instead, the authors implemented most logical axioms as part of domain and range "restrictions", which is a more suitable approach when modelling RDFS ontologies, where the language is less expressive. Notice also that domain and range "restrictions" are not, in fact, restrictions. In the particular case of object properties, they generate additional inferences, such as class assertions, some of them might produce unexpected effects
- Following the discussion about modelling decisions that are not clear to me, I would like to mention a couple of examples. For instance, the authors of the IoT-TTA ontology define the subclasses of the class Application as disjoint classes, where the subclasses of Technology are not. I do not understand why the subclasses of Application are disjoint because a "Big data application" can also be a "Machine Learning application" defined as a "SOA application". Another example of a modelling decision that it is not clear to me is the implementation of several SWRL rules. These rules are not properly introduced and motivated. Adding SWRL rules to an ontology increase computational complexity and require specific tool support that it is currently very limited. Moreover, the latest version of the HermiT reason cannot process a couple of these rules
- A fundamental problem in the design of the IoT-TTA ontology is that the authors reuse very few terms defined by existent ontologies. These ontologies were selected mainly to provide terminology for creating annotations. The authors ignored relevant ontologies for units of measure, sensor representation and IoT specific. To make things worst, the authors followed the MIREOT guidelines that do not guarantee complete and correct inferences when importing terms from other ontologies. OWL 2 language was specifically designed to ensure that the computation of consistency and implicit inferences is tractable. The MIREOT guidelines ignore this fact and, by extension, the authors of the IoT-TTA ontology. I recommend to the authors of this paper that they carefully study how the OBO Foundry (http://obofoundry.org/) implements and organizes their ontologies. The domain is different, but the modelling principles should be suitable for building ontologies in the IoT domain
- The evaluation section requires a substantial redesign. Many of the metrics considered by the authors (some invented by them) are not relevant when testing the quality of the ontology. In my opinion, the evaluation should be guided by a collection of functional requirements to be defined at the beginning of the paper. These requirements should determine, for instance, the "minimum coverage of relevant terms" and how data should be represented and queried. Integration with other relevant ontologies and verification of consistency must be also evaluated.
- The presentation and organization of the paper should be improved. For instance, the resolution of the figures is not good enough. They should be re-scaled or imported using a vector format
DETAILED COMMENTS ABOUT THE PAPER:
Sections 1 and 2:
The motivation for the implementation of this ontology is not clear to me. In particular, the authors did not provide in the paper:
- A clear presentation of the current issues the industry is facing that justify the need of building a new ontology in the IoT domain. This presentation can be enriched with concrete industry projects and applications that could help the authors to bring the discussion into concrete and practical issues
- A description of relevant attempts (with or without ontologies) that aimed to solve the previous list of issues and why they (partially) failed
- Based on the previous analysis, the authors should include a list of functional requirements the IoT-TTA ontology must fulfil to solve the problems that have been identified
Section 3:
The IoT domain is heavily based on industry standards, protocols and recommendations that are not sufficiently mentioned and discussed in the paper. The IoT layers presented in Section 3 are not properly defined and they are not supported by any industry standard or recommendation. For instance, there are other models (such as the OSI model) that provide a different layer configuration. So, I wonder where the model presented in Section 3 comes from and why it was chosen.
I suggest splitting the description of the IoT layers and the discussion about how to model these layers using an ontology into two different sections. The current organization of Section 3 is confusing. It is also not clear how this ontology will be used and what types of data and applications would benefit from having this ontology.
Some modelling decisions presented in Section 3 are not clear to me. For instance, the ontology defines the classes "Big Data application" and "Machine Learning application" as subclasses of "Application" and the classes "Big data technology" and "Deep Learning technology" as subclass of "Technology". However, "Big Data application" and "Machine Learning application" are defined as disjoint classes and the classes "Big data technology" and "Deep Learning technology" are not. The former modelling decision does not make sense to me because a big data application can be implemented using machine learning algorithms.
Section 4:
The relation (defined as mappings or import relations) between the IoT-TTA ontology and other IoT ontologies (e.g., SOSA) and domain independent ontologies (e.g., units of measure, provenance, etc) is not stablished. For instance, Sections 4.2 and 4.3 refers to quantity kinds and units of measure that are defined in prominent OWL 2 ontologies such as QUDT and OM2. It does not make sense to me the authors create new IRIs, labels and definitions for these terms.
The authors of the IoT-TTA ontology only considered importing terms for creating annotations. These terms are defined in specialised ontologies such as DCMI, Vann or Voaf. However, the authors of the IoT-TTA ontology followed the MIREOT guidelines to avoid importing complete ontologies or meaningful modules, which should include not only the URIs and labels of a specific term (e.g., class, object property, individual, etc.) but also other terms and axioms that ensure completeness and correctness. Quoting from the authors of MIREOT [1]: "The MIREOT standard is a trade-off between complete consistency checking and heavyweight importing versus lightweight importing but partial consistency checking. We are aware of and accept that by copying only parts of an ontology there is the risk that inferences drawn may be incomplete or incorrect: correct inference using the external classes is only guaranteed if the full ontologies are imported.". Because OWL 2 language was selected as the modelling language for the IoT-TTA ontology, not ensuring completeness and correctness (e.g., consistency or complete definitions and related inferences) after importing terms defined by other ontologies is not acceptable. Alternatively, the authors could consider the use of tools for computing self-contain modules of ontologies using, for instance, atomic decomposition based on the paper by Cuenca-Grau et. al [3] and available in the tools offered by the OWLAPI [2].
I strongly recommend the authors of this paper that they study the work done by the OBO Foundry (http://obofoundry.org/). In my opinion, this organization has created the state-of-the-art on modelling and organizing collections of ontologies to facilitate interoperability. This should be also the approach to be followed in this paper and in the IoT community in general.
I found confusing the notion of "RDF constraints" in the context of building an ontology using the OWL 2 language. This modelling language offers the possibility to define domain and range restrictions for data properties.
The authors describe in Section 4.4 that they created SPARQL queries to populate ABoxes following the definitions provided by the IoT-TTA ontology. This approach requires additional steps to verify correctness, which might include testing consistency or verifying restrictions using SHACL constraints. Notice also that there are other alternatives to populate ontologies such as ontology templates. OTTR templates [4], for instance, provide an independent modelling language and tool support for building and instantiating libraries of ontology templates.
The authors indicated that the IoT-TTA ontology has been enriched with SWRL rules. The need for implementing these rules is not clearly explained and no examples are provided that illustrate how their application of these rules improve the ontology. In addition, reasoning tasks such as checking consistency of an ontology implement using SWRL is undecidable [5]. The authors did not clarify if they used a tractable fragment of SWRL, and they did not indicate either, which tools were considered to provide reasoning capabilities. Notice that tool support for SWRL is very limited, which can affect usability and applicability of the IoT-TTA ontology.
Section 5:
Based on the selection of ontologies included in the evaluation section, I wonder if this selection is representative enough of the IoT domain or other relevant ontologies should be included. It would be interesting if the authors introduce and discuss the selection criteria of the ontologies to be compared with IoT-TTA.
I am not convinced about some of the metrics used to evaluate the ontologies. Some metrics such as BIndex, RAnno and BP Index are invented by the authors, but it is not clear to me why they are relevant. For instance, adding more individuals or logical axioms increases the BIndex. However, I am not sure why this might improve the ontology (their might cases where it does not make sense). The metric RAnno only considers number of annotation properties and assertions. Why annotation of classes and individuals are not relevant?
The authors also compute other metrics for testing cohesion. Calculate the number of root classes (NoR) or the number of leaf classes (NoL) could be interesting but obtaining higher numbers do not necessarily mean the ontology is better.
In my opinion, there are better metrics to test the quality of an ontology. This might include checking if the ontology is consistent or not, or analysing the coverage (percentage of terms required for a specific domain or application) provided by each ontology. Another relevant criterion is to identify common design patterns and to analyse if the ontology has been modelled to facilitate integration (via ontology mappings) with other relevant (domain independent and specific) ontologies.
COMMENTS ABOUT THE ONTOLOGY:
- Namespaces and prefixes for terms defined by external ontologies are not included in the list of prefixes of the IoT-TTA ontology.
- Very low axiomatization-level is provided in terms of class definitions. Only disjoint classes are explicitly mentioned. I would expect that class restrictions are defined to stablish how individuals on a particular class are related to individuals in other classes via object property assertions
- Instead of class restrictions, domain and range restrictions are defined for property objects. This modelling approach makes more sense on RDFS but not in OWL 2. Moreover, this modelling approach might produce unexpected side effects because these restrictions are not restrictions really. In fact, they might produce unexpected class assertions. See the following resources for details:
+ http://mowl-power.cs.man.ac.uk/protegeowltutorial/resources/ProtegeOWLTu...
+ https://protege.stanford.edu/conference/2004/slides/6.1_Horridge_CommonE...
- It is not clear to me some modelling decisions when stablish class relations. For instance:
Why the class BigData_app is disjoint from Machine_learning and SOA? Would not be possible to define an application designed following SOA and able to handle big data using ML algorithms?
- Hermit returned few syntax errors related to some of the SWRL rules defined in the ontology.
REFERENCES:
[1] Ruttenberg, A., Courtot, M., Gibson, F., Lister, A., Malone, J., Schober, D. and Brinkman, R., 2009. MIREOT: the Minimum Information to Reference an External Ontology Term. Nature Proceedings, pp.1-1.
[2] https://github.com/owlcs/owlapitools
[3] Cuenca-Grau, B., Horrocks, I., Kazakov, Y. and Sattler, U., 2008. Modular reuse of ontologies: Theory and practice. Journal of Artificial Intelligence Research, 31, pp.273-318.
[4] M. G. Skjæveland, D. P. Lupp, L. H. Karlsen, and H. Forssell. Practical Ontology Pattern Instantiation, Discovery, and Maintenance with Reasonable Ontology Templates In: Vrandečić D. et al. (eds) The Semantic Web—ISWC 2018. ISWC 2018. LNCS vol 11136. Springer. 2018.
[5] Horrocks, I., Patel-Schneider, P.F., Bechhofer, S. and Tsarkov, D., 2005. OWL Rules: A Proposal and Prototype Implementation.
| 
