SQuAP-Ont: an Ontology of Software Quality Relational Factors from Financial Systems

Tracking #: 1897-3110

Andrea Giovanni Nuzzolese
Paolo Ciancarini
Valentina Presutti
Daniel Russo

Responsible editor: 
Pascal Hitzler

Submission type: 
Ontology Description
Quality, architecture, and process are considered the keystones of software engineering. ISO defines them in three separate standards. However, their interaction has been scarcely studied, so far. The SQuAP model (Software Quality, Architecture, Process) describes twenty-eight main factors that impact on software quality in banking systems, and each factor is described as a relation among some characteristics from the three ISO standards. Hence, SQuAP makes such relations emerge rigorously, although informally. In this paper, we present SQaAP-Ont, an OWL ontology designed by following a well established methodology based on the re-use of Ontology Design Patterns (i.e. ODPs). SQaAP-Ont formalises the relations emerging from SQuAP in order to represent and reason via Linked Data about software engineering in a three-dimensional model consisting of quality, architecture, and process ISO characteristics.
Full PDF Version: 

Minor Revision

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Karl Hammar submitted on 27/May/2019
Major Revision
Review Comment:

This ontology description paper introduces SQuAP-ONT, an ontology formalizing the authors' previously developed SQuAP model, which covers software quality, process quality, and architectural quality, within financial software (specifically on the Italian banking market).

An ontology description paper shall, per the SWJ reviewer guidelines, be reviewed per the following dimensions: (1) Quality and relevance of the described ontology (convincing evidence must be provided). (2) Illustration, clarity and readability of the describing paper, which shall convey to the reader the key aspects of the described ontology.

-- Review Summary --

The quality of the SQuAP-ONT ontology itself is generally acceptable (with some room for improvement, see detailed comments below). However, the authors do not sufficiently motivate the relevance and impact of this resource in the article; there are also several ambiguities and some faults in the text that require fixing. This article could perhaps be acceptable for publication after a major revision. Note that since the article is only 10 pages long, there is some room for additional text describing those bits that are in the present version not sufficiently covered.

-- Detailed Review --

* I do not see sufficient potential for reuse of the developed ontology. This is essentially a formalized version of a pre-existing quality model, developed in a partly grounded manner from observations in the Italian banking sector (there are quality factors in the model that are explicitly local to Italy.). The explicit purpose of the ontology to support consultancy work in such a context. Reuse outside of that context is not addressed.

* Regarding the underlying SQuAP model on which this ontology is based, there are at least two major issues:
1) While the 28 quality factors have been validated by 124 domain experts, the article does not indicate that the mappings of these factors to the quality characteristics from the referenced ISO standards have been validated in the same manner. So there's arguably little empirical foundation for a large part of the relations in the ontology.
2) The article that describes the development of the SQuAP model's 28 quality factors, in fact, describe only 15 factors. Where the remaining 13 factors referenced int his paper come from is unclear.

* While the SQuAP model design/development process is described over several pages (partially repeating, e.g., 3rd paragraph 1st section and 1st paragraph 3rd section), the SQuAP-ONT design process and methodologies are glossed over in only one paragraph. This is unfortunate, especially as the method employed customizes the XD method with a (to my mind very reasonable) approach to pattern reuse.

* Related to the above, it would for instance be interesting to see a motivation for the use of design patterns in this case, where a basic model already exists to build from. The argument for using patterns and XD is typically that it simplifies and speeds up the development process of new ontologies. Arguably, adding the D&S pattern (which is somewhat abstract) can make the resulting ontology less useful for a practically-minded and applied software developer/engineer/manager; what is the rationale for using it here? Development speed? Quality of the result? Improved interoperability?

* There's some potentially very interesting discussion on the need for and consequences of punning on page 6. But this is glossed over rather quickly, which is a shame; punning such as done in this ontology is in my experience not done often, and if the authors have experience of use cases for this OWL 2 feature, and the consequences of employing it, I think that might be interesting for the community to read about.

* Relating to punning: why use both subclassing for specialization (on class level), _and_ the custom :specializes object property (on individual level)? Are both truly needed? And if so: this property should probably be transitive.

-- Formalities --

* The ontology is published at a w3id and loads in Protégé via content negotiation. However, the text/html-representation promised on page 7 (http://stlab.istc.cnr.it/squap/) results in an Internal Server Error.

* The authors claim on page 7 that the ontology uses the VOID vocabulary to describe license information. That is incorrect, at least for the presently published version, which uses dc:rights to provide a link to the license.

* Figure 2 color coding seems to indicate that :parametrizes is a local object property, when it is (I believe) in fact sourced from an ODP.

* The DL formalization in Section 4.2 is inconsistent with the schema diagram in Figure 2 and the published ontology (not only regarding the abbreviated classes). Also, Figure 2 is inconsistent with the published ontology (e.g., :hasMetric).

* Section 2, paragraph 2: the paragraph's last sentence is incomplete.

* Section 4.2: the first sentence is a tautology.

Review #2
By Charles Vardeman II submitted on 08/Jun/2019
Minor Revision
Review Comment:

This manuscript was submitted as 'Ontology Description' and should be reviewed along the following dimensions: (1) Quality and relevance of the described ontology (convincing evidence must be provided). (2) Illustration, clarity and readability of the describing paper, which shall convey to the reader the key aspects of the described ontology.

Review of "SQuAP-Ont: an Ontology of Software Quality Relational Factors from Financial Systems"

The authors present an ontology constructed to capture, aid and provide interoperability for metrics of software quality, specifically aimed at Financial Software Systems. The ontology is informed by existing ISO standards but provides the extended explicit capability to account for the software development process and architecture. Additionally the ontology provides a meaningful contribution to solving to a domain identified gap in the lack of identified relationships between the three dimensions required for successful being software quality, software development process, and software architecture by explicitly modeling these relationships. The authors construct their Ontology through the reuse of Ontology Design Patterns namely Descriptions and Situations and Parameter Region patterns which are appropriate to apply for this modeling problem producing a higher quality model. Modeling done using domain experts both at the initial modeling phase and a validation phase that assured the empirically derived dimensions of software engineering identified in the model were appropriate. However, in reviewing the model with respect to software development process, it does appear to be slightly more oriented to the more traditional 'waterfall' methodology than the more modern agile techniques. Typically agile methods decision points are centered around a scrum where a team with different 'role' relationships (product owner, scrum master, development team) determine the software development trajectory. The provenance of decision points whether waterfall or agile and,in the case of the later, the formal role relationships (that in the scrum method should represent certain points of view) may be necessary to understanding the underlying software structure and quality. This is outside of the competency questions to which the models are constructed, but probably should be mentioned in the paper as a potential area to be addressed.

The SQuAP-Ont ontology was to have been constructed using techniques to ensure quality and reusability. Given that their intended use case is for software quality, the authors may want to highlight the need potentially 'dogfooding' their ontology for the development of both the ontology and software tools using the ontology to provide complete foundation. At a minimum, the authors should address ontology quality and metrics of their ontology in the text given the intended use case for their models.

Competency questions were relevant and provided a reasonable check. In addition, the authors provide relevant examples in the paper of how the ontology could be populated and used to answer one of these questions. It is particularly useful that the authors provide a rule based example using SPARQL CONSTRUCT given that a potential use case would be to encapsulate the Ontology using SHACL rules. It would be useful if the authors address how SHACL and SHACL rules could aid in validation of these metrics and consider for future work SHACL validation of their parameter models which would further aid in the quality of data that will be instantiated in their model. Axiomatization of the ontology is consistent with the foundational ODPs and provides reasonable constraints for the model. The notation and text descriptions the authors use provide a clear understanding of the ontology axioms.

However, there is an inconsistency between the axiomatization presented in section 4.2 of the paper and the implementation (at least the Github version). 'Measurement Result' in the paper (bottom of page 6) as a much stronger restriction of "accesses only 'Software quality characteristic'" versus the axiom in the OWL version which states "accesses some 'Software quality characteristic'". This discrepancy should be reconciled and possibly an explanation of why the stronger restriction is required and should be addressed before publication.

The OWL release otherwise well constructed. The release license is explicitly present in the file, contains a versioning IRI. It might be useful to include ontology metadata such as versioning using OMV - Ontology Metadata Vocabulary or similar vocabulary. Additionally, the authors use OPLa to provide annotation of the patterns used in the ontology, which is very useful. The authors use a w3id persistent namespace for the ontology which should aid in sustainability. The authors are utilizing linked data principles and providing serializations of the ontology via content negotiation. However, the default response appears to be xml/RDF instead of html and I was unable to access the human readable HTML version as claimed on page 7 of the paper. Content negotiation for turtle and rdf/xml works and was validated using the curl command line tool. This issue should be addressed for publication.

There is a typo on page 8 with the rendering of the accents in Protege.

Review #3
By Paola Espinoza Arias submitted on 11/Jun/2019
Minor Revision
Review Comment:

This paper reports the development of an ontology formalizing the results of a previous study where a model, named SQuAP, was defined. This model is used to assess Financial Information Systems’ quality. More specifically, the SQuAP model describes the relations between the quality, architecture, and process ISO characteristics for assessing quality software.

The SQuAP-ont ontology is a relevant contribution in the area of the evaluation of the software quality allowing to represent in an integrated way the assessment phases until now available in separate ISO’s. Despite this ontology has been developed in an industrial project from the banking system, I like the decision to open it as a way to promote its use it in other software domains. The manuscript is clear and easy to follow. The ontology development has followed a well-established methodology, and the paper provides rich illustrations and supporting example code in order to explain the authors’ work. I really like the use of ODPs and the DUL alignment version.

Having said that, I have some more detailed comments:

- In page 2, you mention “ontology is available online with accompanying documentation” and you provide a URI as footnote. However, when I tried it on the browser I did not retrieve the code neither the HTML documentation as I expected. Instead of that I was redirected to http://stlab.istc.cnr.it/squap/ and I retrieved a “404 error”. It is really important that you solve this issue in order to provide a correct content negotiation strategy . Finally, It is worth mentioning that I was able to retrieve the ontology code using Protégé v5.5.0.
- In page 4, you refer to "SoftwareQualityMeasurementResult" but in Fig. 2 and in the ontology code is defined as "SoftwareQualityResult". Please fix it.
- In page 6, subsection 4.2, you refer to "MeasureQualityRes" for "MeasurementQualityResult", but this concept is not present in the ontology. ¿Did you want to refer to "SoftwareQualityResult"? If so, you need to replace it in the text and in the Description Logic representation.
- In page 7, subsection 4.3, you explain about the content negotiation, but it doesn’t work, as I already mentioned in my first comment. Please, further review the content negotiation. I checked it several times during may/june 2019.
- In page 7, subsection 4.3, you mentioned that you used the VOID vocabulary in order to include the license information about the ontology, but according to the code you are using Dublin Core Metadata Element Set for licensing representation. In addition, VOID is a vocabulary oriented to express metadata about RDF datasets and it doesn’t have a concept/property for licensing representation.
- In page 7, in the example code you also use "SoftwareQualityMeasurementResult" concept instead of "SoftwareQualityResult", please fix it. The same error with the "ArchitecturalAlignmentMeasurementResult" and "ProcessMaturityMeasurementResult" concepts. In addition, the online TURTLE file is not the same than the example explained in the manuscript, so I could not test it. In this code you use concepts ("ProcessMaturityAssessment", "ArchitecturalAlignmentAssessment", and "SoftwareQualityAssessment") not defined in the SQuAP-ont. In order to allow reproducibility you should to fix both online and github repository example files. I really miss the test part of your ontology.
- Finally, I am not sure about why did you decide to use different URIs to define some subclasses?, for example: you have the concept “https://w3id.org/squap/ArchitecturalAlignment” and then as subclass of the ArchitecturalAlignment: “https://w3id.org/squap/ArchitecturalAlignment/ObjectiveCharacteristic” and then as a subclass of ObjectiveCharacteristic: “https://w3id.org/squap/ArchitecturalAlignment/ArchitectureFramework
Why did you adopt this strategy? This is not common in other ontologies, and may suggest people not too aware of ontology engineering that the usage of this schema is sufficient to describe concept hierarchies. What would you do then in case of multiple classifications?

- In the abstract, SQaAP-Ont should be SQuAP-Ont.
- In order to be consistent, In pag 2 you should add “(Section 3)” or mention it at the end of the sentence where you describe it.
- In page 4, subsection 4.1, “We model three types of :SoftwareQualityCharacteristic: :SoftwareQuality, the class:Architectural-Alignment), and the class:ProcessMaturity.” You should to remove this parentheses.
- In page 6, “They are also modelled as classes so as to make the ontology extensible with possible specific axioms).” You should to remove this parentheses.
- In page 8, fix ProÃl’gÃl’. In the same page, add squap prefix to :affectMeasurementOf property in the SPARQL CONSTRUCT example.
- In reference 11, guidelines instead of guidlines.

- You may want to add some useful metadata to your ontology (both squap and squap-dul), for example, ontology version, preferred namespace prefix, last vocabulary modification date, etc. This information is important for reusing purposes.
- In page 4, subsection 4.1, “We model three types of :SoftwareQualityCharacteristic: :SoftwareQuality, the class:Architectural-Alignment), and the class:ProcessMaturity.” Maybe it is not necessary to use the word class two times. In my opinion, when you said “types of” is enough to understand what do you are going to explain.
- Finally, you may want to request the addition of your ontology into a vocabulary repository in order to make it discoverable for others.