Review Comment:
This manuscript proposes the inclusion of human-based decisions in the context of the operations of a blockchain, and its application in the development of ontologies, so that ontologies can be built and evolve considering the changes that are being proposed in a blockchain block and their analysis and evaluation by humans, so that they reflect the consensus usually needed in ontology development. The approach is implemented, and as the authors discuss, it is not only to be considered as a contribution in ontology development but also as a contribution into how blockchain technologies and approaches can be modified when human-based interactions/consensus are needed.
I will first reflect on the part of the contribution that focuses on improving or providing an alternative for distributed ontology development using this approach. I think that this contribution is not sufficiently strong (nor sufficiently evaluated) for being published as a journal paper, for the following reasons:
- The authors make several claims about ontologies without clear references or studies that support them, and that given my experience in ontology development and governance I would rather say that may be considered incorrect. For instance, in the abstract the authors comment that "ontologies are difficult to validate and maintain". And later on, in the paper, the authors say that "since there are currently only a few standards available, these ontologies are under constant development and there is no standardized way to install trust into them yet". And later they claim "There are very little standardized ontologies at the moment, which results in a dynamic further development of existing ones". Given the current state of the art of ontology engineering, I find it difficult to agree with these statements if no further references are provided. I cannot see where the difficulties in evolution and maintainance of ontologies lie, apart from usual difficulties in the evolution and maintainance of any other artefact. I cannot see either the causal relationship on number of standards and constant development, and the need for trust, or the fact that there are very little ontologies (if you take a look at some domains, you may see that this is not really a problem.
- As a result of some of these claims, the authors propose a way to solve those problems (which I cannot consider as real problems in the world of ontology engineering and governance) by using a blockchain approach, which is modified with the creation of a human-based consensus step. I would have find it interesting to have a clear discussion (and corresponding evaluation) of different types of human-based consensus and how they affect the creation and evolution of ontologies. I think that such an approach would be very valid for the audience of this journal. However, this is not the case. Only one approach for obtaining consensus is proposed, and no formal qualitative evaluation of how this approach performs wrt others is provided.
- As a final reflection on this: how is this approach not covered already by the development of ontologies that is being done in standardisation groups (e.g, W3C groups), where all decisions are being recorded in a source code repository (e.g., using git), where the different versions of the implementation code are being stored and can be traced, where the consensus is obtained in group calls, videoconferences and f2f meetings? How is this approach different to that, and how can it ensure that it provides more trust, as the authors claim?
- Finally, the evaluation is not clear at all on which research hypotheses are being evaluated (I would have expected evaluations on the quality of the ontologies generated, with a larger corpus of ontologies and not just with a single one whose implementation is not made available publicly, on the feelings of the participants in the ontology development process, on the amount of conflicts generated, on the most common types of agreements and disagreements, etc.)
- I can also mention that in terms of incentives, I agree that tokens can be provided, but this is not considered to be a problem in current ontology engineering practices. People who participate in ontology development have their name included in the author or contributors list, as a usual practice. Why do tokens solve this?
I think that these weaknesses make the paper unsuitable for publication in this journal.
On the second core contribution, the installation of human-based consensus in the process of blockchain, I like the idea and I think that it is novel (although I am not an expert here), but this core contribution would need further development and at the same time would be out of the scope of this journal.
|
