Loki - the Semantic Wiki for Collaborative Knowledge Engineering

Tracking #: 2791-4005

Authors: 
Krzysztof Kutt
Grzegorz J. Nalepa

Responsible editor: 
Oscar Corcho

Submission type: 
Full Paper
Abstract: 
We present Loki, a semantic wiki designed to support the collaborative knowledge engineering process with the use of software engineering methods. Designed as a set of DokuWiki plug-ins, it provides a variety of knowledge representation methods, including semantic annotations, Prolog clauses, and business processes and rules oriented on specific tasks. Knowledge stored in Loki can be retrieved via SPARQL queries, in-line Semantic MediaWiki-like queries, or Prolog goals. Loki includes a number of useful features for a group of experts and knowledge engineers developing the wiki, such as knowledge visualization, ontology storage or code hint and completion mechanism. Reasoning unit tests for knowledge quality assertion are also introduced. The paper is complemented by the formulation of the collaborative knowledge engineering process and the description of experiments performed during Loki development to evaluate its functionality. Loki is available as free software at https://loki.re.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Reject

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 25/Oct/2021
Suggestion:
Reject
Review Comment:

(1) originality
The paper deals with an interesting topic, but it is not original. This paper follows a series of publications that ended in 2017. The contributions of this specific paper are not separated from previous works, although the comparison of features with other semantic wikis is one of the strong points of the paper.
It is very interesting to have features such as the SPARQ endpoint, the in-page queries (already existing in other wikis), and the prolog logic.

(2) significance of the results
In my humble opinion this is the weakest part. In section 2, a list of requirements (R1 to R13) is shown but, in table 5, we see that Semantic Media Wiki (SMW) only has 3 less requirements than Loki: 2 concerning quality management, and user roles. If I were a system administrator maintaining SMW, I would not consider them enough significative to migrate to Loki.
Concerning the experiments carried out, I miss more details about the profile of the users: It is said "students of computer science", but this is a severe bias if the wiki is intended for a specializad but not computer science specialists. Also experiments 1 and 2 has very few users (8 and 13 respectively) to get a statistical significance of the results.
Also, several experiments (sections 6.2 and 6.3) only produce "development hints" that authors claim that were used to enhance the tool.

(3) quality of writing
It is well written but some sentences are too long and hard to understand. Some typos should be polish.

(4) reproductibility
I cannot find any link to the results of the experiments. Also, I have tried to use the wiki at http://loki.re with bad results:
- I cannot log in, neither to register to get an account. Therefore I cannot see the wiki text source of most functionality.
- The link to the SPARQL endpoint (one of the interesting features of the project) does not work for me (I get a "create page message", but I cannot create any).
- I have not finished the installation of the application but, viewing the installation info it seems that Loki is a set of plugins (over a general wiki) whose source code has not changed since 2017.

Other
-------
- Many opening quotes are low instead of high along the paper.
- The references to colors in Fig. 8 could be easily replaced by dark and light gray. I suggest this is better for b&w printing.
- References to Fig. 10 should explain SUMI scales. In particular, what range of values is considered good.
- Experiment 6, seems a "relaxed version" of experiment 4, and the conclusions are extrang to me: gamification is interesting but student want to pass exams and are not engaged. This explanation is an hypothesis that should be checked more carefully.

Review #2
Anonymous submitted on 19/Nov/2021
Suggestion:
Reject
Review Comment:

The manuscript is centered on an interesting topic given the importance of shaping knowledge in a collaborative way.
However, the manuscript (and the proposed tool as well) lacks of novelty and usability.

Concerning novelty, it is, first of all, strange that some important reference about collaborative ontology modeling tools are missing.
An example is MoKi:
- Mauro Dragoni, Alessio Bosca, Matteo Casu, Andi Rexha: Modeling, Managing, Exposing, and Linking Ontologies with a Wiki-based Tool. LREC 2014: 1668-1675
Second, many important aspects associated with the SW field have not been integrated into the system.
An example is the creation of SHACL shapes and their application for validating knowledge used for populating the created ontology.

Concerning usability, which is the real aim of the tool? Many aspects are mentioned (e.g. ontologies, BPM, etc.) but not a clear focus is provided.
Moreover, the writing of the query is quite hard to follow and it is more difficult to understand with respect to the well-known SPARQL syntax.
Why did the authors do this choice?

The evaluation part is a positive contribution of the manuscript and it has been organized well.
However, is the scientific/technical contribution that, from my perspective is missing.
My main suggestion to the authors is to run a requirements gathering campaign from the community in order to understand the main needs that researchers have concerning the creation of collaborative ontologies nowadays.
Contributions mentioned above, like MoKi, were designed and developed in an era where some paradigms were not present neither defined.
Hence, more and more novelty is required for having a contribution like this accepted in the journal.

As minor things, the manuscript presents some bad English constructions, grammar mistakes, and misuse of articles: a professional language editing service is strongly recommended (e.g., the ones offered by IEEE, Elsevier, and Springer) to sufficiently improve the paper's presentation quality for meeting SWJ’s high standards.
Finally, double-check both definition and usage of acronyms: every acronym should be defined only once (at the first occurrence) and always used afterwards (except for abstract and section titles).
Also, it is not recommendable to generate acronyms for multiword expressions that are shorter than 3 words (unless they are universally recognized, e.g., AI).