Review Comment:
This manuscript was submitted as 'full paper' and should be reviewed along the usual dimensions for research contributions which include (1) originality, (2) significance of the results, and (3) quality of writing. Please also assess the data file provided by the authors under “Long-term stable URL for resources”. In particular, assess (A) whether the data file is well organized and in particular contains a README file which makes it easy for you to assess the data, (B) whether the provided resources appear to be complete for replication of experiments, and if not, why, (C) whether the chosen repository, if it is not GitHub, Figshare or Zenodo, is appropriate for long-term repository discoverability, and (4) whether the provided data artifacts are complete. Please refer to the reviewer instructions and the FAQ for further information.
The paper proposes INK, a solution for mining both task-specific and task-agnostic rules from KGs. The main contribution consists in proposing an internal representation, based on neighbourhood nodes, which ultimately come up to a binary matrix. After that, task agnostic rules are obtained via symbol manipulation whilst task-specific rules are obtained by combining the INK representation with an existing Bayesian Rule miner. A comparative experimental evaluation is provided.
The paper targets an important problem and proposes a solution having moderate novelty but showing interesting results particularly in terms of efficiency. The paper lacks a bit of clarity. This particularly applies to sections 3 and 4, illustrating the core of the work. A careful proof reading is also suggested. The paper is rather weakly motivated and some of the motivations that are provided do not result fully precise and appropriate. Also the analysis of the related works requires improvements (detailed comments are reported below). The further discussion illustrated in section 7 has been overall appreciated.
The source code for the proposed solution is publicly available on GitHub (this is highly appreciated) and the content appears clearly organized and accompanied by readme files.
In the following detailed comments are reported.
Reference [1] has been used to address the issue of providing a definition for a KG. Some recent publications targeted this aspect (among the others), see for instance Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d'Amato, Gerard de Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, Axel-Cyrille Ngonga Ngomo, Axel Polleres, Sabbir M. Rashid, Anisa Rula, Lukas Schmelzeisen, Juan Sequeda, Steffen Staab, Antoine Zimmermann: Knowledge Graphs. Synthesis Lectures on Data, Semantics, and Knowledge, Morgan & Claypool Publishers 2021, pp. 1-257
On page 2, lines 12 - 14 the authors assert "ARM can be applied to semantic data or KGs by converting the internal representation to a set of transactions." This is a quite strong assertion. Indeed, for the best of my knowledge there are cases (listed juts below) where this statement does not apply (as also reported by the authors in section 2.1). I'd suggest to rephrase it considering that this would be a possible approach and not the only one.
- Luis Galárraga, Christina Teflioudi, Katja Hose, Fabian M. Suchanek: Fast rule mining in ontological knowledge bases with AMIE+. VLDB J. 24(6): 707-730 (2015)
- Claudia d'Amato, Steffen Staab, Andrea G. B. Tettamanzi, Duc Minh Tran, Fabien Gandon: Ontology enrichment by discovering multi-relational association rules from ontological knowledge bases. SAC 2016: 333-338
Additionally, the approaches reported above are significantly inspired by the ILP literature but they do not solve task-specific problems even if they can be applied also for that. As such, also the consideration concerning the second category of approaches introduced in section 1 results not very precise.
The presentation of the rules provided in Section 2.1 is not fully precise. The authors should specify that the atoms of the body are binary predicates, as for the case of the head. With the provided representation it seems that atoms of the body can be of different types.
Section 2.3 should in principle discuss related works concerning rule mining (from KG) however, this subsection actually refer to methods for performing concept learning which is not exactly rule mining. Furthermore, when talking about concept learning a more extensive literature should be taken into account. Some relevant related works are reported in the following:
- Nicola Fanizzi, Claudia d'Amato, Floriana Esposito: DL-FOIL Concept Learning in Description Logics. ILP 2008: 107-121
- Giuseppe Rizzo, Nicola Fanizzi, Claudia d'Amato: Class expression induction as concept space exploration: From DL-Foil to DL-Focl. Future Gener. Comput. Syst. 108: 256-272 (2020)
- Franco Alberto Cardillo, Umberto Straccia: Fuzzy OWL-Boost: Learning fuzzy concept inclusions via real-valued boosting. Fuzzy Sets Syst. 438: 164-186 (2022)
- Francesca A. Lisi, Umberto Straccia: A Logic-based Computational Method for the Automated Induction of Fuzzy Ontology Axioms. Fundam. Informaticae 124(4): 503-519 (2013)
- Stefan Heindorf, Lukas Blübaum, Nick Düsterhus, Till Werner, Varun Nandkumar Golani, Caglar Demir, Axel-Cyrille Ngonga Ngomo:
EvoLearner: Learning Description Logics with Evolutionary Algorithms. WWW 2022: 818-828
Details concerning implementation and optimization of related works (described in different subsections of section 2) could be omitted whilst section 2.4, which is actually relevant for the purpose of the paper could be extended a bit more.
The rational adopted for the representation presented in section 3 (and related subsections) reminds to the one proposed in Petar Ristoski, Heiko Paulheim: RDF2Vec: RDF Graph Embeddings for Data Mining. ISWC (1) 2016: 498-514. The authors should consider this aspect and maybe briefly introduce the difference with the approach (and goal) presented in this paper. This comment is applicable also to text reported at the end of section 1, where the authors present the adopted representation as one of the main novelty with respect to the state of the art.
The key parts of the paper (e.g. section 3.1) require improvements in terms of clarity. Some suggestions are reported below (and in the section MINOR).
- End of section 3.1: "tuple" should be specified.
- Additional details should be provided when presenting the example at the end of section 3.2
- In section 3.3.2, the authors refer to a notion of similar relations. However, it seems that here exactly the same relationship is considered. If so, the notion of similarity is somehow misleading.
In section 4, it would be appropriate to report formal and specific algorithms for extracting rules. Additionally, in this section the authors argue that INK can mine not closed rules and that this leads to more and more advanced rules. This argumentation results a bit vague. A more precise explanation on the reason why this could be an improvement with respect to the state of the art and what is its actual value added should be provided. Furthermore, in the sections opening the papers the authors argue on the lack of negative examples in KGs due to their incompleteness and the open world assumption, which motivates, e.g. for AMIE, different ways to compute metrics such as confidence and support. However, in section 4.2 it seems that the closed world assumption is adopted. The authors should discuss and motivate this difference with respect to the arguments reported at the initial sections of the paper. A limited discussion is reported only almost at the end of the paper, in section 7. This aspect is quite important and needs a more extensive discussion and motivations.
As for the experimental evaluation, some more detailed discussion and information should be provided in section 6.1. Some suggestions are reported in the following: a) the meaning of the numbers within the parenthesis in Table 2 should be provided. b) The authors should provide a rough idea of the values of N. c) The number of rules that are discovered by INK is definitely greater than AIME. This data needs to be accompanied by additional information, e.g. it would be useful to know if a further analysis has been performed in order to understand if there are noisy / wrong rules that are generated by INK and or AMIE and if so what is the difference between the two systems. One of the main reasons for the greater number of rules of INK wrt to AMIE should be due to INK returning also not closed rules. As already highlighted above, the authors should clarify if this actually brings a value added or, at the contrary, if this basically inject noise. Some considerations at this regard are reported in section 7.1. I'd recommend to add at least a short text in section 6.1 announcing that further analysis is discussed in section 7.1 (from my personal taste, I'd report the further discussion immediately after the comparative experimental evaluation (that is in section 6.1 and 6.2) but this is subjective).
The comparison between AIME and INK should be performed also in terms of efficiency. Indeed AMIE is well known also for its efficiency. This aspect has not been considered for the experimental comparison with AIME whilst it has been showed when comparing to DL-Learner which is well know that it is less efficient since it basically adopts a purely symbol-based approach and makes of usage of the reasoner.
As regard section 6.2, the provided qualitative analysis is appreciated. Nevertheless, the authors should provide information concerning the criterion that has been adopted for comparing the rules extracted by the two systems, e.g. for each dataset the rule having the highest accuracy is compared or for each dataset a rule (randomly picked) is compared between the two systems or for each dataset the rules that are compared are those referring to the "Prediction of" column reported in Table 3. The Suramin dataset resulted particularly hard for INK. A possible explanation for that should be provided. Some comments at this regard are reported on page 23 (2nd column) but it is definitely too far away from the comparative analysis showed in section 6.2. A similar situation happens for lymphography dataset, even if, in this case, the values are more close each other for the two systems. Table 3 is displayed but no reference to this table has been reported in the text. It should be introduced and briefly discussed at the beginning of section 6.2.
In section 7, the authors argued on the ability of DL-Learner to exploit the background knowledge and reasoning capability. This aspect (targeted for instance in Claudia d'Amato, Steffen Staab, Andrea G. B. Tettamanzi, Duc Minh Tran, Fabien Gandon: Ontology enrichment by discovering multi-relational association rules from ontological knowledge bases. SAC 2016: 333-338) has not been considered at all by the authors when proposing their solution. Some considerations at this regards should be reported when introducing the solution, at least the reason why this is not taken into account should be provided.
MINOR:
- Beginning of section 2.1.1: "(with r(x, y) a the head atom as explained above)" --> "(with r(x, y) the head atom as explained above"
- page 6, 2nd column, row 30 - 35: this sentence should be rephrased. Additionally, the purpose / meaning of this part does not result fully clear, as well as the example reported immediately below the text.
- page 8, 2nd column, row 15, 16: "While the relation§object columns were are by default extracted by INK" --> this sentence should be rephrased
- page 9, 1st column, row 39 - 41: "For the Bayesian Rule Set model, this results in a a smaller number of rules." --> "For the Bayesian Rule Set model, this results in a smaller number of rules."
- page 10, 1st column, row 7 - 9: "If target values are specified together with the nodes of interest, the INK miner assumes a task-specific mining operation can be executed" --> this sentence should be rephrased
- page 11, 1st column, row 12 - 16: "By incorporating INK within this framework, a fair comparison between INK and DL-Learner, which has already been used in many evaluations within the SMLBench framework." --> this sentence should be rephrased
- page 11, 1st column, row 33 - 35: "DL-learner version 1.5 was used with the by SML-Bench default parameters for each learning task:" --> "DL-learner version 1.5 was used with the SML-Bench default parameters for each learning task:"
|