Reason-able Embeddings: Learning Concept Embeddings with a Transferable Deep Neural Reasoner

The paper presents a method to perform approximate subsumption checking for ALC
ontologies using feed-forward neural networks. To the best of my knowledge,
such a form of approximation has not been explored before, but I'm not up to
date with the latest developments for this particular task.

The paper is well-written and introduces (almost) all necessary notions to
understand the contribution. Although some modifications are strongly
encouraged (see below), overall the paper meets the standard of the journal,
both in terms of topics and presentation.

The rest of the review describes some improvements that should be implemented
before acceptance.

- The introduction lacks a proper motivation. I understand that studying
whether subsumption checking can be done with neural networks is
intellectually interesting, but it is unclear whether there are some practical
applications. I think this issue should not be put under the carpet: If the
contribution is entirely curiosity-driven, then let's state it very clearly.
Otherwise, more experiments are needed to show whether the approximation
proposed by the method brings some advantages in terms of runtime. It should
be fairly easy to show an experiment where you report the runtime necessary
if we use an exact reasoner (fact++) and using your method. I believe your
method will be faster (especially if you use a GPU), so the comparison will
be useful.

- The notion of "concept" is never properly defined. This is a problem because
it is unclear how the neural layers which are necessary for, e.g., the
negation, are trained.

- Although the paper is quite detailed, I must admit that the procedure for
training the network is not entirely clear to me. I think it would be good to
report more statistics about the training data. For instance, how many
subsumption queries contain negation? Are some concept names trained more
than others? These details would be useful since a big concern is whether the
network over-fits a specific dataset.

- I'm not sure I would label the proposed neural architecture as "deep". At the
end, it consists only of a few layers with not so many neurons. Also, the
adjective "recursive" which is often used throughout the paper, is misused in
the sense that normally a neural network is called recursive if the previous
state is fed as part of the input (while here it indicates the recursive
construction of the network depending on the formula). I think it would be
good to clarify what you mean with "deep" and "recursive"

- Sometimes the authors make some arbitrary choices, like the number of neurons
in the layers, or a specific optimization algorithm. It would be easy, as a
reviewer, to ask for extra experiments to show how the performance changes if
we vary them. I would refrain from doing so to avoid ending up in an endless
list of changes. There is, however, an experiment that I would like to have
seen. That is, I would like to see what the perfomance is if you change the
hidden layer from 16 to fewer (or more) neurons (line 19). I think that
showing what happens if you do so can help understand whether the network is
memorizing the answers (next to the experiments that have already been
presented).

- The high accuracy of the proposed method leads to the natural question: What
is hard for such a reasoner? That is, are there specific queries for which
the accuracy will be low? The paper does not present many findings on this.
This is a pity because an proper error analysis could be very useful to drive
meaningful future work. It would be good to add an extra section with some
considerations on this.

All in all, it is a nice article which will be interesting for many readers of
this journal.

Summary:
Authors presented a method to create embeddings of knowledge base concepts. They employ a Recurrent Neural Network and a set of embedding layers. The neural network part is trained to classify if subsumption axioms hold (Binary Class Classification). The embedding layers embeds concepts in a vector space so that the classification loss is minimized. The classifier is transferable across different knowledge bases. The deep deductive reasoner is trained with diverse Knowledge Bases. Thereafter, they freeze the reasoner head and train the embedding layers with unseen Knowledge Bases.

Authors generated a synthetic dataset that consists of 60 randomly generated KBs. They also preprocess real-world OWL ontologies to make it compatible with their reasoner. They performed 4 set of experiments where they have assessed the reasoning ability, knowledge transferability and reported Accuracy, Precision, Recall, F1-score, AUC-ROC and AUC-PR. They have reported average accuracy as high as 97% with 0.92 F1 score.

The trait of the reasoner being transferable across knowledge bases makes learning concept embeddings faster and less expensive. It also shows that embedding concepts from multiple domains in a single shared space.

Correctness:
Authors have conducted multiple experiments with both synthetic and real-world data. Dataset settings and synthetic data generation process have also been outlined in the paper. For different set of experiments, the hyperparameter settings such as maximum axiom depth, number of epochs, learning rate have also been provided. The split of datasets into train, validation, and test aligns well as well. In terms of metrics, the coherence relationship between accuracy, precision, recall, and F1 score is also clear.

Novelty & Importance:
Deep deductive reasoning is a difficult problem in the sense that, the model needs to map the inherent relationship that lies between two entities in a symbol-invariant fashion. Designing an architecture that is able to embed the concept between entities as well as able to transfer across different knowledge bases is a significant step towards solving the deep deductive reasoning problem.

Quality of Writing and Presentation:
The paper is well-formatted. It provides appropriate information regarding dataset preparation and settings, loss functions and activation functions. Different experiment settings and respective result analysis are also provided. Methods of result analysis has also been outlined.

The dataset generation code, experiment code, datasets are provided in the given link. It also includes a nicely-written README file to facilitate reproducibility.