Review Comment:
The paper presents a method to perform approximate subsumption checking for ALC
ontologies using feed-forward neural networks. To the best of my knowledge,
such a form of approximation has not been explored before, but I'm not up to
date with the latest developments for this particular task.
The paper is well-written and introduces (almost) all necessary notions to
understand the contribution. Although some modifications are strongly
encouraged (see below), overall the paper meets the standard of the journal,
both in terms of topics and presentation.
The rest of the review describes some improvements that should be implemented
before acceptance.
- The introduction lacks a proper motivation. I understand that studying
whether subsumption checking can be done with neural networks is
intellectually interesting, but it is unclear whether there are some practical
applications. I think this issue should not be put under the carpet: If the
contribution is entirely curiosity-driven, then let's state it very clearly.
Otherwise, more experiments are needed to show whether the approximation
proposed by the method brings some advantages in terms of runtime. It should
be fairly easy to show an experiment where you report the runtime necessary
if we use an exact reasoner (fact++) and using your method. I believe your
method will be faster (especially if you use a GPU), so the comparison will
be useful.
- The notion of "concept" is never properly defined. This is a problem because
it is unclear how the neural layers which are necessary for, e.g., the
negation, are trained.
- Although the paper is quite detailed, I must admit that the procedure for
training the network is not entirely clear to me. I think it would be good to
report more statistics about the training data. For instance, how many
subsumption queries contain negation? Are some concept names trained more
than others? These details would be useful since a big concern is whether the
network over-fits a specific dataset.
- I'm not sure I would label the proposed neural architecture as "deep". At the
end, it consists only of a few layers with not so many neurons. Also, the
adjective "recursive" which is often used throughout the paper, is misused in
the sense that normally a neural network is called recursive if the previous
state is fed as part of the input (while here it indicates the recursive
construction of the network depending on the formula). I think it would be
good to clarify what you mean with "deep" and "recursive"
- Sometimes the authors make some arbitrary choices, like the number of neurons
in the layers, or a specific optimization algorithm. It would be easy, as a
reviewer, to ask for extra experiments to show how the performance changes if
we vary them. I would refrain from doing so to avoid ending up in an endless
list of changes. There is, however, an experiment that I would like to have
seen. That is, I would like to see what the perfomance is if you change the
hidden layer from 16 to fewer (or more) neurons (line 19). I think that
showing what happens if you do so can help understand whether the network is
memorizing the answers (next to the experiments that have already been
presented).
- The high accuracy of the proposed method leads to the natural question: What
is hard for such a reasoner? That is, are there specific queries for which
the accuracy will be low? The paper does not present many findings on this.
This is a pity because an proper error analysis could be very useful to drive
meaningful future work. It would be good to add an extra section with some
considerations on this.
All in all, it is a nice article which will be interesting for many readers of
this journal.
|