Review Comment:
This paper surveys Multilingual Question Answering Systems for Knowledge Graphs (i.e., for the task of mKBQA).
(1) It describes each of the 17 systems found to meet the defined criteria, which is a good starting point for someone working on the area. Yet, to serve as an introductory text on the topic, it could go deeper on explaining the area of KBQA, before mKBQA, provide more details common approaches and challenges, and on the actual application of the referred techniques to mKBQA.
(2) Inclusion criteria are well-defined and coverage of target systems is the best available to my knowledge. Still, some parts of the selection step could be more clear. For instance:
Why were the initially accepted 63 papers further narrowed down to 37?
Why were the sources considered for three languages (specifically, English, German, Russian)?
What were the 12 selected publications that the authors were previously aware of? Why were they not selected with the common procedure?
(3) The paper is well written and well organised. However, I feel that Section 4.1 may be too long and the authors could try to compress it, perhaps by describing similar systems together. The taxonomy in section 4.2 helps to better understand the relations between papers, so authors could try to align section 4.1 even more with the columns of table 4, and even introduce them before describing the papers.
Some descriptions are not completely systematic. For instance, why is it referred for some systems that they are in Java, while the programming language is not given for others?
Moreover, it does not make sense to say "we assume the ... following papers do provide details" and, a few paragraphs later, actually describe these papers. For papers which are several years old, it is also not very natural to describe their future work as something for future. More important would be to confirm whether it has been tackled, either by the same or by other authors.
Finally, would it make sense to refer to the first described paper, from 2011, as some kind of landmark? Some reasons could be further provided for having nothing earlier, and on what triggered mKBQA.
(4) The covered material is definitely important for the Semantic Web community, but also for others, like Natural Language Processing (NLP). I would highlight the website with links to related papers, and the leaderboards for several datasets.
Still, having in mind current advances in NLP, the pros and cons of KBQA towards, e.g., prompting Large Language Models (LLMs) for answer generation should be discussed.
Additional comments:
The Introduction states that "the extraction of the direct answers is enabled by the introduction of the Semantic Web", but it is not exclusive of the Semantic Web, as there are other tasks with similar goal, such as Extractive Question Answering and, more recently, tools like LLMs.
In the taxonomy (section 4.2), I would not put group G3 at the same level as the others, because it is exclusively related to the translation between natural languages.
Section 4.3 describes three characteristics of the methods, but the section could be less theoretic, and better supported by the surveyed papers, also, if possible, speculating on how they could be measured. Otherwise, they are not much more than ideas.
Future directions in Section 6.2 suffer from a similar problem. They do make sense, and some are quite obvious, but could be linked specifically to the identified challenges and supported by conclusions taken in the surveyed works.
Finally, section 5, on benchmarking, could include a deeper discussion on the best approachs for each dataset. Perhaps a summary of the leaderboards, including conclusions on the most suitable / promising approaches.
Possible typos / grammar issues:
p6, l46: there are -> there were ?
p9, l44: will allow to set of up the corresponding services
p11. l24: encoder on the data on a data-rich
p11. l26: This paradigm to be adapted to KGQA to build
p12, ll36: require the other ones to form (which ones?)
p14, l34: in the NLP as
p16, l42: Section 4 (4.1)
p18, l11: worth underlining, that (remove comma)
p21, l8: showed, that the assignment (remove comma)
Plus, there is no introductory text in sections 4.2, 5, 6; and numbers below 13 should be written in full (e.g., four instead of 4).
|