Review Comment:
This paper describes a natural language query system over biomedical linked data. The (controlled) natural language component of the system is implemented in Grammatical Framework and is available in English and Romanian. The system maps the natural language input to the corresponding SPARQL query for execution. The focus of the paper is on the implementation of the biomedical domain specific grammar.
I find the topic (querying biomedical data) important and the technology choice (multilingual CNL implemented in GF) suitable. However the presentation of research results is poor, thus a major revision is recommended.
Originality. Natural language interfaces to SPARQL have been done before (also multilingually and in GF), as the paper's related work section also discusses. The application to the biomedical domain is possibly new but the paper fails to discuss the novel aspects of this application domain, which would not directly follow from the existing general approaches.
The significance of the results is hard to estimate. No actual controlled natural language is presented in the paper (apart from a few example sentences on some figures), i.e. the reader does not get a clear idea of the query language regarding its syntax, semantics, control of ambiguity, and coverage of SPARQL. There is very little evaluation of the developed CNL (how easy it is to read/write/etc?) or the whole translation and query system. There is no link to the (open source) implementation of the system (grammar and the query UI from Fig 12).
Quality of writing. The structure of the paper makes it hard to follow. Also, the English should be improved.
More detailed comments.
I would propose the following structure, which would make the paper easier to follow.
1. Introduction/Goals
2. Related work
3. Controlled English for querying biomedical data: syntax, semantics (as mapping to SPARQL), ambiguity handling
4. Controlled Romanian for querying biomedical data: ...
5. Multilingual implementation of these CNLs in GF, incl. lexicon building from existing resources
6. Usage examples. Possibly user interface considerations (is there a look-ahead editor to help with the query entry, how is ambiguity communicated to the user, etc.)
7. Evaluation of benefits over plain SPARQL input or graphical user interfaces
8. Future work
The current section 3 (specifically 3.1-3.3) is poorly structured and the details remain unclear, especially the role of DL in the system.
The handling of out-of-grammar expressions (in Algorithm 1) by replacing the last word by "XX" and reparsing could be replaced by a more general method. Recent developments in GF support robust parsing which might be the right solution in this case. Also, ambiguity resolution by picking the tree with the "smallest length" needs more motivation. Also, it remains unclear how "smallest length" is defined. Algorithm 1 contains unnecessary technical details ("GF Rest Service"), inconsistent indentation and notation (capitalization, function call notation etc.). Also, avoid "!" as a negation operator. The algorithm is called "English2SPARQL". Would it look any different when applied to Romanian?
Figure/table captions should contain more information about what is shown on the figure/caption.
It would be useful to have a small summary of QALD4 before discussing the evaluation. There should be a separation of development set (set of queries/use cases/etc. for which the system was optimized) and a test set (which the system had not seen before), otherwise the evaluation results are not very meaningful. There could also be an evaluation of how do different formalisms (SPARQL vs CNL) represent the 25 test questions, and insight into why does CNL outperform SPARQL in terms of readability/writability (if it is the case).
The lexicon building section could present the English and Romanian lexicons in parallel, e.g. there could be a table that lists the counts of words and word forms in each language for each concept (drug, disease).
Fig 12. seems to present a developer UI and doesn't even present the query results. In addition (or instead) there could be a figure showing the envisioned end-user query interface. An end-user UI would probably not display two CNLs in parallel, would offer a disambiguation dialog (in case there are multiple abstract trees), would not show any technical details (SPARQL, abstract trees) and would present the query results in a format that is easily relatable back to the CNL input.
In general, the paper could focus more on the multilingual aspects and specifically answer the question of how easy it would be to add an other language to the system considering the available resources (biomed databases for words and GF's resource grammar library for syntactic structures).
Smaller issues
Fig. 5: caption is not in English
Table 2: use more consistent DL terminology, e.g. class expressions and assertions
"object properties relate two concepts" is incorrect, better: "an object property constructs a class from a restriction and another class"
use monospace font for code examples
double "f" (e.g. in "SideEffect") is badly formatted
"rdfs : label" and similar expressions are badly formatted
typo: classese
typo: liniarization
Fig 7: ":" missing on the 2nd line
|