Review Comment:
The paper presents a comprehensive study of several approaches for taxonomy enrichment and an error analysis describing their typical mistakes. To this purpose, the authors introduce two new datasets for this task extracted by WordNet (in English and Russians) and evaluate alternative solutions based either on word representations or graph representations
The paper is well written and clear. The topic is very relevant to the special issues.
This is a very interesting paper that can be a good starting point for future research in the area. In particular, I appreciate Section 7 which does an in-depth error analysis on the different types of mistakes produced by state-of-the-art approaches and offers some interesting insights of their limitations. The most interesting finding is that the methods based on word vector representation appear to outperform graph-based approaches, which are usually performing well on link prediction tasks.
The only major issues of the paper regard the sharing of data and the contributions with respect to previous articles, which I would like to clarify before acceptance.
A good portion of the paper appears to derive from the article:
Irina Nikishina, Varvara Logacheva, Alexander Panchenko, Natalia V. Loukachevitch: Studying Taxonomy Enrichment on Diachronic WordNet Versions. COLING 2020: 3095-3106
While it if fine for the paper to be an extension of COLING 2020, the relationship between the two paper should be stated explicitly. In particular the authors should clarify in the introduction 1) if the paper is an extension of COLING 2020, and 2) what contributions are new.
Regarding the sharing of the data, I commend the authors for sharing code and data regarding the two novel datasets. However, it is not clear to me if the authors are sharing all code and data regarding the study presented in the paper. The linked Github repo (https://github.com/dialogue- evaluation/taxonomy- enrichment) seems to contain only part of the data, but I may be wrong. The authors should make all the data available and clarify better where they can be found.
In the following, please find additional comments/questions on specific sections.
Section 2
The state of the art is quite comprehensive. I would suggest to briefly refer to methods for extracting taxonomies/ontologies/knowledge graphs from the text, such as “Cimiano, P. and Völker, J., 2005. Text2Onto. In International conference on application of natural language to information systems. Springer, Berlin, Heidelberg.”, “Luan, Y., He, L., Ostendorf, M. and Hajishirzi, H., 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. arXiv preprint arXiv:1808.09602.”,“Dessì, D. et al. 2021. Generating knowledge graphs by employing Natural Language Processing and Machine Learning techniques within the scholarly domain. Future Generation Computer Systems, 116”. It may be useful to mention also some of unsupervised approaches developed by the semantic web community such as “Osborne, F. and Motta, E., 2015. Klink-2: integrating multiple web sources to generate semantic topic networks. In International Semantic Web Conference 2015. Springer, Cham” and “Han, K. et al. 2020. WikiCSSH: extracting computer science subject headings from Wikipedia. In ADBIS, TPDL and EDA 2020 Common Workshops and Doctoral Consortium. Springer, Cham.”
Section 4
“This baseline has a shortcoming […]” This reads confusing at the beginning of a new section. I would rephrase with something such as “The solution presented in the previous section has…”
“This approach was presented at the RUSSE’2020 33 Taxonomy Enrichment task” Why is this approach not cited?
Section 6
The evaluation would be stronger if a statistical significance test was presented, in particular for validating some of the major findings of the study, such as the difference between word- based and graph-based methods.
Typos
“non-linear projection transofrmations” > transformations
|