From Manual to LLM-assisted: A Mapping Review on Ontology Construction Methods

The paper fills a gap by updating the landscape of manual ontology construction methods (last surveyed in 2020) and proposing a two-tier classification across manual, semi-automatic, and automatic categories. The SMS methodology is correctly applied with transparent attrition reporting, the F1–F8 feature scheme for manual methods is practically useful for practitioners selecting a methodology, and the dual classification of semi-automatic approaches (embedding-based vs. LLM-based) with a convergence analysis is a nice methodological contribution. The task x strategy matrices for automatic methods (Figures 10–11) give a clear snapshot of what architectures solve which ontology-learning tasks across two time periods.

However, the clustering of ~41 manual methods into k=11 clusters produces many singletons and pairs whose practical value is unclear. The conclusion is very brief and offers no synthesis across categories, no identification of open problems, and no discussion of limitations. Additionally, no data deposit or long-term URL is provided, which the SWJ survey track explicitly requires. Some figures (especially 10 and 11) are difficult to read at print size, and a copy-editing pass is needed for minor errors throughout.

Questions for the authors:

- With k=11 clusters and several singletons (e.g., clusters 9, 15 in Table 16), what is the practical guidance a reader should draw from the clustering?

- Why was Gower's distance not considered for the mixed categorical/binary/ordinal feature space instead of the custom distance function?

- For the semi-automatic LLM-based classification, was any robustness check performed across different LLMs or prompts?

- Could you provide a data deposit (e.g., Zenodo, GitHub, etc.) containing the encoded feature tables, clustering scripts, LLM prompts/responses, and a README, as required by the SWJ survey track?

- Could the conclusion be expanded to discuss cross-category findings, concrete open problems, and limitations of the survey itself?

- The representation of the clusters in Figure 8 should be improved. They are not clearly visible.

This manuscript presents a survey of ontology construction methods, which are organized in a threefold taxonomy: manual, semi-automatic, automatic.
The survey relies on the SMS approach supported by the use of a LLM-as-judge method and is structured in three macro-sections: one for each approach.

While the paper’s aim is relevant and a high amount of work has been done to survey existing literature, several weaknesses affect the manuscript:

1. Potentially out-of target. A survey must be introductory to a certain extent. However, in this case (Sections Introduction and Background Knowledge) the paper appears to be too introductory in defining what is an ontology and a KG. In contrast, a contextualization and scope of the survey at a general level is missing.

2. Engagement with existing literature. The NLP literature presented in the introduction (e.g. LeCun et al, 2015) feels very broad and somehow generic. I expected to find some pointers to existing challenges and approaches at the intersection between SW and NLP to be fully examined in the other sections. Additionally, existing surveys that overlap with this one are not presented. For instance:
- Zengeya, T., & Fonou-Dombeu, J. V. (2024). A review of state of the art deep learning models for ontology construction. IEEE Access, 12, 82354-82383
- Li, J., Garijo, D., & Poveda-Villalón, M. (2025). Large language models for ontology engineering: a systematic literature review. Semantic Web Journal, 20(x), 1-45

3. Scattered methodology. The presentation of the survey method is clear. However, in the following sections a number of additional methodological approaches are adopted to analyze papers without a specific rationale. For instance, clustering methods at page 9. I suggest to create a methodology section that can include all the methodologies adopted in the survey, being clearer about the authors’ design choices.

4. Problematic usage of LLM-as-judge. In two different sections (Review Methodology and Semi Automatic Ontology Creation) authors introduce the usage of a LLM (GPT) to automatize their analysis. This is not necessarily problematic. However, if authors propose a LLM-as-judge approach they should ensure that they properly checked the model’s output and mitigated potential bias in the pipeline. Additionally, using a commercial closed-sourced model makes the methodology non replicable.

5. Scope of the survey. While the general aim of the survey is clear, its scope is not entirely understandable. Specifically, RQs are presented without a specific connection to the objectives of the manuscript. This hinders following the flow of the narrative. For instance, “RQ6: What is the context of the methodology?” is not explained nor logically connected to the overall work leaving me wondering why this question was relevant for the survey. Dedicating more space to contextualize the selection of these RQs is essential to improve the manuscript making its contribution clearer and easier to follow.
6. Thoroughness of discussion. The three sections where results are presented are very descriptive. They briefly answer the questions with lists of items (e.g., type of evaluation approaches) without discussing them. For instance, in the Semi-Automatic Section (page 9), “RQ4: What metrics are available?” abruptly refers to Table 7 without further discussion about these metrics. In other cases (Figure 9, right, page 13) the survey appears to rely on some taxonomical shortcuts creating the category “Specific procedure” that might mean a wide range of things. I suggest rethinking all these sections trying to identify some takeaways that one can get from the survey. E.g., Table 12 (page 14) on automatic methods for ontology construction shows a clear imbalance towards reference-based metrics (F1, precision, recall, accuracy), which in the actual AI landscape are contested since they are not reliable to assess models’ abilities out-of-domain and their biases (e.g., a LLM can generate an ontology that is biased towards specific socio-demographics). This is an interesting insight about some directions that could be taken to design better evaluation metrics.

7. Some colloquial passages. The paper is clearly written but I suggest revising some passages that are too colloquial. E.g. “It is not difficult to imagine that many papers are in common between the different sources”; “as you can see from Figure 8”.

8. No permanent links provided. The paper presents several automatic methods supporting the survey but no link to the code is provided nor a pointer to the list of selected papers.

Given these weaknesses, the paper is not publishable at this stage. I would be glad to revise it again if the authors decide to provide a new version.

This paper presents a classification of ontology construction methods according to the level of automation proposed.

The paper follows a mapping study strategy and describes its steps to some extent. However, several aspects of the process are either insufficiently described or lack methodological clarity. For example:

* Most of the research questions are answered through descriptive reporting of the observed data rather than through insights derived from the extracted and analyzed evidence.

* The exact criteria used to exclude papers during the screening phase are unclear. Additionally, it is not explained how these criteria were applied to papers retrieved from arXiv.

* Did the authors apply snowballing as part of the search strategy?

* In "Step 8. Local Classification", the parameters and metrics used for comparison should be explicitly listed and defined.

* The paper provides numerous tables and figures; however, it is not clear how these artifacts were generated or how they can be traced back to the analyzed papers. All extracted data features should be described, together with the analysis and harmonization procedures applied. For example, what observations from the primary studies led to the generation of Table 2? The same concern applies to the tables and figures associated with Figures from 6 on. The data collection and validation procedures should also be explained.

* It should be clearly stated which data were extracted from the papers (and their corresponding values) to answer each research question.

* How is the term "feature" defined in Table 2?

* Another major issue concerns the predefined categories of manual, semi-automatic, and automatic approaches. Are all retrieved works complete methodologies, or is there a mix of methodologies and methods/techniques targeting specific tasks or activities? If so, are all these elements truly comparable?

* Why is "linguistic" considered a category within Type (F1)? How was the list of features defined?

* It is not clear what is meant by "local evaluations."

* Page 9: "For this, we reduce the 8 features to 3." Why was this reduction performed, and which features were retained?

* In general, the presentation of partial information in tables creates uncertainty regarding whether all papers were analyzed. For example, should readers assume that papers not appearing in Table 5 have no reported limitations, or that the table only presents a subset of the results? Information should be reported for all studies, including the explicit absence of particular characteristics.

* Page 9, RQ5: The proposed description appears somewhat ad hoc with respect to the application of machine learning in this domain. Were automated methods that do not rely on machine learning also considered?

* It is not clear how Figures 6a and 6b were generated, nor how the conclusions derived from them were obtained.

* Page 9, RQ8: The description provided is not sufficiently clear.

* Page 13, RQ8: The objective appears to be the classification of methodologies. Based on the results presented, would "clustering" be a more appropriate term than "classification"?

Overall, to provide a more rigorous and valuable contribution, the paper should:
(a) clearly define the objects of study and how can they be compared, that is are all of them methodologies?

(b) include more analytical research questions in addition to the predominantly descriptive ones currently presented;

(c) explain in detail the data extraction, coding, harmonization, and validation processes, while making the extracted dataset available; and

(d) improve the presentation of results and the derivation of conclusions.