A Framework for Assessing LLM Consistency in Knowledge Engineering

The authors have improved the paper based on the reviewers’ comments. However, I remain unsatisfied with the related work section. In particular, for the ‘replacement test’, there already exists work comparing different LLMs on the same KE tasks, including KG completion and reasoning [1], ontology learning [2], relation extraction [3], ontology generation [4], ontology matching [5]...

[1] Li, Qian, et al. "LLM-based multi-level knowledge generation for few-shot knowledge graph completion." Proceedings of the 33rd International Joint Conference on Artificial Intelligence. Vol. 3. 2024.
[2] Mai, Huu Tan, Cuong Xuan Chu, and Heiko Paulheim. "Do LLMs really adapt to domains? an ontology learning perspective." International Semantic Web Conference. Cham: Springer Nature Switzerland, 2024.
[3] Zhang, Bohui, et al. "Using large language models for knowledge engineering (LLMKE): a case study on Wikidata." arXiv preprint arXiv:2309.08491 (2023).
[4] Llugiqi, Majlinda, Fajar J. Ekaputra, and Marta Sabou. "From Experts to LLMs: Evaluating the Quality of Automatically Generated Ontologies." 2nd Workshop on Evaluation of Language Models in Knowledge Engineering (ELMKE), co-located with ESWC-25, to appear. 2025.
[5] Qiang, Zhangcheng, Weiqing Wang, and Kerry Taylor. "Agent-om: Leveraging llm agents for ontology matching." arXiv preprint arXiv:2312.00326 (2023).

The authors have addressed my comments sufficiently. The content of the paper is clearer and more intuitive to follow.

However, I would like to point out that it was difficult to figure out what exactly was updated and where, since the response letter included limited pointers (e.g. update on page X line Y), and updates in the paper have not been highlighted to guide the reviewer.

For example, "We have now clarified the difference between reproducibility and reproducing (similar to replicating and repeating) throughout the paper." It would be beneficial if you pointed the reviewer to a concrete section (at least).

Minor formatting comment: There seems to be quite a lot of space before and after the abstract. This could be from the template itself, but if not, the page will look better if the authors try to include parts of section 1 on the 1st page.

The beginning of section 5 is still cut off by table 5.

References [28][55][82] are incomplete.
Reference [88] is missing a hyperlink. What was accessed?

The paper presents a framework for assessing consistency in LLM-driven knowledge engineering by considering experiment repeatability, the impact of LLM updates, and the effects of LLM replacement. The topic is timely and clearly presented. Compared to the previous version, the paper has significantly improved, particularly in its introduction of the problem, positioning of the contribution, and explanation of the proposed framework.

As a minor suggestion, the related work section could benefit from the inclusion of a comparative table to better structure differences among existing approaches and make patterns more evident. The results are clearly and thoroughly described. Another minor comment concerns accessibility: the authors should consider using color-blind–friendly color schemes in tables to indicate increases and decreases (e.g., yellow and blue instead of red and green).

The discussion section is the main point of concern, as it reads more like concluding remarks than an in-depth discussion. However, the results are sufficiently rich to support this structure. The authors might consider either more clearly separating results and discussion (e.g., organized by research questions or in relation to prior studies) or merging the two sections.

Overall, I recommend acceptance of the paper as is, with a few minor suggestions for improvement.