Reranking Answers of Large Language Models with Knowledge Graphs

Submitted by Alexander Panchenko on 12/10/2025 - 09:23

Tracking #: 3985-5199

Authors: 
Mikhail Salnikov
Hai Le
Olga Tsymboi
Ivan Lazichny
Dmitrii Iarosh
Egor Cheremiskin
Andrey Savchenko
Dmitry Simakov
Alexander Panchenko

Responsible editor: 
Guest Editors 2025 LLM GenAI KGs

Submission type: 
Full Paper
Abstract: 
Answering natural language questions over knowledge graph data is challenging due to the vast number of facts, which can be difficult to process and navigate. One potential solution for this issue is to use mined subgraphs related to the query, although this process still requires extracting these subgraphs. This research presents a solution for extracting subgraphs related to entity candidates from a question-and-answer set, which can be obtained by inferring a large language model by calculating the shortest paths between entities. The proposed approaches detail various features that can be extracted from the subgraphs and reranking models to select the most probable answers from a list of candidates. Experiments were conducted on Wikidata to evaluate the effectiveness of the proposed approaches. This involved enumerating all the main feature types that can be extracted from mined subgraphs and a detailed analysis of the proposed features and reranking method combinations. In addition, a public web application that provides a useful web tool for studying the graph space between question and answer entities has been developed to work with subgraphs. This includes visualization of the extracted subgraph and automatic generation of natural language text to describe it.
Full PDF Version: 
PDF icon swj3985.pdf
Previous Version: 
Reranking Answers of Large Language Models with Knowledge Graphs
Tags: 
Reviewed
Long-term Stable Link to Resources: 
https://github.com/s-nlp/kbqa

Decision/Status: 
Minor Revision

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 22/Jan/2026
Suggestion:
Minor Revision
Review Comment:

The reviewed manuscript addressed the majority of the concerns expressed. However, some improvements are still needed/some comments were left unaddressed:

Given that the manuscript presents and evaluates in combination different techniques/features for various parts of the reranking pipeline, the conclusion section should include the main take-aways.

The first part of Section 5 contains the discussion of Figures 4 and 5. However, given the importance of the results in the contribution, it feels underdetailed and not completely accurate. A more precise discussion is needed, and particularly commenting on the competitor ranking approach RankGPT (it could benefit from a better introduction, as in the manuscript it is not clear what it does). Importantly, given the high computational/time costs of generating the subgraphs, some explicit discussion quantifying the gains when using graphs features may be beneficial.

Again, throughout the manuscript, there are instances of inconsistent or incorrect formatting when referencing figures, sections, tables, algorithms, and appendices. Also, spell and phrasing checks should be performed (for instance, in the first paragraph of the Methods section).

Review #2
Anonymous submitted on 16/Feb/2026
Suggestion:
Accept
Review Comment:

I have carefully read the revised version of the submitted paper and compared it with the previous version and with the revisions requested in my initial review. The quality has improved significantly, and the authors have made substantial efforts to address the previous concerns and to implement the modifications. For these reasons, I believe the paper can be accepted.