Structured Decomposition for LLM Reasoning: Cross-Domain Validation and Semantic Web Integration

(1) Originality

This paper presents a novel integration of LLMs with Semantic Web standards for rule-based reasoning over natural language. While neural-symbolic integration is already an active research area, the specific architecture represents a distinctive contribution. It positions LLMs as ontology-population engines that generate ABox assertions according to expert-authored TBox specifications, with SWRL-based reasoners providing deterministic verification. The work shows sufficient originality for a full paper and constitutes a substantive extension of prior research through its grounding in Semantic Web standards and its broader cross-domain evaluation.

(2) Significance of Results

- Empirical contributions: The evaluation across three domains, eleven language models, and six experimental conditions, including controlled ablations, provides substantial evidence for the framework’s effectiveness.
- Practical significance: The framework offers several practical advantages for rule-governed domains, notably inspectability, interoperability, and deterministic verification.
- Limitations affecting significance: The evaluated tasks use only conjunctive SWRL rules. The claimed benefits of the OWL/SWRL grounding, such as support for class hierarchies, multi-label classification, and composable imports, are not empirically demonstrated. This limits the strength of the evidence for the advantages of OWL/SWRL specifically. The paper also acknowledges the manual effort required for ontology creation but does not quantify it. Without such estimates, practical adoption barriers remain unclear. In addition, generalisability to multi-class, ranking, or extraction tasks remains untested.

(3) Quality of writing

The manuscript is well organised and follows a logical progression. The description of OWL 2 and SWRL in the section “Semantic Web Technologies for Knowledge Representation” is accurate and appropriately detailed for the SWJ audience. The formalisation of task predicates (Formulas 1–4) is clear and correctly maps to the ontology implementations.

Statistical reporting is generally careful, with paired t-tests and effect sizes (Cohen’s d) reported throughout. However, additional robustness analyses, such as confidence intervals or non-parametric tests, could further strengthen the empirical claims.

In Figure 3, the colour distinction between improvement and regression lines may be difficult to interpret in black-and-white print or for colour-impaired readers; adding directional arrows, patterns, or +/- markers would improve accessibility.

The bibliography is comprehensive and appropriately cites relevant work in neural-symbolic integration, Semantic Web technologies, and LLM reasoning. Recent work from 2023–2025 is well represented. One minor issue is that Citation [1] appears to be unreferenced in the main text.

(4) Assessment of Provided Resources

The authors provide two URLs in the manuscript:
- Dataset generation code: https://github.com/albsadowski/ruleeval-xd
- Framework implementation: https://github.com/albsadowski/structured-decomposition-swj

Both repositories appear to be organised with clear directory structures and include well-written README files. Both repositories are hosted on GitHub, which is explicitly listed in the reviewer instructions as an acceptable platform for long-term resource hosting.

The authors may nevertheless consider archiving a release snapshot on Zenodo or Figshare to obtain a DOI. This would improve long-term citability and archival stability even if the GitHub repository location changes. This is a best practice for academic software, though not a strict requirement for acceptance.

Overall, the provided artifacts appear sufficient for understanding the framework and supporting partial replication. However, some API response logs are included but not comprehensively for all 33 model-task combinations, and API cost and runtime information are not provided. These omissions do not invalidate the resource package, but additional detail would strengthen reproducibility.

(5) Issues and Required Revisions

Issue 1. The authors motivate the transition from SMT to OWL/SWRL partly on the grounds of enabling “richer inference patterns that simpler formalisms cannot express,” including class hierarchies, multi-label classification, and composable imports from domain vocabularies (Abstract, Introduction, Conclusion). However, the evaluated tasks use only conjunctive SWRL rules, effectively corresponding to relatively simple logical combinations of extracted predicates. No evaluated task exploits non-trivial subsumption reasoning, existential restrictions, property chains, or imported ontology modules in a way that demonstrates the added expressive value of OWL. The authors should either add a demonstration task that genuinely leverages OWL expressivity, or temper these claims and reframe the contribution primarily around standardisation, inspectability, and interoperability.

Issue 2. The authors acknowledge that “ontology design required manual effort by the authors” (section “Limitations and Future Work”) but provide no estimate of this effort. For practitioners considering adoption, understanding the cost of creating task ontologies is essential. A qualitative or quantitative estimate of the effort required per task would substantially improve the paper. This could be added in “Limitations and Future Work” or as a short subsection in “Task Formalisation.”

Issue 3. The paper reports that structured decomposition regressed on specific model-task combinations, yet no qualitative analysis of these failures is provided, despite the framework’s claimed inspectability advantage. The authors should include at least one illustrative failure analysis in the Discussion, or in a dedicated subsection, showing where the decomposition failed and how the inspectable intermediate representations help diagnose the error.

The paper proposes an interesting neuro-symbolic framework using LLMS, ontology representation, and SWRL reasoning for structured information extraction and validation. The integration of semantic reasoning with LLM-based extraction is interesting and a hot topic although few approaches have been proposed in that topic like Sahbi, Aya, Céline Alec, and Pierre Beust. "Semantic vs. LLM-based approach: A case study of KOnPoTe vs. Claude for ontology population from French advertisements." Data & Knowledge Engineering 156 (2025): 102392. The authors could improve the paper by discussing further existing approaches and providing a table of SoTA and how they position their work.
The paper is generally well motivated and the proposed pipeline is interesting.

However, several aspects of the work would benefit from further improvements and also the experimental validation. First, the paper does not clearly explain how the ontology schema (TBox) is constructed. Is it made with experts or through LLMs. It raises also if the work is positioned in closed or open world assumption. The integration of the ontology into the prompting strategy should be detailed further (example of a prompt with used prompted strategy).

Since the extraction performance depends on prompt engineering and ontology alignment, it is important to provide representative prompts and extraction results. In addition, it is not clear how the framework handles incomplete, null, or hallucinated assertions generated by the LLM, especially under OWL’s open-world assumption.

The experimental evaluation is also limited does not sufficiently show the robustness on realistic noisy datasets.
Further details concerning the ground-truth construction, the level of human intervention in the pipeline, and comparisons with existing extraction or neuro-symbolic baselines would strengthen the contribution considerably.

This paper presents a neural-symbolic framework for rule-based classification over natural language text. The core idea is to use large language models (LLMs) as ontology population engines: given a task defined as an OWL 2 TBox with associated SWRL rules, the LLM extracts entities and asserts properties (ABox population), while a symbolic reasoner applies classification rules deterministically. The authors extend their prior work to three domains, namely legal reasoning, scientific literature analysis, and clinical trial eligibility, and evaluate eleven language models from four providers under six experimental conditions. The main findings are that structured decomposition with SWRL verification outperforms few-shot and chain-of-thought baselines in aggregate, that the symbolic verification step contributes substantially beyond structured prompting alone, and that complementary predicates, which were beneficial in prior work, prove detrimental in the current evaluation.

Strengths of the Article
* Innovative approach and strong theoretical foundation: the integration of LLMs with symbolic reasoning directly addresses current challenges in explainable AI.The paper clearly explains the complementary strengths and weaknesses of neural and symbolic systems.
* Robust experimental validation: the framework is evaluated across three distinct domains and eleven language models.
* Reproducibility: the authors provide ontologies, scripts, and GitHub repositories for replication.
* The change in results regarding complementary predicates is discussed honestly by the authors. In their previous work, these predicates improved precision, but in this study, they reduced performance overall. The authors suggest that their usefulness may depend on the specific model and may be less important for newer LLMs. This transparent discussion is valuable and also provides useful guidance for ontology design.

Weaknesses and Limitations
* High technical complexity: implementation requires advanced knowledge of LLMs, OWL ontologies, and symbolic reasoning systems. The ontology design process is underspecified. How were the entity descriptions and assertion specifications validated? Were the ontologies designed with knowledge of the test data, and if so, what measures were put in place to fight against overfitting the ontology?
* One important limitation is the relatively small size of the test sets. With a limited number of instances, and only about half labeled as positive, even a small change in predictions can lead to substantial fluctuations in F1 scores. The authors should either increase the size of the evaluation datasets or provide a more detailed discussion of how this limited scale may affect the robustness and reliability of their conclusions.
* The precision/recall trade-off needs a deeper analysis. Table 6 shows that SD achieves a higher recall (84.7%) than few-shot (69.0%) but lower precision (79.4% vs 85.3%). The paper frames this as a net gain in F1. The mechanism behind this trade-off is not analysed. Is the higher recall a consequence of OWA leading SWRL rules to fire when evidence is ambiguous? And if it is the case, how to address this issue?
* The paper notes the open-world assumption of OWL as a theoretical consideration but concludes that its practical impact is limited for the conjunctive binary tasks evaluated. This is probably correct for the current evaluation, but it is also a significant limitation of the framework for more complex reasoning patterns.
* The baselines in this paper are limited to few-shot prompting and chain-of-thought prompting. For the three tasks evaluated, stronger baselines exist: RAG approaches, task-specific fine-tuned models, and prior neural-symbolic systems cited in Table 1. At a minimum, the authors should discuss why these baselines were excluded.

This article represents an important contribution to neuro-symbolic AI research. It demonstrates that LLMs become more reliable when coupled with formal logical verification mechanisms. The use of Semantic Web standards such as OWL 2 and SWRL also improves interoperability and explainability. However, the framework is mainly suited for domains governed by explicit rules and requires substantial ontology engineering effort. Despite these limitations, the work opens promising directions for explainable AI applications in regulated domains such as law, medicine, and compliance.