Review Comment:
# Review of SWJ Submission 4053
Title: SHACL-Guided Small Language Models for RDF Knowledge Graph Population: Semantic Evaluation, Cross-Domain Generalisation, and Long-Tail Mitigation
Authors: Ringwald, Gandon, Faron, Michel, Abi Akl
Recommendation: Major Revision
## 1. Summary of the Paper
This manuscript extends the Kastor framework - a relation extraction pipeline where a SHACL NodeShape defines the target schema - from extracting datatype properties on dbo:Person to jointly extracting datatype and object properties across 16 DBpedia classes. For each class shape, the authors distil a dataset of paired Wikipedia abstracts and DBpedia subgraphs, keeping only triples grounded in the text (via string matching for literals and Markdown hyperlink matching for object URIs, termed "Wikicheck"). These pairs are then used to fine-tune separate BART-base models (139M parameters) that take entity_URI : Abstract as input and generate a single-line factorised Turtle graph.
The paper puts forward five main contributions:
* Joint extraction of datatype and object properties within the pattern-based framework.
* An eight-category error taxonomy (E1-E8) for object properties, paired with four automated metrics: valid URI rate, Wu-Palmer class similarity, wrong relation choice rate (WRC), and an AlignScore-based hallucination estimator (AS).
* The release of KastorKG, comprising 19 SHACL shapes adapted from Text2KGBench, 141,096 distilled text-graph pairs across 16 classes, and 16 fine-tuned BART checkpoints.
* An evaluation on the DBpedia-WebNLG portion of Text2KGBench against 4-shot Alpaca-13B and Vicuna-13B baselines, accompanied by a cross-domain transfer analysis.
* An investigation of long-tail property mitigation on dbo:Person, concluding that guaranteeing at least 1,000 training instances per property ("sufficient-exposure sampling") is more effective than dataset scaling, stratification, or re-weighted loss functions.
## 2. General Assessment and Journal Criteria
The core engineering effort behind this work is substantial, and the resulting resource (KastorKG and the accompanying checkpoints) is a welcome contribution to the Semantic Web community. The error taxonomy and the pattern-level evaluation metrics also offer a practical way to look beyond strict triple matching when diagnosing extraction failures.
At the same time, several parts of the manuscript build on the authors' ESWC 2025 and KCAP 2025 papers - specifically the distillation pipeline, the dbo:Person experiments, the REBEL baseline comparison, and the long-tail study. While the authors acknowledge the KCAP 2025 overlap in Section 1, they also cite a companion SWJ submission currently under review (Ringwald et al., 2025b). To help readers and reviewers assess the exact delta of this journal version, the paper should clearly delineate which tables and experiments are new relative to both KCAP 2025 and the companion SWJ submission.
Furthermore, the paper's central empirical claims - regarding the superiority of SLMs over LLMs on Text2KGBench, the "hallucination-free" nature of the approach, the predictors of cross-domain transfer, and the long-tail mitigation results - suffer from methodological confounds and train-test leakage that I detail below. During my review, I inspected the Zenodo archives, re-ran the evaluation calculations where files were available, and tested the released KastorCity checkpoint locally. While the REBEL baseline numbers in Table 4 reproduce cleanly from the released files, the inspection of the data splits and model behaviour revealed several issues that require a major revision.
## 3. Strengths
* Openly licensed, large-scale resource. The KastorKG Zenodo archive (v2, 2.6 GB N-Quads dump under CC BY 4.0) and the 16 Hugging Face checkpoints resolve properly. Releasing both the intermediate named graphs (inferences, Markdown abstracts, Wikichecked triples) and the trained models makes the pipeline inspectable and reusable.
* Diagnostic value of pattern-level and semantic metrics. Reporting the exact pattern-conformance rate (Eq. 4) and Pattern Extension Capacity (PEC, Eq. 5) alongside micro-F1 and macro-F1 clearly shows that the models tend to under-generate rather than over-generate when uncertain. Likewise, breaking false positives into substitution errors (FP_diff, Eq. 10) and novel errors (FP_new, Eq. 11) and scoring them with taxonomic similarity and AlignScore demonstrates that many strict false positives are either unlabelled true facts in Wikipedia or ontologically close URIs.
* Qualitative discussion of register shift. The analysis in Sections 6.4.4 and 6.4.5 of why classes such as dbo:SportsTeam and dbo:CelestialBody collapse when transferring from Wikipedia prose to WebNLG (which verbalises infobox attributes like orbital epochs or stadium grounds) is well reasoned.
* Reporting negative results. The authors transparently report that stratified sampling, Weighted Cross-Entropy, Focal Loss, and naive template replacement (KR0) failed to improve rare-property extraction in Section 7.
## 4. Major Concerns
### M1. Asymmetric experimental setup in the Text2KGBench comparison (Section 6.4.4, Table 14)
The abstract and conclusion emphasise that 139M-parameter SLMs beat 13B-parameter LLMs on 11 of 16 classes with zero subject or relation hallucinations. However, the evaluation conditions differ between the two model families in ways that heavily favour the Kastor models:
* Given subject vs. open extraction. The Kastor prompt is $entity_URI : $Abstract (p. 8). Because the target subject URI is given in the prompt and the model is trained to emit triples for that single subject, subject hallucination is zero by design. In contrast, the 4-shot LLM baselines must identify and link the subject entities from the raw text and are penalised when they extract triples about secondary entities. The authors mention (p. 19) that they restricted the WebNLG evaluation set to triples whose subject is the main entity and whose relations exist in DBpedia, and state that recomputing the LLM metrics on this restricted subset "yielded no significant differences". Please include the recomputed LLM numbers in Table 14 (or in an appendix) and specify what statistical test was used.
* Closed relation vocabulary per class. Each Kastor model is dedicated to a single class and trained on a pruned shape where uninstantiated properties were removed (Section 6.1, reducing constraints by roughly half). It can only emit the 3 to 18 properties seen during its training. Comparing its relation hallucination rate against few-shot LLMs prompted with the full unpruned ontology is not an equal test of schema adherence.
* Baseline selection and anomalies in Table 14. The LLM baselines are 4-shot Alpaca-LoRA-13B and Vicuna-13B taken from the original 2023 Text2KGBench paper. Comparing 16 specialized, fine-tuned models against 2023 few-shot baselines conflates model scale with training regime. To isolate the benefit of the Kastor architecture, the benchmark needs a fine-tuned baseline trained on the same data (for instance, a single multi-class BART or T5 model, or a LoRA-tuned small LLM). Furthermore, please clarify where the per-class Alpaca and Vicuna numbers in Table 14 come from: the global rows match Table 2 of Mihindukulasooriya et al. (2023) except for Vicuna's recall (0.31 here vs. 0.27 in the original paper), and the row for SportsTeam under Vicuna lists Precision = 0.52 and Recall = 0.91, which cannot yield micro-F1 = 0.42. Finally, City under Kastor has an unusually high standard deviation across folds (0.56 +/- 38 in micro-F1, 0.69 +/- 47 in recall) compared to 99 +/- 0 on KastorKG (Table 11); please explain what happens across the folds on this class.
* Input format and URI matching protocol. Table 3 evaluates both PLAIN and MD (Markdown with hyperlinks) inputs. Since Text2KGBench sentences are plain text without Wikipedia Markdown links, please specify which input variant (PLAIN or MD) was used for the 16 models in Section 6, how the models generate exact DBpedia URIs when no links are present in the input, and whether evaluation requires exact URI string matching or normalised label matching (as used in several Text2KGBench studies).
### M2. Claims of "hallucination-free" and "SHACL-constrained generation" (Section 5, Section 8.2, Table 19)
In Section 8.2 and Table 19, the paper describes the Kastor models as "hallucination-free" and attributes their low hallucination risk to "SHACL-constrained generation". Both statements contradict what the paper's own experiments show:
* No constraints are active at inference time. SHACL shapes are used offline to filter the training data. During decoding, the BART model generates tokens freely via standard beam search without constrained decoding or grammar masks, and the shape itself is not passed in the prompt.
* Object-level hallucination is the primary error mode. While subject and relation hallucinations are prevented by the prompt format and the closed training vocabulary, the paper's own error breakdown on dbo:Person (Fig. 2, p. 11) shows that Hallucination (E8: generated triple not expressed anywhere in the input text) accounts for over 50% of all errors in both PLAIN and MD modes. In Table 8, the mean AlignScore on false positives is 0.56-0.57, and in Table 15 (WebNLG) it drops to 0.09-0.39 for several classes, indicating a relevant object-level hallucination.
* Lack of abstention in practice. I downloaded and ran the released Datartisan/KastorCity checkpoint locally on a few test inputs using the $Entity : $abstract prompt format. On standard Wikipedia leads for cities (Lyon, Turin, Osaka, Tartu, Kraków), the model properly outputs :X a :City; :country :. and nothing else. However, when the country mention is removed from the abstract of Lyon, the model outputs :Lyon a :City; :country :Lyon. (the city as its own country). When given out-of-class entities, it still forces a City graph: :Johnny_Marr a :City; :country :England. and :Mount_Everest a :City; :country :Earth. While these few manual probes do not establish an error rate, they illustrate a structural property of the training setup: because every training example contains at least one valid triple for the target shape, the model cannot abstain when a property is unsupported by the text or when the entity belongs to another class.
* AlignScore validation details. In Section 5.2.4 (Table 7), AlignScore is validated against manually split supported (FP+) and unsupported (FP-) false-positive sets on dbo:Person and then used across all 16 classes. Please report the number of annotated false positives in Table 7, the annotation guidelines, inter-annotator agreement, and the proportion of false positives falling below a chosen entailment threshold (since an average score around 0.50 can mask a bimodal split between 0 and 1).
Please revise the abstract, Section 8.2, Table 19, and Section 10 to state precisely that the models avoid subject and out-of-schema relation hallucinations, while discussing object-level hallucinations and the inability to abstain.
### M3. Mismatch between the formal definition and the per-shape architecture (Section 1, Section 4.4, RQ3a)
Section 1 (p. 2) formally defines the extractor as a function conditioned on both the input text and the SHACL shape, mapping a (text, shape) pair to a graph. In practice, the shape is never passed to the model; instead, the authors train 16 separate BART models, one per class, that take only the entity URI and text as input.
Training dedicated models per class ("extractor factory", Section 9) is a perfectly reasonable design choice, but the formalisation in Section 1 and the phrasing of RQ3a ("generalise across a large collection of SHACL shapes") suggest a shape-conditioned model. Have the authors tested passing a linearised representation of the SHACL shape in the prompt to a single unified BART model trained across all 16 classes? Even a compact ablation comparing a unified shape-conditioned model against the 16 specialist models would directly answer whether the SLM is taking the shape definition in consideration or simply memorising a class-specific output template.
### M4. Methodological issues and train-test leakage in the long-tail study (Section 6.4.1, Section 7, Zenodo 15917325)
Section 7 argues that guaranteeing 1,000 examples per property (D4SE_all, Algorithm 4) is the key to mitigating the long tail. Below I list internal contradictions and experimental concerns in this section:
* Inconsistency with how the 16 KastorKG models are trained. In Section 6.4.1 (Eq. 18), the 16 class models are trained on a random sample of 1,000 entities in total per shape. Looking at the per-property counts across the entire 141,096-entity KastorKG dataset in Appendix C (Tables 22-39), most properties do not reach 1,000 instances even in the full distilled KB, let alone in a 1,000-entity subsample. For example, in dbo:City (Table 27, 4,034 entities total), country has 3,983 instances, while postalCode, utcOffset, and areaCode are underrepresented with only 42, 29, and 18 instances across the entire dataset; this explains why KastorCity only ever predicted country in my local tests. Thus, the 16 flagship models in Section 6 violate the training principle advocated in Section 7, and the sufficient-exposure strategy cannot actually be applied to most of KastorKG using natural Wikipedia data. This contradiction needs to be addressed directly in Sections 6 and 8.
* Single-class scope and fixed exposure threshold. The long-tail experiments are conducted solely on dbo:Person (10 properties), and the exposure threshold is fixed at 1,000 based on Allen-Zhu & Li (2025) - a paper studying factual memorisation during LLM pre-training, which is a different mechanism from seq2seq relation extraction fine-tuning. Sweeping the threshold across smaller values (e.g., 100, 250, 500, 1,000, 2,000) and testing on at least one or two additional shapes from Section 6 would be needed to support general claims about the 1,000-example rule.
* Performance trade-offs and missing size control. In Table 18, MD4SE_all (trained on 3,472 entities) is compared against MD10 (12,000 entities). There is no random baseline of 3,472 entities. There are internal inconsistencies in the reported F1 gaps: Section 7.5 (p. 26) cites a 10-point gain on d_C, while Section 8.1 (p. 28) and Section 10 (p. 32) cite ~7 points; Table 18 confirms 7.4 micro / 11.7 macro. The severe drop on frequent properties (d_+) and unweighted test sets (d_N, d_R) should be discussed candidly as a precision/recall trade-off rather than an across-the-board win.
* Construction of d_C from model errors. Section 7.5 describes d_C in a single parenthetical remark: "(manually annotated 1,470 FP and FN to remove KG noise)". Inspecting the Person archive on Zenodo (15917325), d_C uses the exact same 200 abstracts as d_R, modifying the reference graphs (639 triples kept, 405 added, 132 removed). The file CORRECTED_Random.csv contains 1,355 rows (whereas the union of the seven X*.csv model error files has 1,475 unique errors). Because d_C was built by pooling and judging the errors made by the evaluated models on d_R, any valid triple present in the text that none of the models extracted remains unannotated, and models that over-generate plausible triples benefit more from false-positive flips. In the archive, X10All1000.csv (which appears to correspond to MD4SE_all) has 246 unique novel false positives (FP_new) marked verif = TRUE, compared to 94-149 for X1, X10, X10All, X10synth, and X10synth2. Please document the exact protocol for building d_C: who annotated the triples, how many annotators were involved, whether they were blind to which model produced each error, and why CORRECTED_Random.csv has 1,355 rows instead of 1,470.
* Train-test entity overlap in the Zenodo archive. Section 7.2 states that the four 200-example test sets (d_N, d_+, d_-, d_R) are independent, and Section 7.3 states that results confirm "the absence of any memorisation effect". However, cross-checking entity URIs and abstracts between the training and test directories in Zenodo 15917325 reveals substantial overlap:
* The d_+ test file contains 1,200 entities (not 200 as stated in Section 7.2), and it is identical to the full D1 dataset (all_test).
* The 12,000-entity training sets (X10 and X4, corresponding to D10 and its stratified counterpart) contain around 1,000 of the 1,200 d_+ entities and 68 of the 200 d_- entities (34% overlap).
* The training directory X10ALLext (which has 3,472 entities, matching the size of D4SE_all in the text) contains 25 of the 200 d_R test entities (12.5%, with identical abstracts, which are also the entities in d_C), 21 of the 200 d_N entities, 25 of the 200 d_- entities, and 156 of the d_+ entities.
* Looking at the error file X10All1000.csv, those 25 overlapping d_R entities average 1.00 unique error per entity, compared to 3.47 errors per entity on the 175 non-overlapping entities (while every other model makes more errors on those 25 entities than on the remaining 175).
This leakage inflates MD10 on d_+ and d_-, and inflates MD4SE_all on d_R, d_C, and d_N. All cross-evaluation numbers in Tables 17 and 18 must be recomputed after removing overlapping test entities from the training sets (or by scoring strictly on the disjoint test subsets).
* Mixed cross-validation folds. In Table 17, the standard Cross-Entropy models use 10-fold CV (90% train per fold), while the STRAT models use 5-fold CV (80% train per fold). This confounds the stratification comparison.
### M5. Contradictions and missing statistical tests in the cross-domain transfer analysis (Section 6.4.4, Tables 13-14)
* Contradiction between text and Table 13. On page 19, the text states that the classes where Kastor leads - listing Airport, Artist, City, Food, MusicalWork, University, and WrittenWork - "all exhibit higher pattern overlap" with a pattern Jaccard similarity of at least 0.16 (Eq. 19). In Table 13, however, Airport has a pattern Jaccard similarity of 0.10 and Artist has 0.08. In fact, Artist (0.08) falls below the 0.09 threshold used in the very next sentence to explain why Kastor loses on Athlete (0.09), Company (0.07), and Politician (0.05). At the same time, CelestialBody (0.33) and Building (0.31) have some of the highest pattern overlaps in Table 13.
* Swapped global scores in the text. Sections 6.4.4 (p. 19) and 8.1 (p. 27) state that global micro-F1 is "0.44 vs. 0.30 for Alpaca and 0.25 for Vicuna", whereas Table 14 lists 0.25 for Alpaca and 0.30 for Vicuna.
* Lack of correlation metrics. Claiming that example-specific pattern Jaccard similarity (Eq. 19) and Sentence-BERT cosine similarity (Eq. 20) are stronger predictors of transferability than property-set Jaccard similarity requires a quantitative test across the 16 classes. Please report Spearman rank and Pearson correlations (with p-values) between WebNLG micro-F1 (or the relative micro-F1 drop from KastorKG to WebNLG) and each metric in Table 13 (property-set Jaccard, pattern-set Jaccard, Sentence-BERT similarity, text length ratio, and pattern count).
### M6. Unchecked entity overlap between KastorKG and Text2KGBench (Section 6.3-6.4, Table 19)
Table 19 labels data leakage for KastorKG as "Fully traceable". However, both KastorKG and Text2KGBench-DBpedia draw from DBpedia entities (and WebNLG triples were originally sampled from DBpedia). Because the 16 models must generate exact DBpedia object URIs from plain text, it is important to know how many of the Text2KGBench test subjects and target object URIs appear in the 1,000-example training sets (Eq. 18) or in the broader KastorKG dump. Please report the entity and triple overlap between the KastorKG training sets and the Text2KGBench test sets, along with performance on the strictly disjoint subset.
### M7. Distillation rules, Wikicheck bias, and metric definitions (Sec. 3, 4.1, 6.4.3)
* Inference-rule leakage: Eq. 7's object-property rule adds (s, p, country) alongside (s, p, place) during distillation (p. 2, 7), so the models learn to emit two triples per place mention. Reference graphs apply no such closure. How many Text2KGBench false positives stem from this extra country triple?
* Wikicheck hyperlink bias: Keeping a triple only when its URI is wikilinked in the lead paragraph drops plain-text mentions and demonym-linked entities. How much of DBpedia does this discard, and how does it skew the distilled KB?
* PEC (Eq. 5) is ill-defined: As written, extending minus total deviating graphs is ≤ 0, contradicting the [0, 1] range — the numerator should just be the count of extending graphs. Also, in Tables 11–12, MD_City has zero deviating graphs, so PEC, AlignScore, URI rate, and class similarity are undefined (undefined denominators) and should be marked n/a, not 0.00.
## 5. Minor Issues, Reproducibility, and Text-Table Inconsistencies
* Exclusion of dbo:Monument (Section 6.2 vs. Tables 9, 21, 33). Section 6.2 (p. 13) states that Astronaut, ComicsCharacter, and Monument were excluded due to insufficient DBpedia instances for fine-tuning. While Astronaut (0 distilled entities) and ComicsCharacter (19 distilled entities) fall well short of the 1,000-instance requirement, Monument has 1,707 entities with 100% Markdown abstracts in Table 9, and 1,173 wiki-validated entities in Tables 21 and 33 - more than enough for the 1,000-sample training regime. Why was it excluded?
* Conflicting entity and relation totals (Section 6.3 vs. Tables 20-21). Section 6.3 (p. 15) states that KastorKG vs. Text2KGBench has "116,097 vs. 721" entities and "152 vs. 663" unique relations. However, the Total row of Table 21 (p. 42) gives 141,096 vs. 510 entities, and its property columns sum to 153 (109 OP + 44 DP) vs. 683. Meanwhile, Table 20 (p. 40) lists 101 vs. 356 properties. Please reconcile these numbers and explain whether properties shared across shapes (e.g., dbo:country, dbo:birthDate) are counted once globally or summed across classes. Also, p. 15 refers to "20 entity categories (Table 21)", whereas Table 21 has 19 classes.
* Swapped OP/DP columns in Table 21 (p. 42). Under the Text2KGBench-DBpedia header in Table 21, all properties are placed in the OP column (with DP = 0) for 17 of the 19 classes, but for Athlete (0 OP, 37 DP) and Monument (0 OP, 26 DP) the numbers are placed in the DP column instead. Also, Food lists 24 properties in Table 21 vs. 18 in Table 9.
* Inconsistent class count on Zenodo. The Zenodo record 18862240 states that KastorKG covers "18 DBpedia classes", whereas the paper states 19 shapes and 16 populated classes (or 18 if counting ComicsCharacter and Monument in Table 21).
* BART release date and temporal split cutoff (Section 4.2 vs. Table 10). Page 7 states that D1 is restricted to articles published before 2021, "the BART release date". BART (Lewis et al., 2019/2020) was released in October 2019 (and is missing from the bibliography). Furthermore, Table 10 (p. 15) defines the temporal split graph as wikinew_202004/{shape} (April 2020). Please clarify whether the temporal cutoff is April 2020 or January 2021.
* Definition of micro- and macro-F1 (Section 3, p. 5). Defining micro-F1 as the unweighted average of F1 across graphs deviates from the standard IR/NLP definition of micro-F1 (which pools TP, FP, and FN across the entire corpus before computing F1). Please either rename the metric to "graph-averaged F1" (or "sample-averaged F1") or clarify whether standard pooled micro-F1 was used.
* REBEL evaluation details (Table 4 vs. Zenodo 15917325). Recomputing scores from the RebelResults JSON files in the Zenodo archive reproduces the pooled numbers in Table 4 (P = 82.8, R = 54.3, F1 = 65.6; macro-F1 = 42.2). However, two points should be clarified:
* The 10 per-fold files store cumulative counts across folds; differencing them yields a per-fold standard deviation of roughly +/- 2 points for P, R, and micro-F1, rather than +/- 1 in Table 4.
* REBEL's macro-F1 (41%) is averaged over all 10 dbo:Person properties, including rdfs:label, dbo:alias, and dbo:birthName, for which no alignment was defined in Table 2 (so REBEL automatically scores 0 on 3 of the 10 properties). Conversely, the JSON files show that birthYear and deathYear were derived from REBEL's predicted dates (scoring 97% and 93%), though this rule is not mentioned in Table 2. Reporting REBEL's macro-F1 restricted to the aligned properties alongside the 10-property figure would be fairer.
* Equations 21-22 (Section 7.1.3, p. 23). In the Weighted Focal Loss (Eq. 22), the stratum weight is defined at the sequence level (determined by the single rarest property in the graph, Algorithm 1), while the focal term uses a token-level probability. Please write out the summation over sequence tokens explicitly and explain why a sequence with several frequent properties and one rare property receives the full rare-property weight across all its tokens.
* Hugging Face checkpoint packaging and hyperparameters (Datartisan/KastorCity). The model repositories on Hugging Face currently host raw PyTorch Lightning checkpoints (last.ckpt, 1.67 GB including optimizer states) without a config.json or tokenizer.json, meaning AutoModelForSeq2SeqLM.from_pretrained() fails out of the box (loading requires unpacking the pickle state dict and manually resizing a facebook/bart-base embedding matrix to 50,267). Converting the checkpoints to standard Hugging Face safetensors format with config files and a brief model card would greatly improve usability. In addition, inspecting the hyperparameters saved inside last.ckpt shows warmup_steps = 100, max_steps = 400, patience = 3, and fold 0 (epoch=13), which differs from the 1,000 warmup steps and patience 5 stated in Section 4.4 (p. 8). Please clarify which hyperparameter set was used for Section 6, and confirm that the Text2KGBench evaluation in Table 14 averages across all 10 fold checkpoints per class while Hugging Face hosts fold 0.
* Broken GitHub links and licence mismatch. In the Data Availability section (p. 32), p. 6 (footnote 6), and p. 13 (Section 6.1), the URLs pointing to [https://github.com/datalogism/KastorKG](https://github.com/datalogism/KastorKG) returned HTTP 404 when checked (the shapes actually live under [https://github.com/datalogism/Kastor/tree/main/shapes/txt2kg](https://github.com/datalogism/Kastor/tree/main/shapes/txt2kg)). Also, in datalogism/Kastor, the LICENSE.txt file and badge specify Apache-2.0, while the text at the bottom of the README says MIT.
* Missing and broken references:
* Page 8 (Section 4.4) has a weird question mark in the LaTeX citation: [?Ringwald et al., 2024].
* Several models and standards discussed in the text are never cited in the bibliography: BART (Lewis et al., 2020), Sentence-BERT (Reimers & Gurevych, 2019), XLM-RoBERTa (Conneau et al., 2020), CasRel (Wei et al., 2020), TPLinker (Wang et al., 2020), SPN4RE (Sui et al., 2020), DBpedia, the W3C SHACL recommendation, and the Alpaca/Vicuna models.
* Zhang et al., 2025 is listed without a venue; it appeared in Findings of EMNLP 2025 (DOI: 10.18653/v1/2025.findings-emnlp.671). Arzt et al., 2025 and Sharma & Mehta, 2025 are also missing venues/identifiers.
* In Section 9 (p. 30), when discussing BART's 1,024-token limit for full Wikipedia articles, it would be natural to briefly mention long-context encoder-decoder architectures (such as LED / Longformer-Encoder-Decoder or LongT5) alongside the RAG and chunking proposals.
* Formatting: on p. 38, the heading "B Statistics on Text2KGBench and KastorKG" sits alone on an otherwise blank page before the landscape Table 21.
## 6. Related Work Positioning
Section 2 covers REBEL, GenIE, SPIRES, SynthIE, and Text2KGBench well, but it overlooks recent work on ontology-constrained extraction and fine-tuned models on the same benchmark that appeared prior to this submission:
* Ontology-constrained extraction and verification: Chepurova et al. (TextGraphs-17, 2024, pp. 61-77) and the follow-up Wikontic pipeline (Chepurova et al., arXiv:2512.00590, Nov 2025) perform ontology-aware triple extraction with type and relation filtering against knowledge base constraints. Similarly, Lo et al. (arXiv:2312.00353) formalise "ontology hallucination" against DBpedia class and range constraints, which connects directly to the URI-validity and Wu-Palmer similarity metrics in Section 5.
* Fine-tuned and prompted baselines on Text2KGBench: Beyond the original 2023 Alpaca/Vicuna baselines, subsequent papers have evaluated fine-tuned seq2seq models and structured prompting on Text2KGBench - for example, Meng et al. (Domain Ontology-Driven Knowledge Graph Generation from Text, ACM Trans. Probab. Mach. Learn., Dec 2025, using fine-tuned T5-large) and Nie et al. (VLDB 2024 LLM+KG Workshop, using ChatGLM-6B). Discussing these works (and, ideally, including a fine-tuned baseline under a shared evaluation protocol as requested in M1) would situate the 16 Kastor models much more accurately. Other adjacent works on LLM ontology population include Norouzi et al. (arXiv:2411.01612) and ODKE+ (arXiv:2509.04696).
## 7. Summary of Requested Revisions
To make the paper ready for publication in the Semantic Web Journal, I recommend the following revisions:
* Clarify the novel contributions: Add a clear statement or small table specifying which datasets, experiments, and tables are new relative to ESWC 2025, KCAP 2025, and the companion SWJ submission (Ringwald et al., 2025b).
* Eliminate train-test leakage in Section 7 and release Kastor predictions (M4.4, M4.5): Recompute all cross-evaluation results in Tables 17 and 18 on strictly entity-disjoint test sets (removing overlapping training instances or filtering the test sets). Document the annotation protocol for d_C and E1-E8, explain the 1,355 vs. 1,470 row count in CORRECTED_Random.csv, and include the per-model Kastor prediction/score files in the Zenodo archive alongside a mapping from folder names (X10ALLext, etc.) to paper model names.
* Recalibrate the Text2KGBench comparison (M1, M6): Explicitly state the input type (PLAIN vs. MD) and URI matching protocol used on Text2KGBench; provide the recomputed LLM baseline scores on the restricted subset; report the entity/URI overlap between KastorKG training sets and Text2KGBench test sets; and include at least one fine-tuned baseline (and, if feasible, a unified shape-conditioned model where the shape is passed in the input, M3).
* Tone down "hallucination-free" and "SHACL-constrained generation" claims (M2): Revise the abstract, Section 8.2, Table 19, and conclusion to distinguish between subject/relation constraints and object-level hallucinations (E8), and discuss the lack of abstention on unsupported properties or out-of-class entities.
* Reconcile the long-tail guidelines with Section 6 (M4.1-M4.3): Discuss the fact that the 16 KastorKG models are trained on 1,000 entities total (leaving most rare properties in Appendix C well below the 1,000-example threshold), add a size-matched random control (3,472 entities) on dbo:Person, and sweep the exposure threshold across smaller values.
* Add correlation statistics for cross-domain transfer (M5): Fix the discrepancies between the text and Tables 13-14 (Airport, Artist, and the swapped global Alpaca/Vicuna numbers) and report Spearman/Pearson correlations between the domain-shift metrics and WebNLG performance across the 16 classes.
* Fix resource links, checkpoint packaging, and text-table inconsistencies (Minor 1-11): Fix the 404 KastorKG GitHub URLs, upload standard safetensors/config.json files and model cards to Hugging Face, clarify Eq. 5 (PEC) and n/a values in Table 12, explain the exclusion of Monument, and reconcile the entity/property counts across Section 6.3 and Tables 9, 20, and 21.
|
