AIDOC-AP: An Application Profile for Technical Documentation of AI Systems

This is a timely and interesting description on the problem of the AI-system technical documentation required under Annex IV of the EU AI Act.

i) quality and relevance of the described ontology

The ontology is clearly relevant to the journal audience, and the AP perspective is sensible, as the paper reuses established vocabularies and introduces a targeted profile for compliance-oriented technical documentation. The main reservation I have is that the paper still needs a stronger justification of why this particular methodology was chosen and what is genuinely novel relative to adjacent work. The manuscript says it extends NeOn-GPT with two additional steps, but it does not sufficiently explain why NeOn-GPT was preferred over alternative ontology engineering approaches, nor whether any conversational/interactive aspect of NeOn-GPT was actually important. Relatedly, the claim that this is the first pipeline to combine LLM-assisted alignment and CQ-based coverage estimation should be positioned more carefully, e.g. wrt to works dealing with automated CQ verification and automated terminological evaluation. Furthermore, since the validation is limited, the discussion should lean more heavily on that limitation. In general, the discussion would benefit from a fuller interpretation of the reported metrics before moving to limitations.

ii) illustration, clarity, and readability of the paper

The paper is generally readable and clear along with figures and tables. However, some key methodological details are still underspecified or not reported where the reader needs them. In particular, I do not think the manuscript is yet clear enough about which LLMs were used at which stages, why those models were selected, and how reproducible the outputs really are. The paper reports Gemma 3:27B for alignment and averages coverage over Gemma 3:27B, Llama 3.3:70B, and GPT-OSS:120B; it also says temperature was fixed at 1.0 “to ensure consistency.” That last claim is not convincing as written, especially because the paper itself notes that a lower temperature would reduce stochasticity. This needs a more careful explanation, and the paper should cite model cards or equivalent documentation for all models used throughout the pipeline. A further clarity issue concerns the alignment step. The paper states that only correspondences above 0.75 are retained and that each alignment triple is manually reviewed by experts who can accept, reject, or refine the suggestion, but the manuscript does not report how many mappings were accepted, rejected, or modified, nor how this curation was carried out in practice. Those numbers would make the evaluation much more convincing.

(iii) Assessment of the data/resources under the long-term stable URL
The resource is well organized and the repository is complete. However, there are discrepancies between the paper and the code: the paper describes a 0.75 confidence threshold for semantic alignment and reports Gemma 3:27B / Llama 3.3:70B / GPT-OSS:120B, whereas the current public scripts default to llama3.1:8b (alignment_semantic.py), use CONF_THRESHOLD = 0.5 for semantic alignment, and the coverage script does not explicitly set temperature in the API call. The paper lists the long-term stable resource link as a W3ID URI, and the paper also provides a Zenodo DOI; the public materials are additionally available through GitHub.

Minor comments
1. In the abstract, please expand LLM on first use; in the introduction, use “large language models (LLMs)” consistently.
2. There are small typographic/presentation issues, including wrong quotation marks and some formatting inconsistencies that should be cleaned up.

Thanks for the work (and that this is in a Horizon project so I look forward to more work). Always good to see more semantics used in legal informatics for data/AI etc. My comments are more about grounding the work in what exists and to share lessons learnt from our work so that it helps approaches like yours.

1) AIDOC-AP: the naming is quite similar to the "-AP" style used in SEMIC, I don't know if this was intentional as the paper does not mention DCAT-AP which is used by SEMIC, and which has been extended as MLDCAT-AP for AI models. In particular, MLDCAT-AP is a specification that also models several of the concepts and is something to take a look at for reuse of concepts.
2) The paper mentions use of AIRO and VAIR. Of these, in table 1, AIRO is stated as not having regulatory grounding, which is incorrect. It was in fact the first ontology to model AI Act concepts! We were following the AI Act text from its initial drafts up to the final publication, and you can check the definitions of concepts referencing specific clauses in the AI Act. The table also states that both AIRO and VAIR lack data requirements and dataset concepts, which is also unclear as to how because AIRO does have concepts such as Data and quality characteristics and VAIR has concepts for input/output modelling. Maybe its a partial match. The criteria wasn't clear to me.
3) The paper doesn't include Data Privacy Vocabulary (DPV) https://w3id.org/dpv which IMHO is the most comprehensive vocabulary for EU laws (GDPR, AI Act, etc.) We integrated AIRO and VAIR I think in version 2.1 or 2.2 last year, and now we've released v2.3. Please take a look at the TECH extension https://w3id.org/dpv/2.3/tech/ for a general modelling of technologies, AI extension https://w3id.org/dpv/2.3/ai/ for AI-specific concepts, and most importantly the AI Act extension https://w3id.org/dpv/2.3/legal/eu/aiact/ for AI-act specific concepts. Most of what I see in AIDOC is present in these extensions, and in the interest of interoperability IMO its better to use DPV where available. We have been maintaining this work for 8 years now with updates, and we also cover other laws besides the AI Act so the concepts work across multiple regulations. In the sustainability part of the paper, it is mentioned that you plan to continue the work within the lifetime of the project. I invite you and colleagues to add this work to DPV vocabularies, and where we also discuss creating guides -- and where how to represent technical documentation as a specification would be in scope.
4) In terms of related work, in a previous paper (and in Delaram's PhD) we already looked at creating documentation for AI Act, including the technical document, see https://osf.io/preprints/osf/43rq2_v1 section 4.3. You can also take a look at Delaram's thesis which should have more details/resources.
5) In terms of modelling, I see several concepts as properties, e.g. intendedPurpose is a property with a string as range/value. This is bad practice IMHO, and I also see this is the case in MLDCAT-AP. When modelling things as semantics, we should strive to generate sets/classes/concepts where we can because that's what creates value in the use of semantics. For example, for two intendedPurposes A and B, if both are string, we cannot express any relationship between them, whereas if we express them both as instances of class IntendedPurpose, we can do all sorts of stuff like make links, use skos:broader/narrower, or even create a subclass/SKOS hierarchy. We've done this for GDPR's purpose concepts, see https://w3id.org/dpv/modules/purposes Additionally, the concept Intended Purpose in the AI Act is going to be doing a lot of heavy lifting in determining obligations, including tracking where it has changed. This implies a need to compare two concepts, which is not as rich when using just strings. We have this concepts in DPV's AI Act extension, and also have discussions on the use of it in compatibility assessments https://github.com/w3c/dpv/issues/300 Same applies to other concepts (which are also likely to be present in DPV vocabularies)
6) The paper only focuses on technical documentation as Annex IV, however, the AI Act Article 11 is the primary obligation for the documentation, and you can see specific criteria in para 1. Even if it overlaps with Annex IV, the reference to the documentation should arise from Article 11 in the paper first, and then use the information list in Annex IV. Separately, the Article 11 also notes that Annex IV may be modified in the future, which requires resources to be adaptable -- something to mention/consider in the work as well.
7) I understand the need to have a rigid scope -- in this case only modelling AI Act technical documentation. But at the same time, this same information is necessary elsewhere and one question I always have with hyper-focused ontologies is how will they work with these other compliance/governance approaches. That's also why I suggest using DPV (or something similar) whose concepts are intended to be used across many laws and obligations, so someone else can target another obligation and there will be interoperability between (your) technical documentation and other uses of DPV.
8) For the project, I definitely think you should discuss working to enrich DPV vocabularies. Especially since now its being used in HealthDCAT-AP for EHDS and MLDCAT-AP for AI Act (but AI in general). So all these cases help us make the case stronger for why semantic web and why interoperability (essentially to convince non-semweb usecases to adopt this).

Thanks and happy to answer any questions you may have -- feel free to reach me at harshvardhan.pandit@adaptcentre.ie

In this paper, the authors design the ontological structure of an application profile for technical documentation (based on EU AI Act annex 4) of AI systems. The paper presents also a use case validation to demonstrate the effectiveness of the ontology.
Specifically, the authors have extended NeOn-GPT, an approach recently presented at ESWC. NeOn-GPT is an LLM-base ontology engineering approach, which already develops crucial steps like terminology extraction, axiomatisation and others. In this paper this approach has been further extended with 1) an alignment to reference ontology (using LLMs) and 2) iterative coverage evaluation. The first step (step 6 in the paper) ensures interoperability with existing ontologies within the SW ecosystem. The second (step 7) checks if the current version of the ontology is covering the set requirements (in this case annex 4 of the EU AI Act).

The paper reads well and it can be easily understood by the SW audience. Very hard for other types of audiences.
In general, the approach and the workflow sound appropriate. I just have a few remarks on the technical implementation.
In step 6, the authors have selected correspondences of 0.75 and above. However, this ensures high precision, but it increases the chances of false negatives. Are there cases in which there could have been meaningful matches below 0.75?
On this note, I looked at the prompt, and it is being asked to the LLM to provide a confidence score between 0 and 1. This is generally a not accepted practice as it is well known that LLMs cannot generate confidence scores. Those numbers are just made up. Also, in the prompt you are asking to map two terms with 5 distinct semantic relationships and there is no definition. In this case, you are relying on the inner definition of the relationship embedded in the model weights. Highly risky.
Step 7. You set temperature to 1.0 to ensure consistency. Shouldn’t it be the lowest possible?
In the metric interpretation, I was curious to see whether you performed several runs to checked whether the coverage score would differ. I am pleased to see you performed 3 runs (table 5). However, I am highly sceptical on why the value consistently increases for the three models on successive iterations. Did you perform these calls on a clean interaction, or was it using the previous prompt. In any case, also here my stance is that we should not rely on LLMs own capability to assess confidence.

No GenAI has been used to redact and amend this review.