SHAPEness: a SHACL-driven RDF Graph Editor

Tracking #: 2698-3912

Authors: 
Rossana Paciello
Daniele Bailo
Luca Trani
Valerio Vinciarelli
Manuela Sbarra
Lorenzo Fenoglio
Sara Capotosti

Responsible editor: 
Guest Editors KG Validation and Quality

Submission type: 
Tool/System Report
Abstract: 
The Shapes Constraint Language (SHACL) has been recently introduced as a W3C recommendation to define constraints (called shapes) for validating RDF graphs. In this paper a novel SHACL-driven multi-view editor is presented: SHAPEness. It empowers users by offering them a rich interface for assessing and improving the quality of RDF graphs. SHAPEness has been developed and tested in the framework of the European Plate Observing System (EPOS). In this context, the SHAPEness features have proven to be a valuable solution to easily create and maintain valid graphs ac-cording to the EPOS data model. The SHACL-driven approach underpinning SHAPEness, makes this tool suitable for a broad range of domains, or use cases, which structure their knowledge by means of SHACL constraints.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Reject

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Martin Necasky submitted on 10/Apr/2021
Suggestion:
Major Revision
Review Comment:

The paper presents a tool for asessing and improving the quality of RDF graphs called SHAPEness. It provides a rich user interface which enables users to create RDF graphs, visualize them as a network and validate them. The user interface is generated automatically on the base of provided SHACL shapes. The authors show the motivation for the tool, principles it is built on, its architecture and a user study on a use case of European EPOS-ERIC.

In my opinion, the paper has many serious drawbacks which need to be addressed before having a chance to be accepted. They are listed below in two groups. The first group (1) refers to the quality, importance, and impact of SHAPEness. The second group (2) refers the clarity, illustration, and readability of the paper. In (3) few typos are listed. In the end I summarize my comments.

(1) Quality, importance, and impact of the described tool or system (convincing evidence must be provided).

Section 2
=========

The argumentation for the desktop application is weak. It needs to be explained better.

SHAPEness is a desktop application and the authors argue in Section 2 that web applications need to be installed and configured. This is obivously not true. If a tool is distributed as a web application it usually does not require technical skills of users. On the contrary, distributing a software as a web application is usually the most effective if we speak about required technical skills. I could argue that many users today are not able to install a desktop application - they even do not know what it is.

Another argument for the desktop in Section 2 is the internet connection. This argument is failing in the world of today where connection is available everywhere and with low costs, especially everywhere where people need to edit RDF data. There could be special cases, e.g. a scientist somewhere in a jungle, but this would need a better argumentation. The current argumentation is weak. Moreover, even browser based applications can work even when the internet connection is not currently available.

Section 4
=========

It is not clear whether SHAPEness requires the whole model of the data edited and validate through SHAPEness needs to be expressed as SHACL shapes or only some constraints and the rest is expressed with RDF Schema or in OWL. The example with EPOS-DCAT-AP described later in Section 5 indicates that everything needs to be expressed as SHACL shapes and RDF schema expressions are not used by SHAPEness. If this is not true, it should be explained better in the paper. If this is true then I seriously doubt about such design decision. An alternative is to use the RDF schema / OWL, which exists and is easier to create in my opinion, also as an input for a tool which generates forms to create RDF statements. The shapes would then be only complementary. The authors may not agree with me and I can be wrong. However, such design is also possible and the authors should consider possible designs and argue for their decision.

It is also not clear what happens when the underlying model changes. What happens when new shapes are added? When an existing shape is modified or removed? What if a new shape affects other shapes? This is important in particular with files already created in the application and then loaded back after the model has changed.

Section 5
=========

Section 5 presents benefits and added value of SHAPEness. However, it would be possible to create also a classical application fixed to a concrete model (e.g., the model of EPOS-DCAT-AP). From the usability point of view, it would bring the same benefits to the end users. From the user's point of view, it is not important whether the application is programmed in the classical way or generated from the model (SHACL shapes). I do not argue for the classical way of hardcoding the model into an application. I want to support the low code / no code approach based on model-driven and data-centric approaches. However, the authors should do the same and argue for it in their paper. The audience sceptical to generating UI from the model will ask how this particular work shows that an application generated from a model (RDF schema/OWL/SHACL shapes) is better for users from the point of usability. Usually, tools generated from a model are criticizes for their generic UI with low usability. The authors do not show in any way how their approach helps with usability while they present this as one of their messages.

Section 6
=========

This section presents the results of a user study so it is crucial for demonstrating the quality and impact of the described tool. However, the user study is done only with SHAPEness. It was not compared to other tools mentioned in the related work. 1 or 2 related tools should be included in the user testing.

The user study is based on author's own methodology. However, the goal is to test usability and we have standardized methodologies for usability testing, e.g. System Usability Scale [James R. Lewis (2018) The System Usability Scale: Past, Present, and Future, International Journal of Human–Computer Interaction, 34:7, 577-590, DOI: 10.1080/10447318.2018.1455307]. The authors should explain why they used their own methodology or do also the System Usability Scale testing.

Moreover, it is definitely important to test usability but from the performed tests we do not get answers to the following questions which are important in my opinion:

1) How the usability of an application generated from the model differs from the usability of an application which is hard coded in the classical way?
2) If there is some loss of usability, do we get something else which is also important? For example, the authors talk also about extendability. Is this an attribute which is better in this case?

The paper should also answer these questions.

(2) Clarity, illustration, and readability of the describing paper, which shall convey to the reader both the capabilities and the limitations of the tool.

Section 3
=========

What do the terms "parent node" and "child node" mean in the context of RDF which is non-hierarchical? Is it in the context of a concrete SHACL constraint which can be considered hierarchical? This should be clarified.

Please clarify what does the term "properties compilation" mean and why it is important for users and why they need assistance.

In subsection 3.1, the structure of the presented user interface does not correspond to the structure described at the beginning of Section 2. Form based and tree based visualization is not visible on the screenshot. How these two different descriptions of the user interface map on each other?

The features in the 2nd paragraph of subsection 3.1 are very common and normal features of any tool (either web based one) so it is not so important for me to describe them in a paper.

Please clarify what does the term "close representation" mean. "Close" does not mean "the same". Is it some kind of simplification of the original RDF graph? The following specification shows that the graph view shows the original RDF graph as it is. So these two parts of the paragraph are not consistent.

Please clarify what do you mean by a node type, esp. in the context of 4th paragraph in 3.1. Is it an RDF class instantiated by the node or is it its SHACL shape? At first, I thought you mean RDF class but in the following paragraph you describe the palette view which shows SHACL shapes and I can see here the types you have in the graph view as types for filtering. And iIn the same paragraph, "frequently used" means used by the current user? What do you mean by frequently?

In subsection 3.3 the 2nd paragraph is redundant, it was already said earlier.

In subsection 3.3 you talk about customization. What exactly can be customized? A user may customize concrete visual properties?

4th paragraph in 3.3 also describes common and trivial features which I would not describe in a paper.

In subsection 3.4 you describe how SHAPEness reads SHACL shapes. SHACL is a rich language which can express complex RDF shapes. However, from all descriptions of the Properties View in the paper, it seems that only shapes which somehow restrict only primitive property types, allowed values and cardinalities are supported by SHAPEness. The paper lacks a more precise specification of what classes of SHACL shapes are supported. It is also not clear how exactly a SHACL shape is interpreted to build the user interface. It is not clear how parts of a SHACL constraint are mapped to the UI controls.

In subsection 3.5 you describe the notification mechanism for violations of SHACL constraints. Is it based on features of a SHACL validator which you just reused or you needed to implement your own validator because existing validators do not support this?

In 3.6 you describe that a PNG image of the graph view can be exported. A vector output is not supported? Please clarify.

Section 4
=========

At the beginning of Section 4 you provide the list of requirements. These requirements were defined by the author of the tool or they are the result of some survey among potential users? If they are your own, than I cannot agree with 3 and 4. If the application would be a web application you would not need 4. This needs to be clarified.

The requirements need some supporting evidence. While it is clear that it should have user friendly interface, it is not clear why offline usability or multi-platformness is necessary (it could be a web browser based application).

The requirement 5 is too vague. Does it mean adding new views? New shape constraints languages? New versions of SHACL? Adding SHACL constructs which have not been covered yet? This requirement can be many different things.

After the requirements there is a paragraph which tries to explain that the requirements are fulfiled by SHAPEness. However, it is not convincing. In particular, the descrition does not show that reqs 2, 4 and 5 are covered. Eclipse Rich Client Platform on its own cannot guarantee user friendly inteface. OSGi framework does not mean automatically that your application is easily extensible. Even if I have a perfect framework I can still create an ugly and hard to modify application. The platform means that it is locked in Eclipse which means a single platform which is in a contradiction with 4. By the way, the first sentence of 4.1 this sentence could be read as an argument against using RCP for building an application. As the sentence states, RPS provides a minimum set of plugins to build a rich desktop application. Why you used it when it does not support you?

Subsection 4.2 presents very technical implementation details of how a SHACL shape is transformed to a JAVA bean. However, I am missing something more important - how a shape is reflected in the user interface and its controls? How possible values a user can fill in for a new RDF resource? How object triples connecting different RDF nodes are created by users using which UI controls? These important questions are not answered by the paper in the necessary detail.

Section 5
=========

The diagram of EPOS-DCAT-AP should be in the paper.

(3) Typos

- (section 6) software responds to user expectation -> corresponds?
- Table 1, T5, ttl -> ttl file (to make it consistent with T6)

To wrap up, the paper presents an interesting tool which can be an interesting contribution to the field of low code approaches based on semantic domain models. This field is definitely worth of studying and viable and usable software tools are necessary to get academic results into practice. The authors made a significant step in this field. However, the paper has some serious drawbacks which need to be addressed before the paper can be considered for acceptation. Some of them are related to core design decisions of the authors. In may opinion, some of them are wrong and some of them just need (much) better explanation in the paper. Many of my comments are related too vague and unclear descriptions of various aspects of the tool. However, in my opinion the authors should get a chance to improve the paper as the tool itself seems interesting. I confirm that I tried the distribution of SHAPEness provided from the Github repository linked in the end of the paper. The distribution works and I am able to go through the tasks of the user study described in the paper.

Review #2
Anonymous submitted on 10/Apr/2021
Suggestion:
Minor Revision
Review Comment:

Summary
The paper describes a desktop tool called SHAPEness, which provides users with an environment to edit and visualize RDF data and to validate RDF data with SHACL shapes. The design of SHAPEness was driven by requirements of an Earth sciences research project: the European Plate Observing System (EPOS). The SHAPEness tool was evaluated with users, who considered it a beneficial tool to support the creation and validation of RDF graphs within the EPOS ingestion pipeline.

General comments
The paper is on topic for the special issue, it is generally well written (though some clarifications are needed; see detailed comments), and I believe that the SHAPEness tool described here would be of interest to some of the audience of the journal.

It is not clear how the SHAPEness tool would handle large RDF graphs, both from the architectural point of the view (there were no performance tests), and the effect of the scale of the graphs on the users' understanding of the graphs' structure and semantics, which SHAPEness purports to improve.

The evaluation of the tool is highly specific, since it includes only experts with knowledge of EPOS resources. This does seem to have been by design. So my concern is that it isn't clear how the tool would be of general interest to users outside of the EPOS community. It would be good to add a discussion of the envisioned impact of the tool on general users. Also missing is a discussion of how users with no SHACL experience can benefit from the tool.

The tool's code isn't publicly available for viewing; at least no source code repository is provided in the paper. The tool can be downloaded through a cloud storage service (mega.nz). I would expect that there will be a persistent reference to the tool before the paper is accepted for publication.

Detailed comments

Section 1
- "the content of RDF data graphs (called nodes)" (pg2) - The content of RDF graphs isn't just nodes.

- "graph properties can be edited and visualized using simple forms" (pg2) - Unclear what is considered an RDF graph property. Give example(s) of such properties.

Section 2
- "data are stored in a SPARQL database" (pg2) - These databases are more commonly known as (RDF) triple stores.
- "ActiveRaUL does not provide users with a serialization of RDF data" (pg3) - Unclear whether the tool does not allow exporting data in general, or data in RDF format specifically. If the latter, then what format(s) does the tool output data in?
- "none of these tools combines the form-based visualization with a graph-based representation of data." (pg3) - It isn't exactly clear what it is that current tools are missing out of this combination. Is it that they allow form-based editing of RDF data, but not form-based visualization of (parts of) the RDF graph(s)?

Section 3
- "a graph-based visualization, for helping users to analyze and better understand graphs' structure and semantic" (pg3) - It's not clear how the graph-based visualization will help to achieve the intended goals. In particular, how would it help to understand the semantics of graphs?
- "The application assists users in properties compilation" (pg4) - The term properties compilation isn't defined; what does it mean?
- "by concealing the schema complexity" - How is the "concealing" of the complexity achieved? (Why does it need concealing?)
- "by helping avoid typos" - Typos when typing what?
- "The Graph View is a close representation of an RDF graph" - How close a representation? What is ommitted?

Section 4
- "The design of the SHAPEness application was conducted considering the following requirements" (pg7) - Scalability isn't in the list; what are the goals and expectations regarding that? What size RDF graphs can the tool handle?
- "advanced Java programming techniques were adopted" (pg8) - What techniques precisely, and to what end(s) were they used?
- Figure 8 (pg10) - Unclear why bytecode is provided here, instead of Java code.

Section 5
- "The motivation for developing SHAPEness was triggered by the concrete requirements of a challenging Research Infrastructure (RI) for solid Earth sciences" (pg10) - What was especially challenging about the infrastructure?

Section 6
- "paved the way for additional tests with a wider user-base" (pg12) - Are these the same kinds of tests as were (partially) done in the first release of the software?
- "to the Google web forms questionnaire" (pg13) - Missing link to the questionnaire defined here.
- Figure 10 (pg14) - Explain the axes in the figure caption.

Section 7
- "non-expert users in order to meet SHACL compliance requirements of RDF data faster and more accurately" - How could the tool help those users who are just starting to use SHACL?
- "Future plans include tackling open issues" - What are these open issues?
- "These activities are planned and soon SHAPEness will be released as open-source" - I would expect that this happens before the paper can be accepted for publication.

Review #3
By Eric Prud'hommeaux submitted on 01/Jun/2021
Suggestion:
Minor Revision
Review Comment:

Per reviewer instructions,
--------------------------
> This manuscript was submitted as 'Tools and Systems Report' and should be reviewed along the following dimensions:
> (1) Quality, importance, and impact of the described tool or system (convincing evidence must be provided).

I think SHAPEness is an early example of schema-directed data interfaces. It seems to be well-written I expect it heralds many more to come.

> (2) Clarity, illustration, and readability of the describing paper, which shall convey to the reader both the capabilities and the limitations of the tool.

The example schema doesn't include node constraints beyond sh:datatype. I added a `pattern` constraint t ContactPointShape's schema:telephone: ```sh:pattern "^tel:" ;```. (I know, that should be an IRI, not a string, but it serves as a test.) The input form accepted a new schema:telephone of "asdf" so I don't believe that SHAPEness currently validates field input per the schema. I don't believe the paper or documentation mentions this limitation.

> Please also assess the data file provided by the authors under “Long-term stable URL for resources”.

There wasn't such a heading in the paper, but the Conclusions include "Software availablity" () which I took to fill that role.

> In particular, assess
> (A) whether the data file is well organized and in particular contains a README file which makes it easy for you to assess the data,

Knowing the github.io URL schema allowed me to find the source repo: . I think this woulld be a better source link for the paper. (The other provides downloadable software and user guide.) The online documentation is pretty scant but the downloaded docs are thorough. There are online javadoc-style docs but they are oriented towards developers.

> (B) whether the provided resources appear to be complete for replication of experiments, and if not, why,

The software is easy to install and run. The docs include links to sample data which demonstrate the features. It would be mildly more convenient to have those wrapped up in an example project, but the example data in the docs were very helpful.

> (C) whether the chosen repository, if it is not GitHub, Figshare or Zenodo, is appropriate for long-term repository discoverability,

The repo is GitHub, though it could be useful to include a release tag or date so folks know what was documented/tested in this paper.

> (4) whether the provided data artifacts are complete.

The allowed me to install and start the software and play with the UI. I did not try to use it in a project.
--------------------------

Comments on the manuscript
--------------------------

1. Introduction

"Resource Description Framework 1 (RDF) is the most popular format for..."
I'm not sure that it's documented as being the *most* popular. Unless you can back that up with a reference, perhaps better to s/the most/a/.

"... semantic (meta)data, structured as dynamic and schemaless graphs. However, the RDF flexibility turns out to be the source of many data quality and knowledge representation issues."
Well stated.

"the content of RDF data graphs (called nodes)"
The content isn't really called "nodes". I'd either remove "(called nodes)" or figure out some artful way to introduce the point that shape contraints are applied to the sets of triples into or out of nodes in the graph.

"and as an RDF structure query mechanism."
Indeed, a point often missed.

"target domain concepts (e.g. environmental science)"
Could add a couple more e.g.s to suggest the breadth

"limited knowledge of the domain, of its main concepts and data structures."
This sentence works, but this part is mildly awkward. Maybe "limited knowledge of the domain's main concepts and data structures."

"innovative visual tool in creating"
Perhaps "for creating"?

"section 2 provides an overview"
The rest of the sections in the sentence are capitalized.

"validating RDF data, against a set of constraints"
spurious ','.

2. Related Tools

"Schimatos serializes the created RDF data to RDF/Turtle format"
propose "data as RDF/Turtle" or "data in the RDF/Turtle format". Also, the "RDF/" is probably not helpful.

"..., but only for storing data in a SPARQL database."
As opposed to?

You may also want to include Inrupt's solid-sdk-forms [1] which maps ShEx to a form language. This is derived from annotations prototyped in shex-form [2] and the ShEx value constraints are re-expressed in that form language.

Not necessarily for the paper (your discretion), but if you click the
(layout in schema) button, you see some annotations (underlined in
teal) that provide the typical stuff you want to add for forms (label,
read-only). The prob with that is that the schema is now littered with
stuff that probably conflicts with other peoples litter.

The (layout by path (ShExC)) button moves those into a separate Layout
pane, allowing you to use a shared copy of the schema. The ShapePath
stuff is prototypical but we're working on it in the ShExCG and it
already has a spec (out of date) and a couple implementations.
Hopefully, we'll be done with it in a couple months. Though this might
interest you as it could make it easier to track changes to shared
schemas.

Re my field validation comment early in this review, the shex-form
example was a quick demo for a the solid-sdk-forms folks so it doesn't
have much field validation but the foaf:name has a minLength of 2
(accepts e.g. "ab") and the vc:telephone must match /^tel:\+?[0-9.-]/
(e.g. "tel:123"). Entering field values other than that will give you
a red border and a tooltip.

3. SHAPEness features

"understand graphs' structure and semantic;"
number agreement; either "understand a graph's structure and semantics;" or "understand graphs' structures and semantics;"

Fig. 1.
I don't know how easy it would be to stuff more resolution into the image, but that could help folks who want to zoom in.

3.1 User Interface

"The SHAPEness user interface consists of five views..."
This repeats the caption of Fig.1, but I'm not sure there's any way around this.

"a toolbar which provides"
Could add "a toolbar (upper right-hand corner) which provides"
I assume there's no screen shot of the filter panel but if you're re-doing the screenshot, you could include a context menu.

"and the edges are represented by arrows in order to indicate the subject to the object of the relationships"
Awkward. Maybe "and the edges are indicated by arrows showing the relationship to some object", adjusted according to whether these work for inverse triple constraints?

3.2. Using a SHACL schema to set up SHAPEness

"SHAPEness creates two subfolders for each project folder. One for ... Another one for ..."
Technically, those last to sentences have no verb. ": , ," would be pretty long; I propose "One *is* for" or maybe s/One/The first/.

3.3. Browsing and Authoring RDF graphs

"via graph-based intuitive interface"
"via *a* graph-based intuitive interface"

"Every element of the graph, be it a node or an edge, when clicked is highlighted with a black border."
propose "Clicking an element of the graph, be it a node or an edge, highlights it with a black border."

3.4. Assisting properties compilation

"as well as dedicated widgets to avoid typos"
Is creation of additional widgets done in Java? Are there provisions for user-defined widgets or would they all be added by PR on the SHAPEness project? RCP may provide this, but it also depends on whether the SHAPEness widget library is itself dynamically extensible.

"A help button functionality is provided to assist users in understanding the meaning of IRI labels."
Could be shorter, e.g. "A help button pops up the RDFS label or defintion of IRIs."

Fig 2. caption
"The SHAPEness user interface helps users in node properties compilation"
"The SHAPEness user interface helps users enter valid node properties" is probably less awkward and more informative.

3.5. Validating RDF graphs

"upper side of the view an error"
Needs a ',': "upper side of the view, an error"

3.6. Exporting RDF graphs

"by inserting the Git repository URL,"
Propose "by *providing*(or supplying) the Git repository URL," to clarify that it's not inserted into the graph.

4. Architecture of SHAPEness

"application was conducted considering the"
Propose: "application driven by the"

4.1. Eclipse Rich Client Platform (RCP)

"pushing the serialized graph (i.e. Turtle file) to a Git repository"
Is that normalized to provide minimal/readable diffs? Jena sorta does that (judging from Protégé output), though it can produce spurious diffs so formal canonicalization would be necessary to eliminate that. This info could also go into 4.2.3.

4.2.1. Loading SHACL content
"The shapes graph contains two types of shapes.
Node shapes which declare constraints on a node and Property shape which declares constraints on the attributes of a node through a path property."
Propose either
"The shapes graph contains two types of shapes: Node shapes, which declare constraints on a node, and Property shape*s*, which declare constraint on the attributes of a node through a path property."
or
"The shapes graph contains two types of shapes.
Node shapes, declare constraints on a node, and Property shape*s*, declare constraint on the attributes of a node through a path property."

4.2.2. Generating model classes

"enhanced by Javassist 26 library"
"enhanced by *the* Javassist 26 library"

"Such library allows for"
Probably "This library allows for"

"implement the Serializable interface"
Do they serialize back to SHACL?

"This allows the Engine to use an instance of a subclass wherever an instance of Node is needed and therefore to manipulate, at runtime, objects differently depending on their class type."
Presumably, this is how shape-specific widgets are invoked.

Fig. 8. The Java bytecode of the class Person generated by means
of Algorithm 1 and Algorithm 2.
If there's a corresponding Java class you could compile and get the same bytecode, showing that instead would be helpful to folks who haven't recently implemented a JVM.

4.2.3. Serializing/deserializing the data model to RDF/Turtle

"into a stream of bytes objects"
Maybe "into a stream of bytes"?

5. Application of SHAPEness in EPOS

"Being EPOS committed to provide the integrated"
Propose "Since EPOS *is* committed to provide the integrated"

"challenges in EPOS a methodology"
add ',': "challenges in EPOS, a methodology"

"In the first phases of the project the population"
add ',': "In the first phases of the project, the population"

"In the next phase the RDF graph"
add ',': "In the next phase, the RDF graph"

"in a consistent consolidated knowledge graph"
add ',': "in a consistent, consolidated knowledge graph"

"metadata is validated it requires"
add ',': "metadata is validated, it requires"

"As illustrated in Figure 9 SHAPEness plays"
add ',': "As illustrated in Figure 9, SHAPEness plays"

"In the next section the approach adopted"
add ',': "In the next section, the approach adopted"

6.1. Testing Objectives

"It was decided to perform a user testing"
remove 'a': "It was decided to perform user testing"

"It was decided to perform a user testing, ..., as the structural or behavioral testing [2], both already performed..."
This sentence is long and rambles a bit. Could it be broken up? (Sorry, I don't have a specific suggestion.)

"raised by what user profiles, allows the developer"
remove ',': "raised by what user profiles allows the developer"

"with very low knowledge of the metadata concept"
Maybe "with minimal experience or awareness of metadata"?

6.2. Testing methodology

"the Google web forms 33 questionnaire."
33 refers to general info about Google forms (), not the specific form used for user testing.

6.3.2. Software Usage

"each of which split in several tasks:
a) Creation of an RDF/Turtle file from scratch, including 6 tasks;
b) Creation of a project from an existing RDF/Turtle file, including 7 tasks."
Could use a forward ref to Table 1. Propose: "each of which split into the six tasks listed in Table 1
a) Creation of an RDF/Turtle file from scratch (includes all 6 tasks);
b) Creation of a project from an existing RDF/Turtle file (includes the 6 tasks plus ???."
I didn't figure out what the 7th task as.

"as in the case of the spatial coverage"
Maybe "as was the case for the spatial coverage"?

Table 1:

"not clear why and ID is needed"
s/and/an/

General

The technical description of the functionality and interface of SHAPEness was detailed and well-written. As a write-up of an implementation, this paper is a strong candidate for publication.

The user study was underpowered, with apparently a total of six respondents. I recognize this as a common result of minimal resources and/or a highly-specialized user base. Expressing the stats as "33%" glosses over this point. Writing them instead as "2/6" would at least confront this paucity of data head-on. Applying this tool to more domains would provide much more confidence by expanding the user base and ensuring that the tool was repurposable.

6.1 abstractly described the user feedback forms, which appeared to be focused on how well this tool was implemented. Without access to the actual form, I'm guessing that the questions didn't include more meta questions like "was the schema validation helpful?" (vs. being redundant against dedicated widgets). In order to evaluate the efficacy of such a tool (vs. the implementation of this tool), it would be helpful to ask green-field functionality questions like "Given the tasks you've just performed, can you imagine any different approach that would have made them easier and less error-prone?", i.e. "How best could the machine help you?". The real proof of this approach comes when no one imagines something more helpful approach, like a DSL or a domain-specific program. With such feedback, the authors could evaluate whether that wish-list could be reallized simply with information available in the schema, or would it require additional configuration.

Sometimes a little verbose about process, e.g. 6 describes the development of the testing protocol, rather than just the final protocol.

1 https://github.com/inrupt/solid-sdk-forms
2 https://ericprud.github.io/shex-form/?manifestURL=examples/manifest.json