YASGUI: How do we Access Linked Data?

The paper presents YASGUI, a webclient to create SPARQL queries for different semantic web endpoints. YASGUI has been designed under specific consideration of usability.
The paper focuses on the evaluation of the usefulness of YASGUI. Usefulness on the one hand as general usability. Usefulness on the other hand as the appropriateness of the tool to be used in the context of semantic web specific tasks. As a foundation of the task detection, a taskonomy is derived from earlier work in the domain of tasks and activities relevant for semantic web usage.
A preliminary questionnaire study and a logging based analysis was conducted. The study includes information about YASGUI and additionally information about general semantic web usage. The paper tries to make a connection between YASGUI specific usage patterns and general semantic web usage. However, it was difficult to understand the connection when reading the paper.
The development of usable clients for the semantic web is an important activity. Especially the analysis of the usage of semantic web clients and SPARQL constructs is an interesting perspective.
In this sense, I completely support the road map of the paper and think it contains very useful for the semantic web community.
Nevertheless, I propose a major rework of the paper.
Reason 1: The focus of the paper somewhat shifts within the paper: while initially YASGUI is focused, the later presentation has a bias towards a general analysis of semantic web usage. The separation between these two aspects needs to be made more explicit.
Reason 2: The section which introduces the used taskonomy should be improved (see the following walkthrough for details).
Minor: Some spelling errors need to be corrected.

Walkthrough:
Section2:
Provides a good overview of SPARQL client STA.
Section 3: The YASGUI SPARQL Client
The client is described from a technology and service driven perspective (which technologies are used and how). More details about the original structure of the client, the user interface and the way the features are embedded, accompanied by one architecture section+graphic with technology and service details would be nice.
Remark: The section 3.1 title “Architecture” is more a section about a technology and service selection, not so much about the architecture.
Section 4: A SPARQL Taskonomy
The section derives a taxonomy of tasks probably relevant for SPARQL clients. This taskonomy is the foundation of the following sections.
It remains unclear to which extent the work of [5], [15] and [23] is based on a comparable understanding of the terms activity, task as well as which user group was considered and which method was used to derive the respective taxonomies. However, such an understanding is required to defend the process of integrating [5], [15] and [23].
Another aspect is the derivation of the taskonomy for SPARQL client use. It remains unclear which process of selection was used to derive the final taskonomy used within this paper.
Important: the section states that YASGUI was built without specific functional requirements in mind but more or less based on perceived usefulness of functions during the development process. It would be useful to get an understanding of the background of the developers and the tasks they had in mind while building the system: the system was – even with the developer as source of requirement - built with specific intentions which would already indicate an answer to the question asked: “What tasks is YASGUI deemed more useful to than the other clients?”
Due to the relevance of this taskonomy for the remainder of the paper a rework of this section is required from my point of view.
Section 5: Reported SPARQL Use & Section 6: Actual SPARQL Use
For me the idea behind these sections is the core contribution of this paper as a journal paper: analyzing the actual use of SPARQL clients – considering YASGUI is a good thing but while reading the passage I was even more interested in the general usage of such clients. Nevertheless, the separation between a more general and a more YASGUI oriented analysis was not always clear to me.

Some formatting, language aspects (only for the first pages):
p. 3
- ‘ Query Book’ (wrong space)
- triple-stroe Ideally (missing .)
p.4
- of-fline (sign separation)
-elaborate open source project (missing plural s)
p. 5
- “on the CKAN SPARQL endpoint and Mondeca Endpoint Status endpoint for endpoint URL auto-completion and search.” (accumulation point for the term endpoint)
- The only scenario where YASGUI fails to connect to an endpoint is where (1) a locally installed endpoint is unreachable from the web, (2) [it is] operating on a different port than YASGUI, and (3) CORS-disabled. (the sentence for (2) is incomplete)
p. 7
-“Service composition it is” (no it)
-merged terminology” Semantic enrichment (missing .)
p.8
- table 2 caption should provide more information about the table

The article proposes a new SPARQL user interface called YASGUI in order to facilitate writing/executing SPARQL queries on local/remote endpoints. The authors perform a feature-based analysis of the existing SPAQRL UIs and try to fill the deficiencies of them in the implementation of YASGUI.
The subject discussed in the article is relevant for the special issue and the authors target a demanding and worthy area (Usability of SPARQL interfaces) in the Semantic Web domain.
The strong points of the paper are:
*The paper provides a comprehensive review of the state-of-the-art in the area of SPARQL user interfaces and demonstrates the need for a more usable UI very well.
*The SPARQL taskonomy presented in the paper (although some parts might be overlapping!) can serve as a basis to measure the functionality and usability of SPARQL in general.
*The tool itself is a strong point which is already available online and is progressing well.
The weak points of the paper are:
*Although the paper is technically deep in investigating/analysing existing SPARQL UIs, the technical depth of the parts discussing the YASGUI application itself are not very high. The main innovation of the paper from my point of view is combining Web 2.0 techniques with SW to provide a more usable SPARQL interface but the authors fail to elaborate on this aspect very well.
*Most of the paper is focused on evaluating the functionality of SPARQL in general and not specific to the tool. I think having a broader usability evaluation elaborating more on the target users and employing the existing usability factors in HCI community, would improve the value of the paper in this aspect as well.

In overall the paper is written well but there are some typos and grammatical errors which need to be corrected for the final version.
What I see as a gap in the paper is related work in the domain of HCI. Having a quick look at the references clearly shows that there is an imbalance between usability and functionality discussed in the paper. I recommend authors to take a look at the usability studies in other related domains (e.g. SQL editors in DB community) and advance their work in this direction as well.

The paper presents YASGUI, a Web client to access SPARQL endpoints. This tool has been developed trying to solve the limitations of the current state of the art in SPARQL User Interfaces. The authors also present an analysis of the common tasks performed with SPARQL and relate them to the use of YASGUI.

My first impression after reading the paper was that it was is clearly divided into two parts, which in my opinion are not much related. In the first part, YASGUI is well presented and compared to other SPARQL clients and similar tools. The features implemented in YASGUI are interesting and can be useful for Semantic Web developers who need to perform SPARQL queries or maybe even for lay-users without experience in Semantic Web technologies. The comparison with other tools is broad and complete and it shows that YASGUI is the tool with more features.

However, there is not any formal evaluation to prove that users can perform better with YASGUI than with other clients. In the second part, an analysis of SPARQL queries performed with and without YASGUI is presented. Unfortunately, this analysis does not serve as an evaluation of YASGUI and is not enough. It is not possible to prove whether the functionalities implemented in YASGUI suppose an improvement for end users or even if users understand its User Interface. Moreover, as the authors already mention, the number of queries analyzed is not statistically significant and in some cases it is not possible to compare the usage of YASGUI with other clients.

I recommend the authors to ask themselves which is the main contribution of this paper and then revise it. If YASGUI is the main contribution, which seems to be, then it should be formally evaluated with end users performing real tasks. My suggestion is to make a usability study of YASGUI and compare it to the most popular similar tools, for example the SPARQL client integrated in Virtuoso. I would establish different scenarios, one for each tool, with similar tasks to perform with each tool. The taskonomy proposed can serve as basis to select which tasks to perform. Then, it will be possible to compare YASGUI to other tools using formal usability metrics such as efficiency, effectiveness or users' satisfaction, and check whether these features help users and they can perform more complex tasks.

Minor remarks:
- page 2: another advantages --> advantage
- page 2: without having to execute it --> without having to execute them
- section 2.3: triple-store Ideally --> a "." is missing.
- section 2.3: triple-stores Saving --> a "." is missing.
- section 5.1: Table 3 shows the we questions --> Table 3 shows the questions we

The article is about YASGUI, a web application for accessing linked data. YASGUI supports different features, where the requirements for these features are derived by the authors’ own work with linked data. YASGUI is then compared with other SPARQL clients. A taskonomy describes typical SPARQL task, which might be solved with YASGUI and other tools. Within a questionnaire users were asked for their actual SPARQL use, especially regarding YASGUI (if it is known) and the taskonomy. Also an analysis of the YASGUI server logs compared to the DBpedia logs with respect to the query structure is presented.

Structure:
To stress the bottom-up approach it would be more appropriate to introduce YASGUI first, analyze the tasks and the reported use, and compare YASGUI to the other systems in one of the last steps.

Content:
In the abstract the authors describe the requirement for a strong focus on usability regarding a Linked Data accessing tool. This sentence arouses expectations for a usability study which analyses the efficiency, effectiveness and user satisfaction concerning the proposed interface (cf. [1] for an example, cf. [2] for more general information). With the analysis of logs and the questionnaire, the article does not satisfy that expectation completely.
Besides the argument that the authors have taken interfaces which “range from very basic to elaborate” as comparatives, there are no further arguments why, e.g., TopBraid is missing. Focusing on these 12, which have very similar interfaces leads to the fact that useful functions like “Did you mean?” as proposed in LODatio [3] were not considered (referring to Section 6.3.2 Queries 1200 invalid queries). Also different approaches like graphical query composition or inspired by, e.g., GQL, Zigist, or ViziQuer, were not taken into account, nor substantiated why not, especially regarding usability.

Table 1:
Why did the authors mix up the order of the features and did not (for consistency reasons) show the features under their groups (Syntactic, Applicability and Usability Features)?
At the first sight the meaning for the sign “+/-“ is not clear, because it resembles software description with different versions, e.g., where the commercial version offers features, the freely available does not. Reading the footnote and text helped to get the definition, but wouldn’t it be easier to understand the intention of a table without reading the text? So an additional row(s) instead of using so many footnotes is a suggestion.
Section 2.3. “The only application that does not support the downloading of results is the FLINT SPARQL editor”. In the table also SparQLed does not support downloading.

Interface:
The icons in the upper (menu) bars are squeezed together, which are then hard to understand on the first sign. A visual grouping with spacing would support a user (cf. http://www.uxmatters.com/mt/archives/2006/07/label-placement-in-forms.php).
Also a hover function would support the user, but the hover function is only available for greyed-icons.
The initial explanations of the buttons are only available for IE. Without the quick tips while hovering or the option to get the initial quick tips twice, features like “double-click to rename tab” will get lost.
There are also differences between IE and FF in the error handling, e.g., FF threw a “Method not allowed error”, which intends a user have done an error in formulating the query. IE instead caused a 404 error, which was in this case correct, as the DBpedia servers had been under maintenance (17th September around 11 a.m.).

Questionnaire and Evaluation:
A link to the original questionnaire - especially regarding the combination of different tasks should be provided.
Furthermore, it is more comparative to see the true data/distribution of for example the way of accessing SPARQL endpoints. The same note applies to the re-arranging task. It is not even clear how the task is accomplished in detail or the weights a task get based on its new position; in detail: Was a user able to delete a task from the new list, if the task does not fit in the user’s opinion? Is the weight a task gets equally graduated or are the first three tasks for example higher weighted than the last tasks?
Figure 3: As the authors want to clarify if there are task which are more frequent accomplished with YASGUI than with another tool, an analysis per task, user familiar, unfamiliar to YASGUI and users of YASGUI combined in one, in particular using one scale (maybe better in percentage) is more meaningful. Since the authors did not indicate how many users are un- and familiar with YASGUI, the statistic in its current state is not that transparent. There are 12 respondents, who used YASGUI recently, but then the amounts of every task in (c) do not fit, neither do the amount of task Integrating in (b) fit to the amount of the others (19 respondents, the other task have got 27).

Actual SPARQL Use:
Table 4: Do the values for syntactically valid queries for YASGUI (DBpedia) and DBpedia also contain duplicate queries?
Table 5 and Query Complexity: “The YASGUI DBpedia logs show a larger number of queries with at least 1 join than the DBpedia server logs.” – Do the authors mean the YASGUI logs? Otherwise, there is an error in the table 5. Besides that uncertainty and the missing indication if all three columns integrate duplicate queries, using percentage as representation might confuse the reader.

Summary:
First, it is a good idea and grounded research analyzing and comparing existing SPARQL interfaces the authors’ own implementation included. Defining a set of features based on that analysis and the authors’ needs and consequently extracting a SPARQL taskonomy, enables the authors to distinguish the different purposes someone might want to use a SPARQL interface for. Comparing that taskonomy to opinions of users with a questionnaire and finishing the research with an analysis and discussion about the actual use, based on YASGUI’s logs compared to those of DBpedia, supports the authors in finding inferences and lets them on the other hand conclude that still a lot of work will have to be done. Nevertheless, there were a few unanswered questions and notes to make.
Main Strength: There are implemented promising aspects in YASGUI, which other tools do not support. The discussion and inclusion of related work within the most sections allow a straight comparison to other systems and gives the reader an impression about the actual usage of SPARQL clients and requirements users might have.
Main Weaknesses: The analysis aspects, especially regarding usability and HCI, which are mentioned in the abstract in keywords are partially missing or not accurately. Furthermore inconsistencies in structure and citations can be found. There are also a lot of typos.

[1] E. Kaufmann and A. Bernstein. 2010. Evaluating the usability of natural language query languages and interfaces to semantic web knowledge bases.J. Web Sem., 8(4):377-393, 2010
[2] A. Field and G. Hole. 2003. How to design and report experiments, Sage publications Ltd, London ; Thousand Oaks, Calif
[3] T. Gottron, A. Scherp, B. Krayer, and A. Peters. 2013. LODatio: using a schema-level index to support users infinding relevant sources of linked data. In Proceedings of the seventh international conference on Knowledge capture (K-CAP '13). ACM, New York, NY, USA,