Visual data exploration for understanding and sharing of knowledge in large semantic datasets

Tracking #: 1630-2842

Authors: 
Dmitry Mouromtsev1
Peter Haase
Dmitry Pavlov
Yury Emelyanov
Ariadna Barinova

Responsible editor: 
Guest Editors IE of Semantic Data 2017

Submission type: 
Full Paper
Abstract: 
The problem of understanding knowledge hidden in large datasets relates not only to information modelling and data identification but also to the processes of data exploration and consumption. Indeed, the major part of published data, especially Linked Data, has well-known schemas, metadata, and related descriptions. However, quite often the underlying knowledge of these datasets remain concealed for users. At the same time, it is well known that visual representation is the easiest and the most efficient way to dig into the meaning of data. Hence, the role of visual data exploration has become very important for understanding and re-using of knowledge. In this paper we introduce a visual step-by-step method and tool for dataset exploration through diagrammatic approach. We introduce the term Diagrammatic Question Answering to refer to the process of answering question addressed to a knowledge graph using visual means only. Also we present the exploratory system that uses the Wikidata as a knowledge graph, metaphactory as a knowledge graph platform and Ontodia library as visual tool for data interaction. We evaluate our approach through a user study and assessment of a diagramming process with experiment where two independent groups were involved. The first group worked with developed tool and created a number of diagram representing knowledge of Wikidata. And the second group tried to understand the meaning of these diagrams. The result showed the efficiency of diagrammatic approach for data exploration aimed at knowledge understanding and sharing.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Reject

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Roberto García submitted on 31/May/2017
Suggestion:
Major Revision
Review Comment:

The paper presents a visual tool for data exploration in large semantic datasets. The tool is presented as a generic tool and compared with existing visualization tools based on diagrammatic representations. Then, from an evaluation point of view, it is compared with existing Query Answering tools, most of them based on natural language processing. Then, it is made clear that the tool just supports a quite limited range of queries as seems just possible to start exploration from a single starting node. For instance, how to solve the query “Main influencers of Dutch painters?

Moreover, though presented in the scope of large semantic datasets, it is not clear how incremental visual exploration as presented by the authors scales to thousand or even hundreds of resources. The only way to deal with that is the initial search that narrows the scope to a limited set of resources. However, even when starting from a single node, it is not clear how the tool will deal with instances connected with tens of hundreds of other resources. Though filtering is provided, there does not seem to be overview components. For instance, when browsing instance properties in the tens or hundreds, is it possible to filter them or display the most relevant ones first following some sort of criteria?

On the other hand, is it possible to distinguish the direction of the properties? It is not the same who influenced van Gogh than who was influenced by him and the presented example recovers both from/to van Gogh. Also, in that example, why “student” and “student of” are shown if they have not been selected by the user?

Moreover, authors should clarify how customized templates might be generated beyond the one they mention for Wikidata for countries and persons. Otherwise, the default one is used. Is it possible for a user to customize the shapes and colors in the templated from the graphical representation and then store them as a template for future explorations? Or is this something that has to be coded and thus not available for lay users?

Regarding evaluation, it is too preliminary to drive any conclusions about the usability of the presented tool. First of all, the set of queries supported is very limited and this is not made clear from the beginning of the paper. The authors should clarify better, even from the abstract, the kind of queries the proposed system supports. Moreover, they do not evaluate their system against existing ones, for instance tools from the QALD community or the ones analyzed in the related work. The first impression is that for the kind of queries at hand, existing tools based on natural language queries might be more efficient and effective. The authors should show their system outperforms state of the art.

Future work seems to point in the same direction than natural language-based query answering approaches. It should be clarified better what the added value is in the long term of following and alternative approach based on diagrams.

Finally, the paper requires a profound review of English. Below are just some detailed comments about sentences in the different sections that require rewriting or further details:

Abstract:
“…the process of answering question addressed…” questions addressed
“…diagramming process with experiment…” with an experiment
“…created a number of diagram…” diagrams

Clarify that the diagrams used by the second group of the experiment are the diagrams created by the first group.

1. Introduction:
“…that help keep a focus on…” that help users focus on
“…find data patterns that are of the interest to them” find data patterns they are interested in
“…underlying data models or schema.” schemas
“…interactive and customized diagrams.” customizable
“… OWL ontologies.” missing reference of further details
“Evolving diagrammatic and interactive aspects of user interfaces to the datasets help users navigate easier and share information naturally.” Difficult to follow, maybe you mean something like: “Applying diagrammatic and interactive aspects of user interfaces to datasets helps users explore them and share information more naturally”.

It should be better justified the comparison of the proposed approach with Question Answering. Then, follow this comparison also during evaluation and compare results with QA state of the art systems.
The authors refer to “Knowledge Graphs” in general but the description they provide is just constrained to Semantic Web standards and technologies. They should avoid this terms, mention their scope is “Semantic Data”, “Linked Data”, “Ontology” and “Semantic Web Knowledge Graphs”.

2. Related Work
“… a form such that they can be evaluated…” “…a form that can be evaluated…”
“… user interacts with with the knowledge…” with
“DiscoveryHub is the tool we could declare as being the most…” “DiscoveryHub is the tool our review highlights as the most…”
“which she could want to limit results…” “which she might use to limit results…”
“…search, YouTube video recommender on query subject.” “…search and a YouTube video recommender based on the query subject”.
“…slow at times in performance especially…” “…slow at times, especially…”
“…the tool utilizes freebase…” Freebase, also consider link or reference
“…in our opinion the operation of such high-level algorithm…” “…in our opinion, this feature…”
“…listed in suggested gold standard.” Additional context required to fully understand the scope of these claims.

3. Diagrammatic approach…
“…get lost in variety and volume of information.” “…get lost in the variety and volume of the information at hand.”
“…Martin Epper and Remo Burkhard…” “Epper and Burkhard”
“…has a context menu. the data fed to it via corresponding SPARQL queries…” “…has a context menu and the data fed to it via SPARQL queries…”
“in our terminology diagrammatic reasoning” it should be explained better as it just seems classical data and schema exploration.
“functionality allowing the to see…” the user?
“We click the “Ontodia” tab to go to the next stage” Not very informative for users that are not aware of what Ontodia means. Maybe better something like “Browse” or “Details”…
“The integration points exists when a diagram is being initialized,…” Please, clarify this sentence.
“…retrieved incrementally and populate respectfully Data Model and Schema Model.” “…retrieved incrementally and populate the Data and Schema Models.”

4. Evaluation
This section needs a lot of rewriting, especially subsection 4.1.2. Look for an alternative way of presenting query selection, using tables, query characteristics, etc.

5. Discussion
“…this result overcomes initially expected 30-40 minutes time interval” “…this result improves the initially expected 30-40 minutes”. Authors should clarify where these expectations come from and why this is really an improvement, for instance comparing with related tools.

6. Conclusions
“…share knowledge and make inferences by means of direct manipulation…” What does inference mean here?

Review #2
By Aba-Sah Dadzie submitted on 20/Jun/2017
Suggestion:
Major Revision
Review Comment:

The paper proposes the use of Diagrammatic Question Answering as a more usable, visual support for answering natural language queries, to remove the need for the use of formal query language and an understanding of the knowledge base from which answers are retrieved. The topic discussed is practical and relevant.
However, the paper as is is not ready to be published. The biggest issue is presentation, the paper switches between a lot of background detail and skimping on required detail about the work actually done. Some parts are difficult to interpret because of basic, preventable errors. The evaluation methodology leaves a lot of questions unanswered, and the selection of results to be presented allows the reader to obtain only limited insight into participant experience and feedback. Importantly, the authors do not carry out a very objective assessment of the results presented, but rather push for a conclusion that appears to have been decided on with only some regard to the actual results.

Wrt novelty, the visual representations are node-link graphs that simply follow the data structure of Linked Data - there is existing work using visual hierarchies in both the Vis and SW fields - do a simple search, you will find them. Novelty/significance MAY lie in the application to QA specifically. However, the authors haven't really shown that their work adds any value over traditional, text-based QA - see more detail on this point below. The study IS useful, I would class the vis as proof of concept. Overall, as is, the contribution is probably just yet another demonstration of application of a simple visualisation add-on.

***

Why Wikidata, metaphactory and Ontodia were used for implementation is never discussed. Further, they weren't even weighed against other options. This is critical esp. as Wikidata was also used as the knowledge store for evaluation. Importantly, also, apart from the brief descriptions on p.12 - which actually don't say much - there is no discussion of their features. The reader therefore doesn't have much on which to judge them as is and their suitability for this exercise.
Are these the authors' own tools? Are they third party? Is there some link to or report or paper with further information? Esp. with reference to S3.3.4 - it is not clear what the last two points are saying - how did Ontodia acquire "an ability to save diagrams" and how did the (new) method for drawing diagrams obtain support from Ontodia?

The authors don't mention till a good way into the paper, and even then only in passing, that their study does not consider statistical/numeric data. First of all, why not? Secondly, this decision excludes a good amount of data. Further, when is factual data not numerical, and the reverse?

A number of statements are made that are at best inaccurate. E.g., S1.3 "Changing the existing approach in KM featured with spreadsheets, lists, and other documents to design knowledge graphs and managing data without coding or writing queries is a key to visual communications. " - no, this is not KEY to visual comms. It is a good contributor but only from the UI and depends on the user and task. Visual representations alone, without means for the user to at least inspect, and preferably interact with and modify the underlying algorithms, analysis or whatever approach is used to generate the visualisation itself, are difficult to use in general and as a basis for drawing confident conclusions. Even non-tech users should be given some means for interacting with vis, especially as size increases, in order to increase understanding and confidence in the results seen, as well as validity of the underlying data.
What is really necessary, and I'm assuming the point the authors wanted to make - is removing the NEED (not capability) for the end user to have sufficient technical know-how to carry out the underlying data processing and analysis.

"We focused in our review on tools that work online (web-based). " - WHY? I have no issue with doing this, but you need to justify this and ensure it doesn't bias your conclusions. Ditto restricting to most recent. For a start, what is your definition of recent? What about non-recent tools that are regularly updated? And what does this filtered set have that the others don't? Are these working tools or prototypes? Who do they target?
Also - online is not synonymous with web-based - a desktop tool may still need a network connection.

The literature review is a bit unbalanced. LD ViZWiz, for instance, has a fairly lengthy point by point description of functionality. Further, parts of the lit review read more like an advert than an unbiased comparison of features.

A lot of decisions are simply stated - without explaining what the reasoning is behind them or what impact they have (and how this is dealt with). E.g., in listing the visual aesthetics - "Most often the direct class dependency is shown, but in some cases we render the names of several classes going up the class hierarchy." - why? Further, doesn't this lead to clutter?

EVALUATION - this is one of the key contributions of this paper. Unfortunately, a lot of emphasis is put on background material rather than on a fair, objective discussion of the results. The sample results shown appear to have some been chosen selectively, more informative examples could have been chosen. Importantly - see also below - the conclusions are biased. The problem being that this makes it difficult to take any learnings from this study away with any degree of confidence.

The second evaluation question is VERY general - and can lead to any of a number of inconclusive or misleading answers. Please be more specific - state what kind of diagram you are evaluating and in what context.

S4.1.2, excluding Table 3, doesn't belong in this section - I'm not really sure where would be best in this paper, based on current structure. The first section is very close to a literature review. But it should probably be somewhere under methodology or similar.

What are "regular users"?
Does it matter that you have x number of male or female participants? Especially considering how unbalanced this was, the only reason for reporting this is if it was expected to impact the experiment or results. Considering this point is not addressed the information is not useful. Importantly, the BACKGROUNDS of the participants is not provided - for evaluation of a technical subject this is what really matters - in terms of domain, technical ability and experience, knowledge of the SW in general, NLP and QA in the context of the SW. How were participants selected - again, this matters.

"Major part of respondents were occupied within academia, however 7 users were employed in industry. " - "however" here is incorrect - suggest replace with "and" - being in industry is different from but not the opposite of being in academia. Wrt to my point above, what impact did area of work have on the results?
In the same vein "as a result" in the last sentence is incorrect - the description of the users did not result in the number of diagrams - I would suggest deleting this.

S4.1.2 - "Lexical gap between the vocabulary of the user and that of the ontology contains." - not a complete sentence - and I can't figure out what it's supposed to be saying.
"Complex queries that can only be expressed using complex queries containing …" - take a look at this sentence and reconsider using the same phrase to explain itself.

Why are the precision, recall and f-measure calculations taken over 4 answers? 4 out of all? 4 repeated per participant? ???

"Based on Precision/Recall measurements as well as given comments on working experience we conclude that the DQA approach satisfies the stated …" - first, what is "working experience"? And whose - the participants' or the authors'? Second, no, I really can't see that any conclusions can be drawn here… there is no discussion of the PRF scores preceding this statement. There is a broad summary after - but more space and effort is spent on discussing the theory on which QA and the QALD approach is used than on discussing the results.
WHAT, WHICH and WHO show similar scores, but there is a large drop for HOW and even more for NAME. I would expect at least a discussion of this variation.
I would suggest moving Table 6 to this section, and providing a bit more detail on what went into those calculations, and how questions were allocated. I would actually suggest merging tables 5 and 6 - as is, the reader needs to cross-reference QType across two pages - these. Most of the other columns in 5 are needed to interpret the PRF scores and draw informed conclusions.

Concluding S4.2.1 - "Also, a few reported obstacles were fixed immediately so that participants could follow the experiment flow and complete the questionnaires easily." - please elaborate. And state clearly what impact this had on the experiments and the results. What was done with the questions affected, esp. before and after whatever it was was fixed?

S4.2.2 - "4. The description contains a correct answer but is verbose and providing more information than the gold standard has. An example of redundant answer for the question "In which time zone is Rome?" could be "The location of Rome on the world map with the indication of the time zone"." - compare this to Fig. 10 and the example of verbosity in the text is labeled as the gold standard answer!

Table 4 - please provide a simpler, easier to interpret label for item 3, col. - reading the paper for the 3rd time I was stumped again by this header.

End of S5 - How were the estimated timings calculated?

No snapshots are provided for the second part of the experiment - this is actually even more critical than for the first - as third parties are being asked to perform QA based on a random selection of examples. And where the (characteristics and background of) participants generating those samples are essentially unknown - to the reader and the other participants.

Table 4 - the third example is interesting. How it is dealt with is however disappointing. Neither the discussion nor the suggested solutions deal with the actual problem. Further, what is the problem with Ontodia visualising a list? Let me rephrase that - this is a very basic requirement, why is it not possible? Importantly, also, did the authors not check that all questions COULD be answered with their test tool? Why frustrate participants with a question they had no chance of answering correctly using the tool?
Wrt to this "We were aware that some of the questions might constitute a challenge for our evaluators, but we were willing to verify our hypothesis in order to draw clearer conclusions" - I can't see what willingness to verify a hypothesis has to do with deliberately making things difficult for participants - who are in essence helping you out. What results in such cases is a cross between frustration and potentially unreliable results - in which case you're not verifying a hypothesis based on accurate feedback.
The conclusion of that paragraph leaves a lot to be desired - essentially, it is saying the results were not good, but will be ignored and the conclusion desired pushed forward regardless.
This continues in the following paragraph - the authors pick what suits them in order to present the conclusion they have decided to present. Rather than carrying out a clear, informative analysis of the results. More than half the questions being answered precisely really says nothing - very simple examples were given in the text and which would have required little effort. Such an approach is fine as a warm-up exercise, but you cannot judge effectiveness and value based on overly simplistic exercises. Especially when compounded with the other extreme - impossible or near impossible questions to answer using the tool provided.

In the conclusion - "First of all, the user information needs should explicitly state the question focus that the user is looking for. " - what does this mean - is the burden being placed on the user to frame a question properly in order that the tool can provide an answer? If so, how is this an improvement over formal querying? Or esp. over (deliberately) terse NL querying?

"The use of cognitive principles of aesthetics and custom visualisation templates make diagrammatic representation selfdescriptive and do not require any additional textual information for knowledge sharing. " - really? If so why does Table 4 repeatedly suggest the provision of additional text to help users answer the questions?
Also, "selfdescriptive" is two words, not one.

*** Other points

In the abstract:
"At the same time, it is well known that visual representation is the [EASIEST] and the most efficient way to dig into the meaning of data." (FYI - emphasis mine) - actually not true - most efficient - maybe, depending on what you're doing and what the data is like. EASIEST - the problem here is the words - easy is relative. There are many circumstances under which simple statistics would win hands down. Again, as above, it depends. I would suggest write intuitive or supportive or some such. Without using absolute claims. Visualisation is a plus under most circumstances but nothing is ever the easiest or the best all the time.

"user-friendly" is no longer recommended, but "usable" rather, just being user-friendly is not enough. And often is difficult to interpret.
As an example, on p.7 - "The initial idea behind Ontodia is to enable a user to see what is inside an unknown dataset and to present the exploration results in a user-friendly environment. " - what exactly is "a user-friendly environment"?

p.3
"An ontology is a complex and formal collection of terms and definition of the types, properties, and interrelationships of the entities that really or fundamentally exist for a particular domain of discourse or concern. In our work we talk only about OWL ontologies."
What is meant by " … really or fundamentally exist …" - what is the difference between the two words?
Further, are OWL ontologies sufficiently distinct from other ontologies that this needs to be made here? In any case, what is it about OWL ontologies specifically that this study addresses?

"… are knowledge bases created with W3C’s Semantic Web Standards (RDF) and knowledge models (OWL). … that can be executed with a simple semantic (SPARQL) query." RDF is not equivalent to Semantic Web Standards, nor OWL knowledge models, not semantic query SPARQL. They may each be an example, but the statement essentially says this is what each term refers to.
Similarly, "For more specific technological problems of data exploration and applications we use "Linked Data", especially when discussing querying datasets," - Linked Data does not cover all such cases. Data in such cases MAY be encoded as LD, but the reverse isn't true.

S1.3
"Interaction and visualization are among the top trends in knowledge and information management today. " - may be true, but a sufficiently strong statement that it needs to be substantiated by verifiable references.

"a model for diagrammatic representation of semantic data with [SUPPORT TO DATA INTERACTION] based on embedded queries;" - do you mean "… support FOR data interaction …"

At the end of S2.1
"At the same time, existing ontology visualization and exploration are not supposed to create personalized diagrams on top of raw data using visual exploration and publish and share designed representations on the web for reuse and sharing." - I can't see how this is mapped backwards to the work cited - it reads like the requirements spec for this work - such a statement is useful ONLY if it has some direct bearing on the literature being cited. Further, this is an open statement. What is the reader meant to do with it?

"QueryVOWL … Even though it delivers some navigation insights along with its main use case its main focus still lies in query construction rather than in question answering. It does not provide any means of sharing the results of the findings, only a SPARQL query can be exported from the tool."
This paragraph goes on to say what can be reused from QueryVOWL - which is good. However, I cannot see what is gained by pointing out things the tool does not do - that it was not designed to do (as for the other tools mentioned). This is on a par with saying an apple does not fulfil its purpose because it is not a pear. It's fine to state what you aim to achieve, highlight what you've learnt from previous work and show how you plan to extend it. Preferably without negative unwarranted criticism of other work - as in the quote I paraphrase - judging a fish by the requirements to ride a bike will result in it feeling endlessly stupid.

"We assume that the approach the authors of QueryVOWL took for constructing queries can be transferred to the process of defining join operations in DQA …" - this is a scientific paper, please be precise - did you check this - if so please remove "We assume". If not, I'd suggest you do find out if this is possible - based on the example at the end if not then you probably need to look at other alternatives?

What makes LODLive unique?

Wrt Aemoo - "This happens because the authors of the tool made an arbitrary decision on what relations could be navigated. " Really? Can you substantiate this comment? Yes, the relations are restricted, but the decision on which is not arbitrary.

"InWalk stands out by its special algorithm of thematic aggregation of linked data employed to create an abstraction called "cluster"." - what exactly does "special" mean? Further, clustering is not particularly unusual - why is it in quotes here?

What is the "suggested gold standard" (p.6)?

" The user does not have to know how to write data queries and even visual queries as well. " - this is an unusual statement - formal querying requires the use of specific languages, but VISUAL queries are non-uniform and do not conform to any particular standard (not that there are any defined). The statement that skill in visual querying is not required is therefore at best redundant. Further, the point in visual querying is that it does not require any expertise in querying.

"For large knowledge graphs … it is crucial for the user to be able to arrive quickly at the most relevant data … otherwise one can instantly get lost in variety and volume of information." - instantly is a bit of an exaggeration - to get lost requires at least some initial navigation.

In listing the requirements for DQA - "2. Purpose: "A picture is worth a thousand words"." - what follows conflicts with the saying… please double-check what the saying means and either remove the header or edit the requirement to match it. Probably the former.

"4. Communicative situation" - if a (key) aim of DQA is to improve the sharing of the results of querying then it should share in at least the key most reusable forms. Restricting sharing to diagrammatic representations alone REDUCES ability to reuse results.

"Labels for nodes could be displayed depending on the user’s choice of language." - why should language influence label display?

The discussion about templates on p.8 is confusing - what do the authors define as a template, and how do they implement this? What feeds into each - you can't both say all features available and then list different templates.
I cannot see the point in the "organisation" template - a label "org" is not much more useful than the empty "default" template.

"… which prevents the risk of loosing the user by overwhelming …" - LOOSE -> LOSE. Importantly, also, is it about the system losing the user - which is a different thing, or the user getting lost in the data?

Maybe a bit pedantic, but because this is talking about layout, you should probably clarify what is meant by "… a parallel navigation keeping the previous one in view." - is parallel here saying the two paths are related to each other, or that there are simply two paths drawn on a canvas that are not necessarily related to each other?

"… while the connections panel lets the user control the edges between nodes." - how?

"We were particularly aware that production of such visual statements will force the user to process the data, extract the knowledge out of it applying knowledge personalization …" - what is a "visual statement", and what does "knowledge personalization" here mean?

FIGURES & TABLES

I would suggest using a dark font for the text in Fig. 3 - the white on the palest background is a bit difficult to read. Contrary to the usual the colour version is even more difficult than a monochrome printout because the colour contrast is worse.

S3.3.2 - I would suggest pointing to examples of each feature as described - the point of visualisation is to remove the need for the user to carry large amounts of information in (working) memory. At the end of the list there is a sample question - I would suggest moving this to the start, then step through answering it to illustrate how each of the features described works. If it is possible to group them under (or classify as, if multi-category) the 5 lifecycle events that would be even better.

In stepping through the van Gogh influence example a lot of redundant information - aka noise - is provided. E.g., that you "… hit "Add Selected" button" …" is irrelevant - what matters is the actual action and result - placing new, relevant nodes on the graph.

Col 4 in Table 3 is redundant - the max count is 3!!! - see my point above about noise.

Pages 12 and especially 17 have small chunks of text hidden between diagrams. This makes them difficult to find, esp where a sentence starts on the previous page. This makes it especially difficult to read while cross-referencing between the different diagrams. Whatever editor you're using you can force sets of little figures to the top or bottom of a page - do this so all the text sits together.

Fig.14 has only 4 slices - I'd suggest using a line to link each to the corresponding item in the legend. Or, better, move the labels to the slices. Esp in a monochrome printout, only the lightest slice is easy to link to a legend item. Also, there are 4 slices but 5 lines in the legend - it appears as though something is missing - there is more than enough white space to prevent the spillover. Regardless, the second option makes for the lowest cognitive load interpreting the diagram.

CITATIONS & REFERENCES

"It is proven in psychology that people remember …" - please provide references to academic literature or other appropriate technical reports or white papers, and cite with a proper reference - this statement is one of the foundations on which this work is built.

Would suggest using Author [refNo] where the citation is the subject, it's a bit strange to refer to a human being by a number. E.g., "[1] provides an overview over the research area, while the QALD initiative [11] …" - strangely this is done for the second non-human reference but not for the first citation that starts the sentence!

Please capitalise acronyms correctly. If using LaTeX enclose in {} to preserve capitalisation.

PRESENTATION & GRAMMAR

Double check the encoding for writing out your equations - some of the non-alphanumeric characters do not print properly.

Gender neutral (they) or inclusive him/her (order irrelevant) is preferable to using one gender. Switching to feminine does not really solve the bias in more predominant use of masculine. Worse, gender use in the paper is inconsistent.

A very large number of errors that would be caught by an auto-check. Regardless of first language this is a simple exercise that makes reviewing much easier and helps the authors - if I'm wasting time and effort trying to decipher what was written because of careless/preventable mistakes I'm going to have less energy left to spend on assessing actual content. Plus, readability is important, so this requires me to lower my overall score.

A small sample of examples:
"to refer to the process of [ANSWERING QUESTION] addressed " - either "QUESTION ANSWERING" or "ANSWERING QUESTIONS"

"with a mean" -> "with a meanS"

"They are aimed to make …" and similar -> "They aim to make "

"in order to find the answer to the outstanding question …" - outstanding does not mean what was intended here. Probably "focus question" or "question of interest".

"Tools comparison table" -> tool without the 's'

"could be drag-and-dropped directly to the workspace" -> "could be dragGED-and-dropped directly ONto the workspace"

"Context menu existent on demand …" delete existent

"We removed the least relevant items from the diagram leaving out only those elements that are essential to the question." - LEAVING OUT actually means you REMOVED those elements - delete "out"

"The hypotheses is knowledge being represented" - hypotheses is plural - either "hypothesIs is" or "hypotheses ARE"

"Questions focus helps to manage" -> "Question focus …" - without the 's'

Some inconsistency in formatting - e.g., in the lit review section alphabetical and numerical lists are used randomly and interchangeably.

A large amount of incorrect use of definite and indefinite articles - mostly omitted, but also included where they shouldn't be. E.g.,
"Also we present the exploratory system that uses the Wikidata as a knowledge graph, metaphactory as a knowledge graph platform and Ontodia library as visual tool for data interaction. " ->
"Also we present the exploratory system that uses [DELETE the] Wikidata as a knowledge graph, metaphactory as a knowledge graph platform and [INSERT the] Ontodia library as [INSERT a] visual tool for data interaction. "

"Resulting Precision, Recall and F-measure values are represented in the Appendix B." - delete "the" before Appendix. Also represented -> presented.

"The description is correct in case it contains all relevant nodes description." -> "in THE case" - in fact, in this case leaving out "the" changes the meaning of the sentence.

"The question could be answered with SPARQL over Wikidata query service." -> "The question could be answered with SPARQL over THE Wikidata query service."

Several instances of redundant use of commas - these should either separate parts of list, sentences or be used only where there is a natural pause. Otherwise it distorts meaning or at best makes reading difficult. E.g.,

"The stripe at the top of a node and its border color-code the class, to which this node belongs." - there should be no comma here.

COMMENTS ON THE TOOL

I did only a cursory scan from the link to the van Gogh influences example - even reading that there was a little button to bring up the context menu I still automatically did a right-click - that IS how you bring up a context menu. Which didn't work - till I found the little button. I'd suggest you also trigger this menu on right-click - apart from being intuitive - this is the norm for this kind of interactive layout - and that IS the trigger for a context menu. Otherwise it is simply a menu triggered by a button click.

Review #3
Anonymous submitted on 13/Aug/2017
Suggestion:
Major Revision
Review Comment:

The paper presents a visual step-by-step method and tool for the exploration of semantic datasets using a diagrammatic approach. The authors coin the phrase "Diagrammatic Question Answering" to refer to the process of answering questions to a knowledge graph using visual exploration only. They attempt to address the following research question: "How to carry out an interactive data exploration process using only visual means with respect to the user’s information needs and without troubling the user with knowledge of query languages and underlying data models or schema." They consider their main goal "as to enable people to think visually and make inferences through visual interaction with data." The proposed approach has been implemented in a tool based on OntoDia and using Wikidata as a knowledge base (graph). It has been evaluated with two user groups on example questions using Wikidata.

The paper fits well into the scope of the special issue. It provides a comprehensive description of the work, encompassing research question and goal, challenges, requirements, related work, implementation, architectural details, use case, and evaluation. It contains some good ideas and observations (e.g., "incremental exploration"), and the implemented tool appears to be well-designed (apart from some minor visual issues) and effective to be used on semantic datasets (apart from the issues and limitations discussed in the paper).

However, similar research questions and related concepts of visual exploration can already be found in other works (e.g., OptiqueVQS, gFacet, etc.), limiting the research contribution and insights gained in this piece of research. The description of the method and its implementation in a tool are quite straight-forward and miss some innovative ideas, insights and novel lessons learned. The paper appears more like a system paper, describing the developed "Diagrammatic Question Answering" system but providing limited research insight. As such (i.e., as a system paper), it is of interest and could fit well into the special issue, but it would be less suitable as a research paper in its current form. In any case, the paper lacks a comprehensive, precise and clear description of the research context, including related concepts like "faceted search / exploration", related approaches and tools like OptiqueVQS, gFacet, RelFinder, or SparqlFilterFlow and a generalization of the approach, listing the research insights and lessons learned more comprehensively.

## Further issues:

Some statements are too general and/or bold. They need to be rephrased and/or backed with literature references, e.g.:
- "Users widely benefit from data visualization software for data exploration, because it allows them to view the most relevant properties and easily understand the meaning of their datasets quickly."
- "Interaction and visualization are among the top trends in knowledge and information management today."

Arguments for the following decisions are missing; without arguments, they appear quite arbitrary:
- Section 2: "We focused in our review on tools that work online (web-based)." -> This excludes related and possibly relevant tools, such as OptiqueVQS, SparqlFilterFlow, etc. (see above)
- Section 2: "...we gave a priority to the most recent tools and we indicated the operational status of each tool together with its description." -> This seems to exclude older related works like gFacet, etc. without a good argument.

The following is unclear to me:
- "The stripe at the top of a node [...] colorcode the class," -> What exactly is meant here and what about color-blind people (in more general)?
- "The main node label provides a node’s name. Such labels are usually extracted from rdfs:label or other fields that usually carry naming strings." -> What about "nodes" without such attributes? What is shown in their cases?

## Language Issues:
I would strongly encourage the authors to let a native or trained English speaker proof-read the paper to resolve any language issues. Pay particular attention to missing or incorrectly added articles (e.g. "the Wikidata", "with developed tool"), correct use of singular and plural forms (e.g., "a number of diagram") and commas (e.g. "understand its meaning otherwise one"). You should also adhere to the common style of scientific writing and avoid, e.g., to start sentences with "And..." or use words like "obvious" (especially if it is NOT "obvious"). Numbers up to 12 are usually written as words in the text (e.g. "5 persons / people"), and it is usually recommend to write e.g. "Lopez et al. [1] provide an overview..." instead of "[1] provides an overview...", i.e. not to use reference numbers as subjects of objects of a sentence but the authors of the works or alike. If you refer to author names, ONLY add the surname, not the first name of authors (e.g. "Martin Eppler"). Please also check for duplicated words (e.g. "... with with.." or "...encountered a quite a large...") and incorrectly added words ("... the to see...").

## Minor issues:
- The HP marketing material referenced in Footnote 1 is NOT the actual scientific source for the given numbers. Please refer the original source as a scientific reference!
- Remove the blank spaces before the footnotes.
- I would suggest to add a subsubsection heading before the paragraph starting with "In order to clarify the dataflow".

## suggestions:
- A link to a demo video would help to better understand the approach and use case


Comments