A Fine-Grained Evaluation of SPARQL Endpoint Federation Systems

Tracking #: 884-2094

Muhammad Saleem
Yasar Khan1
Ali Hasnain1
Axel-Cyrille Ngonga Ngomo

Responsible editor: 
Axel Polleres

Submission type: 
Full Paper
The Web of Data has grown enormously over the last years. Currently, it comprises a large compendium of interlinked and distributed datasets from multiple domains. Running complex queries on this compendium often requires accessing data from different endpoints within one query. The abundance of datasets and the need for running complex query has thus motivated a considerable body of work on SPARQL query federation systems, the dedicated means to access data distributed over the Web of Data. However, the granularity of previous evaluations of such systems has not allowed deriving of insights concerning their behavior in different steps involved during federated query processing. In this work, we perform extensive experiments to compare state-of-the-art SPARQL endpoint federation systems using the comprehensive performance evaluation framework FedBench. In addition to considering the tradition query runtime as an evaluation criterion, we extend the scope of our performance evaluation by considering criteria, which have not been paid much attention to in previous studies. In particular, we consider the number of sources selected, the total number of SPARQL ASK requests used, the completeness of answers as well as the source selection time. Yet, we show that they have a significant impact on the overall query runtime of existing systems. Moreover, we extend FedBench to mirror a highly distributed data environment and assess the behavior of existing systems by using the same performance criteria. As the result we provide a detailed analysis of the experimental outcomes that reveal novel insights for improving current and future SPARQL federation systems.
Full PDF Version: 

Minor Revision

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Juergen Umbrich submitted on 01/Dec/2014
Review Comment:

I want to thank the authors for addressing my comments/concerns and the ones of the other reviewers.
I am happy to accept the the version as it is.

Review #2
By Carlos Buil Aranda submitted on 02/Dec/2014
Review Comment:

I think the authors answered all my questions.

Review #3
Anonymous submitted on 04/Dec/2014
Minor Revision
Review Comment:

This manuscript was submitted as 'full paper' and should be reviewed along the usual dimensions for research contributions which include (1) originality, (2) significance of the results, and (3) quality of writing.

I thank the authors for their responses. Nevertheless, they did not precisely address all my questions.

a) Although the authors have observed an “insignificant” difference between the results produced by the different methods used to measure the reported time, there is an error in measurements. This issue needs to be explicitly mentioned in the paper.

b) There is a misunderstanding and misleading idea about the query processing capability of producing results incrementally. In the context of adaptive query execution engines, this capability does not mean that there is an API that implements the “iterator model” and is able to output one answer when the getNext method is invoked. Contrary, the capability of producing results incrementally means that the engine outputs the results as soon as they become available from non-blocking operators. The authors should include the reference where is stated that Sesame does implement non-blocking operators. Additionally, it is required an explanation of why FedX, LHD, and DAW implement N.B.O. and A.Q.P while others are not adaptive (statement in Page 7: “Ten of the considered systems support adaptive query processing with DHT, SPLENDID, QWIDVD, and DARQ being the only systems that do not support this paradigm.”).

c) There are still typos in the paper that need to be fixed. Some of them:
Page 2:
“Our main contributions are summarized as followS:”
“We present (to the best of our knowledge) THE most..”
Page 3:
Betz et al. [6] identify. The same for: Gorlitz el al., Ladwig et al., Umbrich et al., Montoya et al.
CHANGE: “FedX[33] compares its performance… “ BY “Schwarte et al.[33] compare FedX performance with AliBaba…”
CHANGE: “LDH compares…” BY “Wang et al.[37] evaluate the performance of LHD, FedX, and SPLENDID using the Berlin SPARQL Benchmark (BSBM) [7]”
Page 4:
Avoid repeating some many times the word “insights”
Use the font of BSBM and SP2Bench consistently.
FedBench [31] is designed to evaluate (NOT SIMULATE)
Page 5:
Rephrase: Result completeness: “Assuming that the SPARQL endpoints from which **you**??”

Page 6:
Rephrase: 2. Query federation over Linked Data: … The set of data sources which can contribute ***results***?
Page 7:
Use on-the-fly consistently.

Page 11:
Section 6.1.1
Order the references CHANGE [33,8,22,29] BY [8,22,29,33]

Consistently use British or American English. Do not mix both, e.g., behaviour versus behavior. Similarly, consistently follow the punctuation rules for the numbers, e.g., 10693 versus 3,670.9.

Page 20:
Section 7.1
Order the references CHANGE [8,1,33,26,22] BY [1,8,22,26,33]