Review Comment:
The paper has been improved in this version and the authors have made an effort going through all the comments from the reviewers and addressing them as well as they could.
Despite the improvements, the paper still seems to require additional work. There are a very large number of unclear statements in the text, it is also unclear if the experimental setup was fair (e.g., was SaGe allowed to use significantly less resources than other systems?), if the experiments can be reproduced, if the code of smart-KG+ is included in the provided repository.
While a public repository is available, it lacks sufficient documentation to help potential users. For instance, no instructions are provided in the repository to run each of the different systems that were used to obtain the results presented in Section 5.
I could not find the code that implements the algorithms presented in the paper.
The files in the repository look very similar to the files in these repositories: https://github.com/WiseKG/WiseKG-Experiments and https://github.com/WiseKG/WiseKG-Java
Section 5 presents results for different systems (TPF+NP, brTPF+NP, smart-KG+, Wise-KG^{Typed-Family}), but it is unclear how could I obtain results for each of these systems using the code available in the repository. The code in the repository seems to be the one of Wise-KG.
Page 2, footnote 1
There seems to be a typo in "In act"
Page 3, lines 48-49
The text "We refer to the components (i.e. subjects, predicates, and objects) of a single RDF triple t \in G as subj(G), pred(G), obj(G), where the RDF graph G is a finite set of such triple" seems wrong and to mix the two notions introduced in the following paragraph: subj(G)/pred(G)/obj(G) and subj(t)/pred(t)/obj(t)
Page 4, line 6
The text "in the context of the later introduced frameworks" lacks references that make clear which frameworks are meant there
Page 5, line 38
The text "whenever the set of bindings Ω is not considered (i.e. only empty binding sets Ω = \emptyset are expected)" AND definitions 2.2 and 2.3, seem to indicate that when \Omega is an empty set so should be s(G, Q, \Omega) and s*(G, Q, \Omega). It seems that there should a mistake in the definitions or the text that follows them
Table 1, Data Dump
\Gamma_1 should be \Gamma_0, given the values of l and o and the text on page 6, line 1
Table 1, SPF
The text "(iterating over increasing offsets o := o + l)" seems to indicate an algorithmic solution instead of a formal definition
Table 1, SPARQL Endpoints
\Phi lacks its parameters, but assuming they are as for other interfaces, it seems unclear why the first parameter "n" would be 1. Maybe that is the case in some possible correspondences, but it is not the case in the one I could expect: as many triples as triple patterns are in Q
Table 1, SAGE
It is unclear why is \Gamma_0 not included
It is also unclear why the first parameter of \Phi is 1 (similar as for SPARQL Endpoints)
Page 7, line 47
It is unclear why is \Gamma_0 not included (it was mentioned in line 1, page 6)
Page 8, lines 10-12
It is unclear is there is any relation between 'n' and 'm'
Page 8, line 27
typo or missing word in the text "on and LDF-server"
Page 9, line 30
It is unclear what is meant by "p = pred(Q) ∩ pred(G)" the right side is a set and the left side is a predicate
Page 10, lines 2-3
"we assume partitions per n object ranges (e.g. from a histogram) can be split into a set of ordered values {v0, ... , vn}" it is unclear if there are n or n+1 ranges
Page 10, line 44
The text related to "the modulo operator" likely comes from an early version and it doesn't seem to be used in line 46
Page 11, line 23
the text "we see various options here" could be supported by adding a few details and references
Page 13, Figure 2
Maybe using F_1, F_2, F_3 in the right side is a mistake (those are defined in the left side)
There is an inconsistency, the predicates should be IRIs as in figure 1 (e.g., :director instead of director)
Page 13, lines 20-21
The text is inconsistent with the predicates and classes used in the example given just before
The statements in lines 46-46 of page 2 and lines 24-25 of page 13 are slightly inconsistent, was it 88% or 90%?
Page 19, lines 6-7
The definition of Q seems to have been written twice in the same sentence
Page 19, line 30
The text "If the result is trivially empty (lines 4-5), it returns an empty plan." is unclear.
There are some inconsistencies in Algorithm 2 and the text that describes it, in the algorithm it is stated that TPF is used to evaluate some subplans, but the text mentions brTPF. It might be due to the use of "TPF" to denote brTPF requests
Control has been used as: "a control parameter to transfer intermediate bindings together with query patterns, or a control to specify the page size..." (page 5), but later is used as "TPF control" (page 19, line 50), "brTPF LDF API control TPF(Q_s, \Omega)" (page 20, line 49). It is unclear what is meant by control and if it is meant the same in these three places. On page 19, it seemed to denote keywords "TPF" or "SKG". The use of TPF and \Omega and the connection to the brTPF selector seems misleading (in Table 1 it was stated that the main difference between the selectors of TPF and brTPF is that \Omega is empty for TPF).
Page 21, lines 1-2
The 's' seems to disappear and Q_s becomes Q. It is unclear if it means the entire query, possibly with more than one star
Page 22, Algorithm 3
It is unclear if the use of solution mappings from previous stars is restricted to the last star. For instance, for a plan with four stars that use brTPF for the first 3 stars and SKG for the last one, it seems that the algorithms would not use the solution mappings when evaluating the 2nd and 3rd stars labelled with brTPF even if those are available from the evaluation of previous stars using brTPF.
Page 23
The text: "We implement server-side query planner to re-order the star-subqueries and triple patterns based cardinality estimations available at the server." is likely missing one or more words.
In [31], it was shown that SaGe performs better when 4 workers are available (compared to having 1 worker), but it is unclear if 4 workers a good choice for the setup in this paper given the server specifications, 32 workers could have been a better choice for SaGe
Page 24
It seems odd that the 3 sub-workloads watdiv-stf have so different sizes. It is unclear how these sizes and queries were chosen. From 156 queries to 6, 14 or 23 seems like a very big change. Workloads with so few queries do not seem to be stress testing. A possible cause could be that the predicate rdf:type doesn't occur so often in the original workload
typo in watdiv-stfboth
Pages 25-26
The value of \tau_{class}_{high} for DBpedia is inconsistent in the text and table 5 (the text states 0.01/100 as value and the table states 0.1/100)
While it is good that the DBpedia Analysis information is provided in the repository, it lacks some description to help others understand what was done
It is unclear how the three different configurations used in the Ablation Study can be obtained / tested using the code in the repository
Tables 6 and 7
None of the columns is labeled as smart-KG+. It is unclear if all the systems used to obtain those results are still missing an important component of smart-KG+
Page 26, lines 43-44
The text "This is attributed to the query execution strategy of smart-KG+, which always pushes intermediate results to brTPF, instead of joining the intermediate results entirely at the client-side." significantly differs from what is stated in Algorithm 3 (where intermediate results seem to be ignored for all but the last star)
Maybe the client could be a bit smarter and only use solution mappings when beneficial. That would likely also help with what is mentioned on page 35: "while brTPF has a poor performance since some of the triple patterns are non-selective (even though the entire star-query is highly selective)"
Section 5.7 omits results for watDiv1B
Section 5.7.3 also omits results for watDiv100M
smart-KG+ seems to have three main improvements with respect to existing systems: i) typed-partitions; ii) server-side planning; iii) use of brTPF instead of TPF. But Section 5 seems to mainly focus on ii and iii, or i, and it is unclear if all three improvements have been considered for some of the experiments. For example, Section 5.3 considers two versions (one with ii and one with ii and iii / but it is unclear if they had i), Section 5.4 does not consider i, Section 5.7 only considers i. It is unclear if Section 5.5 uses i.
The last sentence in the caption of Table 11 seems redundant
Page 36, line 21
Maybe there is a typo in "watdiv-btf_{unbounded}" and it should be "watdiv-stf_{unbounded}". In any case, this text does not seem to be consistent with Table 11
Numbers in Table 11 do not seem to support the following lesson learned about WiseKG: "The results show that typed-family partitioning significantly reduces data transfer". When using typed-family partitioning reduces the data transfer, it seems to remain in the same order of magnitude. But in the case when it is increased, it seems to be increased by an order of magnitude. Maybe if larger datasets had been considered it would be clearer how much using typed-family partitions improves or worsens the performance of WiseKG.
I disagree with the reply to comment 49: showing that a property holds for an instance of size 2 is not equivalent to having done a proof by induction.
| 
