A decade of Semantic Web research through the lenses of a mixed methods approach

This paper presents a study using mixed methods for investigating the topics and trends in Semantic Web research. The authors identified top-down research topics from four seminal papers and used three data analysis tools to extract bottom-up research topics from Semantic Web research papers. The top-down and bottom-up topics are then compared and the results are discussed.

Pros:
This paper presents an original and interesting study on the main topics in the field of Semantic Web using mixed methods. The insights summarized and discovered from the seminal papers and data-driven analysis can provide useful references for scholars in this field. This paper is also well-written.

Cons:
There are two major issues with this paper. First, the authors didn't discuss the impact of parameter selection on the data-driven analysis results. Do the three tools require some input parameters, such as the number of topics? What are the parameters selected for your experiments and why do you select these values? It is also possible that the tools may not require any parameters from the users. In either case, the authors may need to provide some explanations to enhance the reproducibility of this work. Second, this paper overall is presented in a qualitative manner although it took a mixed methods approach. Is there anyway that the authors can, e.g., quantitatively compare the rankings of the topics output by the three tools (e.g., using Spearman's correlation coefficient), and explain why the ranking is different?

Some more detailed comments are listed as below:

- Abstract: "Overall, we provide a reflectional study on the past decade of Semantic Web research, however the findings and the..." should be put into two sentences: "...past decade of Semantic Web research. However the findings and the..."

- Page 3: "Another common solution is the adoption of probabilistic topic models, such
as Latent Dirichlet Analysis (LDA)" LDA refers to "Latent Dirichlet Allocation" not "Latent Dirichlet Analysis".

- PoolParty seems to be a commercial service. Did you purchase their service for this research or they have a free academic version? Some explanation is necessary here.

- Figure 10: what do the different colors of the nodes represent? If you use a metric (e.g., degree of centrality) to define the colors, you will need to add a legend here.

- Page 19: "Among the topics experiencing strong variations through time, Web Service is a declining one. After experiencing a peak of use in 2008 with a 40% distribution in the documents, it then dropped to less than 20% in 2015." It is important to note that fewer explicit mentions do not necessarily mean less popular. At the beginning of Web, people often say "World Wide Web" but nowadays people just say "Web". One more sentence explanation should be added here.

- Paper length: overall, this paper is too long. The authors may consider shortening some discussions.

The paper presents empirical work on the deveopment of the semantic web filed in terms of the coverage and importance of topics. In constrast to previous work, the authors make an attempt to combine a meta survey with the application fo different automatic topic detection methods.

In principle this approach is interesting as it promises to test the claims made by the 'experts' about the development of the field. Being an empirical approach, it has to stand a number of questions:
(1) are the hypotheses novel and important
(2) is the methodology well founded
(3) does the data support the claims

In order to be accepted, these questions have to be answered in a positive way.

(1) Question one already reveals one of the major weaknesses of the approach: there are no hypotheses. The study remains almost completely descriptive and does not really aims at drawing substantial conclusions about either the state of the field or the vailidity of the previous surveys. I think the paper would be much stronger if the authors would formulate a number of hypotheses that are tested.

(2) The overall idea of the proposed methodology is interesting and promising. Contrasting expert opinions with statitiscal results is very interesting. A well founded approach however, requires these two ways of anaylyzing the data ar eas independent as possible. In the current way the study is designed, I am not so sure if this is always the case. In particular, the results of the automatic analyses - at least for some approaches - strongly depend on manual input by the authors. This means that there could easily be a bias in the choices made. Maybe it would eb more clean to either resort to completely automatic methods or first consolidate a shared taxonomy that is the basis for all analyses manual and automatic.

(3) Getting the data right is clearly a critical step in this study. I acknowledge that this is a difficult problem and there are clearly different ways to take, but stil I am not really happy with the choices made by the authors, both with respect to the selection of survey papers to compare and the document corpus to analyse. Wrt. the surveys the authors mix two types of papers. The first type are vision papers that are not based on factual information but rather aim at convincing people to follow a certain path. The second type are actual surveys that describe the research field a posteriori. Mixing these two types is very questionabls. The problems become apparent when looking at the vision paper from the database community that names completely different topics than all the other papers. The selection of teh corpus is also not unproblematic in my opinion. The authors use papers from ISWC and ESWC which is somewhat reasonable as these two conferences can be seen as the core events of the community. I don't really understand why the 'semantics' conference has been included. I don't think it has a status that is anywhere near ISWC and ESWC. Further, it is unclear why other sources have not been included (e.g. the Journal of Web Semantics or the Semantic Web TRack at WWW). I miss clear selection criteria. This makes the selection look rather arbitrary.

In summary, I think the idea of the paper is very interesting and worthwhile, but there are a number of weaknesses (s.a.) that should be addressed by the authors before publication.