Solicited review by Agnieszka Lawrynowicz:
I recommend rejection of the paper for the following reasons:
1) The paper is written in a very careless way. The language is very poor. There are plenty of typos, gramatically incorrect/unfinished sentences that make it very unclear in many places. This is unacceptable for a journal paper.
Some examples:
"APFEL [20] it is based on the general observation that alignment methods like QOM [18] or PROMPT [23] and extracts additional features by examining the ontologies for overlapping features, including domain-specific features."
"A new compound measure SSFPOA[29] has been proposed by us which uses 12 different matchers to find semantics."
2) It is unclear from the paper what the authors did, and why, what are the contributions of the paper.
The "SSFPOA (Semantically Similar Frequent Patterns extraction using Ontology Algorithm) for extracting and clustering semantically similar frequent patterns", called "measure" in the paper is never properly introduced. Instead of very vaguely describing it at the end of "literature survey" section, it could be better to describe it properly in a preliminaries section.
What does it exactly mean, the step 5 in the method: "5. Cluster the mapping results in the range of 0-1"?
There are also other things left without an introduction, e.g. the test cases.
It is unclear what are the goals of experiments, nor it is clear what the results mean (e.g. Figure 5).
3) Instead of "using ontology", "Ontology is a logical system that…" it would be better to use "an ontology" that would mean an engineering artefact, and not a subdiscipline of philosophy.
In summary, technical soundness and presentation make the paper impossible to be published in a journal.
Solicited review by Jerome David:
This paper presents an analysis of the performance of several supervised learning methods used with a similarity measure (SSFPOA). Compared learning methods are SVM, Bayesian networks, and multilayered perceptron.
In general, the paper misses some important details to be understandable. Results of other matchers (fig5) does not agree with those provided by OAEI.
The SSFPOA measure could be more precisely presented in this paper. It is also not clear which are the measures evaluated in the charts (1to11, 1 to 3, etc.) On Figure 5, what are the differences between SSFPOA and compound metric ?
The training step should be carefully explained. If the learning process has been made on the 3xx tests, it is obvious that results will be good. The paper does present details about this important step of supervised learning.
In order to analyse if learning methods are interesting or not, a comparison with classical alignment extraction strategies would be welcome: threshold, maximum weight matching, stable marriage, etc.
The definition of F-measure could be simplified. There is no need to introduce such a complex and generic notation.
Since reference alignments between 3xx ontologies are not provided, the paper should explain how they are made.
On OAEI 2011, 3xx tests are not used anymore (there is no results with these tests). It is said that these alignments are not perfect and are here only for comparability reasons with previous years (see: http://oaei.ontologymatching.org/2011/benchmarks/). How the authors can find results of PRIOR+, ASMOV, etc given that they are not provided in OAEI 2011? Furthermore, if we look to the results of previous years, results of other matchers given in the paper does not correspond to those provided by OAEI in 2010, 2009, 2008. They are perhaps those of 2007, but if we look to next OEAI, results have been enhanced a lot.
According to the OAEI policy, the paper have to compare its result with all the participants of OAEI 2011, and not only some of them (see: http://oaei.ontologymatching.org/doc/oaei-deontology.2.html).
All this reasons call into question the validity of the presented results and analysis.
Solicited review by Kate Revoredo:
The paper proposes to evaluate the use of a compound measure (SSFPOA) to solve ontology alignment problem through the use of data mining techniques. The evaluation is done by comparing the SSFPOA with seven other approaches in the literature. Although, the results show an improvement in F-measure of the proposed approach over other seven approaches, it is not clear what are the measures aggregated by SSFPOA and it is seems that a significance test was not considered, leaving doubts about the actual improvement.
Moreover, an evaluation of different similarity measure were performed using 3 classifiers (Bayesian network, Neural Network and Support Vector Machine). Four ontologies were considered for this evaluation and the results were presented using Precision, Recall and Accuracy. It is not clear the goal of this evaluation: verify
which classifier performs better? It is mentioned that 12 "matchers" are considered, but which one concerns the SSFPOA? Why F-measure was not used? A significance test was not considered.
Experimental methodology must be well defined and applied. For instance:
- No explanation is provided for considering 80% for training and 20% for test.
- What is the structure of the dataset given to the learning algorithms? What is the role of the cluster column in the learning?
- An explanation for the chosen parameters configuration for each algorithm must be provided.
- Figures are shown in different scales (e.g. Precision for 301 vs 304 and 302 vs 303).
- Significance test must be used.
Text must be completly revised, specially the English. In many situations it is hard to understand what the authors mean. Some general comments concerning the text:
- The structure should be revised. Section 2 is very confusing. For instance, page 2 mentions "...test case 303 Vs 304 and m(Proceeding, Porc, =, .36)...301 Vs 304...", without proper explanations. Moreover, Section 2 should focus on approaches based on data mining, since they are the ones related to the approach evaluated on the paper.
- There are citations missing (e.g. Machine Learning, decision tree, Neural Network, Support Vector Machine, Bayesian Network) and for others I suggest a review. For instance:
Ontology --> Gruber, T.R., Toward Principles for the Design of Ontologies Used for Knowledge Sharing. International Journal of Human and Computer Studies, vol. 43,issues 5/6. pp. 907–928 (1995), instead of [33]: www.wikipedia.com
Ontology alignment --> Ehrig, M., Ontology Alignment: Bridging the Semantic Gap. Springer (2007) or
Euzenat, J.; Shvaiko, P. Ontology Matching. Springer (2007).
Information Retrieval --> Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, Cambridge University Press. 2008 instead of [32]: Venkat Gudivada, Vijay V.Raghavan, William I Grosky, Rajesh Kasanagottu, "Information retrieval on the world wide web", IEEE Internet Computing Sep 1997 1089-7801/97.
- I also recomend a review in the text regarding the use of terms such as alignment/matching/mapping. For that considers Ehrig, M., Ontology Alignment: Bridging the Semantic Gap. Springer (2007)
- The bibliography entries are incomplete, without standards and some of them do not represent the state of the art.
- Figures and tables are inadequate. The text mentions tables and figures, but what are presented are not exactly a table or figure (e.g. Figure 1, Table 1 and 2). They should be revised.
- Decision tree is confused with Bayesian network (Page 4 --> Table2).
