Review Comment:
Overall evaluation
Select your choice from the options below and write its number below.
== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
== -2 reject
== -3 strong reject
2
Reviewer's confidence
Select your choice from the options below and write its number below.
== 5 (expert)
== 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)
3
Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4
Novelty
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4
Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4
Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 not present
4
Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
== 2 poor
== 1 very poor
4
Review
The paper describes an approach that uses distant supervision for identifying relations between entities or entities and the class they belong to in text. The authors propose an approach that improves related work by applying simple statistical models to filter unreliable data, leave ambiguous data out of their seeds and using more negative training data.
The paper is very well written and the approach and results provide a decent contribution to the field.
There are a few parts that could be improved:
In `related work': the difference between Open IE and distant learning is not as big as the authors suggest: Open IE, for instance, uses distant learning as a first step. They also make use of machine learning techniques, so the claim that they are 'rule based' is incorrect. If the authors mean to address a specific part of the approach, this should be specified.
The work by Wu and Weld (among others [1]) on Kylin should be cited.
In `Evaluation': 'relative recall' is not quite as well established as 'recall': please explain the concept in one sentence.
How do the results of the reimplementation of Mintz et al.'s system compare to the original?
The difference in recall and positive hits and the (probable) cause by the use of the additional NE class could be explained better. On the one hand, there are more hits, on the other hand less recall. At a first glance, it seems odd that a step that can add more candidate data would reduce recall. The issue at hand is, of course, that the systems work (1) with different sets and (2) with different features, resulting into different sets. The fact that relative recall (and not recall) is reported is extremely relevant here, because additionally identified entities will not be found by approaches not using the additional NE identifier and can thus not help improve recall. If my description above is correct, I managed to figure it out, but it should be spelled out.
Minor comment:
Compliments for the careful writing, with no noticeable errors and typos.
In section 4.2, footnote 1 should immediately follow the comma which should immediately follow `API'.
[1] Wu, Fei, and Daniel S. Weld. "Autonomously semantifying wikipedia." Proceedings of the sixteenth ACM conference on Conference on information and knowledge management. ACM, 2007.
| 
