Integrating Heterogeneous Knowledge with FrameBase

Tracking #: 1239-2451

Jacobo Rouces
Gerard de Melo
Katja Hose

Responsible editor: 
Guest Editors ESWC2015

Submission type: 
Full Paper
Large-scale knowledge graphs such as those in the Linked Data cloud are typically represented as subject-predicate-object triples. However, many facts about the world involve more than two entities. While n-ary relations can be converted to triples in a number of ways, unfortunately, the structurally different choices made in different knowledge sources significantly impede our ability to connect them. They also increase semantic heterogeneity, making it impossible to query the data concisely and without prior knowledge of each individual source. This article presents FrameBase, a wide-coverage knowledge-base schema that uses linguistic frames to represent and query n-ary relations from other knowledge bases, providing also different levels of granularity connected by logical entailment. This altogether provides for flexible and expressive seamless semantic integration from heterogeneous sources. It also opens possibilities to draw on natural language processing techniques for querying and data mining.
Full PDF Version: 

Major Revision

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Andrea Giovanni Nuzzolese submitted on 16/Jan/2016
Major Revision
Review Comment:

The paper presents FrameBase, a resource that, according to the authors' claims, provides a broad-coverage schema for integrating datasets by
using N-ary relations modelled as frames according to a neo-davidsonian representation. FrameBase addresses the challenging problem of providing
a reference schema for supporting knowledge integration from heterogeneous sources. Hence, the paper introduces a novel solution that relies on linguistic frames from FrameNet used for interpreting the knowledge soup consisting of the Semantic Web with different representation schemata, ontologies, vocabularies, etc.
The overall quality of writing is good.

--- Originality
The paper submitted to the Semantic Web Journal (SWJ) is an extension of the paper presented at the 12th Extended Semantic Web Conference (ESWC 2015). With respect to the ESWC paper there are more details about:
* the referencing of roles in State of the Art section
* the schema induction procedure. Namely, the item about the creation of intermediate nodes is more detailed. Nevertheless, some concrete
examples would help the reader to better understand the method. This holds in general for the schema induction part.
* the automatic Reification-Dereification mechanism. This mechanism enables the automatic dereification of n-ary relations consisting of FrameNet
frames to simple binary predicates and, vice-versa, the reification of binary predicates to FrameNet frames. It is worth saying that this part is more
exhaustive than that described in the ESWC paper. However, not all the ReDer rules presented in the boxes are described in the text properly.
* Section 6 about knowledge integration. This part is the most original contribution of this paper (considering the ESWC paper as reference for
comparison). In fact, it provides details about how to generate integration rules for integrating datasets by using FrameBase.
In my opinion, the extension submitted is still quite limited and something more can be done especially concerning the State of the Art and the evaluation with a more elaborated discussion. In fact, the focus of the paper should be moved a bit more on knowledge integration, i.e., the title, the authors' claims are about knowledge integration, however some crucial parts of the paper (i.e., State of the Art and Evaluation/discussion) are not.

--- State of Art
The State of the Art (cf. Section 2) provides a detailed and fair description about the variety of methods used for representing n-ary relations. Nevertheless, the authors claim that (in the authors' own words) "FrameBase (is) a wide-coverage knowledge-base schema that uses linguistic frames to represent and query n-ary relations from other knowledge bases". Hence, n-ary relations are used by FrameBase as a means for dealing with knowledge integration, but no existing solution about information/knowledge integration is reported in State of the Art.
Consequently, the authors should clearly position FrameBase with respect to the State of the Art.
Additionally the ReDer mechanism has some contact point with Legalo [1], thus a comparison can be provided.

--- System Overview
Section 3 provides a good description about the main contributions of the paper, i.e., the FrameNet-WordNet mapping, the schema induction solution and the ReDer mechanism.
The authors should explain in more detail how they obtained the optimal values for the two parameters a and b of the mapping function.
In the Sub-section 4.2 (General frames) the two specific properties for inheritance and perspectivization that inherit from rdfs:subClassOf remain unnamed as they are never introduced.
More concrete and coherent examples would be very helpful for appreciating the schema induction procedure in each of its steps.

--- Automatic Reification-Dereification Mechanism
A bit more details should be provided (just few sentence) about LUs and their role in FrameNet. This comment also holds for GF labels that in are only mentioned in Section 5.2. As already stated, the boxes presenting the creation rules need to be better explained in the text. Also it is
not always clear the relation between the creation rules and their corresponding ReDer rules.
The authors state the they used the Kuhn-Munkres algorithm for obtaining one-to-one assignments and, then, achieving idempotency in ReDer rules. Firsly, a reference to the Kuhn-Munkres algorithm should be added. Then, (i) few more details more about the algorithm should be provided and (ii) an
explanation is due about why this algorithm was adopted.

--- Evaluation
The results presented about the FrameNet-WordNet mapping do not show anysignificant evidence about the advantages of using new mapping over
other already existing mapping. Thus, a further discussion motivating such a choice is required.
There is no evaluation about the effectiveness, complexity and sustainability of FrameBase for supporting knowledge integration.
This is the main weakness of the paper. In fact, the construction of integration rules seems to be quite complex and costly. For example, what kind of user can deal with the design of integration rules?
Additionally, once the problem of writing integration rules is addressed to what extent FrameBase is effective in knowledge integration by also taking into account already existing solutions?

--- Minor issues (typos, etc.)
page 2: "These representations will me discussed" -> "These representations will be discussed"
page 4: Use a footnote for referring to
page 4: The sentence "For example, has a domain SportsTeam and Person as domain and
range respectively" is unclear. Rework it.
page 5: the last object in the sentence "object is a person that plays in the team denoted by the object"
is probably a subject.
page 6: "FrameNet provides an reasonable level" -> "FrameNet provides a reasonable level"
page 6: The title of Subsection 3.2 is meaningless
page 6: "to transform the original resources for into FrameBase's lightweight" -> "to transform the original resources into FrameBase's lightweight"
page 8: Footnote 3 "and in the context if knowledge bases" -> "and in the context of knowledge bases"
page 9: DBPs is introduced without prior explanation
page 9: Probably the term "pair" is more appropriate than "set" in the context of the sentence
"We term the set of a reification rule and its converse dereification rule as a ReDer"
page 14: Rework sentence at the beginning of Section 6 "Knowledge from other KBs such as Freebase..."

[1] V. Presutti, S. Consoli, A. G. Nuzzolese, D. R. Recupero, A. Gangemi, I. Bannour,
and H. Zargayouna. Uncovering the semantics of wikipedia pagelinks. In
K. Janowicz, S. Schlobach, P. Lambrix, and Eero Hyvönen (Eds.), Proceedings of
the 19th International Conference on Knowledge Engineering and Knowledge
Management (EKAW 2014), Lecture Notes in Computer Science 8876:413-428.
Springer, Berlin, Germany, 2014.

Review #2
By Bonaventura Coppola submitted on 21/Jan/2016
Minor Revision
Review Comment:

The paper addresses thoroughly and from several perspectives the very relevant issue of representing complex events with an unrestricted number of participants (i.e. n-ary relations) by the convenient means of standard RDF triples. The relevance of such topic is supported by the current, surprisingly high number of papers in the Semantic Web community which, while addressing very interesting tasks (as e.g. on-the-fly novelty detection) fail short of an effective representation model and rely on toy assumptions like the frequent "one triple per event" oversimplification. Hence, a first important contribution of this paper is in its preliminary survey of the several current, standard methods for projecting arbitrary n-ary relations onto consistent sets of triples. All of such methods are discussed and their pro's and con's considered. The eventual choice of the authors towards the Semantic Frames representation is extremely well motivated by considerations about expressiveness, space complexity, and reliance on sound, widely supported theories of meaning (Frame Semantics) from the linguistic-theoretical counterpart. Then, the practical creation process of a frame-based knowledge base (KB) schema ("FrameBase") from the established lexical-semantic resource "FrameNet" is described and discussed in detail, including an adequate number of practical examples. Also, the method for integrating several popular heterogeneous KBs into a single instance of FrameBase.

For such many mentioned reasons the paper deserves appropriate attention, and its underlying development project would greatly benefit from extensive feedback by a growing community of users. Nonetheless, it shows a few consistent weak points which can't allow a seamless acceptance for publication in a journal. First, the supporting KB is still under development and its current status and quality evaluation are not made completely clear. Also, it does not provide consistent improvements from a previous conference version. However, since it was extensively expanded and clarified with a number of details and documented examples, I think that the relevant practical advantages it would bring to the field do sufficiently motivate the opportunity of a revision process and a possible successful outcome.

I would consider the following set of requested revisions as "minor" in the sense that no further computations are required (and indeed probably not reasonably possible). However, several specific parts of the paper are surprisingly and disappointingly poor and were probably not even proof-read before submission. For these reasons, I strongly recommend the authors to take their chance for a thorough rework of the paper. A few points below (which get a global numbering across sections for easiness of further reference) will be critical for a possible advance in the acceptance process, including and not limited to: R22, R29, R31


R1: As already said, I suggest a thorough rework for smoothing the consistent unbalance of style, which goes from very descriptive and sometimes redundant (e.g. the examples discussions) to extremely synthetic to the limits of understandability (e.g. most of the quantitative evaluations, and the methods described in the end of Section 5.2)

R2: Check the alignement of stated goals (abstract, introduction) and practical results obtained.

R2.1 Seamless semantic integration without knowledge of each individual source. Do examples in 7.4 show this? Not sure. -- Avoid "seamless" along the whole paper.

R2.2 Emphasize at least in one practical (querying) example how different levels of granularity turn useful

R2.3 Include a concise enumeration of resources currently offered: a) Schema: available resources (definition, reder rules, direct binary predicates) with their status and maturity/stability; b) KB integration (integration rules, integrated KBs) with their status and maturity/stabilit. The current "Integration" Section seems to be a set of selected examplars of "how this could be done". No automatism. No evaluated integrated KBs. c) Please consider that on the FrameBase web site a few resources are still unavailable


R3: Caption of Table 2 should be fixed. Also consider the following item R12.

R4: Figure 1 not understandable, colors not useful in BW printing.

R5: ReDer rules in Section 5.2, better using individual subsection with separators, one per e

R6: Integration rules in 6.1 (use subsection separators, number examples, comment them before rather than after the code)


- Section 1

R7: Since you mention the IBM Watson QA System, it is worth considering the actual paper which documents its exploitation of structured KBs and even Semantic Frames:

A. Kalyanpur et al. 2012. Structured data and inference in DeepQA. IBM Journal of Research and Development, 56, 3 (May 2012), 351-364.

R8: I suggest adding the estimation of the number of average predicate arguments in FrameNet's frame definitions (should be about 6) to make the point stronger.

R9: Typo: "These representations will me discussed"

R10: In item 1 "When linking data": I would avoid the word "pattern" here (first appearance in the paper) which makes less clear the reference to the preceding discussion

R11: In item 3, I would mention that this is a (very relevant indeed) part of the preceding item 2

- Section 2

R12: While Table 2 shows an important aspect of the paper (space efficiency), it remains a bit out of the discussion in the main text. My advise is to move most of the caption content into the main text of Section 2. This will also reduce the format problem of such caption.

R13: Still on Table 2, I have problems in matching it to the representations of the example "John married Mary in 1964" shown Table 1. It seems to work for the Neo-Davidsonian representation with n=3 and k=0, but not for the other ones. It should be my fault, but for the sake of clarity please 1) double check the table content, and 2) give explicit values for n and k for the other representations as well.

R14: In general, consistency and easiness of reading would improve a lot if any best effort would be done to keep Section 2 completely aligned to Table 1 for terminology and examples. A few cases below

R15: Section 2.2 uses "subject, predicate, object" for RDF Reification while Table 1 uses "subject, property, object"

R16: Minor examples mentioned along 2.2 like bornAtPlace, bornOnDate, Paris isConnectedTo London, could be made consistent with the actual triples in Table 1

R17: The case of roles (Section 2.4) must be added in some way to Table 1. Ideally, it should possibly keep expressing the same event.

R18: In 2.4, an additional point worth to consider is that, while with the current representations in Table 1 link additional participants to the event predicate, the roles usage shown would link them to the existing event participants. This makes an important difference (at least) from a linguistic perspective.

R19: Typo in 2.4: delete the words "a domain" in the last sentence of page 4

- Section 3

R20: In 3.1: minor punctualization: while FrameNet's semantic role set remains the most rich and appealing one, historically the SRL task acquired most of its popularity in NLP in the simpler PropBank setting (CoNLL-2004 and 2005 Shared Tasks). The interest in FrameNet-based shallow semantic parsers (and their applications) stemmed from those workshops and had its first autonomous task in SemEval 2007. Reference:

Collin Baker, Michael Ellsworth, and Katrin Erk. 2007. SemEval'07 task 19: frame semantic structure extraction. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval '07). Association for Computational Linguistics, Stroudsburg, PA, USA, 99-104.

R21: Section 3.1 sounds slightly apologetic about FrameNet. Issues in coverage, granularity, homogeneity could be mentioned as well.

- Section 4

R22: I see an organization problem in Section 4. Since its core goal is the creation of a class hierarchy of frames, this concept should first be properly presented and motivated in a dedicated paragraph. Instead, the Section goes very fast into algorithmic details (WordNet mapping) and the first mention of "hierarchy" is just incidental in 4.2. The motivations for a hierarchy are present (e.g. in the beginning of 4.2.3) but I strongly advise to gather them in a clear initial presentation (Section 4.0) where all of the 4 types of frames are introduced all together, saying "what" and "why". Then the rest of the Section should just take care of the "how". Also see the next point about Figure 1.

R23: A good practical example of when the frame hierarchy turns useful should be first included in its motivations

R24: Figure 1 should play the role of leading the reader step-by-step across the whole Section 4. Instead, its first and only mention is just incidental in 4.2.3. I would properly introduce it in the beginning of Section 4, and constantly refer to it.

R25: Figure 1 is almost useless when/if the paper is printed in black and white. I strongly advise to change it not relying on colors anymore.

- Section 5

R26: Move the DBP acronym expansion (5.1) to its first mention (in 5.0)

R27: Figure 3: wrong caption. It is not the general pattern (Figure 2) but a sample instance.

R28: Section 5.1. Sentence starting with "Besides the plain" hard to understand, probably ungrammatical.

R29: Section 5.2. The enumeration of creation rules and their related examples are extremely interesting. Nonetheless, there is a major problem here. The core topic of the paper is promoting the usage of n-ary relations with n > 2 to represent events. Here, it is compellingly, long, expected an example of how the creation of a 3-ary or 4-ary relation (frame) works, and such example is not here. Since all the shown rules are defined on a single input triple, I SUPPOSE there should be a mechanism where multiple rules fire at the same time on a set of triples and create a single frame instance. But such mechanism is not explicitly mentioned. Even if this is not the case, it is still totally necessary to show here a bright exemplar of frame creation with n > 2. I consider this point not negotiable for the whole consistency of the paper. To give you three different suggestions I can advise either to show 1) the creation of a simple "Buyer+Seller+Goods" Commercial_Transaction frame from two (or three?) triples, and/or 2) how did you create the ":frame-Leadership-leader.n" frame (n=4) referred in the beginning of second column of page 14, and/or 3) if applicable, show how the Creation Rules 3 and 4 in 5.2 could fire together to create a "merged" ":frame-Destroying-destroy.v" which you currently show broken into two pieces.

- Section 6

R30: Section 6 should be the strongest part of the paper according to its title, AND its motivating extension for this journal version. Instead, it's frustratingly rough and opens with an ungrammatical sentence to start with. My advise is to put a serious effort in reworking it.

R31: The beginning of Section 6 must be rewritten. The paper title is about integrating KBs. What is the goal of such integration? To provide a methodology? To provide a populated, integrated KB? To provide a few examples? From current content, the only evidence is the third one. In the other two cases, I would expect either numbers about current population status (rules, frames, triples, coverage percentages) or at least an estimation of how difficult it would be for the interested reader to start manually writing per-frame integration rules as shown in the examples. On the other hand, from other sources (framebase web site, previous discussions, etc.) I believe that a serious effort in automatic population of the framebase schema from other KBs is on its way. To my understanding, Yago2 and FreeBase should be there already. This is a relevant project which would motivate this paper in any case and at whatever stage it is. Hence, Section 6 must include and describe the plain current status of such integration process, with a discussion of the weaker points. Ideally, it should also mention a few guidelines for the evaluation of the integration process (which is understandably not present in the paper). Presently, the per-case, very specific examples provided and their related discussions are interesting but they can't make the section alone.

R32: Section 6.2.1.b : The standalone reference to your "organized crime taxonomy" can be removed.

- Section 7

R33: Typo in the first sentence, "showS".

R34: Typo in title of 7.3, "Derefication".

R35: In general, a thorough quantitative evaluation of the articulated FrameBase schema creation process would be a very complex task because several of structural/representation choices must (and have been) done a priori and cannot be compared against alternative choices which haven't been realized. The overall quality of FrameBase will only be clear (and possibly improve) when a task-based evaluation will be viable, and the project is clearly not there yet. The current figures provided necessarily cover extremely specific steps of the process (e.g. the FrameNet-WordNet Alignment) whose global impact cannot be measured end to end, or they cover so small parts of the schema induction (100 frame pairs out of 18357) that cannot be considered meaningful. This is not authors' fault. Therefore, I would appreciate a real Introduction to Section 7 where they plainly motivate what can be evaluated in the current state, what not, and what makes no sense at all to be evaluated with a synthetic/sampling methodology. Instead, the whole Section tone sounds rather defensive in making extremely assertive quantitative statements whose overall relevance could be widely discussed, and I will not. My suggestion is to smooth this overall position in order to let the reader better focus on the real contribution of the FrameBase schema in its current stage.

R36: As a single examplar for the above discussion, I only mention the paragraph in 7.2 starting with "An average precision of 87.55%". It makes no sense to fire numbers before describing the evaluation setting and data. Please, at least bring back the ESWC version of that paragraph.

Review #3
By Valentina Presutti submitted on 10/Mar/2016
Major Revision
Review Comment:

The paper describes FrameBase: a new semantic web resource that provides a frame-base schema for representing knowledge. Together with the resource, the authors contribute with a method for automatically generating the FrameBase schema, as well as an alignment between FrameNet and WordNet. The claim is that such resource can be used for allowing easier integration of multiple heterogeneous sources. The authors support this claim by showing examples of integration rules, and by producing actual integration with some of the most popular semantic web knowledge bases, such as DBpedia, Freebase, and Yago.
An evaluation of the work is also discussed as well as potentialities and limits of the current status of the work.

Overall comment:

The article extends a conference paper, and I consider such extension valuable, as far as suitability for publication on the semantic web journal is concerned.

Although there are some aspects that need clarification and improvement, the work has merits in its potential impact on future research in semantic technologies. The resource alone deserves to be disseminated in a more complete article than a conference paper.

The main weaknesses of the paper are its presentation and the evaluation (at least in the way it is reported), which motivate my request of major revision. I think that the community should have detailed insights and easy understanding of the work, hence presentation is key.

I also recommend to have a native speaker to proof-read the paper.

Often, the author use terms such as "above", "below", "later", to refer to examples or formulas. I recommend to embed such examples/formulas in some environment that allows for precise references (e.g. number for the forumlas and the examples). Such current generic references make the paper hard to follow.

In general, the paper would benefit from including more examples. The ones (not few) provided need to be better included in the narrative. They should immediately follow or precede the more theoretical/formal text, which explains their underlying concepts. Some complex concepts are described only abstractly and in-text, while a narrative such as: informal definition -> example -> formal definition, would make the paper much more readable and effective.

In addition, the paper sometimes lack details. The authors should write as if the reader wants to replicate their work, hence providing her with enough details to succeed.

In the current state, a non-NLP-expert would find some passage very hard to understand as there are certain things that are not explained, sometimes even not cited (see detailed comments for specific references). I recommend the authors to do their best to make the paper self-contained as the target readers of this journal are not always NLP experts.

Section 5 and Section 6 need restructuring. In Section 5 the authors may want to have a paragraph for each text-design issue and associate it with the relevant examples, rules/solutions and their explanation. Section 6.1 and 6.2 should be merged and restructured in a similar way.

I think it would be important to have a discussion about the coverage of FrameNet and it's impact on FrameBase, especially during integration with external KBs. Have the authors met some situation where FrameBase schema would not align to some class or property in one of the KBs? How did they deal with it? Is this a marginal issue (i.e. it happens very rarely)? Whatever they have to say about this aspect is relevant and interesting to the reader (at least this one).

Detailed comments:

Table 1 needs to be accompanied by explanation in text, and should also include the roles modelling, which is later discussed.

p.5: Provide examples of CVT

Does CVT stands for "compound value types" or "composite value types"? At p.5 both are used, and it's confusing.

p.5: the following paragraph needs rephrasing and must be accompanied by an example
"While CVTs do not represent
frames or events per se, from a structural perspective,
they can be regarded as isomorphic to a neo-
Davidsonian representation with specific roles (see
Table 1). However, Freebase places a number of
restrictions on CVTs. For instance, they cannot
be nested, and there is no hierarchy or network of
them that would for example relate a purchasing
event to a getting event."

p.5: A more clear definition and an example of a FrameNet frame should be included in the paper as early as possible, considering that FrameBase's core inspiration and principle are linguistic frames.

p.5: When citing references 17, 32, 33, the authors may want to elaborate more on the synergies and differences between these works and FrameBase.

p.6: The first paragraph of Section 4 would benefit from rephrasing

p.7: When discussing LU disambiguation, please show an example to explain the process and the design choices (e.g. one-to-one mapping)

p.7: The S(l|a,b) function should have a formalised definition and be accompanied by an example.

Section 4.2: It's not clear why the authors use and refer to RDFS instead of OWL. They should justify this choice. Especially considering that it's clear that they are exploiting OWL semantics (also when using rdfs:subClassOf and certainly when using transitivity and symmetry). The description of the schema semantics must be rigorous, especially considering that the main purpose claimed by the authors is KB integration.

As for "perspectivization" the author should elaborate more on this concept and explain its semantics, and then motivate their design choice. owl:equivalentClass seems to me more appropriate than rdfs:subClass in certain cases. Also, it is not evident how the inversion of roles work: is this information provided by FrameNet? Is it extracted in some way?

p.8: Figure 1 should be closer to the text referring to it.

p.8: Please provide examples of LEMON annotations.

In the state of the art section and/or at the end of Section 4, the authors should consider referencing BabelNet and aligning this resource to FrameBase as it includes WordNet and address multilingualism:

"R. Navigli and S. Ponzetto. BabelNet: The Automatic Construction, Evaluation and Application of a Wide-Coverage Multilingual Semantic Network. Artificial Intelligence, 193, Elsevier, 2012, pp. 217-250."

Section 5:

The work on ReDer rules has some similarity with
this similarity, although the two works target at different goals, showed in several parts of the paper. The authors may want to look into this article and consider commenting on synergies in the related work section.

p.9: Explain the DBP acronym the first time it's used.

p.9: Figure 2 is presented as a dereification rule: doesn't it show a ReDer rule?

p.9: when showing examples of ReDer rules, the author should comment and explain in detail what is happening there, also by referring to how the terms of the rules are identified (e.g. by saying that the FrameNet frame is "Statement", the LU is "write", etc.).

Section 5.2:
I suggest to start the explanation of ReDer rules with an explained example.

p.11/12/13: all the examples should be spread within the text accompanied by detailed explanations.

p.11: please clarify what the dots are for, e.g. :frame-Forming_(...)-divorce.v

p.13: what prep, cop, dobj, nsub, are for? They should be defined and explained in detail.

p.13: please, cite and explain Kuhn-Munkres algorithm.

Section 6:
- the first two paragraphs are really hard to understand. Please, provide a formal definition for integration rule.

- it is not clear if integration rules have been defined manually. If so the authors should provide an estimation of the effort needed and the criteria for selecting/discarding the entities involved.

- p.15: the discussion about more specific and more general elements is unclear. It's not clear also what sentences such as "the substitutions for ?f that fire the rest of the examples..."

A large part of Section 6 is devoted to discuss limits/problems of integration. The authors seem not to have a viable solution at the moment to such problems. Hence, they should move this discussion in a dedicated section, preceding only the concluding section. Such a section should be structured in order to discuss limits, possible solutions, and the results of evaluation.

Considering that support to integration is the main claim of the authors, it sounds odd to read that tranformations are hard to do automatically. The authors should better support their claim, or make it less strong, in the light of this consideration (I'm referring to the discussion in Section 6.2)

Suggestion: As far as difference in representations choices is concerned, don't the authors think that conventions and guidelines on how to use FrameBase would help reducing this problem? If they agree, providing such guidelines would enrich their contribution and make it stronger (especially for supporting the integration-support potential).

Section 7:
The evaluation needs to be better presented, more rigorously and more detailed.
What are the evaluation tasks? What is asked to raters? What are their guidelines? What measures are used (define them, indicate intervals and their meanings), what precision/recall means in the context of each task?

Please, use tables for presenting a summary of the results.

Only two raters, being authors of the paper, where involved in the evaluation. This is a bit weak per-se. But even if one wants to overlook this weakness, at least the procedure followed, how they solved disagreement, and the degree of agreement should be reported.

As earlier mentioned, the authors should add a "Discussion" section, where they comment on the evaluation results and elaborate on the limits of the approach (e.g. moving part of Section 6).


I suggest to use the present tense instead of the future.

The author should avoid to use "so-called": for example at p.5 "so-called neo-Davidsonian representations" should simply be "Davidsonian representations"

p.2: will me -> will be

p.4: I suggest to remove Footnote 2 and include its content within the main text

p.5: adapt -> adopt

p.5: the semantic role -> their semantic role

p.6: adopted -> adapted

p.6: an reasonable -> a reasonable

p.6: might be reified on its own: strange phrasing

p.6: (3.2 b)) the mappings -> their mappings

p.6: for into -> into

p.6: the KB -> a KB

p.7: connected -> interconnected

p.7: semantic pointers -> semantic relations

p.7: better match -> to better match

p.7: the particular events -> specific events

p.9: Figure 2 and Figure 3 have the same caption

p.9: avoid sentences such as "perfect annotations" unless you can provide a contextualised definition for "perfect"

p.9: -s, -o -> -S, -O