ImageSchemaNet: Formalizing Embodied Commonsense Knowledge Providing an Image-Schematic Layer to Framester.

Summary:

The paper describes ImageSchemaNet which aligns FrameNet and other linguistic resources with a representation of image schemas (including spatial primitives). The paper defines a semantic-web representation for image schemas based on a pre-existing ontology (ISSAC) and map a subset of common image schemas to FrameNet entities.

To create this mapping, the authors use queries over Framester as well as manual curation. To evaluate the mapping, they use off the shelf FrameNet parsers to predict activated IS(s) via the mappings to the FrameNet frames.

Quality and Relevance:

A persistent challenge using FrameNet is that frame elements do not have consistent meaning across the resource. Grounding FrameNet components in spatial primitives via image schemas could go a long way to alleviating that challenge and, I believe, has both practical and theoretical benefits.

The evaluation results did not convince me of the resource’s quality, but there are several factors that make it difficult to draw conclusions: The dataset only annotates for a single IS; the test set was curated; FrameNet parsing is unreliable. I do think these results could provide a baseline for future work, but I’d like more analysis of the resource itself e.g.

-The authors map 6 image schemas. How representative is this set? How many more do the authors expect to need? Are there any common schemas that were left out, and if so, why?

- What percentage of FN frames mapped to at least one image schema? Were there any similarities among the kinds of frames that the mapping approach missed?

- In mapped frames, how often were frame elements mappable to spatial primitives? Do those elements tend to be core or non-core? Is it possible for a frame to have IS mappings that are incompatible?

While the resource is publicly available, even some high-level discussion of topics like these would go a long way towards giving a sense of the quality/depth of the resource.

Clarity and Provided Data:

The paper is clearly written, and the appendix provides all queries used to construct the resource. Both Framester and ImageSchemaNet are publicly available and have an active endpoint. Building from existing resources like ISSAC also makes it easy to integrate.

Other Suggestions:

- Is there any indication of FRED/OpenSesame’s FrameNet parse accuracy for the test set? It’s possible that the resource is accurate but that the parsers performed poorly. Recent parsers based on models like BERT and T5 may give better results.

- The authors may be interested in work on “Embodied Construction Grammar” by Nancy Chang and others. Ditto work on “Decompositional Semantics”. Both ground FrameNet-style representations in a more primitive representation.

1 Quality and Relevance
=======================

This paper contributes an ontology for image schemas, ImageSchemaNet,
which is an image-schematic layer in the Framester hub.
ImageSchemaNet links to existing resources like FrameNet, WordNet, and
VerbNet. The authors describe the ImageSchemaNet and its vocabulary,
and perform an evaluation using OpenSesame and FRED (frame-based
parsers).

The authors include reusable queries and evaluation materials in the
Appendix. While there is not a README file, the authors provide links
to an endpoint which can be tested. Since the contribution is a
semantic web ontology layer, this is the appropriate resource.

2 Illustration, clarity and readability
=======================================

The paper is well written and organized. The authors provide a nice
overview of schemas and frame, although they may also want to include
Borchardt's frame representation that describe space transitions,
which is closely related to image schemas:
- Borchardt, G. C. (1994). Thinking between the lines: Computers and
the comprehension of causal descriptions. Cambridge, MA: MIT Press.

Some of the key points of the paper could be strengthened with
figures. For example, one of the strengths of the contribution is
that it extends FrameNet. One thing I was wondering about was all the
components of this IS layer. Perhaps the paper could be strengthened
by a flow diagram indicating the different components and
dependencies.

2.1 Suggestions
~~~~~~~~~~~~~~~

- In 3.1, perhaps having an example of how playing with shape puzzles
represents early experiences of CONTAINMENT. Perhaps the authors
could include a specific example.
- The 0's in the Figure 1 and two are confusing, perhaps they can be
blurred out or be a different color? Similarly, it would be helpful
to include some sort of total on top.
- One novel contribution of the paper is the explainability of
ImageSchemaNet. This contribution is mentioned a few times in the
paper, but the authors did not provide an example of the
explainability. Perhaps it is worth adding an example to the
Appendix.
- I think this paper could be strengthened with a more detailed guide
to building the resources (especially for readers that may be new to
semantic web technologies and contributions).

3 Questions
===========

- One thing I was wondering is about the role and impact of colloquial
language. For example, the authors go through the "bottling up"
example, but what about other idioms and colloquial phrases like
"their mind was racing?"
- When the authors describe the schemas, how can these be connected to
the existing work in frames? Is a schema in the image realm just an
abstract frame?
- What's the major difference between ImageSchemaNet and Image Schema
Logic? Perhaps this could be evaluated on in Section 2.
- The authors elaborate on the 99 sample sentences for the evaluation,
but it still seems to be rather small. Was this sample size
sufficient for the evaluation?
- The authors note that the distribution of sentences isn't uniform
across the image schemas. The authors describe that this is to
reflect the image schema frequency in the original dataset. But I
imagine this might introduce other biases, how are the authors
trying to avoid those?
- What happens to the IS-Annotated Sentences that are not successful?
- Is that from the second manual evaluation?
- Why is the most confusing image schema CONTAINMENT?

4 Typos and Writing Suggestions
===============================

- line 41 page 1: "Commonsense knowledge we deal." -> The commonsense
knowledge that we deal
- line 10 page 3, "to propose a sense cluster-based" might be missing
a preposition?
- line 10 page 8 "Activation assertions to FrameNet frame elements is
extended" -> Activation assertions to FrameNet frame elements are
extended
- page 13 line 40 paragraph indent missing?

This manuscript was submitted as 'Ontology Description' and should be reviewed along the following dimensions: (1) Quality and relevance of the described ontology (convincing evidence must be provided).

This paper provides an ontology for image schema using Framester semantics. The ontology can be queried from Framester's endpoint. Image schema is a common and important cognitive instrument for sentence making and creating language expressions. Hence, this newly created ontology can be highly useful and relevant to the readers of this journal.

I can not quite evaluate the quality of this ontology. See my comments below.

(2) Illustration, clarity and readability of the describing paper, which shall convey to the reader the key aspects of the described ontology.

While there are almost no typo or grammar errors, I found the paper quite hard to follow. This is because of 4 types of reasons.
1) Related work is discussed, but they are not compared with the current work. For example, from reading the paper, I cannot tell how this work is different from and superior to Image Schema Logic and ISAAC.
2) Some sections do not flow well. For example, in Section 2--Related work (line 17 on page 3), the current work is introduced as an experimental evaluation. In Section 3.1 (line 18 on page 4), the discussion on the definition of image schemas resumes after introducing the schemas implemented in this work. It is unclear how these works are related to the ontology created in this work.
3) Most importantly, I don't fully understand how the ontology was created. Based on Section 5, some of the procedures are manually done. For example, line 10 on page 7 mentions a manual revision. Line 25 on page 8 mentions a manual exploration, and a manual check of coherence. However, there are no further descriptions of these manual processes, e.g., who did it and the principles to be followed. Furthermore, the evaluation in Section 6 makes it sound like the entire process can be automated. So I am confused as to whether the ontology is created automated or with manual effort. It will be very helpful if a diagram can be provided for describing the overall ontology creation process.
4) Mintor writing issues. The term DOL (line 18 on page 4) is used without explanation. The link to Image Schema Database is broken (line 50 on page 8).

Please also assess the data file provided by the authors under "Long-term stable URL for resources". In particular, assess (A) whether the data file is well organized and in particular contains a README file which makes it easy for you to assess the data,
There is a readme file in the github repo with example queries. No data file is provided, but the entire ontology is accessible through Framester's hub.

(B) whether the provided resources appear to be complete for replication of experiments, and if not, why,
If the ontology creation process does not require manual work, then the provided resources should be sufficient for replicating the experiments. If manual processes are involved, then additional descriptions and guidelines are needed.

(C) whether the chosen repository, if it is not GitHub, Figshare or Zenodo, is appropriate for long-term repository discoverability,
yes.

and (4) whether the provided data artifacts are complete. Please refer to the reviewer instructions and the FAQ for further information.

The current work only includes 6 types of image schemas. While there are other types of image schemas, at least we have a clear idea of what is being covered.