Experiential Observations: an Ontology Pattern-based Study on Capturing the Potential Content within Evidences of Experiences

Tracking #: 2761-3975

Authors: 
Alessio Antonini
Alessandro Adamou
Mari Carmen Suárez-Figueroa
Francesca Benatti

Responsible editor: 
Special Issue Cultural Heritage 2021

Submission type: 
Full Paper
Abstract: 
Modelling the knowledge behind human experiences is a complex process: it should take into account, among other characteristics, the activities performed, human observations, and the documentation of the evidence. To represent this type of knowledge in a declarative way means to support data interoperability in the context of cultural heritage artefacts, as linked datasets on experience documentation have started to appear. With this objective in mind, in this paper, we describe a study based on an Ontology Design Pattern for modelling experiences through observations, which are considered indirect evidence of a mental process (i.e., the experience). This pattern highlights the structural differences between types of experiential sources, such as diaries and social media contents, providing a guideline for the comparability between different domains and for supporting the construction of heterogeneous datasets based on an epistemic compatibility. We have performed not only a formal evaluation over the pattern, but also an assessment through a series of case studies. This approach includes a) the analysis of interoperability among two case studies (reading through social media and historical sources); b) the development of an ontology for collecting evidences of reading, which reuses the proposed pattern; and c) the inspection of experience in humanities datasets.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Major Revision

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 14/Jul/2021
Suggestion:
Major Revision
Review Comment:

In this paper, the authors study the modelling of knowledge behind human experiences. Traditionally, semantic models to represent experiences (understood as a personal perspective of the person “owning” the experience) have been simplistic and have adopted a pragmatic approach, in which experiences are merely represented but no further described in semantically meaningful ways. This hinders interoperability and comparability between experiential sources, which are of paramount interest in cultural heritage and digital scholarship. To address this, the authors propose an ontology design pattern called Experience & Observation (E&O), a content pattern that models the observation of experience as a result of a direct engagement of a person in an activity, and reflected in an observation prompted from the person engaged in that activity. The design of E&O is fueled by the requirements of 3 competency questions, and contains various classes and properties that exist others previously proposed. The pattern is evaluated through a set of SPARQL queries that give answers to the competency questions; a usability evaluation; a number of applications of E&O in various Digital Humanities projects (READ-IT, and retrospectively to RED and LED); and an analysis of 9 recurring configurations of the pattern, which can be seen themselves as (smaller) knowledge patterns.

The paper is extremely well written, and is highly relevant to the special issue as it proposes semantic models to formalise an area of knowledge that is a constant subject of discussion among digital humanists and cultural heritage researchers. Many of its design choices, like the n-ary relationship ‘Engagement’, unveil unexpected but very much needed new views on how individuals interact with activities, and how the depth and intensity of such interactions lead to ‘prompts’, engagements that produce actual observations and which are fundamentally different from those that do not. In addition, the authors do a really good job at publishing all their resources on GitHub, assign namespaces, etc. to ensure that the pattern can be reused by others. So, in this sense the paper is exemplary.

However, I think the paper has a number of limitations that do not make it suitable for publication in its current state. Primarily those are:

- Rooting in the state of the art. The paper makes a number of interesting descriptions and statements about experiential observations and evidences, but too few of these have a backing reference. The paper is also very verbose on those theoretical definitions; while this works in some parts of the paper to give the reader an ‘intuition’ of the subject matter, I feel more formality is needed here. Especially in Sections 2 and 3, where authors survey existing ontology design patterns that partly cover observations, activities, transitions, etc; I would have certainly expected a much broader survey with theories from afar fields, e.g. psychology, neuroscience, sociology, philosophy, etc. that could help the reader understand a variety of points of view over what others have thought about experiential observations; and how those views can be condensed and unified under a consistent ODP. I thought the work at [1] does a really good job at precisely this, i.e. gathering, summarising and using a small survey for the purpose of developing an ODP, and I think a similar effort is needed in this paper.

- Evaluation. While I deeply appreciate the effort that the authors have put into evaluating the ODP in several settings (CQs, user study, actual application, recurring configurations), I missed a clear and explicit set of research questions that drive this specific evaluation choice. For example, while I could somewhat assume that the goal of the CQs, user study and actual applications was to prove that the design of E&O is correct, useful and applicable, I could not understand what the motivation behind recurring configurations is? I think this is useful, but an argument justifying its inclusion in the paper is missing. In addition, Section 4.5 does not mention what ontology design methodology was used to instantiate the pattern as an OWL ontology. Similarly, [12] is chosen as pattern evaluation methodology; but the paper does not argue why this particular methodology was chosen instead of other, plausible alternatives

- Competency questions. I thought the explicit origin of these CQs (regardless of coming from project requirements, literature review, etc.) was missing. In addition, I think CQ3 is problematic because it assumes a “closed world assumption” on the usage of the CP. In principle, the OWA would make it possible for activities that do not explicitly have observations to be instances of the class Prompting as well. This extends to Listing 3.

- User study. While the reader appreciates the transparency of this section, the results of this and their interpretation are not clear. What is the background of these students? What is their knowledge of e.g. digital humanities and text criticism? I feel the results of the survey could have been very different depending on the background of these students. A figure summarising the survey results could potentially help on this

Minor issues:
- Many footnotes (e.g. PROV, CIDOC, Transition Pattern, News Reporting Event, Social Reality, etc.) should be references to the extent possible
- Fig. 5 should include namespaces in its classes and properties
- Section 4 would benefit from a running example instantiating the pattern
- Namespaces: http://modellingdh.github. 33 io/ont/odp/term/ and http://ontology.eil.utoronto.ca/icity/ActivitySpecification/ should perhaps be moved to purl or similar for greater stability?
- “It is customary (...) to reformulate these as queries in the SPARQL language” Customary according to? I think a methodological reference is needed here
- I missed a clear reference on whether the pattern has been inserted into the ODP wiki (http://ontologydesignpatterns.org/wiki/Category:ContentOP)
- In Listing 2, the authors may want to include the person related to the observation?

[1] Tiddi, I., d'Aquin, M. and Motta, E., 2015, October. An ontology design pattern to define explanations. In Proceedings of the 8th International Conference on Knowledge Capture (pp. 1-8).

Review #2
Anonymous submitted on 27/Sep/2021
Suggestion:
Major Revision
Review Comment:

The article presents an ontology design pattern intended to cover experiental observations – i.e., the data/observations that describe the experiences people have had while consuming or partaking in some cultural artefact/activity. Examples given by the authors include how a listener may think that a particular song that they are listening to is “groovy”, and either they or an independent observer may want to note this down in a formal manner.

I am torn on the feasibility of such modelling, and particularly, of at-scale data integration of such experiences – the complexity inherent in building a formal model over such an ambiguous area, so tightly interconnected with and dependent on individual human variance and experiences, seems to be overwhelming. Any model of this must by its very nature be a simplification – hopefully a useful one. I suppose that a generic ODP might be a useful approach to try to capture that simplification.

The paper is language-wise very well-written, but in parts very dense. I would personally have preferred some more illustrations describing the rather heavy concepts discussed in the first four pages (which I had to re-read several times to fully grasp). Some of that rather dense text is packed with beautiful analogies, e.g., “ontological coordinates in the topology of an experience” – but some of it would benefit from more tangible examples and perhaps slightly less convoluted language (e.g., large parts of Section 2).

The main issue with this article as I see it is that the first half builds up a substantial expectation; a grand challenge of AI is about to be tackled, how to formalize and describe the inner cultural/emotional notions of experience; and that the latter half does not actually follow through on this grand promise. The latter half describes the E&O design pattern, which has a reasonable but certainly not blindingly clever design, and covers in some detail its individual relations, before describing a functional suitability and usability evaluations, and some cases of the ODP being applied with legacy data. This evaluation portion of the paper is the least convincing; the user study is lacking in quantitative analysis (sample size is not mentioned, nor is generalizability), and the case studies seem to indicate the authors themselves utilizing the E&O pattern to represent data, rather than independent modellers being tasked to utilize it. However, this is unclear, as many relevant characteristics of the cases and processes adhered to within them (e.g., how was the modelling team composed; what were their experiences/expertise; how were design patterns selected; etc) are lacking from the description.

For this article to be accepted for publishing, I propose that major revisions are needed, particularly as pertains to the evaluation. The quantitative evaluation needs to be made much more rigorous, and the case studies need to be re-interpreted and rewritten as case studies, with an emphasis on clear and transparent descriptions of case context. Until then, while the goal of the work is ambitious and interesting, I fear the significance of the work is too low.

Semantic Web Journal review criteria:
Originality: High
Significance: Low
Quality of writing: High

Review #3
By Valentina Presutti submitted on 27/Sep/2021
Suggestion:
Major Revision
Review Comment:

The paper addresses a theoretical problem with a high intrinsic complexity: representing the effect of subjective experiences such as reading a book, listening to music, or admiring a painting. The authors claim that this pattern enables interoperability between different research use cases and show how this can be achieved by means of refactoring the models of two existing ontologies, from two projects that are analysed in a retrospective fashion.
I think the proposed pattern is interesting and has potential reuse, but there are some important issues to be addressed to make the paper ready for publication. As for the interoperability between research use cases, I fail to see a special point as this is the role of all ontologies and ODPs, when they address general concepts such as this one.

The main issues that are reflected in my suggested decision are:
- key concepts are not clearly and rigorously defined, until the pattern is presented
- readability: the text is far too complex than needed and too abstract: I believe the paper could be reduced of 1/3 and made much clearer with examples using concrete data samples. the authors dedicate too much space to the three projects READ-IT, RED, and LET
- missing relevant work that should be discussed and considered for reuse
- the usability evaluation presents some concerns

General comments:

By reading the paper I felt frustrated because I could perceive what is the problem addressed, but never had a confirmation or a clear picture of it, because the text is very hard to read: very long sentences, mostly theoretical / abstract concepts, lots of metaphors, synonyms. I recommend to minimise the amount of metaphors and synonyms for terms that are core to the definition of the problem and of the solution.
The paper would greatly benefit if the authors would add a running example with samples of data that one would like to produce as compared to what can be done with current means.

For example: "...a Classical
Studies scholar comparing the reception of Roman
texts based on marginalia in a Latin codex with social media from online reading platforms, without requiring any understanding of, for example, the differences in reader profiles and reasons for reading classical literature." -> this does not give any intuition to me about the problem at hand.

Another example: "having a common ontology of experience is limited to supporting the interoperability of research outputs (e.g., annotation of sources), while the interoperability of research requires addressing sources as a key factor in the design of a case study." I don't get the problem raised here. Concrete examples are needed.

The authors rephrase their goal and the addressed problem many times in different parts of the paper. Every time the reader (at least myself) changes its understanding, which is very annoying. Again, please use a running example and simplify the explanation of the problem so that it becomes easy to both appreciate its importance and to evaluate the proposed solution.

The two case studies presented are interesting, but it looks like the authors present the projects rather than use them to demonstrate a concrete usage of the ODP: the part dedicated to the projects is far too long and does not even include examples with concrete data.

Some existing relevant ontologies and patterns are missing in the analysis of the state of the art. It is not only a matter of citing them but to discuss their relationship with the proposed pattern and motivate why they are not reused in this context (see details below).

The authors should reflect on avoiding reusing single entities from a ODP, but to reuse it as a whole as a template or by taking inspiration from it, if the whole modelling solution does not fit their requirements.

Detailed comments:

Abstract: "To represent this type of
knowledge in a declarative way means to support data interoperability in the context of cultural heritage artefacts, as linked
datasets on experience documentation have started to appear" this is referred to modelling human experiences: why this should only required interoperability in CH and not in other domains? Maybe the title should clarify that the authors refer specifically to experiencing CH.

Intro/Motivation:

The sense of the term "observation" in this paper must be clarified, as early as possible. I associate this term, in the context of ontology engineering and science in general, to a set of measurements, hence to an objective view. See for example [pattern observation]. Then, more than one "interpretation" may describe that observation e.g. according to different theories or methods.

"E&O models the observation of experience as resulting from a direct engagement in an activity and reflected
in an observation prompted from the person engaged in the activity": the term "observation" here is used twice. Are you referring to same concept? Does it mean you model a feedback from a person that experienced an activity? See my previous comment.

The authors refer to "sources", without defining what they are. They use many terms, possibly as synonyms, without clarifying their meaning in this context, hence causing confusion to the reader. Some sentences need to be read twice or three times as they are more complex than needed: I recommend to use shorter sentences and go straight to the point, and avoid synonyms to refer to concepts that are key to the domain, unless they are previously defined.

In Section 1.1 the term observation is again used to refer to the information content of a source, which is different from the "feedback" from who engages in an activity: the way it was (to my understanding) used before.

Figure 1 should be explained and commented. From the caption: "..opportunity to express the observation (evidence)." I do not understand this definition of "evidence". Evidence usually refers to facts or information that allow to infer the truth value of a proposition.

Both the introduction and the motivation section need to be enriched with examples. Reading them is difficult as everything is described at theoretical level with terms that can sometimes be ambiguously interpreted.

At the end of Section 1.1 the authors say that the aim of this pattern is to support data interoperability between research use cases. This comes out of the blue, what type of research use cases are they talking about? My impression is that they have specific examples in mind but do not define them properly.

Section 2/Backgroung
"sources of evidence of experience" -> what is that? can you make an example

In Section 2.1 finally there's an example that clarifies the problem that the authors have identified. They refer to "underlying condition of sources", which in my opinion is not a good way of defining what they are looking at. More simply, if my understanding is correct, I would refer to implicit knowledge that scholars may have and use as background knowledge for interpreting the content of a source.

Again in Section 2.1, after the example, the authors jump to the issues of data-driven approach to interoperability. I cannot see the connection with the previous text and it is not clear to what integration they are referring to and what they mean with "facets" of a phenomenon. Figure 2 and its caption provide a better explanation than the text used in the paper. I recommend to move the caption in the main text and to make sure that a possible running example (a very simple one) focuses on the reuse of data among two different use cases.

"Intuitively, human experience is the result of reflection while, similarly, the generation of an evidence requires also a reflection on the reflection which can, for example, trigger a revision, and therefore a change in the experience." -> I do not find this intuitive. What does it mean to "generate an evidence"? Again, to make it intuitive please use examples or provide clear definitions.

In Section 3 the authors reports on the SoA of ontologies and models targeting observations and other related concepts. From their analysis it is clear that the term "observation" is used in current literature to refer to phenomena that can be measured. The term observation is ambiguous in natural language, but as this is an ontology engineering paper I would recommend to use a specialised term e.g. subjective observation, or something better/different, to distinguish from existing, spread use of the term and avoid confusion.

The authors claim to reuse a class of a pattern because "it provides the necessary elements for representing temporal distance and significant events (as change of states) reflected within the observations." it is the ODP that provides such specification, not the class. I mean, ODPs shall be reused as whole, by using them as templates and then possibly extend them to a specific case.

I think the authors should review the Description and Situation ODP and discuss its relation with their work, and why this ODP is not reused as foundational reference for it.

Another relevant ontology that is missing in the SoA analysis is ArCo, which includes an Interpretation pattern, which is instantiated for modelling information that led to some conclusion/thesis e.g. about authorship attribution of an artefact.

Section 3.7
Figure 4 is of no use to me. The concept of prompting is suddenly used without any explanation/definition but a reference to the figure, which does not provide any formal or rigorous definition of it. Nor the Figure is explained or commented.

Section 4

I disagree on not including a class "Experience" in the pattern or at least the motivation provided by the authors does not convince me. Having a class does not mean that the experience is captured by a single entity. All the entities that contribute to its constitution will be related to it and be key components of the ODP. The authors should discuss why an experience cannot be modelled as a frame (e.g. by specialising Description and Situation), which is a complex structure by definition. A frame "Experience" would include a named class to identify it uniquely, which is something that should be possible, in my humble opinion.

Figure 5: what is the meaning of the dashed line?

Please clarify and simplify the description of the "Prompting" class.

The axiomatisation of the ODP should be fully reported in the paper.

I don't see how this pattern addresses the contextual information that influences observations, which is discussed as key part of the problem in the previous sections.

Functional completeness: SPARQL unit tests, such as the ones used here, should be run with sample data. The authors refer to existing datasets that encode information that may be compatible with this ODP, once properly transformed. Was this considered? Using sample data helps to identify possible gaps in the model.

Consistency: did the author check the consistency of their ODP by importing the reused entities?

Usability of the ODP: from the user study it emerges a difficulty in understanding the different concepts and the need for examples of usage. The population involved in the evaluation was not expert in ontology design, I think that this is a flaw in the evaluation. This is also further demonstrated by the fact that 1/3 of them reused individual entities, which is not a good practice. I appreciate that the feedback was useful for improving the documentation, but if the authors want to have a conclusive analysis of the usability evaluation should run a new study with ontology-expert users.

It is unclear what "reusing a class in different fashion" means. Do the author accept that a different meaning is assigned to the ODP classes, in different contexts?

Section 6
In all figures, the individual e1 is repeated twice. the examples are generic, please model concrete scenarios, e.g. by using sample data from the READ-IT datasets.

Minor:

using such tools as Protégé -> using Protégé
"While evaluating an ontology design pattern shares
some methodological elements with the evaluations
of ontologies tout-court, the associated quality criteria
and evaluation framework lie on even less trodden
ground than ontology evaluation itself." -> I'd remove this sentence.