Improving Readability Of Online Privacy Policies Through DOOP: A Domain Ontology For Online Privacy

1. Originality

Originality is moderate. Several privacy ontologies have been proposed, and several studies on identifying key aspects of the policies exist. The main contribution seems an automated and ontology-based way of highlighting the parts of privacy policies that more directly relate to user concerns. Lack of details makes it difficult to assess more precisely the novelty of the approach.

2. Significance of the results

From a technical and scientific perspective the paper has a few important drawbacks.

a) It does not cite all related work. There are several privacy ontologies in the literature. Some of them might even have the level of abstraction required in this work (although the authors say they don't know about previous attempts to capture the vocabulary of privacy policies). So at least the following should be discussed in related work:

- the HL7 ontology http://wiki.hl7.org/index.php?title=Security_and_Privacy_Ontology

- Towards an Ontology for Privacy Requirements via a Systematic Literature Review. ER 2017: 193-208
[this paper has also an extensive set of interesting references]

- "On the application of ontology to privacy protection", in 2016 2nd IEEE International Conference on Computer and Communications (ICCC)

- Privacy Ontology Support for E-Commerce. IEEE Internet Computing 12(2): 54-61 (2008)

- Privacy Support and Evaluation on an Ontological Basis. ICDE Workshops 2007: 221-227

b) The DOOP ontology is not described in detail, and apparently it is not available, so it is not easy to assess it. Apparenty it does not cover some of the crucial aspects of privacy policies, such as which data are collected, for what purpose, how are data processed, where are data stored. All of these details have become mandatory by the GDPR and most of them occurred in relevant vocabularies, such as P3P's and ODRL's. It is not even specified in which language is DOOP encoded (RDFS? OWL? other?). All the paper provides about DOOP is numbers, but even from such incomplete information the impression is that 15 classes and 35 individuals do not suffice to capture the main privacy related concepts (cf. the size of the vocabularies of P3P and ODRL, for instance).

c) The three queries/competence questions on p.5 do not cover other natural privacy related questions, such as:

- which data are collected about me?

- are they anonymized?

- why are they collected? what is the web site doing with them?

The criterion for identifying and selecting the CQ is not clear.

d) The paper does not explain how the ontology is used to extract/highlight the relevant parts of the policies. This makes it harder to understand how the experiments have been carried out and assessed. An algorithmic description of the extraction/highlighting process should be included.

e) The paper does not discuss the false sense of safety that may result from false negatives (i.e. the algorithm's failing to return relevant sentences). This may induce a user to accept a policy that in fact does not fulfill the user's expectations. Do you foresee any approaches to address this issue?

Further comments:

* Sec. 5.1: SPARQL queries might not be enough if the ontology uses a rich fragment of OWL, so it is important to explain better in which language is knowledge encoded.

* Sec 6.1, point 1: By law, the country in which data are stored is more important than the country in which an organization operates or is based.

* Sec 6.1: When you say that something is a "type" then it seems that the appropriate encoding should be a class, not an individual, because a "type" is a feature that may be shared by multiple individuals (such as "int" in a programming language, that denotes the set of all integers).

* Sec 6.3: In general I disagree that the nature of ontologies is individual and subjective. Please support your statement.

* Fig 4: supports the concern that DOOP might be too coarse grained

* page 8 and Sec 6.3.3: what does it mean that DOOP "returns keywords" ? And why does it return keywords that do not occur in the given policy?

* page 9, Sec.7: The numbers in the description of Experiment 2 do not match those in table 5. Moreover, instead of *assuming* that a paragraph is roughly 10 sentences, this should be verified by analyzing the actual policies.

* Table 5: did you check how many of the discarded sentences are actually relevant to the user's privacy question? This is relevant to investigate the risks associated to the false sense of safety.

* Table 6: shouldn't KF+NF be equal to the set of all keywords?

* page 10, point 2: what is a "document" here? why are policies repeated in a single document?

* end of Sec.7: table 12 and fig. 9 are not discussed

3. Quality of writing

English is correct, but the description of work is often unclear and lacks details.

This paper presents the design of an ontology of privacy policy-related concepts (mostly keywords and categories) and its use to highlight sentences that provide answers to privacy concerns in privacy policies. The motivation is mentioned as that web users tend not to read privacy policies as they are too long and complex. While that sounds reasonable, some reference would have been useful. There seem to be some validation of the idea that highlighting "important" sentences would help reduce the effort of reading privacy policies in the literature cited, but not really that it would encourage people to read them more, or that it would help comprehension.

The paper is well-written and easy to read, but lack details in many parts.

I'm not aware of other approaches that have attempted to do something similar, especially using ontologies, other than what is cited. The article could however make further references to approaches (some ontology based) to text classification, topic extraction, etc. that, even if applied in other domains, could have been tried and compared to here.

The methodology used to build the ontology is reasonably described, but the results and the details seem inadequate. While the related works are said to be applicable only to pre-determined classes of concerns, the ontology construction/validation is driven by only three competency questions that represent three very specific concerns. The resulting ontology is also clearly very simple (it is said to be consistent for 3 reasoners, but it couldn't be otherwise since not construct that could have led to inconsistencies has been used) and still not quite satisfactory. The idea for example that Cookie is a sub-class of keyword seems wrong. A Cookie is not a keyword. That instances of cookie are types of cookies is also very much wrong, if cookie is a subclass of keyword. It seems also to me to be lacking in precision to say that Policy Document is a subclass of Privacy Policy.

What is also not quite clear is how doing this with an ontology is actually bringing any advantage compared to simply mapping keywords to privacy categories (which seems to be the only role of the ontology).

The validation of the ontology/annotation is also not clear. It seems to find around 75% of the sentences it should. Is this good? There is not reference or comparison to any other approach. It also seems that such incompleteness would be an issue in the specific scenario addressed. That it would make people have less to read is one thing, but if what they have left to read is incomplete, wouldn't that be an issue? This particular point generally applies to the whole approach, i.e. that it is not clear whether it is (legally and socially) appropriate to automatically reduce privacy policies even if it means that it makes a difference in whether or not they read them. An evaluation of how much the reduction would affect comprehension would seem to be required here.

This paper is relevant for the readers of this journal because it aims to provide an ontology for online privacy policies that will help to reduce the amount of text a user has to read in privacy policies by selecting only the information important for the user. This research domain is underdeveloped in the sematic web community, so such papers, if prepared well, are very good contributions.

The paper is a good attempt to construct and evaluate a domain ontology for privacy policies, however, it requires restructuring and revision of all the content. Special attention should be paid to the revision of the sections that describe the process of ontology construction, ontology evaluation and results analysis.

The title and the keywords are appropriate. The abstract, however, does not provide a full and clear description of the content of the paper.

Several claims in the introduction require justification and literature support. For example, “…personal data is sold and shared frequently with third parties that use it to profile users and track them across domains”, “…provides a method to describe the vocabulary in terms of the privacy categories that are widely used by Federal Trade Commission and directives proposed by other commissions in Europe and Canada.”

There is an attempt to articulate a statement of novelty in the introduction, however it does not receive enough support throughout the paper.

A review of related work is too vague. The authors should provide a more comprehensive overview of the existing literature in this research domain.
The authors are using the term “privacy concern” without explaining what they understand under this term. This becomes clearer only on page 5. However, the authors do not provide information on how users will be able to communicate their consent or how they envision their system (“browser extension”) to identify users’ concerns to provide only relevant information to them.
More attention should be paid to the argumentation of a research gap. The statement “These findings suggest that highlighting relevant text with appropriate keywords can provide some feedback to users inclined to read shorter policies” cannot be derived from the argumentation above it.

The following claim in the motivation section “research shows that policies which highlight sections that directly address the user’s concerns tend to be read more often, as it reduces the reading cost” seems not to be based on the literature. If it is, the authors should provide the references.

The ontology engineering section should be merged with the methodology section, since the authors start describing methodologies already in the ontology engineering section. “Functions”, as “a special case…”, should be explained in more details. What does “special case” mean? The paragraph “There are many different types of ontologies that differ based on not only their purpose but also their content. The purpose of the ontology is determined by how widely it is meant to be used and the content is determined by the richness of the term definitions.” is redundant.
More explanation should be provided to justify why Ontology 101 and NeOn were selected by the authors for their research. It would be good to have an introductory sentence on ontology evaluation methods, because the second method comes as a surprise in the end of the section on ontology engineering.

The paper does not provide enough information on the whole process of creating this ontology. The authors also do not justify the decision to use the hybrid construction approach. The identified classes are based on FIPP and OECD principles. However, the authors omit the General Data Protection Regulation that substituted OECD guidelines.
It is not clear how the three competency questions used to construct the ontology were selected. The authors do not provide any argumentation for their assumption that the first question is the most common concern reported by users. If this claim is based on literature, the references should be provided to support that.

The authors do not provide any details on how the mapping between their 7 categories and the 10 categories of OPP-115 was done and what methodology was applied.

The following text is redundant in the section 6.3 because each of the experiments is described in their own subsection: “1. Correctness: Compute the number of matched privacy categories for the same sentences from both DOOP (based on the keywords the sentence contains) and OPP-115. 2. Policy coverage: Compute the number of sentences that the reader has to theoretically read to understand the risks associated with his concerns. 3. Completeness: Compute per policy keywords that existed in the ontology but not in the policy. 4. Correctness: Compute cases where the keyword’s assigned category in the ontology did not match the OPP-115’s annotation’s assigned category.” This text could be rephrased and integrated into subsections 6.3.1 – 6.3.4.
The names of the experiments should be adjusted because the reader can easily confuse the first and the fourth experiment, as both of them are named “correctness”.
The methodology for each experiment is missing in the paper.
It is not clear what is meant by “their” in the following sentence: “The results are presented as a CSV file for each privacy policy in the consolidation directory as indicated by their manual”.
To be consistent, the explanation of the column description “date” should be provided in the section 6.3.1 as the explanation was provided for all other column descriptions in the CSV file.
The authors mention „ score percentage“ when describing the columns of the Table 4, however, no explanation is provided what is meant by the score.

In the section seven of the paper the authors start to present some analysis based on the results presented in the tables (e.g.: “In Experiment 1 (Table 4), a mean of 76.16% match
for privacy categories with a standard deviation of 12.41 was achieved”, “This experiment demonstrated (Table 5) that the user has to read on average 11.09 sentences with a standard deviation of 14.70, or about 12.09% of a policy with a standard deviation of 16.06…” and so on) but it is not clear how they calculated the percentages and the standard deviation for each table. No such numbers could be found in the tables the authors refer to in the seventh section.
No details about additional analysis of the individual results that were mentioned in the seventh section are provided in the paper. Moreover, the authors list clues provided by “a qualitative analysis” but this analysis was not described anywhere in the paper.
The following sentence contains wrong numbers or a wrong reference to the table: “In general, the total number of sentences dramatically increased from 364 to 1503 (Table 8)”. There are no such numbers in Table 8.

Every figure and every table in the whole paper should be tighter coupled with the text. For example, Figure 1, Table 1, Table 2 are not explained in the paper at all. Some information is mentioned about Table 4 – Table 11, however, the authors do not elaborate on the results presented in the tables.

In addition to the above suggested changes, thorough proofreading is recommended to eliminate various grammar and orthographic mistakes as well as many inconsistencies (e.g.: “iteratively built in iterations”, “a corpus 631 privacy policies”, “OECD’s guidelines for (Organization for Economic Co-operation and Development)”, “by by defining as many axioms needed to answer”, “target audience, user access, and user access”, “for privacy policies already processed”, “coolie”, “Compute the the number” etc.). The major problems concern abbreviations. Although the terms and their abbreviations are common to the sematic web domain, one still has to write the full text version the first time they mention the term (OPP, SPARQL, CSV, OWL-DL, JSON, URL, etc.). The authors should readjust the placement of tables and figures. Table 12 and Figure 9 should be removed from the references.