Trusted Tiny Things: Making Devices in Smart Cities More Transparent

In this paper the authors present an approach that can provide users of IOT devices with information about how the data captured by the devices is being used, by interrogating the devices. Two mobile applications have been implemented: an NFC tag used at bus stops to provide a means to access real-time bus timetables, and a telemetry device installed into vehicles by insurance companies to track driving behaviour.

The paper is very well written, easy to read and articulates the problem it tries to address. However as it currently stands, it is weak in crucial aspects for a journal paper dedicated to SW/LD technologies - only 30% of the content could be considered relevant to the journal. Specifically the paper suffers from the following weaknesses:

(1) Limited related work: The paper lacks a dedicated section on related work. It is bundled in section 4 along with a description of the framework. Smart cities is now a highly researched area. There are several projects which have looked at the applications of SW/LD for improving user interaction with sensors within the context of a smart city. The authors need to provide references to these approaches and clearly compare why they believe their approach is better in terms of scalability, usability, the use of ontologies, making the datasets openly available besides other metrics which are relevant in this context. The evaluation should factor these things in as well. Right now, it looks like a conventional application developed over SW/LD. Either the paper should contribute to the state-of-the-art in SW/LD for IOT or it should improve the domain specific requirements
(2) The authors have developed an ontology which extends a few concepts from PROV and also provide SPIN rules for rule based reasoning however there is no reference to a serialisation of the ontology provided in the paper. Given that the ontology is one of the core contributions that the paper tries to make beside the applications, it should be made de-referenceable with proper documentation or at least made available as a Github repo. Three SPIN rules have been presented in the paper, but the authors mention five. These should also be provided within a Github. The authors do not mention anything about whether the datasets are openly available or if they plan to open them up for reproducibility. This should also be discussed. Further, it is not clear whether the approach includes integration of datasets from multiple sources
(3) Evaluation: The purpose of the evaluation should be to investigate how SW/LD can address a problem which is limited in the way it can be solved by other technologies. Instead the paper puts a lot of emphasis on describing the problem, detailing user studies and describing the complete application and evaluation setup. While these are of course important, it is hard to see the connectivity between how the ontologies address the questions raised. Specifically for experiment 1 about the use of IoT devices for public transport, the authors identify three questions they wanted to address. But these questions are very generic. How did the ontologies or the rules contribute towards answering these questions or how did SW/LD help in answering these questions which could have been limited if other technologies were used is not discussed. It is also hard to see the benefits in experiment 2, unless a scalability analysis is carried out.

Overall, the authors need to make it explicitly clear how their extended ontology has made a difference by showing its impact in their experiments and scalability. A separate section on related work that includes applications, other ontologies, the notion of trust as considered by the authors in this paper against what is perceived by the smart city community should also be discussed.

The paper describes a novel approach to representing provenance in IoT with the aim of providing trust to end-users through transparency. The paper is very well written and is sound from a methodological perspective. It considers existing ontologies and builds on these, describes an architecture and implementation, details realistic experimentation and evaluation (from both a user-centred and technological perspective) and looks at potential improvements and future research.
The presentation to end-users via the mobile app is well received.

However, in some areas, further information would make for a better paper:
1. I am intrigued to understand how reasoning results are presented to the user (and how they interact to allow/ deny certain activities). Further screenshots and/or additional text should be included.
2. The technical evaluation and the overhead of the adopted approach is also of interest and some discussion should be included of the overheads of a non-semantic approach (and indeed authors opinion on whether such an approach would even be feasible – perhaps not).
3. Finally, although the need to scale is noted it would be interesting to learn about specific activities that are planned for wider adoption including other routes to exploitation (e.g. standards activities, open source, etc.).

== Problem ==
The paper investigates the problem of trusts in the context IoT devices. As IoT devices are increasing and becoming more and more pervasive, making aware people about which data is collected, by whom and for what is used is getting crucial. The problem is relevant and, as shown in the rest of the paper, Semantic Web technologies can play a key role to cope with it.

== Solution ==
To tackle this problem, authors present a platform, Trusted Tiny Things, to manage the trustability of IoT devices. The idea is to annotate IoT devices with a description of their features, their owner, etc., and reason over them to let the user know what they do, and let the user decide if to interact with it.

The decision of what information to collect about the devices and how to use it have been elicited through a survey involving 77 participants. An ontology, T^3, has been built on top of PROV-O. The reasoning is managed through SPIN.

The paper does not provide any link to the platform, and in general, it is not clear if it is made available. This should be the core contribution of the paper, but it has already been presented in a previous publication of the same authors: http://ceur-ws.org/Vol-1280/paper9.pdf. The content is almost the same - the two sections are switched, but the text does not add any information and does not describe any new feature. I think that it would be possible to add more details, for example explaining the technical decisions (e.g., why SPIN to manage the reasoning? why TDB?).

== Evaluation ==
This section is not in the previous workshop paper. Two experiments have been conducted.
In the first, a group of users have been asked to use the platform in the context of public transportation. Authors were interested in understanding if users were wanted to have more information about the IoT devices they interact with, to control the data generated and to get feedback about the interface.
In the second, time performance is evaluated.

Authors use mainly percentages to analyse the results (in particular in the second experiment). However, percentages offer a partial view of the results. The lack of absolute values limits the understand of the obtained results.
Moreover, Experiment 2 is not reproducible, since the experimental settings are not described, and the data and software are not available.

To summarise, I believe that at the current status, this paper is not suitable for publication. The evaluation section, as is, is not enough to let the paper be published as a new contribution. Sections 4 and 5 should be extended to increase the amount of contribution, and Section 6 needs to be largely revised. It would be interesting to perform some experiments to study the scalability of the platform. Finally, the related work section should be improved, since at the moment the number of references is very limited (11 references).