Review Comment:
(1) originality: This paper is lacking in describing how it compares to other related work, especially those of Chari et al., who proposed an explanation ontology: https://link.springer.com/chapter/10.1007/978-3-030-62466-8_15 that models user-centered explanation types, that the authors' design style templates for.
(2) significance of the results: The results are comprehensive in an application suited for the cybersecurity domain. However, beyond the cybersecurity domain, I am not sure how the explanation templates would be able to represent explanations in other domains more generally. I think the authors should provide results in other domains to demonstrate the general-purpose capabilities of their tool.
(3) quality of writing: Overall, the manuscript is easy to understand, but the writing can be improved to reduce redundancy of content and writing style ( for example, by eliminating hyphens ).
This paper is yet another step toward an important contribution of providing the ability to allow users to populate user-centered explanation types from different sources, including knowledge base content and database knowledge. While this ability to populate explanations is not new and has already been covered earlier in the explanation ontology, the authors take a different approach in terms of formalism and define logical rules/style templates in Datalog to populate seven user-centered explanation types. I think the authors could include justification on why an ontology-based approach doesn't work well for their use case and why they choose Datalog instead.
They have also developed methods to combine explanation types based on findings from a user study. However, the results from the user study are not described beyond the content in Fig. 5. It would be helpful if the authors could provide more insight into how they chose three explanation types as fundamental ones to root the combinations.
The abstract and contributions mention that an important part of this work is the application of the explanation styles to the cybersecurity domain. However, the cybersecurity section of the paper is quite slim. Also, I cannot tell how the explanation styles are combined with content from the HEIST application framework. Hence, I request that the authors provide more examples and use cases to demonstrate the utility of their explanation tool. The use case section of the paper can be placed right after the technical descriptions so that the readers can apply their understanding of the tool to a working example.
I also find that the paper mentions related work peppered throughout. It can often get confusing to understand how the related work applies to the contributions in the paper. Instead, I suggest that the authors reserve most of the related work descriptions for the background and associated sections. On a similar line, I think the introduction has a lot of related work, reducing the emphasis on the main takeaways or need for the work. I suggest that the authors spend some time reorganizing the introduction to convey why explanations are needed in general in the cybersecurity domain, what kinds of applications in the cybersecurity domain demand explanations, briefly state why existing techniques don't work, and then move on to describing contributions. Also, from my understanding of this paper, the method developed is mainly for the cybersecurity domain. If that is the case, the authors should clarify that this is a domain-specific implementation in the contributions. If not, the authors should spend some time in a discussion section on how they expect readers to apply this framework outside of the cybersecurity domain.
Finally, I glanced through the code on the HEIST Github: https://github.com/jnparedes/HEIST. All the code files described in the paper seem to be present; however, due to a lack of a guide on how to run them, I am not sure how I can test the implementation. I request that the authors include details on this in their README file.
In addition to my high-level review comments, I also include a few remarks on specific sections of the paper below.
Comments:
- In Fig. 4, instead table 4, the FO column is not referred to in the text before it is mentioned here. Also, it is confusing to have to go back to understand what the columns mean. Consider expanding the abbreviations used in the columns in the caption.
- The findings from the study that yielded results on the valid combination styles for explanation types are not described well enough. It is hard to understand the patterns that the study participants were interested in from the results in Fig. 5 / Tab. 5.
- The related work section needs reorganization. There is a mention of ontologies in the data and hybrid explanations sections and not in the knowledge-based explanations section, where they belong better. Also, in the same section, there is a mention of the term network knowledge-based explanations that have not been introduced earlier. I think the definition of this term belongs in the introduction.
| 
