A Review of Argumentation for the Social Semantic Web
This is a revised manuscript following an accept pending major revisions, now accepted. The reviews below are for the revision, followed by those for the original submission.
Solicited review by Fouad Zablith:
The authors made substantial effort in the revised version of the paper,
and is ready to be published. The narrative now flows better, and has been
rewritten more around the theme of Social Semantics. The separation of the
tools into an appendix made it easier to read.
Reviews for the original submission:
Solicited review by Simon Buckingham Shum:
This is a well written survey paper, introducing the reader to rapidly growing field of argumentation and debate on the web, with specific reference to how it relates to social and semantic strands. The survey demonstrates that this is not only a topic of interest in research labs, but also a pressing concern for many institutions who are developing web-based platforms to meet their own needs, eg. in education and government. I anticipate that this will become a standard reference for the field.
The paper orients the reader to the diverse theoretical backgrounds from which current initiatives derive, at an appropriately general level for a survey article. The reader will not understand them in any depth, but is made aware of relevant sources for more detail.
As appropriate for this journal, the paper pays particular attention to work on semiformal/formal representational schemes associated with modelling disagreement and argument, before providing a comprehensive survey of many tools. While some of these are demonstrator research prototypes no longer available, many of the systems are online for the reader to try, confirming the rapid growth in this field in recent years.
Several helpful taxonomies are used to classify the tools, which help distill the survey into overview form.
The concluding comments highlight some relevant issues that emerge, such as the tension between socially-oriented web apps with very lightweight ontologies, designed to make it as easy as possible for many people to contribute, versus more detailed schemes for serious analysts who need finer-grained control of their argument networks. A brief section on usability might relevant here, in which usability is defined, at least in part, with respect to the target audience. An e-democracy tool targetting the public at large will set the adoption bar very low, compared to a tool for in depth literature, philosophical, or intelligence analysis.
In this context, there is little discussion of adoption levels of the tools. The reader is not given many clues as to whether they are reading about a small-scale demonstrator or a widely used system. However, it is recognised that such data may in fact be unrealistically hard to obtain, and it may be beyond the bounds of the article to mine this data. As mentioned, many of the systems are online for readers to gauge adoption and usability for themselves.
However, having read about so many tools -- making web argumentation one of the largest and apparently rapidly growing fields in the social-semantic web -- the reader would value a closing comment on the maturity of this field. Would the authors argue that it has moved firmly from the academic lab, where it was 10 years ago, into the mainstream?
Detailed points:
Omitted from the survey:
* Dan Suthers' CSCL system Belvedere
* Michael Hoffman's work, using CMaps: http://www.spp.gatech.edu/aboutus/faculty/MichaelHoffmann
* Theodor Scaltsas mapping philosophical arguments:
http://www.philosophy.ed.ac.uk/people/view.php?name=theodore-scaltsas
Many of the figures and some of the tables are low-resolution screenshots which should be improved
The screenshots from tools are helpful but small, given the 2-column format. These should be high resolution to allow the reader to zoom in on the images in the final PDF, but the authors might consider providing a link to a set of slides with larger versions.
Strange chars ÔwinningO
Solicited review by Iyad Rahwan:
I read this paper with great interest. The topic is very timely, since the idea of large-scale argumentation support on the Web has been brewing in a variety of fields (AI, Semantic Web, decision-support, informal logic etc.). There have also been many attempts by developers to build systems to support Web-based debate. But no single authoritative review is available that summarizes what is available. So I thank the authors for taking up this task.
The survey is very comprehensive, covering pretty much any online system or relevant paper I could think of. The authors are to be commended for this effort; they did a great job on breadth.
Having said that, I believe the paper is not yet ready for publication. The main reason is that the survey is not analytical enough. To me, a high-quality survey gives, not only a comprehensive overview of the state-of-the-art, but also critical analysis that brings everything together in an overarching framework, making it possible to identify where the gaps and next major challenges are. While the authors have gone to some effort to provide such critical analysis in some parts (e.g. Table 1), I still think the papers needs to extend this kind critical analysis to other places. Below, I give some comments that shed more light on this issue, and outline some ideas for how to deepen the analysis.
In section 2, you summarize multiple theoretical models that form a basis for the different technologies out there. You look at IBIS, Toulmin's scheme, and Walton's schemes. However, it would be useful to compare these models, especially in terms of their potential advantages and disadvantages in supporting argumentation on the social Web. For example, Walton's schemes provide a finer grained classification than Toulmin's single schema. Is this good or bad? For what kinds of systems would one system be better than the other? This kind of analysis would provide useful guidelines for evaluating the suitability of these models for different applications.
The same can be said about section 3. Again, what are the pros and cons of these different linguistic approaches to understanding argumentation? Which of these models are useful for what kinds of purposes in social web argumentation? What tradeoffs do they provide? Can we build a system based on multiple such models?
In fact, I felt that the separation between sections 2 and 3 is a bit strange. In section 2, you look at theoretical models of what an argument is. It seems to me that coherence (3.1), speech act theory (3.2), the language/action approach (3.3) and rhetorical structure theory (3.4) all provide alternative theoretical models, though perhaps focusing on different levels of abstraction. But don't they belong in section 2 with the other theoretical models?
On the other hand, sections 3.5-3.8 are about computational linguistic methods for supporting argument annotation, discovery, etc. There are also bits and pieces of this in sections 3.1-3.4.
Given all the above, I think it might make more sense to dedicate a section purely to theoretical models (describing them in isolation from their applications). Here, you can talk about IBIS, Toulmin, Walton, speech acts, language/action, coherence, and rhetorical structure theory, all in isolation, and purely as theoretical models for capturing argument structure. This section could include some high-level comparison between these models. Later, when you talk about specific computational models/applications, you start drawing connections to the theoretical models underlying them. (e.g. IBIS RDF is based on IBIS, ArgDF is based on Walton's schemes, and so on). The authors could, of course, find an even better way to do this separation, but it's just one suggestion for making the breakdown more meaningful.
Another issue is the separation between sections 4 and 5, which I found a bit unnatural. The authors clarify (in 4.1) that those discussed in section 5 originated in the argumentation community and focus on representing arguments in more detail. To me, it would make more sense to have just one section, and to separate the models, for example, by the underlying theoretical model they use (e.g. Toulmin-based vs. IBIS-based, etc.).
As mentioned above, Table 1 is very good. But I think it would be good to expand a bit on the text in section 6. It would be useful to explain the table in more detail, explaining what the authors mean by things like polarity, taxonomic, concept vs relation centered, etc.
Now, let's move to section 7, in which you provide an extensive description of existing tools. It's clear that the effort that has gone into producing these summaries was enormous. However, the descriptions are a bit monotonous and factual. But I still think it is useful to have. Therefore, I suggest this entire section, together with the matrix comparing them, be moved to an appendix. It would then serve as a reference for people to look up a specific system (much easier than going to the web site and trying to figure out how it works).
Now, to replace section 7 (which I hope gets moved to an appendix), I think there is a room/need for a shorter section that compares ``classes of tools'' more broadly. One way to do this is to describe the general features that these tools must provide. Here's an example of these features (partly based on the comparison matrix), and each of them could serve as a sub-section:
- visualization
- annotation / representation style
- search
- social networking capability
Under each, you can describe a few example systems, and refer the reader to the appendix for more detail. This way, you would engage the reader more, rather than having him/her sift through a description of one system after another, only to provide an analysis of the features briefly at the end.
In summary, I think this has the potential to become a great survey, and most of the hard work has been done. It just needs a bit of shuffling around and some high-level contextualization.
MINOR COMMENTS:
The heading of Section 4.1 is unnecessary.
Page 10, first line of section 4.14, [99] [100] -> [99, 100]
Reference [107] is not published yet, so the year should not be listed.
The definition of WWAW, quoted from [5] is repeated too many times. Might be a good idea to remove a couple of copies and just keep the reference.
Solicited review by Fouad Zablith:
The article provides a survey about the argumentation field, in the context of the Social Semantic Web. The review includes a (a) theoretical overview of existing argumentation models, (b) approaches from the linguistic and communication theory, (c) work based on semantic web models and (d) and tools representing argumentation on the web.
The article gives a very good overview of the argumentation field, with theoretical background as well as applied models/tools. It will serve as a good reference for research in the argumentation field. The coverage of the survey is very good, especially at the level of argumentation tools.
The main issue that the authors should address is the focus of the paper. The focus should be more around the Social Semantic Web theme, and refers back to it whenever possible. There is good analysis performed in some of the sections (I highlight them below), but it would be better to put the aims of the review and motivation in terms of the Social Semantic Web at the introduction level, and use them as reference later in the article. It shows that the authors have made an effort to achieve that, but further work is probably needed. The article contains many key ideas that can potentially relate to social web. I will give some suggestions below.
One way to improve the presentation and analysis is to have a section about the social semantic web, and how the interaction of users around the web has been shifting from closed web systems, towards more open and interlinked social applications. Then argue how important it would be to capture the arguments of the discussions in such environments, along with the requirements needed to achieve this. The requirements can include: argument representation, elicitation, exchange, mappings, mining from text, authorship provenance, etc. The requirements can be used later in the paper as pointers to show how existing approaches fill such requirements, or fail in doing so.
It is worth having in the introduction some of key references showing that the research in the field of argumentation within the social web is getting more important. Some examples:
a- How Ritter et al. model Twitter conversations (A. Ritter, C. Cherry, and B. Dolan, "Unsupervised modeling of Twitter conversations," in Proceedings of Human Language Technologies: The 11th Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2010)
b- The work done about argument schemes for Amazon reviews (S. Heras, K. Atkinson, V. Botti, F. Grasso, V. Juliana, and P. McBurney, "How argumentation can enhance dialogues in social networks," in Computational Models of Argument -Proceedings of COMMA 2010, no. 216 in Frontiers in Artificial Intelligence and Applications, pp. 267–274, IOS Press, 2010.)
...
This helps in motivating the article and review domain.
- An introduction for Section 2 is needed. It could answer questions e.g. why are theoretical models of argumentation important? Worth mentioning that various approaches draw on Toulmin's work to model computer-based argument structures. Also that IBIS influenced several ontologies (it's good to see how the last paragraph of Section 2.1 connects to other parts of the paper, worth doing the same for later sections when appropriate).
- It sounds more natural to swap the order of section 2.1 and 2.2. As section 2.2 gives the kind of starting point and origins of the interest in argumentation, and how Toulmin's work fed into knowledge systems.
- Section 2.3 last paragraph: "For our purposes, "the Walton model" is that a dialogical argument uses one or more dialogue types and one or more argument schemes and has an opening, a middle (argumentation) phase, and a closing." More clarification is needed. What are you achieving through your choice?
At this level, you also talk about relevance, cooperativeness and informativeness. This sounds relevant to the social web, and might be worth to elaborate a bit more on this?
- Section 3, Introduction: "Approaches from linguistics and communication theory are relevant to argumentation in at least two ways." True. What is the impact on social web? E.g. generation of specific ontologies? As you mention that such approaches help in detecting relationships between texts, can you relate it to the social web? (Maybe inferring relationships between people? discussions?...)
- Section 3.3: "model online conversations to classify them and create visual maps, used for information retrieval." Worth relating to the social web.
- Section 3.4: The idea of nucleus and satellite spans of text, sounds important as well to the context of the paper, might be worth highlighting it? For example it could be a potential analysis of arguments running over tweets, what type of diversions might results out of core conversations, etc...
- Section 4.10, last paragraph: This is good! More of such arguments would make the paper even more interesting.
- Section 4: What are your main conclusions?
- Section 5.3: given that your main introduction and motivation starts with the WWAW, and about the need of having large scale argumentation systems, would you give the Argument Interchange Format (used in WWAW), a better edge over the other described formats to serve the purposes of the social web context? More reflections on the social aspect could be useful.
- Section 6: Again it would be easier to have a list of requirements in order to map and support your choice of the table 1 criteria.
- Section 7.2: Here you are talking about the scope of the tools, which is good to push as well to the introduction level, and make it clear in the scope of the whole paper.
- Section 8, before last paragraph: "To understand their current integration with the Social Web, we record whether they use a site-specific login, or allow external credentials (such as OpenID, Twitter, or Facebook). We further..." Here you listing some of the features you are interested in for the social context. Worth adding it to the top of the paper as well, and list it as part of your requirements.
- Section 9.3: needs refinement, also consider concluding the paper on the 3 interesting main conclusions presented in Section 9.2, which are probably more valuable than the categories of the argumentation tools.
Typos and other fixes:
- Some figures are not referenced in the text (mentioned below). Consider dropping some screenshots, if there's not much to say about.
- Abstract: Last sentence too long to parse.
- Section 1, paragraph 2, line 6: farflung --> far-flung
- Section 1, paragraph 5, line 1: "Yet, Social Web does not yet have widely-used argumentative ontologies" --> "Moreover, Social Web does not yet have widely-used argumentative ontologies" ?
- Section 1, last paragraph, line 4 from bottom: "In we compare these..." --> "In § 6 we compare these..."
- Section 4.7, line 1: DILGENT --> DILIGENT
- Section 4.11, paragraph 2, line 1: Figure 4.11 --> Figure 7?
- Section 4.11, paragraph 2, line 7: "Tools using SWAN-SIOC include as PDOnline (§7.32)" --> "Tools using SWAN-SIOC include PDOnline (§7.32)"
- Figure 6 not referenced in text?
- Section 4.14: Figure 4.14 --> Figure 8?
- Section 5.1 and 7.1: are the only introductory sections with title, either add titles to all other intro sections, or remove these two.
- Section 6, line 3: "Topics addressed include are whether each ontology is centered on relations or concepts and whether it is IBIS-like (contains concepts functionally equivalent to IBIS' 'Statement', 'Issue', 'Position', and 'Argument')" Needs rephrasing
- Section 7.1, paragraph 4, line 1: "This tools coverage in this paper differs..." --> "The tools coverage in this section differs..."?
- Table 1: Some columns can be made narrower for table to fit in the page.
- Section 7.5, line 3: ArgDB --> ArgDF?
- Figure 11 not referenced in text?
- Some of the referenced figures appear very far from the reference in text. Some figures can be made smaller to fit in one column, and appear after the reference?
- Figure 13 not referenced in text?
- Section 7.11, line 10: Avcienna --> Avicenna
- Figure 19 not referenced in text?
- Section 7.18, line 9: Figure 7.18 --> Figure 7.23?
- Section 7.21, paragraph 1, line 3 from bottom: [(1)] unclear if this is a reference?
- Section 7.21, paragraph 2, line 6: "for for people" --> "for people"
- Section 7.24, line 3: iDebate. Please add reference
- Section 7.24, line 5: Debateabase. Please add reference
- Figure 28 not referenced in text?
- Section 7.26, line 4: "populated by hand-annotation by activists". Please rephrase.
- Section 7.28, line 3: "in Figure 31 A tree view..." --> "in Figure 31. A tree view..."
- Figures 30 and 31 appear after figure 33 in text. Why not shrink them and put them within the columns?
- Section 7.30, last line: "Google Ware discussion bot Arvina show". Reference needed. [145]?
- Section 7.32, last line: "it is unique it that it uses scientific argumentation" --> Please fix
- Figure 36 not referenced in text
- Figure 7.38, Line 3 from bottom: Figure 39(a) --> Figure 39(b)?
- Table 3: Add label.
- Table 3: Funnelling --> Funneling (as used in the text)
- Section 9.3, line 12: "ÔwinningÕ", character encoding
- Section 9.3, paragraph 2, line 2: "peoplesÕ", character encoding