Deployment of Semantic Social Media Analysis to Call Center

Tracking #: 713-1923

Authors: 
Takahiro Kawamura

Responsible editor: 
Guest Editors EKAW 2014 Schlobach Janowicz

Submission type: 
Conference Style
Abstract: 
The number of inquiries to call centers regarding product malfunctions has been increasing in recent years, and thus manufacturers are struggling with their responses. The Consumer Affairs Agency in Japan stated that the initial response to an inquiry is especially important, since flaming directed toward the company may immediately occur on the Web, and may greatly affect the reputation and sales of the product if the response is inappropriate. However, when a call center accepts the first inquiry, an operator cannot determine whether the malfunction is due to a problem of a model or to a user's way of using the product. Therefore, we have developed a system to automatically determine if the inquiry content is the tip of an iceberg by graph-matching the inquiry content to a Linked Data network, which represents the reputation information of a product on social media. Moreover, by tracing causal links in the network, the system also determines if the inquiry is connecting to users' dissatisfaction and discontent, and then notifies the inquiry to a quality control section with high priority to prevent flaming. In this paper, we first present our approach for converting social media information to Linked Data, and show that an experiment achieved 94% accuracy. We also explain the matching between the inquiry content and Linked Data and its accuracy, and a method of extracting the causal link to the users' complaints.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
[EKAW] reject

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 01/Sep/2014
Suggestion:
[EKAW] reject
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
X -1 weak reject
== -2 reject
== -3 strong reject

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
X 3 (medium)
== 2 (low)
== 1 (none)

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
X 3 fair
== 2 poor
== 1 very poor

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
X 3 fair
== 2 poor
== 1 very poor

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
X 3 fair
== 2 poor
== 1 very poor

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
X 4 good
== 3 fair
== 2 poor
== 1 not present

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
X 2 poor
== 1 very poor

Review
Overall, the paper presents a system, based on the very real problem of dealing with call center complaints of products, to match complaints to a database of common answer. The system is in many regards a classic AI system that matches natural language complaints to a graph structure.

The paper has a few very good points - the problem is exceptionally relevant. The use of RDF "triples" doesn't really make sense in this context, since data is not coming from hetereogenous sources and thus URI-based disambiguation is not needed, and also since the format used is only vaguely related to RDF, being effectively a semantic net with some custom vocabularies. Thus, one cannot see how one could easily SPARQL the data-set. The results over nearly 200 questions show only a 20% total failure rate for the system (no matches) but only a 33% recall, which is quite low. This is likely due to issues with natural language parsing that should be corrected and the small data-set needed to build probabilities. Nonetheless, the system is interesting and this work should be pursued, and the authors should be congratulated for accurately testing their system (even with poor results) over real-world data.

Review #2
Anonymous submitted on 03/Sep/2014
Suggestion:
[EKAW] reject
Review Comment:

Overall evaluation
Select your choice from the options below and write its number below.

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
X -2 reject
== -3 strong reject

Reviewer's confidence
Select your choice from the options below and write its number below.

== 5 (expert)
== 4 (high)
X 3 (medium)
== 2 (low)
== 1 (none)

Interest to the Knowledge Engineering and Knowledge Management Community
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
X 3 fair
== 2 poor
== 1 very poor

Novelty
Select your choice from the options below and write its number below.

== 5 excellent
X 4 good
== 3 fair
== 2 poor
== 1 very poor

Technical quality
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
X 2 poor
== 1 very poor

Evaluation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
X 2 poor
== 1 not present

Clarity and presentation
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
== 3 fair
X 2 poor
== 1 very poor

Review

This paper introduces the idea of using a Linked Data graph to represent reputation information for product mentioned on online social media. Using a “Call Center” as the application platform, the authors developed a method for searching this “reputation Linked Data graph” for common issues across product makes and models. The input to this searching platform is information gained from complaints submitted through the Call Center. While quite interesting conceptually, the paper falls short in a number of ways.

The primary goal of this paper appears to be focused on matching complaints through telephone calls to product discussion on online social media and help forums. This appears to be a fairly standard natural language processing task which would simply look for commonalities between the products mentioned, the defects or issues mentioned and some temporal component. Approaching this problem by developing an ontology and setting up a linked data graph seems somewhat excessive to me.

In this vein, the development of an ontology was mentioned, but I fail to see how it was implemented and the set of predicates discussed (and shown in Figure 4) are quite vague. For example two social media or forum entries are linked through the “Because” relationship. “Because” signifies a directional relationship yet no direction is depicted in the graph. Additionally, vague temporal units such as “Summer” are misleading. Since this paper reflects an actual working prototype of a system, I would expect to find a more detailed explanation of how properties such as “Time” are actually implemented.

Lastly, the strength of the system appears to hinge on the Call Center operators' ability to accurately triply a caller's complaint. The authors report a Triplification Error of 13.6% indicating that the vast majority of call center operators had no problem converting a complaint in to a triple upon which the Linked Data graph could be queried. I would like to see much more of an evaluation section as it relates to how human operators were able to so consistently enter strongly typed < S, V, O > relationships.

Review #3
Anonymous submitted on 03/Sep/2014
Suggestion:
[EKAW] conference only accept
Review Comment:

Overall evaluation: -2
Select your choice from the options below and write its number below

== 3 strong accept
== 2 accept
== 1 weak accept
== 0 borderline paper
== -1 weak reject
X -2 reject
== -3 strong reject

Reviewer's confidence: 4
Select your choice from the options below and write its number below.

== 5 (expert)
X 4 (high)
== 3 (medium)
== 2 (low)
== 1 (none)

Interest to the Knowledge Engineering and Knowledge Management Community: 3
Select your choice from the options below and write its number below.

== 5 excellent
== 4 good
X 3 fair
== 2 poor
== 1 very poor

Novelty
Select your choice from the options below and write its number below: 2

== 5 excellent
== 4 good
== 3 fair
X 2 poor
== 1 very poor

Technical quality: 3
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
X 3 fair
== 2 poor
== 1 very poor

Evaluation: 3
Select your choice from the options below and write its number below.
== 5 excellent
== 4 good
X 3 fair
== 2 poor
== 1 not present

Clarity and presentation: 4
Select your choice from the options below and write its number below.
== 5 excellent
X 4 good
== 3 fair
== 2 poor
== 1 very poor

Review
This paper presents a system that analyzes textual inquiries about products and compares them with entries on social media sites to determine the nature of the reported problems. The approach is to store texts from social sites as triplets, against which inquires are matched to verify if this is a model problem or not. WordNet is used to calculate entity similarities in case there is no perfect match.

The research is well motivated with a good use case. The structure of the paper is fine, and it is well-written and easy to read.

The main contribution of the paper lies in the combination of techniques (POS tagging, Linked Data, triplets, WordNet similarity scores, etc.) to address a clear business need. I also like the fact that they do not complicate their approach unnecessary, but focus on the end result of their system.

However, my main concern is the lack of a substantial research contribution. There are some interesting challenges in entity linking and triplet construction, but the paper does not go into sufficient detail. It is of course fine to concentrate on the overall approach, but then there should be a more extensive evaluation that compares their approach to some keyword-based search solution, or measures some overall benefit over current practice.

The evaluation in Section 4.2 is not clear to me. Since they use WordNet and entity linking to relate entities in triplets, I assume that there may be both perfect matches of entities and entities that are sufficiently related to count as matches to some extent. The results in Table 5, thus, heavily depends on how this entity linking is done. It would be nice if they define "match" properly and explain how this is affected by the quality of entity linking.

Also, there is also an issue of the reliability of social media content. If the system is deployed in full scale, there would also be inconsistent or wrong content out there that would matched against the inquiries. How do they plan to deal with noise in these RDF-based graphs?

Minor issues to be fixed:
- The second format presented in Section 2.2 should be deleted, as it is not used in the rest of the paper.
- Figure 2 is too small
- Page 5: The sentence "The figure is also available..." should be deleted