When to Support and when to Bot
Nemi Pelgrom1 and Daniel Ihrmark2
∗
1- Linnauesuniversity - Dept of Computer Science
Växjö - Sweden
2- Linnauesuniversity - Dept of Cultural Sciences
Växjö - Sweden
April 29, 2024
Abstract
In this study, we analyse a dataset comprising of 8,700 customer support chat lines to discern patterns that differentiate simple from complex
problems, and discuss the implications for automated customer support
systems. We identify a distinct threshold: messages of 20 words or fewer
typically present issues straightforward enough for bots to understand.
Conversely, messages exceeding 21 words often detail multi-step problems,
or personalised problems, challenging for bots to address effectively. This
pattern was found on manual annotation of 1026 messages spanning 18-24
words. We also report on a trial of using an LLM to automate the process
to annotate 8,700 messages ranging from 18 to 30 words. Our findings
show a critical role of message length in determining the suitability of
automated support and offer insights into optimising bot deployment in
customer service environments.
1
Introduction
The distinction between tasks that are appropriate for bots versus those that
require human intervention is a critical aspect in streamlining customer support
services[1]. The aim is to find solutions that are both computationally efficient,
and also able to give the customer a satisfying support experience. The accurate
identification of a customer’s primary issue is central. Issues that demand an
elaborate explanation are typically of a complexity level beyond the bots’ limited
ability to accurately understand the context of the issue, and in many cases
the issues are of a kind that requires human interaction (such as personalised
features or system bugs). The common problem of the customer being unable
∗ We thank Fortnox AB for sponsoring this research. We also clarify that only the dataset
is from Fortnox AB, none of the discussion has any connection to the company.
1
to identify by themselves what support they need to solve their issues, is central
to customer satisfaction [2], which is a hindrance for deploying bots [3]. This
problem necessitates an effective and efficient system within customer support
operations to assess and direct issues to the suitable resolution channel, whether
it be a bot or a human agent. We contribute to this by presenting a pattern
found in initial messages to several support chat channels of a Swedish company.
We will discuss this pattern in the context of evaluating efficiency and customer
satisfaction.
From a linguistic standpoint, the idea of a topic is often seen as a defined
cluster of words that frequently co-occur across the sample being explored. One
popular method for topic modeling tasks is based on Latent Dirichlet Allocation
[4], which has previously been used to explore topic segmentation in the context
of costumer support. There are also several LLM based trials on categorising
topics, these are based on much larger corpuses of training and of prompting
than what is relevant to our aims[5], as our the focus is not on the definition of
the topics themselves, but rather on the number of topics included in a short
text segment.. Making detailed definitions and lists of topics is useful when there
is a known and limited amount of possible questions. With larger companies
that offer a wide variety of services, the amount of possible problems is large
and complex, which makes such solutions time consuming to administrate. The
issue from the user, needs to be communicated to the support system or support
worker, the solutions needs to be identified, and the solutions needs to be either
fixed by them, or communicated back to the user. This makes customer support
be heavily dependent on the quality of communication[6]. Some problems are
very straight forward; ”Where do I find the price list for this service?”, and they
can be solved by a bot that is able to extract the particular request and connect
it with the right link on the company web page. And some problems are more
complex; ”I think I accidentally sent an invoice to the wrong receiver, and didn’t
notice until after I lost the option to take it back. Could you help me remove it
or contact the recipient?”, and these need a support worker to be solved. If we
could identify which messages goes into which category, we both avoid making
the first category of users wait for an available support worker, and the second
category of users have to talk to a bot before they reach a person.
2
Methodology
While doing an unrelated processing of support chats, we noticed a pattern of
a cut-off point for complexity of requests from the users in the messages. We
decided to explore it, to see if we could show that there was a real pattern. We
collected a dataset of real messages from users to various support channels at
Fortnox AB. We wanted to use a zero-shot LLM to help annotate the dataset,
so that we could have more data to validate the hypothesis, however, the model
was not able to learn the pattern based on 240 annotated examples. Since
the pattern was clear in this first round of manual annotation, we decided to
complete the trial manually, and so we annotated more messages in our selection
2
of words counts, until we reached over 1000 annotated messages, with an even
distribution of amount of messages from each category of word count.
2.1
Preprocessing the dataset
The original dataset was 89.000 lines. It contained all messages from all conversations, and we are only interested in initial messages, so we removed all lines
that were not clearly introducing a problem. Approximately half of the messages we used has the Swedish word for ”hello” as the first word, but this was
more seldom used in the longer messages so we decided to not limit ourselves
to only using them only. Then we removed all messages containing identifying information (such as email addresses, links, full names, organisation IDs).
To avoid language barrier biases, we removed the few chats that were in other
languages than Swedish. We removed messages that are citing texts since this
made the word count wrong.
2.2
Criteria for chat message categorisation
We want to see if there is a correlation between how many words there are
in a message and how many separable topics are discussed in them. To do
this, we need a line that separates topics so that we can apply this line to our
messages and see where they fall on it. Since our chat data is directed towards
topics limited to the services the company provides and there are furthermore
many topics mentioned by names particular to the company, we found that the
available topic segmentation algorithms were not effective on our data. While
we acknowledge that topic segmentation is a wide field of its own, it is not
of primary interest for this study, so we have used a manual approximation
to estimate how many topics there are in any particular message. We are not
experts in the topics mentioned in the messages, which makes it more likely that
we have false positives (categorised as 2, but should be 1), than false negatives
(categorised as 1, but should be 2) in the dataset.
We separated the topics on the following basis: If it is possible to reformulate
the content of the message to be one single step question, then we categorised
it as 1. If there are several steps, or several not clearly connected questions,
we categorised it as 2. If the question was about something particular to that
user, such as a particular invoice, or person, we also categorised it as 2, on
the basis that one would not be able to answer the question without first locating the particular invoice or person, which makes it a two-step process to
answer. Messages that were of the form ”I need help with understanding this
product/service” were categorised as 2, since we understand that as a request
of more than one simple question.
Most of the messages that were difficult to categorise were of the form of a
question, followed by a specification of where they were accounting a problem
or why they were asking the question. The difficulty was of judging if the
specification was necessary for answering the question or not. If it was necessary,
we categorised it as 2, otherwise as 1. To avoid bias on only using messages
3
that were easy to categorise, we annotated each message in the final dataset.
While this means that it is possible some difficult-to-categorise messages could
be categorised differently, the difference should only be marginal, and this was
decided to be preferable over changing the dataset it self to have more bias.
We only did the binary categorisation of sorting the data into category 1
or 2. While some messages could be considered to have more then 2 topics in
them, we were looking for a cutoff point between 1 and 2, so while further detail
in the categorisation process could result in slightly more accurate results, it
would not affect the applicability of the results and was deemed unnecessary.
2.3
Examples of categorisation
Two topics:
• Hello! I have a customer who has offset a credit against the wrong invoice.
Can you help me match the overpayment against the underpayment? This
involves two steps: first understanding how to handle credit against the
wrong invoice, and then deciding if and how overpayment can be matched
against underpayment.
• I went in a while ago and entered on the invoice that payment has been
made, has it not reached you yet? This takes two steps: identifying the
invoice, and checking if it has been paid.
One topic:
• Hello, I need to get a certification list for how the certification is set up in
Fortnox for an auditor. How do I get it out? This is one step because it
only involves a specified data extraction.
• Hello, I wonder if I can see if the employer declaration has already been
sent to the tax authority. I am unsure if this has been done. This is one
step because it only requests how to locate a specific part of the webpage
2.4
Trial with zero-shot BERT model
We did a trial on automating the categorization of the messages into our two
categories. The trial was conducted using the zero-shot BERT model created
by KB Labs [7]. The model has previously struggled with text components
such as punctuation, but was considered an interesting possible venue due to
being able to parse based on context, as compared to the content-agnostic approach represented by word count measurement. We did a trial of automating
the annotation of the messages. Our trial was performed on 8700 messages of
length 18-30, but we got poor results. There was a significant over representation of messages being categorised as several topics, even when it was clear
from manual analysis that it was one topic that was discussed. We drew the
conclusion that the model could not identify the pattern of criteria for chat
message categorisation that we are using.
4
3
Results
Figure 1: Plot of the percentage of one or more topics against the amount of
words
Topics
1
2
18
88
0
19
223
0
20
118
57
21
72
131
22
24
111
23
9
100
24
8
85
Table 1: Amount of messages containing one or more topics, according to
amount of words in the message, manually annotated
4
Discussion
These results give us a new insight into a correlation between the amount of
words in a support request and the complexity of the request. This could be
used as a basis for decisions on what parts, or how much, of customer support to
automate. Since many chat bots have difficulties understanding when there are
several tasks mentioned, or when there are complex contexts to consider, it is
useful to be able to separate out such issues from the simpler ones. Implementing this separation of incoming messages to support chats, could increase the
experience of both the user and the people who are working with the support.
The users with complex problems do not have to spend extra time answering
questions from a bot before they are connected to the support they need, and
the people working with the support should get fewer of the simple problems
assigned to them. While the results have not been checked to be replicable in
support chats from other companies, or other languages, we believe that the
insights from our results still have value, and it is possible to easily recreate our
5
trial on any similar dataset for more trials before possible implementation of
this discussed division of messages.
5
Conclusion
We found that there is a very clear patter on the complexity of the issues users
bring up in their initial messages to support chats to Fortnox AB. The pattern
is that messages 20 words or shorter are clear in what they are requesting,
and messages 21 words or longer usually have complex or several-step requests.
While it is not strange that shorter messages are easier to handle, it is surprising
that there is such a clear cut-off point with simple word count. This insight could
be used as part of any system that aims to make AI-aided customer support
more efficient.
References
[1] Debayan Banerjee, Mathis Poser, Christina Wiethof, Varun Shankar Subramanian,
Richard Paucar, Eva A. C. Bittner, and Chris Biemann. A system for human-ai collaboration for online customer support, 2023.
[2] Alexander Rossmann, Alfred Zimmermann, and Dieter Hertweck. The impact of chatbots
on customer service performance. In Advances in the human side of service engineering: Proceedings of the AHFE 2020 Virtual Conference on The Human Side of Service
Engineering, July 16-20, 2020, USA, pages 237–243. Springer, 2020.
[3] Xinyi Yang. Understanding chatbot service encounters: Consumersâ satisfactory and
dissatisfactory experiences. Master’s thesis, X. Yang, 2020.
[4] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal
of machine Learning research, 3(Jan):993–1022, 2003.
[5] Derek Greene, Derek O’Callaghan, and Pádraig Cunningham. How many topics? stability
analysis for topic models. In Toon Calders, Floriana Esposito, Eyke Hüllermeier, and Rosa
Meo, editors, Machine Learning and Knowledge Discovery in Databases, volume 8724 of
Lecture Notes in Computer Science, pages –, Berlin, Heidelberg, 2014. Springer.
[6] Mark Anthony Camilleri and Ciro Troise. Live support by chatbots with artificial intelligence: A future research agenda. Service Business, 17, 11 2022.
[7] Love Börjeson, Chris Haffenden, Martin Malmsten, Fredrik Klingwall, Emma Rende,
Robin Kurtz, Faton Rekathati, Hillevi Hägglöf, and Justyna Sikora. Transfiguring the
library as digital research infrastructure: Making kblab at the national library of sweden.
2023.
6