842486

research-article2019
HKJ0010.1177/1024907919842486Hong Kong Journal of Emergency MedicineYu et al.

Original Article

Hong Kong Journal of Emergency Medicine

Triage accuracy of online symptom 2020, Vol. 27(4) 217–222

© The Author(s) 2019

checkers for Accident and

Article reuse guidelines:
sagepub.com/journals-permissions
https://doi.org/10.1177/1024907919842486
DOI: 10.1177/1024907919842486
Emergency Department patients journals.sagepub.com/home/hkj

Stephanie Wing Yin Yu1 , Andre Ma1, Vivian Hiu Man Tsang1,
Lulu Suet Wing Chung1, Siu-Chung Leung2 and Ling-Pong Leung3

Abstract
Background: Overutilisation of the Accident and Emergency Department is an increasingly serious healthcare challenge.
Online symptom checkers could help alleviate this challenge by allowing patients to self-triage before visiting the Accident
and Emergency Department.
Objectives: This study aimed to assess the triage accuracy of online symptom checkers, which would help determine
the potential roles of symptom checkers in an Accident and Emergency Department setting.
Methods: A total of 100 random Accident and Emergency Department records were sampled from the Queen Mary
Hospital in Hong Kong. The inclusion criteria were patients over the age of 18 attending the Queen Mary Hospital Accident
and Emergency Department in 2016. Symptom checkers by Drugs.com and FamilyDoctor were selected as representative
tools. One triage recommendation was generated by each symptom checker for each case record. Each symptom checker’s
triage accuracy was then evaluated using a few outcome measures: overall sensitivity, sensitivity for emergency cases and
specificity for non-emergency cases, when compared with the triage categories assigned by the triage nurses.
Results: The results showed that Drugs.com had a higher overall triage accuracy than FamilyDoctor (74% and 50%,
respectively), but both checkers are inadequately sensitive to emergency cases (70% and 45%, respectively) with low
negative predictive values (43% and 24%, respectively).
Conclusion: In their current states, symptom checkers are not yet suitable as alternatives to Accident and Emergency
Department triage protocols due to their low overall sensitivities and negative predictive values. However, symptom
checkers might serve as useful Accident and Emergency Department adjuncts in other ways, such as to provide more
information prior to a patient’s arrival to streamline the triage and preparation process at the Accident and Emergency
Department.

Keywords
Emergency service, hospital, medical overuse, online symptom checkers, triage

Introduction 1LiKa Shing Faculty of Medicine, The University of Hong Kong,

The Internet is an easy source of health information, and Pokfulam, Hong Kong SAR
2Accident and Emergency Department, Queen Mary Hospital, Pokfulam,
many patients have been found to research their own dis-
Hong Kong SAR
comforts online before seeking professional advice. Data 3Emergency Medicine Unit, Li Ka Shing Faculty of Medicine, The

have shown that almost three quarters of adults in the University of Hong Kong, Pokfulam, Hong Kong SAR
United States had searched for health information on the
Internet over the past 12 months,1 while more than one- Corresponding author:
Stephanie Wing Yin Yu, Li Ka Shing Faculty of Medicine, The University
fifth of adults in the United Kingdom have self-diag- of Hong Kong, 2/F, William MW Mong Block, 21 Sassoon Road,
nosed through the Internet instead of visiting a healthcare Pokfulam, Hong Kong SAR.
professional.2 Email: yusteph@hku.hk

Creative Commons Non Commercial CC BY-NC: This article is distributed under the terms of the Creative Commons
Attribution-NonCommercial 4.0 License (http://www.creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial
use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and
Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).
218 Hong Kong Journal of Emergency Medicine 27(4)

Generic keyword online searches yield too much Selection of symptom checkers
information for digestible use.1,3 More constructive and
clinically relevant programs, known as symptom check- The inclusion criteria for symptom checker selection were
ers, have been created with the aim of providing patients free and publicly available programs that provide triage
at home with differential diagnoses and triage advice recommendations across all specialties for adult patients. A
based on self-reported symptoms. Various organisations, previous study found 15 such symptom checkers and
such as the National Health Service (NHS) and the Mayo ranked the checkers by their triage accuracies based on
Clinic, have launched their own symptom checkers, with standardised clinical vignettes.4 Among the top 10 most
the NHS symptom checker reporting up to 15 million vis- accurate checkers, HMS Family Health Guide (Harvard
its per month.4,5 Symptom checkers’ advice has the power Medical School, USA), Steps2Care (Paramount Care Inc.,
to influence patients’ health-seeking behaviours and USA), FreeMD (DSHI Systems, Inc., USA) and EarlyDoc
health outcomes, and these checkers’ technology can be (EarlyDoc, Netherlands) have since been discontinued, and
leveraged to alleviate various challenges in healthcare. Healthy Children (American Academy of Pediatrics, USA)
One such healthcare challenge is the growing rate of was excluded due to its focus on paediatric cases. The
inappropriate utilisation of Accident and Emergency remaining symptom checkers were Symptify (LLC 2013,
Department (A&E) services. A study in the United States USA), Symptomate (Infermedica, Inc., Poland), Doctor
estimated that USD 38 billion is being wasted every year Diagnose (AppColliders, USA), Drugs.com (The Drugsite
due to inappropriate A&E usage.6 This overutilisation of Trust, New Zealand) and FamilyDoctor (American
emergency services leads to decreased overall quality of Academy of Family Physicians, USA). The first three are
care for both urgent and non-urgent users of the A&E.7 mobile apps with 5000–10,000, 100,000–500,000 and
Symptom checkers could be a method to alleviate the 10,000–50,000 downloads, respectively, while Drugs.com
overcrowding problem at A&Es, by allowing patients and FamilyDoctor are websites that receive over 6 million
unsure about the urgency of their condition to self-triage and 330,000 unique visitors per month, respectively.13,14
and instructing patients with non-urgent conditions to These five checkers were independently tested by three
refrain from visiting the A&E. The idea of triaging investigators on five sampled A&E cases. Drugs.com and
ambivalent patients before they present at A&Es is not a FamilyDoctor were ultimately chosen as the representa-
novel concept – the NHS developed a telephone triage tions of online symptom checkers because of their accu-
line to ensure that patients are receiving medical care in racy, popularity and suitability for our study design given
appropriate settings.8 With continual technological devel- the limited information documented on A&E charts.
opment, symptom checkers could serve the same purpose
at much lower financial and manpower requirements. Selection of patient records
Before recommendations for the widespread adoption Patient records were obtained from the Queen Mary
of symptom checkers can be made, it is imperative to Hospital of Hong Kong, whose 24-h A&E serves an aver-
thoroughly assess the reliability of these programs’ triage age of 300–400 patients per day. The inclusion criteria were
advice. A number of studies have been conducted in recent patients over 18 years old attending the A&E between 1
years on the accuracy of online symptom checkers,4,9–12 January and 31 December 2016. The minimum sample size
but to date no study has evaluated the accuracy of these required for the analysis, based on a stringent prevalence of
symptom checkers with real-life cases and in an A&E set- emergency cases set at 50%, power of 80% and statistical
ting, which was the literature gap that this study aimed to significance of 0.05, was estimated to be 100 cases.15
fill. The study’s primary objective was to determine the Figure 1 displays the flow of record sampling in this study.
accuracy of online symptom checkers in the triage of real Upon arrival at the A&E, every patient is assigned one
emergency and non-emergency cases. Our results can of the five triage categories outlined by the Hong Kong tri-
contribute to the analysis of whether symptom checkers age guidelines: Category 1 (Cat 1) for ‘critical cases’,
could serve as a major adjunct in triage protocols for Category 2 (Cat 2) for ‘emergency cases’, Category 3 (Cat
patients unsure of their need for A&E care. 3) for ‘urgent cases’, Category 4 (Cat 4) for ‘semi-urgent
cases’ and Category 5 (Cat 5) for ‘non-urgent cases’. In this
study, for each of the 20 randomly selected days, one eligi-
Methods
ble patient chart from each triage category was randomly
The study protocol was approved by the Institutional sampled for both checkers. For each A&E patient chart that
Review Board of The University of Hong Kong/Hospital was incomplete, illegible or whose chief complaint was
Authority Hong Kong West Cluster. Written informed con- unavailable on a specific checker, a replacement case was
sent was not necessary because no patient data have been sampled from the same triage category on the same day for
included in the article. that checker only, or from a different randomly selected day
Yu et al. 219

‘emergency’ advice, while instructions that included one or

more of the phrases ‘seek medical help today’, ‘make an
appointment with a specialist’, ‘call your doctor to make an
appointment’ or ‘self-care’ were classified as ‘non-emer-
gency’ advice. Two investigators independently classified
each symptom checker’s triage advice into ‘emergency’
and ‘non-emergency’, and a third investigator indepen-
dently resolved any disagreements in classifications.
Checkers’ instructions that were similar but not exactly
equal to the ones listed above were classified at the investi-
gators’ own discretion. Triage recommendations that
required users to apply their own judgement to assess
urgency, such as ‘if you think the problem is serious, call
your doctor right away’, were universally considered incor-
rect, because these recommendations are not useful for self-
triage purposes.

Data analysis
All analyses were performed using SPSS Statistics v22.0
(IBM Corp., USA). Descriptive statistics was used for
Figure 1. Flowchart of the study procedure.
demographic data, and quantitative data were reported as
mean ± standard deviation (SD), or percentages with 95%
if no more cases are eligible. Hence, in total, 100 cases confidence intervals (CIs), as appropriate. The percent
from at least 20 different days in 2016, with 20 cases from agreement of each symptom checker with A&E triage nurse
each of the five triage categories, were inputted for analysis was calculated for all cases, emergency and non-emergency
into each symptom checker, with some overlap in cases cases and cases of each triage category. Overall sensitivity,
between the two checkers. specificity, positive predictive value (PPV) and negative
predictive value (NPV) were also calculated. The intraclass
correlation coefficients (ICCs) were computed for level of
Assessment of symptom checkers agreement between independent investigators, with
Each medical record was transcribed by one investigator ICC > 0.7 considered adequate by convention.
into a standardised format containing the patient’s age,
gender, presenting complaints, vital signs, past medical
history and physical examination findings, with the triage
Results
category blinded and recorded separately. Two investiga- Each symptom checker was tested with 100 A&E charts
tors independently inputted information from the tran- with an equal distribution between the five triage catego-
scribed records into both symptom checkers and recorded ries. A total of 51 charts were used to test both checkers,
the checkers’ triage results. A third investigator indepen- with two different sets of 49 charts used to test only one
dently resolved any disagreements between the original checker, making up a total of 149 charts sampled. Among
two investigators’ results. the cases sampled for Drugs.com, the mean age was
56.6 years, with 42% being male. Among the cases sampled
for FamilyDoctor, the mean age was 55.4 years, with 45%
Outcome measures
being male. There was no significant difference in the aver-
For this study, emergency levels were defined as such: Cat age age of sampled cases between the two checkers (t = 0.57,
1–4 cases were classified as ‘emergency’ patients who p = 0.57).
appropriately visited the A&E, while Cat 5 cases were clas- Table 1 displays the triage accuracies of the symptom
sified as ‘non-emergency’ patients who did not need A&E checkers. Drugs.com was more accurate than FamilyDoctor
services. Triage category assigned by the A&E nurse was (74% vs 50%) and had a lower under-triage rate. All ICCs
considered accurate in all cases. for the independent assignments of emergency statuses
Symptom checkers’ triage advices are provided through exhibited adequate agreements (>0.7).
direct instructions addressing the user. For this study, Table 2 displays the overall sensitivities, specificities,
instructions that included one or more of the phrases ‘visit PPVs and NPVs of the two checkers in identifying urgent
the ER now’, ‘seek medical help immediately’ or ‘you cases. Drugs.com performed better than FamilyDoctor in
likely have a life-threatening condition’ were classified as all four parameters.
220 Hong Kong Journal of Emergency Medicine 27(4)

Table 1. Triage accuracies of Drugs.com and FamilyDoctor.

Drugs.com FamilyDoctor
% (n = 100) 95% CI (%) % (n = 100) 95% CI (%)
Accurately triageda 74 64–82 50 40–60
Under-triageda 24 16–34 45 35–55
Over-triageda 2 0.24–7.0 5 1.6–11
Total (%) 100 100
Cases by emergency status Drugs.com resultsb FamilyDoctor resultsc
Emergency Non-emergency Emergency Non-emergency
Emergency (total n = 80) 56 24 35 45
Non-emergency (total n = 20) 2 18 5 15

CI: confidence interval; ICC: intraclass correlation coefficient.

aEmergency level of checker’s result when compared to that of triage category documented on A&E chart.
bICC (emergency status assignment) = 0.85 (95% CI: 0.80–0.89); ICC (recommendation assignment) = 0.73 (95% CI: 0.60–0.82).
cICC (emergency status assignment) = 0.86 (95% CI: 0.82–0.90); ICC (recommendation assignment) = 0.75 (95% CI: 0.63–0.83).

Table 2. The overall sensitivities, specificities, PPVs and NPVs of When considering whether or not symptom checkers’
Drugs.com and FamilyDoctor. level of sensitivity and specificity suffices to justify incor-
Drugs.com FamilyDoctor poration into an A&E’s triage protocol, one must consider
the accuracy of the A&E’s existing triage procedure. A
% 95% CI % 95% CI review of the literature found that triage accuracies vary
Sensitivity 70 59–80 44 33–55 across centres. Studies have estimated an overall triage
Specificity 90 68–99 75 51–91 accuracy of 59.2% among three community hospitals in
PPV 97 88–100 88 73–96 United Arab Emirates, Brazil and the United States,16
NPV 43 28–59 25 15–38 59.6% among four Swiss hospitals,17 82.9% in a single
Brazilian centre18 and 62.2% among the paediatric units
PPV: positive predictive value; NPV: negative predictive value; CI: for four Australian hospitals.19 Even though online symp-
confidence interval.
tom checkers are theoretically less costly to implement
than triage nurses, checkers must perform at least as well
Table 3 displays the triage accuracies of the two symp- as the triage protocol in place before they can be consid-
tom checkers according to each triage category, with accu- ered as replacements to A&E nurses. Applying our data
rate being defined as assigning Cat 1–4 cases to ‘emergency’ locally, where the A&E triage accuracy has been esti-
or assigning Cat 5 cases to ‘non-emergency’. Drugs.com mated to be 78%,20 we conclude that online symptom
was more accurate than FamilyDoctor in every category, checkers are not yet of sufficient quality to replace triage
and both symptom checkers performed better for non- nurses in our locale.
emergency cases than the emergency ones. On top of considering the checkers’ overall accuracy,
it is also important to analyse how the checkers fare for
each triage category. In this study, both checkers were
Discussion
predictably most sensitive for cases in Cat 1 (critical
Both Drugs.com and FamilyDoctor performed subopti- cases) and Cat 5 (non-urgent cases), which represent two
mally in overall triage accuracies (74% and 50%, respec- extremes of the urgency spectrum where the appropriate
tively). According to the audit study performed by Semigran course of action is more obvious (Table 3). However, all
et al.4 in 2015, the overall triage accuracies of Drugs.com cases in Cat 1–4 require A&E attention, and both Drugs.
and FamilyDoctor were 60% and 54%, respectively. The com and FamilyDoctor displayed generally low sensi-
discrepancy between our data and Semigran’s is likely due tivities for emergency cases (70% and 44%, respec-
to our use of real patient records compared to their expert- tively), especially when compared to their specificities
written vignettes, since the two methodologies are other- for non-emergency cases (90% and 75%, respectively;
wise largely similar. As opposed to hypothetical patient Tables 1 and 2). This agrees with the observation that
scenarios, our test samples represent a more diverse and both checkers tended to under-triage rather than over-
realistic range of patient presentations and offer more triage (Table 1), leading to a much higher PPV (97% for
insight into how symptom checkers perform in real clinical Drugs.com and 88% for FamilyDoctor) than NPV (43%
situations. for Drugs.com and 25% for FamilyDoctor; Table 2). One
Yu et al. 221

Table 3. Triage accuracies of Drugs.com and FamilyDoctor would help the checkers be more user-friendly and provide
according to triage category. timelier triage advice. Regardless of whether or not symp-
A&E triage Drugs.com FamilyDoctor tom checkers can aid or replace A&E triage nurses, given
category the increasing trend of laymen utilising the Internet for
Accuracy (%) 95% CI Accuracy (%) 95% CI health-related information, more research should be con-
Category 1 95 75–100 65 41–85 ducted to improve symptom checkers’ triage algorithms to
Category 2 65 41–85 35 15–59 generate more accurate results.
Category 3 65 61–85 30 12–54
Category 4 55 32–77 50 27–73
Category 5 90 68–99 70 46–88
Limitations
One major weakness of this study design is that the refer-
CI: confidence interval.
ence standard was taken as the triage nurse’s triage cate-
gory, but different A&E nurses come with different
could argue that, for a symptom checker to safely serve backgrounds, experiences and, thus, triage abilities.
as a major adjunct in the A&E, sensitivity to emergency Hence, A&E nurses’ triage decision may not be the best
cases is more important than to non-emergency cases, surrogate for appropriateness of A&E use in some cases,
because under-triaging the former could lead to prevent- and future studies should consider using a stricter stand-
able deaths. In this study, Drugs.com under-triaged 30% ard, such as a triage consensus among several senior
and FamilyDoctor under-triaged 56% of the emergency nurses, to compare symptom checkers against. Moreover,
cases (Table 1). These accuracies are sub-par compared the symptom checkers’ source of information is solely
both to local A&E nurses, who have been estimated to based on the A&E staff’s assessment and documentation,
under-triage about 15% of A&E cases,20 and to some tel- which, if inadequate, could have led to an underestimation
ephone triage services, which have been found to under- of the checkers’ accuracies. On the other hand, cases with
triage about 19% of urgent cases.21 Urgent patients chief complaints not available on the symptom checkers
mistriaged by symptom checkers would either not seek were replaced by more compatible cases, which likely
medical care or need to exercise their own layperson resulted in an overestimation of the checkers’ accuracies.
judgement, both of which leave room for otherwise Since only 51% of the tested A&E cases overlapped
avoidable morbidity and mortality. Therefore, we con- between the two checkers, comparing the checkers’ accu-
clude that online symptom checkers, at least in their cur- racies with our study design may not have been ideal, but
rent states, are unsuitable to serve as the sole triage tool the conclusions drawn from each of the checker’s results
for potential A&E cases. and accuracy are still valid.
Since the replacement of triage nurses by automated
programs was likely an overly aggressive goal that involves
both practical and ethical issues, there are other, perhaps Conclusion
less grandiose, niches that symptom checkers could fill in Online symptom checkers are currently inappropriate to
our current A&E system. For example, symptom checkers serve as a primary triage tool in the A&E, due to their low
could serve as an adjunct in prehospital care to optimise overall accuracies and NPVs. More aggressive triaging
A&E resources. Patients, family members or ambulance guidelines and additional improvements in function are
staff can fill in the symptom checker questionnaire en route necessary for symptom checkers to achieve a higher level
to the hospital – if this information can be connected to of sensitivity and specificity. Potential applications of cur-
A&E staff, it could help cut down on the time needed to rent checkers in A&E settings include providing a tentative
take history and make treatment-related preparations after triage level prior to the patient’s arrival to the A&E to help
the patient arrives. A rough guide into the urgencies of with resource preparation. Further research must be under-
incoming cases can also allow A&E staff to better stream- taken to improve symptom checkers’ triage algorithms to
line their triage process and ensure that care is provided in generate more accurate results.
the appropriate sequence.
Improvements must also be made to online symptom
Declaration of conflicting interests
checkers before they can be widely applied to A&E set-
tings. The low NPV of both checkers suggest that triage The author(s) declared no potential conflicts of interest with
advice should be more aggressive to capture all potentially respect to the research, authorship and/or publication of this
life-threatening conditions. In addition, incorporating article.
regional and seasonal epidemiology and the past medical
history of patients would allow a clearer analysis of the pre- Funding
senting symptoms. In the context of A&E cases, having The author(s) received no financial support for the research,
offline programs and a search option for chief complaints authorship and/or publication of this article.
222 Hong Kong Journal of Emergency Medicine 27(4)

Availability of data and materials 8. Turner J, Coster J, Chambers D, et al. What evidence is there
on the effectiveness of different models of delivering urgent
Data and materials can be made available to the public through
care? A rapid review. Southampton: Health Services and
direct requests to the corresponding author.
Delivery Research, 2015.
9. Bisson LJ, Komm JT, Bernas GA, et al. Accuracy of a com-
Ethical approval puter-based diagnostic program for ambulatory patients with
The study protocol was approved by the Institutional Review knee pain. Am J Sports Med 2014; 42(10): 2371–2376.
Board of The University of Hong Kong/Hospital Authority Hong 10. Farmer SE, Bernardotto M and Singh V. How good is Internet
Kong West Cluster. self-diagnosis of ENT symptoms using Boots WebMD symp-
tom checker? Clin Otolaryngol 2011; 36(5): 517–518.
Human rights 11. Poote AE, French DP, Dale J, et al. A study of automated
self-assessment in a primary care student health centre set-
No human subjects were included in the study. ting. J Telemed Telecare 2014; 20(3): 123–127.
12. Wolf JA, Moreau JF, Akilov O, et al. Diagnostic inaccuracy
Informed consent of smartphone applications for melanoma detection. JAMA
No human subjects were included in the study. Dermatol 2013; 149(4): 422–426.
13. Alexa Internet, Inc. drugs.com Traffic Statistics, https://www
.alexa.com/siteinfo/drugs.com (2018, accessed 26 October
ORCID iDs 2018).
Stephanie Wing Yin Yu https://orcid.org/0000-0003-0674-1515 14. Alexa Internet, Inc. familydoctor.org Traffic Statistics, https://
www.alexa.com/siteinfo/familydoctor.org (2018, accessed 26
Siu-Chung Leung https://orcid.org/0000-0003-0169-9899
October 2018).
15. Bujang MA and Adnan TH. Requirements for minimum
References sample size for sensitivity and specificity analysis. J Clin
1. Fox S and Duggan M. One in three American adults Diagn Res 2016; 10(10): YE01–YE06.
have gone online to figure out a medical condition. Pew 16. Mistry B, Stewart De Ramirez S, Kelen G, et al. Accuracy
Research Center, http://www.pewinternet.org/2013/01/15/ and reliability of emergency department triage using the
health-online-2013/ (2013, accessed 26 October 2018). emergency severity index: an international multicenter
2. Push Doctor. One in four people in the UK admit to self- assessment. Ann Emerg Med 2018; 71(5): 581.e3–587.e3.
diagnosis of an illnesses rather than making time for a 17. Jordi K, Grossmann F, Gaddis GM, et al. Nurses’ accuracy
doctor’s appointment, https://www.pushdoctor.co.uk/digital- and self-perceived ability using the Emergency Severity
health-report (2015, accessed 26 October 2018). Index triage tool: a cross-sectional study in four Swiss hospi-
3. North F, Ward WJ, Varkey P, et al. Should you search the tals. Scand J Trauma Resusc Emerg Med 2015; 23: 62.
Internet for information about your acute symptom? Telemed 18. Hinson JS, Martinez DA, Schmitz PSK, et al. Accuracy of
J E Health 2012; 18(3): 213–218. emergency department triage using the Emergency Severity
4. Semigran HL, Linder JA, Gidengil C, et al. Evaluation of Index and independent predictors of under-triage and over-
symptom checkers for self diagnosis and triage: audit study. triage in Brazil: a retrospective cohort analysis. Int J Emerg
BMJ 2015; 351: h3480. Med 2018; 11(1): 3.
5. Gann B. Giving patients choice and control: health infor- 19. Allen AR, Spittal MJ, Nicolas C, et al. Accuracy and inter-
matics on the patient journey. Yearb Med Inform 2012; 7: rater reliability of paediatric emergency department triage.
70–73. Emerg Med Australas 2015; 27(5): 447–452.
6. New England Healthcare Institute (NEHI). A matter of 20. Fan MM and Leung LP. Validation of the Hong Kong acci-
urgency: reducing emergency department overuse. NEHI dent and emergency triage guidelines. Hong Kong Med J
Research Brief, https://www.nehi.net/publications/6-a-mat- 2013; 19(3): 198–202.
ter-of-urgency-reducing-emergency-department-overuse/ 21. Giesen P, Ferwerda R, Tijssen R, et al. Safety of telephone
view (2010, accessed 26 October 2018). triage in general practitioner cooperatives: do triage nurses
7. Nonurgent Use of Hospital Emergency Departments. United correctly estimate urgency. Qual Saf Health Care 2007;
States Senate, Session 112-789. Printer’s No. 81-788. 16(3): 181–184.