Assessing Violence Risk in Threatening Communications
Kimberly Glasgow
Johns Hopkins University
Applied Physics Laboratory,
and
College of Information Studies,
University of Maryland
kimberly.glasgow@jhuapl.edu
Abstract
primarily for the purpose of detecting symptoms
of thought disorder or evidence of impending violent behavior. Language technology is rarely utilized in these efforts, yet it could be a valuable tool
for detecting evidence of illness and increased violence risk in verbal and written communications.
We analyzed a unique data set of threatening
communications sent to judges. Examination of
these written communications indicate that, for
this sample, explicit threats are rare, but evidence
of mental illness is common. We applied two types
of computational methods to the communications
in the sample—topic models, and a simple computational text analysis method: LIWC (Pennebaker
et al., 2001). The results point towards a useful
role for such methods in the analysis of threatening communications, as well as limitations. Advances in language technology methods, as well as
the availability of more data, may both be needed
to make substantial progress.
Violence risk assessment is an important
and challenging task undertaken by mental health professionals and others, in both
clinical and nonclinical settings. To date,
computational linguistic techniques have
not been used in the risk assessment process. However they could contribute to the
current threat assessment process by allowing for early detection of elevated risk,
identification of risk factors for violence,
monitoring of violent intent, and determination of threat level. We analyzed a sample of communications to judges that were
referred to security personnel for evaluation as constituting potential threats. We
categorized them along multiple dimensions including evidence of mental illness,
presence and nature of any threat, and
level of threat. While neither word countbased or topic models were able to effectively predict elevated risk, we found topics indicative of persecutory beliefs, paranoid ideation, and other symptoms of Axis
I and Axis II disorders.
1
Ronald Schouten
Harvard Medical School,
and
Department of Psychiatry,
Massachusetts General Hospital
rschouten@mgh.harvard.edu
2
Violence Risk Assessment and Mental
Health Professionals
Assessment of the risk of violence is a task that
belongs to a diverse group of mental health professionals (MHPs): those who provide clinical care,
forensic MHPs specializing in mental health issues related to the legal system, and those who engage in the even more specialized field of threat
assessment. Other disciplines involved in threat
assessment include law enforcement, security professionals, and intelligence analysts.
Violence risk assessment is a routine aspect of
the work of mental health professionals treating
Introduction
Mental health professionals are called upon to assess the risk of violence in many different settings,
from the determination of the need for hospitalization or increased treatment to consultations for
the criminal justice system (Skeem and Monahan,
2011). These assessments include examination of
the verbal content of a subject’s communications,
38
Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 38–45,
Baltimore, Maryland USA, June 27, 2014. c 2014 Association for Computational Linguistics
COVR (Monahan et al., 2006), and VRAG (Quinsey et al., 1998). None of these instruments consider linguistic factors. They utilize actuarial determinations of violence risk. These instruments
do not provide strict cutoff scores that differentiate between nonviolent and violent individuals.
Rather, they serve as adjunct tools to clinical judgment. As a result, the current best practice in violence risk assessment consists of structured clinical judgment, a process in which actuarial risk assessments are combined with clinical judgment to
reach a determination regarding a specific individual’s risk.
Whereas treating clinicians primarily rely upon
examination of the patient in assessing the risk
of violence, forensic MHPs are expected to go
beyond the clinical examination and incorporate
information from a variety of collateral sources,
such as medical and mental health records, psychological testing, legal documents, police reports, and criminal histories in order to increase
the objectivity and “scientific” basis of their opinion. As in clinical care, language is an important
part of the mental status examination. More detailed review of the evaluee’s communications is
more common in forensic work, as it may provide
insight into the writer’s emotional state, motivation, and intention, as well as thought processes.
The content, syntax, and grammar of communications, as well as the page layout, variations in font
size, use of color, and graphics may all be considered in assessing for presence of a mental disorder
and indications of violence risk.
people with mental illness. While violence against
others on the part of people with diagnoses of
mental illness is far less prevalent than is popularly
thought, the increased risk attributable to these illnesses is barely statistically significant (Steadman
et al., 1998; Swanson et al., 1990). This increased
risk is largely attributable to a small group of individuals who have a history of childhood or adult
antisocial behavior in combination with substance
use disorders and psychotic illness (Elbogen and
Johnson, 2009).
2.1 Methods and Practice of Violence Risk
Assessment
Treating clinicians are responsible for evaluating
their patients to determine if they pose a risk of
violence and adjusting treatment accordingly, or
arranging for hospitalization, as needed. The risk
of violence, as evidenced by threats or attempts to
harm self or others, are two of the bases for hospitalizing people with mental illness against their
will. This assessment primarily relies upon information obtained through interviewing and observing the patient, as well as information from collateral sources when it is available. The patient’s
language is taken into account largely as a part
of the mental status examination, in which attention is paid to the content and form of the patient’s
thoughts, which are characteristically disrupted in
certain illnesses. Clinicians look at many factors
to determine if someone poses a risk of violence,
but a patient’s written communications is typically
not one of them.
MHPs who practice in the field of forensic mental health do so as an even larger component of
their work. Many are routinely asked to assess the
risk of violence in both the civil and criminal justice systems. In the civil justice system, for example, they may be called upon as expert witnesses
in civil commitment proceedings or as consultants
on such matters. In the criminal justice system,
they may be asked to assess the risk of violence
in conjunction with the issuance of restraining orders, determination of conditions of bail and probation, and sentencing. While judges make the ultimate decisions, they generally rely highly upon
the clinical judgment of MHPs with regard to diagnosis and assessment of the risk of violence.
In recent years, a number of tools have been introduced to assist in the assessment of violence
risk, such at the HCR-20 (Webster et al., 1982),
2.2 Threat Assessment
Threat assessment is a discipline that relates to, yet
is separate from, clinical violence risk assessment.
Meloy, et al. distinguish between the two fields,
noting that violence risk assessment is consultative in nature, and generally aimed at assisting legal decision-making and managing a particular individual over the long term. They note that threat
assessment is operational, rather than consultative,
in nature and is aimed at protecting victims by determining the level of risk that they face at a given
moment in time (Meloy and Hoffmann, 2013).
Although the emphasis is different, both take into
account the likelihood that a given individual will
act in a violent fashion. Threat assessment goes
beyond the determination of risk of physical violence and extends to insider threats such as sabo39
tage, espionage, hacking, harassment, and attacks
on reputation. Language assumes an even greater
role in the analysis of threat than it does in violence risk assessment.
The Risk Assessment Guideline Elements for
Violence (RAGE-V) produced by the Association
of Threat Assessment Professionals lists a wide
range of behaviors and risk factors to be considered in assessing the threat of violence. It contains
no reference to the analysis of written materials or
communications, other than suicide notes. (Available at www.atapworldwide.org).
3
natural, purposeful communications from a sender
to at least one judge or court official. They were
perceived as threatening, and referred to court security officers for risk assessment. These referrals were usually made by judges, though District Attorneys and Clerks of the Court can also
report threats to court security. The documents
represented all cases that contained written material (not just verbal threats) from the two largest
districts within the purview of the office responsible for trial court security for this region. Judges
may refer a potentially threatening communication
based on a perceived risk of harm to self, or to the
security of the courtroom.
The Language of Threat
Analysis of language is an important aspect of
threat assessment and has traditionally been utilized in much the same manner as in forensic evaluations. That is, it has largely involved ad hoc,
impressionistic assessments of communications.
Efforts towards a more methodical approach to
linguistic analysis of threatening communications
have been made. However, many of these still rely
primarily on human judgment of content. Smith
and Shuy describe closely examining language as
evidence for clues to race, ethnicity, or gender
of a perpetrator, for identifying false allegations,
and for related law enforcement tasks (Smith and
Shuy, 2002). Scalora describes analyzing threatening language towards members of Congress in
terms of several thematic areas relating to presence
and types of demands (such as policy changes or
personal favors) (Scalora et al., 2003), and Calhoun (Calhoun, 1998) examines threatening or inappropriate communications and assaults against
federal judicial officials based upon factors such
as the directness or immediacy of the threat.
In other related work, efforts to predict case
outcomes for a set of 96 FBI cases involving
threatening communications have incorporated interviews and automated text processing (Smith,
2008) Computational methods have also been applied to the communications of terrorist or radical religious extremist groups to detect aggressive
or violent intent, using function word categories
(Pennebaker et al., 2008) or frame analysis (Sanfilippo, 2010).
All documents were in English. All documents
underwent optical character recognition (OCR),
and the output of the OCR process was reviewed
to correct errors in the text. Handwritten portions
of documents were manually transcribed.
4
A high, medium, or low judgment for risk of violence was made in the manner common in threat
assessment practice, i.e., an overall impression
based upon the intensity of emotion conveyed, the
presence of paranoid ideation directed toward the
Each document was manually annotated for the
presence of atypical formatting or text features,
(e.g., the inclusion of magazine cut-out words or
images, or the use of unusual bolding or italics,
centering, or large point size in text), or presence
of handwritten comments in addition to the text.
These documents include legal documents, letters,
faxes, cards, and other printed materials, as well
as hard copies of emails.
Documents were also coded for indications of
psychotic symptoms, Axis I mental disorders such
as mania, depression, anxiety and psychotic disorders, or Axis II disorders such as personality disorders, developmental disabilities or autism spectrum disorders, utilizing the multi-axial diagnostic scheme contained in the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV-TR)
(American Psychiatric Association, 2000). Psychotic symptoms are characteristic of a number of
Axis I disorders, but were coded separately due
to their special significance in the conveyance and
determination of violence risk. Where indications
of one of these types of disorders were present, the
strength of the evidence was rated as significant, or
very compelling. Forty-eight of the 60 documents
showed significant or very compelling indications
of at least one of these disorders.
Data
Our data consisted of 60 documents that were
sent to judges in a major metropolitan area in the
United States. These documents were genuine,
40
Indications of Mental Illness
Absent
Present
Psychotic
34
26(7,19)
Axis I Disorder
24
36(15,21)
Axis II Disorder
29
31(3,28)
Table 1: Indications of mental illness appeared in most of the threatening communications. When indications were present, these were shown as counts of total number of document, and further broken down
into counts of (very compelling, significant).
able from a single sender, the documents were examined individually, with an effort to isolate each
document from its companions, in order to maintain a focus on language used in the document itself, and enable clearer comparison with the automated methods used later.
In the actual practice of threat assessment, if
multiple documents were attributed to a single
sender, and the case was not referred for assessment until after multiple documents had been received, the documents would be assessed together
as a pattern of communications. Our approach
more closely parallels the situation faced in assessing anonymous threatening communications,
where knowledge of personal, historical, or clinical factors of the sender is not available. Assessment in these circumstances must rely more
heavily on linguistic factors of the communications (Simons and Tunkel, 2013).
The fact that a single assessor reviewed all the
documents is a limitation of the current study,
which can be addressed in future work.
This research was approved as exempt by the
Partners Institutional Review Board, with the provisions that the confidentiality of materials and the
privacy of individuals be protected.
recipient, and specificity and nature of any threat.
This annotation was performed by one of the authors, who is a board-certified forensic psychiatrist
with over 20 years’ experience in both violence
risk assessment and clinical practice.
The presence of an actual threat in the document, and the nature of that threat, were also
recorded. Interestingly, while all documents were
referred out of concern for the personal safety of
at least one judge or court official, in or outside
the courtroom, only a minority of the documents
threatened violence. Just three of the 60 documents made clear threats of violence, while another five contained vague or ambiguous threats.
Fewer than half (26) contained threats of any kind,
and most of these were threats to take legal action. Other documents expressed threats to reputation – they purported to “expose” or embarrass
the judge in some way. Some threatened to file
an ethics complaint. Other threats were more fanciful and clearly outside the power of the author
to effect. For example, they threatened to report
the judge to a non-existent “people’s committee,”
or threatened punishment from God. Some documents contained more than one threat.
Type of Threat
None
Violence
Legal Action
Ethics Complaint
Reputation
Other
No. of Documents
34
8 (3 clear, 5 vague)
16
4
8
2
5
Methods
The potential for computational text analytic
methods to contribute to violence risk assessment
and threat assessment has been noted (Meloy and
Hoffmann, 2013). We apply two such methods,
LIWC and topic models, to our sample of threatening communications.
Word count-based methods, such as LIWC
(Linguistic Inquiry and Word Count) are widely
used. LIWC’s central premise is that words people use reveal their psychological or emotional
state, and may provide insight into their perceptions and intentions. LIWC has been applied to
assessing text for a range of psychological phenomena (Pennebaker et al., 2001), and recently
has been used for detecting indications of decep-
Table 2: Actual threats of violence are uncommon.
Most communications do not contain a threat.
Based on application of the standard threat assessment methods described above to each document, the perceived risk was rated low for twothirds of the documents (41), moderate for 18, and
high for only one document. These methods consisted of examining each document in isolation.
Where two or more communications were avail41
tion, and of aggression and hostility in the communications of terrorist groups (Pennebaker et al.,
2008; Chung and Pennebaker, 2011).
LIWC is organized into a set of dozens of categories that contain words and word stems. These
may be grammatical categories such as prepositions or pronouns, or they may be more psychologically informed categories such as “anger” (attack, battle, angry, enemy, violent, etc.). LIWC
calculates the percentage of words in a document
that belong in each of its categories.
We also employ topic models, which are probabilistic models for illuminating an underlying semantic or thematic structure within a set of documents (Blei and Lafferty, 2009). As an unsupervised method, a topic model is not based on
some predetermined set of associated words, as is
LIWC, with its dozens of categories for function
words, emotion words, and so on. Instead the topics emerge based on the statistical properties of the
documents themselves. This is a consequence of
documents that are about different things typically
using different words with different frequencies.
When the most frequent words in a topic cohere, it is relatively simple to infer what the topic
is “about.” For example, applying topic modeling to over twenty years of the Yale Law Journal
yielded topics appear to relate to various areas of
the law, such as labor (labor, workers, employees,
union, employer) and contract law (contract, liabilities, parties, contracts, party, creditors) (Blei,
2012).
To help avoid overtraining the model, location
names were removed from the documents. Names
of individuals were replaced with tokens for last
name (LN), male first name (MFN), female first
name (FFN), or middle initial (MI). References to
famous historical figures (e.g., Abraham Lincoln,
Hitler, Winston Churchill) were not altered.
We run a Latent Dirichlet Allocation topic
model (Blei, Ng, and Jordan 2003) using MAL LET (McCallum, 2002) (McCallum 2002) on the
set of threatening communications. In addition
to ignoring the standard English stopwords in our
documents, we also ignore a small set of extremely common words in the documents (district,
court, judge), the “LN” (last name) token, and the
months of the year.
Despite the relatively small size of our document corpus, a number of intriguing topics
emerge. We observe topics relating to corruption,
misconduct and ethics, conspiracy or other delusional beliefs, and family and community relationships.
6
Findings
Expressions of Anger and Negative Emotion
and Violence Risk Expression of anger and
negative emotions has long been considered a factor in violence risk assessment and threat assessment. It has been observed that acts of targeted
violence commonly arise from a grievance on the
part of the perpetrator, such as a perceived injustice (Calhoun and Weston, 2003). Chung and
Pennebaker also find significantly elevated rates of
anger words in the language of Al Qaeda leaders
compared to controls (Pennebaker et al., 2008). In
our threatening communications to judges, however, we do not observe a comparable effect with
respect to perceived violence risk. Words reflecting anger, death, or negative emotions are not used
more frequently in documents that indicate elevated risk. Nor do they vary significantly across
documents reflecting Axis I, Axis II, or psychotic
symptoms.
This may reflect a limitation of any tool such
as LIWC that uses word lists to capture emotion. The expressive capacity of natural language is much greater. For example, one threatening communication that contained no terms from
LIWC’s anger, death, or negative emotion categories, called others “animals” and “CRIMINAL
TRASH!”, who would be “held accountable” for
their actions.
Themes Induced through Topic Modeling Unsurprisingly, given that these threatening communications were sent to judges, often by litigants, terms referencing the judicial system appear
prominently in many topics. A closer look reveals
themes relating to claims of judicial misconduct or
ethical violations, conspiracies and fundamentally
sinful or evil acts (“malum in se”). Such topics are
suggestive of symptoms such as persecutory beliefs, paranoid ideation, hyperreligiosity, and hypermorality that can be found in both Axis I and
Axis II disorders. Tellingly, these themes emerged
from the corpus, not from an a-priori categorization of terms.
Not all topics show potential links to detectable
psychopathology. Another topic relates to family
and emotional attachment, and may be indicative
of child custody or child welfare issues. Topics
42
Risk Level
Elevated
Low
All
Number of Documents
19
41
60
Anger
1.22 (1.02)
1.06 (0.74)
1.11 (0.83)
Death
0.21 (0.40)
0.20 (0.40)
0.20 (0.40)
Negative Emotion
2.42 (1.35)
2.50 (1.45)
2.47 (1.45)
Table 3: Threatening communications judged to show an elevated risk cannot be distinguished from low
risk documents, based on LIWC categories of anger, death, or negative emotion. Means and standard
deviations based on LIWC scores are reported.
from this 10-topic LDA include
of redressing their grievances, e.g. appealing the
decision, seeking other legal remedies, or more
rarely, filing complaints of judicial misconduct.
Others express their disagreement and disappointment in a more direct fashion, either by choice or
because they cannot restrain themselves from doing so, in some cases by communicating implied
or direct threats to judges. In doing so, they cross
the boundary of respect for judges and the legal
system that prevents the majority of litigants from
personalizing and pursuing their grievances.
Some such communications are referred by
their recipients to a protective service responsible for the court in question. The ensuing threat
assessment process yields a determination of the
level and type of violence risk, and the need for
any protective measures. The majority of the communications referred for examination are determined to represent low risk of violence. Others,
however, are considered to represent significant
risk of harm and to require actions to eliminate or
diminish the threat.
Since the office responsible for court security
has not yet cataloged its threatening communications, we cannot ensure that this sample is perfectly representative of all threatening communications received by the courts. Plans to implement such a database are under development. In
addition, we do not have a sample of communications to judges that the recipients themselves did
not find sufficiently threatening to refer for assessment, nor do we know the prevalence of such communications.
This pilot study represents an attempt to use
computational linguistic analysis to explore what
aspects of written communications to judges result in the perception of threat and the determination of risk level. We analyzed a sample of documents referred by their recipients as potentially
threatening. In this sample we found evidence of
direct or implied threat of violence in a small minority of examples. An expert rater categorized
• Relationships, family, and community: love
children years told thing drug wife family conviction make date person community
felony simply letter dss
• Conspiracy and injustice: criminal filed order attorney trial conspiracy federal justice conduct made constitutional dr se abuse
malum
• Misconduct, ethics: judicial complaints appointed justice case attorneys federal commission attorney misconduct ethical conduct
complaint respect integrity.
Efforts to build predictive models for identifying documents containing indications of Axis I,
Axis II, or psychotic symptoms based solely on
topic distributions were not entirely successful.
For example, a logistic regression model using
features based on a 10-topic LDA outperformed
chance on a test set at predicting presence of Axis I
symptoms, achieving excellent recall, but low precision. This may have been due to the small size
of the document collection. Additionally, the overlap of symptoms between Axis I and Axis II may
have lead to topics that do not effectively distinguish between them.
7
Discussion
It is not surprising that judges can be the object of
considerable ire and attention directed at them by
disappointed litigants, family members, or others
who have concerns about legal and social issues.
They sit at the apex of a system that resolves interpersonal conflicts and administers justice, but with
no shortage of disappointed parties.
Because of the important role they play in our
society, judges are normally accorded considerable respect and deference. The majority of disappointed litigants use socially acceptable means
43
The next steps for this work include examination of a larger number of communications referred for assessment of possible increased risk
of violence. Communications addressed to other
public figures, as well as organizations and their
personnel, can be analyzed and compared to those
received by judges. Progress on automating the
extraction of text features that were manually annotated, including distinctive orthographic features (contextually inappropriate use of capitalization and emphasis), and number and titles of recipients would be valuable. In addition, it will be important to have the presence of indicators of mental illness and level of risk, rated independently by
multiple experts in the field of threat assessment
in a two part process. First, the documents will
be rated in the absence of any contextual information. Second, evaluators will be provided with
additional information regarding the individual’s
background and asked to rerate the communications.
only one communication as indicating a high level
of threat. Evidence of mental illness on the part of
the senders was found in the majority of examples
(80 percent).
Possible explanations for the disparity between
the universal perceptions of threat by recipients
and expert assessment of threat may include a
combination of the following:
1. The very act of sending an argumentative or
hostile communication to a judge represents
a breach of normative behavior, and suggests
that the sender may have difficulty controlling hostile impulses and maintaining appropriate boundaries.
2. The popular belief that mental illness is associated with a high risk of violence may
increase the likelihood that communications
containing evidence of psychotic beliefs and
other forms of disordered thinking, but no evidence of threat, get referred by court personnel for further investigation.
8
3. Over-assessment of mental illness by the expert rater, in spite of efforts to be conservative
in those ratings.
Conclusion
Mental health professionals are asked to assess
the risk of violence on a regular basis and in a
wide variety of settings. The accuracy and reliability of this complex and challenging task increases with the amount of information available
to the evaluator. To date, those charged with conducting these assessments have not utilized automated approaches for linguistic analysis to inform
their assessments. The results of this pilot study
suggest that such analysis may be a useful addition to the traditional tools currently used in violence threat assessment. The availability of such
a tool could increase the accuracy and objectivity of currently applied threat assessment methods.
However, more data is needed to train and build
models, and fully test their utility. Supervised machine learning approaches, or more sophisticated
topic models, may be needed to tackle the complexities of supporting violence risk assessment
through language technology.
4. Under-assessment of violence risk by the expert rater, however it should be noted that
documents spanned a period from 1995 to
2013 and there have been no episodes of
violence against judges in that jurisdiction
to date. Whether that represents the true
level of actual risk or the successful efforts
of court security personnel in managing the
threat cannot be determined.
The purpose of the current pilot study was to
explore if language technology could be used to
identify those aspects of a communication that
render it threatening to its recipients or correlate
with expert assessment of the level of violence
threat they present. We applied these tools to a
relatively small group of 60 written communications sent to judges. A single forensic psychiatrist, experienced in threat and violence risk assessment, rated each document individually for the
study factors. The results were promising, yet not
dispositive, with regard to the ability of language
technology to identify those factors that render a
communication “threatening,” are predictive of increased risk, or indicative of mental illness.
Acknowledgments We thank the anonymous
reviewers and Jordan Boyd-Graber for their insightful comments.
References
American Psychiatric Association. 2000. Diagnostic
44
André Simons and Ronald Tunkel. 2013. The assessment of anonymous threatening communications. International Handbook of Threat Assessment, pages 195–213.
and statistical manual of mental disorders: DSMIV-TR R . American Psychiatric Pub.
David M. Blei and John D. Lafferty. 2009. Topic models. Text mining: classification, clustering, and applications, 10:71.
Jennifer L Skeem and John Monahan. 2011. Current
directions in violence risk assessment. Current Directions in Psychological Science, 20(1):38–42.
David M Blei. 2012. Probabilistic topic models. Communications of the ACM, 55(4):77–84.
S Smith and R Shuy. 2002. Forensic psycholinguistics: using language analysis for identifying and assessing offenders. FBI Law Enforcement Bulletin,
71(4):1621.
F Calhoun and S Weston. 2003. Contemporary threat
management. San Diego, CA: Specialized Training
Services.
Frederick S Calhoun. 1998. Hunters and howlers:
Threats and violence against federal judicial officials in the United States, 1789-1993. Number 80.
US Department of Justice, US Marshals Service.
S Smith. 2008. From violent words to violent deeds:
assessing risk from FBI threatening communication
cases. Stalking, Threatening, and Attacking Public
Figures: a psychological and behavioral analysis,
page 435455.
Cindy K Chung and James W Pennebaker. 2011. Using computerized text analysis to assess threatening
communications and behavior. Threatening communications and behavior: Perspectives on the pursuit
of public figures, page 332.
H. J. Steadman, E.P. Mulvey, J. Monahan, and et al.
1998. Violence by people discharged from acute
psychiatric inpatient facilities and by others in the
same neighborhoods. Archives of General Psychiatry, 55(5):393–401, May.
E. B. Elbogen and S. C. Johnson. 2009. The intricate
link between violence and mental disorder: Results
from the national epidemiologic survey on alcohol
and related conditions. Archives of General Psychiatry, 66(2):152–161, February.
Andrew Kachites McCallum. 2002. Mallet: A machine learning for language toolkit.
Jeffrey Swanson, Charles Holzer, Vijay Ganju, and
Robert Jono. 1990. Violence and psychiatric disorder in the community: evidence from the epidemiologic catchment area surveys. Hospital &
community psychiatry, 41(7):761–770, July. PMID:
2142118.
J. Reid Meloy and Jens Hoffmann. 2013. International
Handbook of Threat Assessment. Oxford University
Press.
Christopher D Webster, Kevin S Douglas, Derek Eaves,
and Stephen D Hart. 1982. Assessing risk for violence, version 2 (hcr-20). Sigma, 1993:1997.
John Monahan, Henry J Steadman, Paul S Appelbaum,
Thomas Grisso, Edward P Mulvey, Loren H Roth,
Pamela Clark Robbins, Stephen Banks, and Eric Silver. 2006. The classification of violence risk. Behavioral sciences & the law, 24(6):721–730.
James W Pennebaker, Martha E Francis, and Roger J
Booth. 2001. Linguistic inquiry and word count:
LIWC 2001. Mahway: Lawrence Erlbaum Associates, page 71.
James W Pennebaker, Cindy K Chung, et al. 2008.
Computerized text analysis of al-qaeda transcripts.
A content analysis reader, pages 453–465.
Vernon L. Quinsey, Grant T. Harris, Marnie E. Rice,
and Catherine A. Cormier. 1998. Violent offenders:
Appraising and managing risk. American Psychological Association.
Antonio P. Sanfilippo. 2010. Content Analysis for
Proactive Protective Intelligence. Pacific Northwest
National Laboratory.
Mario J Scalora, Jerome V Baumgartner, Mary A
Hatch Maillette, Christmas N Covell, Russell E
Palarea, Jason A Krebs, David O Washington,
William Zimmerman, and David Callaway. 2003.
Risk factors for approach behavior toward the US
congress. Journal of threat assessment, 2(2):3555.
45