ECONOMIC

DEVELOPMENT
ADMINISTRATION
Malware Infections
on EDAs Systems
Were Overstated
and the Disruption
of IT Operations
Was Unwarranted
FINAL REPORT NO. OIG-13-027-A
JUNE 26, 2013
U.S. Department of Commerce
Office of Inspector General
Office of Audit and Evaluation
FOR PUBLIC RELEASE
UNITED STATES DEPARTMENT OF COMMERCE
Office of Inspector General
Washington, D.C. 20230
June 26, 20 13
MEMORANDUM FOR: Matthew Erskine
Deputy Assistant Secretary of Commerce
for Economic Development
Economic Development Administration
Simon Szykman
Chief
FROM: Allen Crawley G A
Assistant Inspector General for Systems Acquisition
and IT Security
SUBJECT: Malware Infections on EDA's Systems Were Overstated
and the Disruption of IT Operations Was Unwarranted
Final Report No. OIG-13-027-A
Attached is the final report of our audit of EDA's information security program and cyber
incident response. In accordance with the Federal Information Security Management Act, we
evaluated EDA's incident response and recovery activities in relation to EDA's fiscal year 20 12
cyber incident. We (I) assessed the effectiveness of EDA's IT security program, (2) determined
the significant factors that contributed to its incident, and (3) evaluated both completed and
planned activities to recover its information systems to support critical operational
requirements.
We found (I) EDA based its critical incident response decisions on inaccurate information, (2)
deficiencies in the Department's incident response program impeded EDA's response,
and (3) misdirected planning efforts hindered EDA's IT system recovery.
In response to the draft audit report, EDA and the CIO concurred w ith all of our
recommendations. We summarized the responses in the report and included the full response
in the appendixes. We will post this report on the OIG website pursuant to section 8L of the
Inspector General Act of 1978, as amended.
Under Department Administrative Order 213-5, you have 60 calendar days from the date of
this memorandum to submit an audit action plan to us. The plan should outline actions you
propose to take to address each recommendation.
We appreciate the cooperation and courtesies extended to us by your staff during our audit.
Please direct any inquiries regarding this report to me at (202) 482-1855 and refer to the
report title in all correspondence.
Attachment
cc: Thomas Guevara, Deputy Assistant Secretary for Regional Affairs, EDA
Rod Turk, Director, O ffice of Cyber Security, and Chief Information Security Officer
Chuck Benjamin, Chief Information Officer, EDA
Deborah Neff, Audit liaison, EDA
Cara Huang, Audit Liaison, Office of the Chief Information Officer










































Report In Brief
J UN E 26, 2013
Background
The Economic Development Admin-
istrations (EDAs) mission is to lead
the federal economic development
agenda by promoting innovation and
competitiveness, thus preparing Ameri-
can regions for growth and success in
the worldwide economy. To fulfill its
mission, EDA uses six regional offices
to provide services specific to each
regions needs.
In accordance with the Federal Infor-
mation Security Management Act of 2002
(FISMA), we evaluated EDAs incident
response and recovery activities in relation
to EDAs fiscal year 2012 cyber incident.
Why We Did This Review
On December 6, 2011, the Department
of Homeland Security (DHS) notified the
Department of Commerce that it detect-
ed a potential malware infection within
the Departments systems. The Depart-
ment determined the infected compo-
nents resided within IT systems operat-
ing on the Herbert C. Hoover Building
(HCHB) network and informed EDA and
another agency of a potential infection in
their IT systems.
On January 24, 2012believing it had a
widespread malware infectionEDA
requested the Department isolate its IT
systems from the HCHB network. This
action resulted in the termination of
EDAs operational capabilities for enter-
prise e-mail and Web site access, as well as
regional office access to database applications
and information residing on servers connect-
ed to the HCHB network.
Given the Departments limited incident
response capabilities and the perceived
extent of the malware infection, the
Department and EDA decided to aug-
ment the Departments incident re-
sponse team. Additional incident re-
sponse support was provided by DHS,
the Department of Energy, the National
Institute of Standards and Technology,
and the National Security Agency, as well
as a cybersecurity contractor. In early
February 2012, EDA entered into an
agreement with the Census Bureau to
provide an interim e-mail capability, In-
ternet access to EDA staff, and Census
Bureau surplus laptops for EDA staff.
ECONOMIC DEVELOPMENT ADMINISTRATION
Malware Infections on EDAs Systems Were Overstated
and the Disruption of IT Operations Was Unwarranted
OIG-13-027-A
WHAT WE FOUND
Reviewing EDAs IT security program and the events surrounding its December 2011 cyber
incident and recovery efforts, we found that:
EDA Based Its Critical Cyber-Incident Response Decisions on Inaccurate Information. Believing
(a) the incident resulted in a widespread malware infection possibly propagating within its
systems and (b) its widespread malware infection could spread to other bureaus if its IT
systems remained connected to the network, EDA decided to isolate its IT systems from
the HCHB network and destroy IT components to ensure that a potential infection could
not persist. However, OIG found neither evidence of a widespread malware infection nor
support for EDAs decision to isolate its IT systems from the HCHB network.
Deficiencies in the Departments Incident Response Program Impeded EDAs Incident Response.
These deficiencies significantly contributed to EDAs inaccurate belief that it experienced a
widespread malware infection. Consequently, the Department of Commerce Computer
Incident Response Team (DOC CIRT) and EDA propagated inaccurate information that
went unidentified for months after EDAs incident. We found that DOC CIRTs incident
handlers did not follow the Departments incident response procedures, that its handler for
EDAs incident did not have the requisite experience or qualifications, and that DOC CIRT
did not adequately coordinate incident response activities.
Misdirected Efforts Hindered EDAs IT System Recovery. With its incorrect interpretation of
recovery recommendations, EDA focused its recovery efforts on replacing its IT
infrastructure and redesigning its business applications. EDA should have concentrated its
resources on quickly and fully recovering its IT systems (e.g., critical business applications) to
ensure its operational capabilities. Our review of EDAs recovery activities found that
(a) EDA decided to replace its entire IT infrastructure based on its incorrect interpretation
of recovery recommendations and (b) EDAs recovery efforts were unnecessary.
The Department, using already existing shared IT services, returned EDAs systems to their
former operational capabilities (except for access to another Departmental agencys financial
system) in just over 5 weeks of starting its effort.
WHAT WE RECOMMEND
We recommend that the Deputy Assistant Secretary for EDA:
1. Identify EDAs areas of IT responsibility and ensure the implementation of required
security measures.
2. Determine whether EDA can reduce its IT budget and staff expenditures, through the
increased efficiencies of EDAs involvement in the Departments shared services.
3. Ensure that EDA does not destroy additional IT inventory that was taken out of service
as a result of this cyber incident.
We recommend that the Departments Chief Information Officer:
1. Ensure DOC CIRT can appropriately and effectively respond to future cyber incidents.
2. Ensure incident response procedures clearly define DOC CIRT as the incident response
coordinator for the bureaus relying on DOC CIRTs incident response services.
3. Ensure that DOC CIRT management has proper oversight and involvement in cyber
incidents to ensure that required incident response activities take place.

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Contents
Introduction ....................................................................................................................................................... 1
Findings and Recommendations .................................................................................................................... 3
I. EDA Based Its Critical Cyber-Incident Response Decisions on Inaccurate Information .....3
A. Inaccurate Analysis and a Misunderstanding Caused EDAs Perception of a
Widespread Malware Infection..................................................................................................... 4
B. EDAs Belief That Its Malware Infection Was Spreading Heavily Influenced Its
Decision to Isolate Its IT Systems................................................................................................ 6
C. EDAs Severely Deficient IT Security Program Gave Credibility to the Purported
Widespread Malware Infection..................................................................................................... 7
D. EDA Sought Validation of a Sophisticated Cyber Attack .......................................................9
E. External Incident Responders Found No Evidence of a Widespread Malware
Infection or Extremely Persistent Malware ...............................................................................9
Conclusion .............................................................................................................................................. 10
II. Deficiencies in the Departments Incident Response Program Impeded EDAs
Incident Response .............................................................................................................................. 11
A. DOC CIRT Did Not Follow Incident Response Procedures ............................................. 12
B. DOC CIRTs Inexperienced Staff Hindered EDAs Incident Response ........................... 13
C. DOC CIRT Did Not Adequately Coordinate EDAs Incident Response Activities ...... 13
Conclusion .............................................................................................................................................. 14
III. Misdirected Efforts Hindered EDAs IT System Recovery ....................................................... 14
A. EDA Acted on Its Incorrect Interpretation of Recovery Recommendations ................. 14
B. EDAs Recovery Efforts Were Unnecessary .......................................................................... 15
Conclusion .............................................................................................................................................. 16
Recommendations ............................................................................................................................................ 17
Summary of Agency and Departmental Responses and OIG Comments ........................................ 18
Appendix A: Objectives, Scope, and Methodology ................................................................................ 19
Appendix B: Detailed Timeline of EDAs Cyber Incident Response and Recovery ....................... 21
Appendix C: Agency Response................................................................................................................... 24
Appendix D: Departmental Response ...................................................................................................... 28
FINAL REPORT NO. OIG-13-027-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

FINAL REPORT NO. OIG-13-027-A 1
Introduction
The Economic Development Administrations (EDAs) mission is to lead the federal economic
development agenda by promoting innovation and competitiveness, thus preparing American
regions for growth and success in the worldwide economy. To fulfill its mission, EDA uses six
regional offices
1
to provide services specific to each regions needs.
On December 6, 2011, the U.S. Computer Emergency Response Team (US-CERT)a part of
the Department of Homeland Security (DHS)notified the Department of Commerce
Computer Incident Response Team (DOC CIRT
2
) that it detected a potential malware
infection
3
within the Departments systems. DOC CIRT determined the infected components
resided within IT systems operating on the Herbert C. Hoover Building (HCHB) network.
Accordingly, DOC CIRT informed EDA and the National Oceanic and Atmospheric
Administration (NOAA) of a potential infection in their IT systems. NOAAs Computer
Incident Response Team analyzed the information provided by DOC CIRT and identified the
infected component. NOAA remediated the malware infection and placed the remediated
component back into operation by January 12, 2012.
By contrast, on January 24, 2012believing it had a widespread malware infectionEDA
requested the Department isolate its IT systems from the HCHB network. This action resulted
in the termination of EDAs operational capabilities for enterprise e-mail and Web site access,
and regional office access to database applications and information residing on servers
connected to the HCHB network.
Given DOC CIRTs limited incident response capabilities and the perceived extent of the
malware infection, the Department and EDA decided to augment the DOC CIRTs incident
response team. Additional incident response support was provided by US-CERT, the
Department of Energy (DOE) Computer Incident Response Team, the National Institute of
Standards and Technology (NIST) Security Implementation and Incident Response Team, and
the National Security Agency (NSA). In addition, EDA retained the services of a cybersecurity
contractor.
In early February 2012, EDA entered into an agreement with the Census Bureau to provide an
interim e-mail capability, Internet access to EDA staff, and Census Bureau surplus laptops for

1
Regional offices are located in Atlanta, GA; Austin, TX; Chicago, IL; Denver, CO; Philadelphia, PA; and Seattle,
WA.
2
The DOC CIRT provides computer incident response support to most of the Departments operating units that
use the Herbert C. Hoover Building networkan Office of the Chief Information Officer-managed infrastructure
that many of the bureaus, like EDA, connect to for Department services, Internet connectivity, and communication
infrastructure support for internal system operation. Incident response services include interfacing with and
reporting incidents to and from the US-CERT, performing malware analysis, interfacing with the Departments
network and security operations centers to coordinate changes in network configuration or monitoring resulting
from an incident, and providing remediation guidance.
3
Malware is software used by attackers to disrupt computer operation, gather sensitive information, or gain access
to computer systems. In EDAs incident, the notification indicated the presence of fake antivirus (FakeAV)
software, which deceives a user into executing an application masquerading as antivirus or a malware removal tool.

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
EDA staff. See appendix B for a detailed timeline of events for EDAs cyber-incident response
and recovery.
In accordance with FISMA,
4
we evaluated EDAs incident response and recovery activities in
relation to EDAs fiscal year (FY) 2012 cyber incident. We (1) assessed the effectiveness of
EDAs IT security program, (2) determined the significant factors that contributed to the
incident, and (3) evaluated both completed and planned activities to recover its information
systems to support critical operational requirements. See appendix A for details regarding our
objectives, scope, and methodology.
4
The Federal Information Security Management Act of 2002 (FISMA), 44 U.S.C 3541 (2002), et seq., requires
agencies to secure systems through the use of cost-effective management, operational, and technical controls. The
statutes goal is to provide adequate security commensurate with the risk and extent of harm resulting from the
loss, misuse, or unauthorized access to or modification of information collected or maintained by or on behalf of
an agency. In addition, FISMA requires inspectors general to evaluate agencies information security programs and
practices by assessing a representative subset of agency systems, and results are reported to the Office of
Management and Budget, the Department of Homeland Security, and Congress annually.
FINAL REPORT NO. OIG-13-027-A 2

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
Findings and Recommendations
As part of our annual FISMA work, we reviewed EDAs IT security program and the events
surrounding its December 2011 cyber incident and recovery efforts. We found that (1) EDA
made key incident response and recovery decisions with inaccurate information, (2) DOC
CIRTs insufficient incident response efforts degraded the quality of EDAs incident response,
and (3) EDAs misdirected efforts hindered the recovery of its IT systems.
I. EDA Based Its Critical Cyber-Incident Response Decisions on Inaccurate
Information
EDA believed the incident resulted in a widespread malware infection that was possibly
propagating within its systems. Furthermore, EDA believed that its widespread malware
infection could spread to other bureaus if its IT systems remained connected to the
network, so EDA decided to isolate its IT systems from the HCHB network.
OIG found no evidence to support EDAs beliefs. Specifically, we found no evidence of a
widespread malware infection. Further, we found no evidence to support EDAs decision to
isolate its IT systems from the HCHB network.
The perception of a widespread malware infection and EDAs incident response decisions
are attributable to several factors:
DOC CIRTs inaccurate analysis and a misunderstanding caused EDAs perception of a
widespread malware infection.
EDA believed that the malware infection would spread to other bureaus on the
HCHB network.
Serious long-standing deficiencies in EDAs IT security program gave credence to
EDAs belief that it experienced a widespread malware infection.
EDAs belief in its widespread malware infection led it to seek validation of a
sophisticated cyber attack.
5
EDA based its recovery decisions on its belief that it faced a widespread malware
infection that included extremely persistent malware.
6
5
A sophisticated cyber attack typically involves the use of attack techniques, such as exploiting previously unknown
vulnerabilities, to successfully compromise a component.
6
Extremely persistent malware cannot be eradicated by reimaging the infected systems hard drive (e.g., malware that
infects a devices firmware in order for the infection to persist).
FINAL REPORT NO. OIG-13-027-A 3

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
A. Inaccurate Analysis and a Misunderstanding Caused EDAs Perception of a Widespread
Malware Infection
EDA believed that a cyber attack resulted in an extensive malware infection affecting
over half of its components.
7
This belief originated on the first day of incident response
activities when DOC CIRT sent EDA inaccurate information concerning the extent of
the malware infection, which overstated the number of components involved.
Additionally, EDA misunderstood DOC CIRTs follow-up communications, which
accurately described the limited extent of the infection. Even though additional
communications occurred between DOC CIRT and EDA, each organization continued
to have a different understanding of the extent of the malware infection.
DOC CIRTs first incident notification was misleading. On December 6, 2011,
US-CERT alerted DOC CIRT to suspicious activity, which involved EDAs systems,
on the HCHB network. In an effort to identify infected components, DOC CIRTs
incident handler requested network logging information. However, the incident
handler unknowingly requested the wrong network logging information (see finding
II, subfinding B, for more information on the incident handler). Consequently, on
December 7, 2011, DOC CIRT sent an e-mail incident notification to EDA (in
response to US-CERTs alert) that inaccurately described the extent of the potential
malware infection. Instead of providing EDA a list of potentially infected
components, the incident handler mistakenly provided EDA a list of 146
components
8
within its network boundary. Accordingly, EDA believed it faced a
substantial malware infection.
DOC CIRTs mistake resulted in a second incident notification. Early on
December 8, 2011, an HCHB network staff member informed DOC CIRT that the
incident handlers request for network logging information did not identify the
infected components. Rather, the response merely identified EDA components
residing on a portion of the HCHB network (i.e., the listing of 146 components
initially provided to EDA). The HCHB network staff member then performed the
appropriate analysis identifying only two components exhibiting the malicious
behavior in US-CERTs alert. With this new information, DOC CIRT sent EDA a
second e-mail incident notification.
DOC CIRTs second incident notification was vague. DOC CIRTs second
incident notification did not clearly explain that the first incident notification was
inaccurate. As a result, EDA continued to believe a widespread malware infection
was affecting its systems. Specifically, the second incident notification
Began by stating the information previously provided about the incident was correct.
EDA interpreted the statement as confirmation of the first incident
7
EDAs IT system was comprised of approximately 250 IT components (e.g., desktops, laptops, and servers).
8
The first incident notification contained an attachment with 146 distinct potentially infected components. DOC
CIRT, EDA, and external incident responders reported numbers ranging from 142 to 148 components, but the
accurate count from the incident notification is 146 components.
FINAL REPORT NO. OIG-13-027-A 4

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
notification, when DOC CIRTs incident handler simply meant to confirm EDA
was the agency identified in US-CERTs alert. Nowhere in the notification or
attachment does the DOC CIRT incident handler identify that there was a
mistake or change to the previously provided information.
Contained an attachment name that further obscured any clarification. Although the
incident notifications attachment correctly identified only 2 components
exhibiting suspicious behaviornot the 146 components that DOC CIRT
initially identifiedthe name of the second incident notifications attachment
exactly matched the first incident notifications attachment, obscuring the
clarification.
DOC CIRT and EDAs misunderstanding continued. Over the next 5 weeks,
additional communications occurred between DOC CIRT and EDA. However, each
organization continued to have a different understanding of the extent of the
malware infection. DOC CIRT believed the incident affected only two components,
whereas EDA believed the incident affected more than half of its components.
Several factors contributed to these different interpretations:
DOC CIRT assumed EDA understood that its second incident notification
superseded the first incident notification and that there were only 2 potentially
infected componentsnot 146. However, DOC CIRT did not follow up to
establish whether EDA understood the new information.
EDA responded to the second incident notification by providing a sample of
two components (on the list identified in the first incident notification and that
were exhibiting malicious behavior) for forensic analysis. DOC CIRT believed
the sample to be the same two components identified in the second incident
notification.
When DOC CIRT confirmed that the sample of 2 components was infected
with malware, EDA believed that DOC CIRT had confirmed the malware
infection for all 146 components listed in the first incident notification.
DOC CIRT did not retain the first incident notification showing 146
components or document initial incident response activities. Therefore, when
DOC CIRT management became involved in the incident response activities,
they could not see that a misunderstanding had occurred.
When DOC CIRT asked EDA to carry out typical containment measures (reimaging
9
the infected components), EDA informed DOC CIRT there were too many
components involved making typical containment measures unfeasible. DOC CIRT
assumed EDA performed an independent analysis to identify additional infected
components (even though EDA lacked the necessary capabilities) and assumed EDA was
now dealing with a widespread malware infection. Likewise, EDA assumed DOC CIRT
was aware of the incidents magnitude, given that DOC CIRT provided the list of
9
Reimaging is the process of reinstalling the operating system and applications on a hard drive, as well as restoring
the necessary information from known good backups.
FINAL REPORT NO. OIG-13-027-A 5

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
infected components in its first incident notification. Now, EDA and DOC CIRT were
operating with the samealbeit inaccuratebelief.
Unfortunately, both organizations continued to propagate the inaccurate information
(the basis for the widespread malware infection) during the incident response activities.
DOC CIRTs representation of the extent of the malware infection was accepted by
DHS and not independently validated in its draft report. DHSs draft report stated,
over 143 systems infected with common fake anti-virus and 50 percent of EDAs
network is infected,
10
which portrayed a widespread malware infection. The NSA
report stated that the EDA network was extremely inundated with malware and the
extent of the compromise and the state of the overly infected network will make it very
difficult to deconflict the vast amount of indicators.
11
NSA did not independently verify
incident information, but it presented similar information to that presented by DHS as
fact. As a result, EDA believed these incident reports
12
supported its conclusion
regarding the extent of the malware infection.
The misunderstanding went undetected by EDA until December 18, 2012and by the
Department until December 19, 2012when OIG completed its validation of events
and informed both organizations of its initial conclusions.
B. EDAs Belief That Its Malware Infection Was Spreading Heavily Influenced Its Decision to
Isolate Its IT Systems
On January 24, 2012, EDA, at the recommendation of EDAs current chief information
officer (CIO), decided to isolate EDAs IT systems from the HCHB network. EDAs
CIO believed that (1) EDA experienced a widespread malware infection, (2) the
malware infection was spreading within EDAs IT systems, and (3) the malware infection
could spread to other bureaus residing on the HCHB network. Specifically, EDAs CIO
believed that an antivirus scan of EDAs primary e-mail server indicated multiple
malware infections and the malware infection could propagate to other bureaus on the
HCHB network. However, we found no evidence to support these beliefs. Specifically,
There was no widespread malware infection. EDA based its conclusion on inaccurate
information (see finding 1, subfinding A).
There was no indication of an infection in the e-mail server. Our analysis of the e-mail
servers antivirus logs showed that the antivirus software was up-to-date (e.g., with
10
U.S. Department of Homeland Security, National Cyber Security Division, February 7, 2012. Strategic Remediation
Strategy for Department of Commerce/Economic Development Administration, Draft Version 1.0. Washington, DC: DHS
National Cyber Security Division, 1. DHS did not issue a final version of its report.
11
National Security Agency, Computer Network Operations Countermeasures Division, Information Assurance
Directorate, May 15, 2012. IAD Intrusion Response of Department of Commerce Economic Development Administration,
I3331-004R-2012. Ft. Meade, MD: NSA, 4.
12
NIST did not issue an incident response report. DOEs incident report addressed the results of an assessment
on one componentanalysis indicated trace evidence of an attempted infection but no extremely persistent
malwarenot the incident as a whole.
FINAL REPORT NO. OIG-13-027-A 6

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
the most current software version and latest malware definitions), was scanning
weekly, and had not identified any malware. Not only was EDAs CIO unable to
substantiate his assertion with credible evidence, EDAs IT staff did not support
the assertion of an infection in the e-mail server.
The e-mail server did not pose an increased risk. EDAs outbound e-mail traffic does
not pass through any other e-mail systems before reaching the Internet; therefore,
the infection would not have spread the way EDAs CIO believed. Further, e-mail
traffic in general does not pose a risk to an e-mail server, as infected e-mail
attachments typically require user interaction. Additionally, the Department has
security measures to address infected e-mail attachments. Thus, EDAs e-mail
server did not pose an increased risk, even if it had been infected.
C. EDAs Severely Deficient IT Security Program Gave Credibility to the Purported Widespread
Malware Infection
Since 2006, OIG has identified significant deficiencies in EDAs IT security program.
NSAs 2009 review
13
further emphasized these deficiencies with the discovery of
multiple common malware
14
infections. We reviewed EDAs IT security program after
its incident and found that many of the deficiencies identified in past reviews remained
unremediated for more than 4 years (see table 1 below for examples of deficient
security measures).
13
In 2009, NSA reviewed security measure implementations on IT systems operating on the HCHB network.
14
Common malware (e.g., spyware, virus, or Trojans), although typically malicious and potentially harmful, can be
removed using common cleaning tools and processes (e.g., reimaging).
FINAL REPORT NO. OIG-13-027-A 7

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
Table 1. Examples of EDAs Long-Standing Security Deficiencies
Security
Measure
Definition Significance
Deficiencies
Identified In
OIG and
NSA
Reviews
a
Secure
Configurations
The processes an organization
uses to define how to secure its IT
products (e.g., operating systems,
databases, and web applications)
limiting the functions of a
component to minimal operations
Without effective secure
configurations, an organization
will not effectively limit
unauthorized use of its
components. Securely
configuring IT products is a
fundamental and critical security
measure (one of DHSs and
NSAs key recovery
recommendations to EDA).
2006, 2009,
2010, 2012: EDA
had not defined
or implemented
this security
measure.
Patch
Management
The processes an organization
uses to track and correct software
(e.g., operating system and
application) vulnerabilities
Without effective patch
management, vulnerabilities can
remain unremediated, leaving
components vulnerable to
compromise and information
less secure.
2009, 2010,
2012: EDA did
not reliably
track
b
or correct
vulnerabilities
(some for many
years).
Auditing and
Monitoring
The processes and tools used to
detect the use of systems and
information by an unauthorized
user or external attackers
Without effective auditing and
monitoring, an organization may
not be able to track
unauthorized access to
components and information,
follow an attackers activities, or
reconstruct what happened
when an incident occurs.
2006, 2012: EDA
did not monitor
for suspicious
activity in its
systems.
Security
Assessments
Assessments performed to
determine the extent of security
mechanism implementation
Without the appropriate
assessment of security
mechanisms, organizations will
not have an accurate picture of
the risks to the system and
management will not have the
2006, 2010,
2012: EDAs
assessment
methodologies
did not
appropriately
identify
information necessary to make
appropriate risk-based
decisions.
deficiencies or
convey risks to
operations and
information.
Source: OIG FISMA reviews from 2006, 2010, and 2012 and NSAs 2009 review
a
Not all security mechanisms were assessed in the course of each OIG FISMA review or in NSAs 2009 review.
b
Prior to May 2011, EDAs systems had not been scanned for almost a year. When scans resumed, they identified
over 35,000 potential vulnerabilities. Scans performed in December 2011, just prior to the incident, indicated that
EDA was struggling to remediate these vulnerabilities. OIGs post-incident review found 37 percent (56 of 151) of the
vulnerabilities highlighted by the NSA in 2009 still existthe NSA asserted in its incident response report that EDA
did not address remediation recommendations from the NSAs 2009 assessment of EDAs IT systems.
EDAs current CIO joined the organization in April 2011. The CIO inherited an IT
security program suffering from longstanding and significant security deficiencies. For
example, the CIO briefed EDA leadership that (1) EDA IT staff lacked appropriate IT
security skills, (2) system configuration management and secure configurations were not
implemented, and (3) systems were not appropriately monitored.
FINAL REPORT NO. OIG-13-027-A 8

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
The Department and EDA
15
knew of EDAs many IT security program deficiencies;
therefore, they more readily believed there was a widespread malware infection.
Further, when external incident responders analyzed the incident, they too observed
pervasive deficienciesthe result of too few implemented IT security mechanisms.
Their observations further reinforced the credibility of a widespread malware infection.
Furthermore, the pervasive deficiencies led the Office of the Chief Information Officer
(OCIO) and EDA not to question the accuracy of the extent of the malware infection,
despite a lack of supporting evidence.
D. EDA Sought Validation of a Sophisticated Cyber Attack
EDA hired a cybersecurity contractorin addition to other external agencies already
responding to the incidentto perform an in-depth evaluation of the malware infection
in its systems. EDAs CIO and senior leadership were specifically concerned about
nation-state actors
16
and the presence of extremely persistent malware that would
prohibit typical containment measures, such as reimaging infected components for
immediate use.
On January 30, 2012, EDAs cybersecurity contractor began looking for suspicious
activity and malware infections. Preliminary analysis found indications of extremely
persistent malware and suspicious activity on EDAs components. EDA immediately
acted upon this preliminary information and began an investigation of its entire IT
component inventory for potential infections.
E. External Incident Responders Found No Evidence of a Widespread Malware Infection or
Extremely Persistent Malware
Within 2 weeks of beginning its incident response activities, EDAs cybersecurity
contractor found the initial indications of extremely persistent malware were false
positivesnot actual malware infections. However, EDAs CIO sought guaranteed
assurance that the components were infection-free and no malware could persist.
External incident responders were unable to provide the assurance EDAs CIO sought,
because doing so involved proving that an infection could not exist rather than that one
did not exist. By April 16, 2012, despite months of searching, EDAs cybersecurity
contractor was unable to find any extremely persistent malware or indications of a
targeted attack on EDAs systems. Further, the NSA and US-CERT did not find nation-
state activity or extremely persistent malware.
On May 15, 2012, EDAs management determined that the forensics investigation was
unlikely to yield new evidence and instead focused on cleaning its data
17
and other
15
The Departments annual internal IT reviews have identified IT security deficiencies in EDAs IT security
program.
16
Nation-state actors are hackers acting on behalf of a nations government to engage in nefarious activity, such as
cyber war and theft of intellectual property.
17
Cleaning involves using several antivirus products to scan data files for indications of an infection.
FINAL REPORT NO. OIG-13-027-A 9

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
recovery activities. Ultimately, incident responders identified only six components
18
with
malware infections. These malware infections could have been remediated using typical
containment measures (e.g., reimaging), which normally have a minimal operational
impact. Additionally, EDAs cybersecurity contractors data cleaning efforts did not
identify any additional components with a malware infection (the contractor did identify
the existence of common malware contained in archived e-mail attachments and
temporary Internet browser files
19
). Typically, antivirus software prevents common
malware from executing; as a result, the contractor did not consider the malware a
threat to EDAs components.
Given EDAs history of common malware infections (the NSA identified common
malware on EDAs IT systems in its 2009 review), there was a high probability that
external incident responders would find some malware infections when investigating
EDAs incident. In fact, EDAs lack of implemented IT security and the significant number
of easily exploitable vulnerabilities negated an attackers need to use costly attack
techniques (sophisticated cyber attacks) to compromise EDAs systems. EDAs deficient
IT security posture made it likely that external incident responders would find common
malware. In the end, nothing identified on EDAs components posed a significant risk to
EDAs operations.
However, EDAs CIO concluded that the risk, or potential risk, of extremely persistent
malware and nation-state activity (which did not exist) was great enough to necessitate
the physical destruction of all of EDAs IT components.
20
EDAs management agreed
with this risk assessment and EDA initially destroyed more than $170,000 worth of its
IT components,
21
including desktops, printers, TVs, cameras, computer mice, and
keyboards. By August 1, 2012, EDA had exhausted funds for this effort and therefore
halted the destruction of its remaining IT components, valued at over $3 million. EDA
intended to resume this activity once funds were available. However, the destruction of
IT components was clearly unnecessary because only common malware was present on
EDAs IT systems.
Conclusion
Since EDA did not validate the information (e.g., number of infected components and
potentially spreading malware infection) it used to make its key decisions, it unnecessarily
expended a large portion of its IT budget and many months investigating its incident and
planning for the recovery of its IT systems. Despite only finding common malware
18
External incident responders identified six infected components, two with rootkits (software that enables a
persistent infection) and four with common malwareincluding the two components DOC CIRT identified.
19
Web browsers store on the IT components hard drive the information downloaded from each Web page
visited to enhance browser performance. Although the industry labels this information temporary, the
information remains on the components hard drive until manually deleted.
20
Prior to the incident, EDA purchased laptops intended as replacements for its current desktop and laptop
environment. Because these new laptops had not been operational, EDA could incorporate them into its new IT
systems.
21
EDA tracks the acquisition value, rather than the depreciated value, of its components.
FINAL REPORT NO. OIG-13-027-A 10

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
infections, EDAs management and CIO remained convinced that there could be extremely
persistent malware somewhere in EDAs IT systems.
To recover from its perceived widespread malware infection, EDA took the following
significant recovery steps:
Employed a cybersecurity contractor to investigate the malware infection and ensure
its important data was free of malware
Entered into an agreement with the Census Bureau to provide EDA with an interim,
minimalistic IT solution
22
Physically destroyed IT components to ensure that a potential infection could not
persist
Employed a contractor to assist in the development of a long-term recovery solution
EDA expended more than $2.7 millionover half of EDAs FY 2012 IT budget (see table 2
below for expenditures and finding 3 for further discussion of recovery activities) in pursuit of
these recovery activities. EDAs persistent mistaken beliefs resulted in an excessive response
and ultimately unnecessary expenditure of valuable resources.
Table 2. Significant Recovery Activity Expenditures
Activity Expenditure
a
Cybersecurity contractor investigation of malware infection and data
cleaning
$823,000
Temporary infrastructure, pending long-term IT solution $1,061,000
Destruction of IT equipment
b
$175,000
Contractor assistance for a long-term recovery solution $688,000
TOTAL EXPENDITURES $2,747,000
Source: Contracts from EDAs recovery efforts
a
All values in the table are rounded.
b
EDA paid $4,300 to destroy $170,500 in IT equipmentthese are rounded values.
II. Deficiencies in the Departments Incident Response Program Impeded
EDAs Incident Response
Deficiencies in HCHBs incident response program (DOC CIRT) significantly contributed to
EDAs inaccurate belief that it experienced a widespread malware infection; consequently,
DOC CIRT and EDA propagated inaccurate information that went unidentified for months
after EDAs incident.
We found the following deficiencies in DOC CIRTs incident response activities:
22
EDA did not intend for the Census Bureau to provide a final IT recovery solution. Instead, the Census Bureau
provided an interim solution that met EDAs minimum operating requirements until EDA could develop a
permanent solution.
FINAL REPORT NO. OIG-13-027-A 11

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

DOC CIRTs incident handlers did not follow the Departments incident response
procedures.
DOC CIRTs incident handler for EDAs incident did not have the requisite
experience or qualifications.
23
DOC CIRT did not adequately coordinate incident response activities.
A. DOC CIRT Did Not Follow Incident Response Procedures
When responding to EDAs incident, DOC CIRT staff did not appropriately follow
incident response procedures. Specifically, DOC CIRT staff did not (1) properly
document the initial incident response activities, (2) establish the extent of the malware
infection, and (3) perform a required containment procedure.
DOC CIRT did not properly document the initial incident response activities.
We found DOC CIRT did not document its communications with EDA or record
pertinent incident details like requests, actions taken, or analysis results. For
example, the incident handler deleted the first incident notification showing 146
potentially infected components and retained only the second incident notification
showing 2 potentially infected components. Had the incident handler documented all
information, per the Departments incident response procedures, it would have
been more likely that other DOC CIRT staff or external incident responders could
have identified the misunderstanding regarding the extent of the malware infection.
As a result, EDA, the Department, and external incident responders would not have
needed to expend resources to resolve a widespread malware infection that did not
exist.
DOC CIRT did not accurately establish the extent of the malware infection.
We found that DOC CIRT staff did not appropriately establish the extent of the
malware infection prior to proceeding with other incident response activities (e.g.,
conducting forensic analysis). The Departments incident response procedures
require that incident handlers establish the extent of an infection before proceeding
with other incident response activities so that all involved in the incident response
efforts can formulate realistic containment and mitigation strategies. Since DOC
CIRT did not accurately establish the extent of EDAs incident, EDAs
misunderstanding (e.g., EDA thought there were 146 infected components instead of
only 2) influenced everyones perception of the incident and contributed to EDAs
unnecessary recovery and remediation activities.
DOC CIRT did not appropriately perform a required containment procedure.
When HCHB network staff correctly determined that US-CERTs alert involved two
components, DOC CIRTs incident handler should have followed the Departments
required containment procedure. Specifically, the incident handler should have
23
The Space and Naval Warfare Systems Command (SPAWAR), a Department of the Navy organization, provided
the OCIO cybersecurity technical supportincluding an incident handlerspecified in an interagency agreement
that ended on February 8, 2012.
FINAL REPORT NO. OIG-13-027-A 12

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
directed HCHB security operations center staff to block HCHB network activity
associated with the malicious address identified in US-CERTs alert. Furthermore, on
December 15, 2011, EDA reminded DOC CIRTs incident handler to block the
malicious address. However, DOC CIRT did not initiate this action until January 24,
2012, the same day EDAs systems were isolated from the HCHB network.
B. DOC CIRTs Inexperienced Staff Hindered EDAs Incident Response
DOC CIRTs inexperienced staff and inadequate knowledge of EDAs incident response
capabilities
24
hindered its ability to provide adequate incident response services. DOC
CIRTs incident handler managing EDAs initial incident response activities had minimal
incident response experience, no incident response training, and did not have adequate
skills to provide incident response services. The lack of experience, training, and skills
led the incident handler to request the wrong network logging information (i.e., perform
the wrong incident analysis), which led EDA to believe it had a widespread malware
infection, and deviate from mandatory incident response procedures. The Departments
Office of the Chief Information Officer should have ensured that all DOC CIRT staff
met the Departments minimum incident response qualifications.
In addition, DOC CIRT staff did not understand that there was a preexisting
expectation of specific incident response services, as outlined in the service level
agreement (SLA) between the DOC CIRT and EDA. This agreement clearly states DOC
CIRTs obligated incident response services (e.g., investigation, forensics, and reverse
engineering) and defines EDAs incident response responsibilities (e.g., reporting
incidents and dealing with quarantined or deleted malware). Since DOC CIRT staff did
not understand this agreement, they inaccurately assumed EDA was capable of
performing its own incident analysis activities (e.g., determining the extent of the
malware infection).
C. DOC CIRT Did Not Adequately Coordinate EDAs Incident Response Activities
DOC CIRT is responsible for coordinating incident response efforts (e.g., dissemination
of information and coordination of incident response activities). However, DOC CIRT
did not effectively coordinate EDAs incident response activities. The inadequate
coordination resulted in haphazard communications, in which external incident
responders received minimal direction. As a result,
External incident responders performed redundant forensics analysis on the same
components. External incident responders unnecessarily and wastefully expended
resources to develop the same conclusions.
The quality of EDAs incident response suffered. DOC CIRT and EDA did not use
external incident responders technical knowledge and experience to their fullest
potential.
24
Each Departmental bureau has designated incident responders and its own set of internal incident response
capabilities. The skill level (as gauged by an incident responders training, certifications, and previous incident
response experience) and the tools available within each bureau differ.
FINAL REPORT NO. OIG-13-027-A 13

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
Gaining a full understanding of EDAs incident was difficult. Inadequate coordination
resulted in undirected incident response efforts and uncoordinated distribution of
pertinent incident information, making it difficult to gain a holistic and unbiased
view of the incident.
Conclusion
OIG briefed the Departments CIO on weaknesses within the DOC CIRT that we identified
during our review of incident response activities. Accordingly, the Department has taken
actions to correct DOC CIRTs weaknesses. Specifically, the Department is taking steps to:
Ensure staff receive appropriate training
Update incident response procedures
Review services offered (including the needs and capabilities of each bureau)
Develop agreements with external agencies to provide incident response expertise
Hire experienced incident handlers
III. Misdirected Efforts Hindered EDAs IT System Recovery
Based on EDAs erroneous belief that it had a widespread malware infection, and its
incorrect interpretation of recovery recommendations, EDA focused its recovery efforts on
replacing its IT infrastructure and redesigning its business applications. EDA should have
concentrated its resources on quickly and fully recovering its IT systems (e.g., critical
business applications) to ensure its operational capabilities.
Our review of EDAs recovery activities found the following:
EDA decided to replace its entire IT infrastructure based on its incorrect
interpretation of recovery recommendations.
EDAs recovery efforts were unnecessary.
A. EDA Acted on Its Incorrect Interpretation of Recovery Recommendations
EDA received similar recovery recommendations from NSA and DHS that focused on
quickly recovering IT services (e.g., reimaging infected components), implementing
security mechanisms and best practices, and monitoring its recovered IT systems for
suspicious activity. These recovery recommendations, conventional practices used to
recover from a cyber incident, were appropriate for EDAs recovery.
EDAs continued belief in the necessity of permanent remediation actions (i.e.,
destroying its IT components) and a significant malware infection contributed to EDA
incorrectly interpreting the recovery recommendations. EDA erroneously interpreted
one of DHSs draft recommendationsa complete network rebuild is
recommendedas both prescriptive guidance and direct support for its decision to
FINAL REPORT NO. OIG-13-027-A 14

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
replace its entire IT infrastructure. However, DHSs full draft recommendation advised
EDA to reimage all IT components and implement required security measures,
effectively rebuilding its network. Neither DHSs nor NSAs recommendations provided
a basis for EDAs decisions to replace its IT infrastructure and destroy its IT
components.
B. EDAs Recovery Efforts Were Unnecessary
Despite recovery recommendations from DHS and NSA advising EDA to focus on
quickly and fully recovering its IT systems, EDA focused instead on building a new,
improved IT infrastructure and redesigning its business applications. In September 2012
(8 months after isolation), EDA leadership presented to the Commerce IT Review
Board (CITRB) a request to reprogram funds to carry out its recovery efforts; the
CITRB did not approve EDAs request.
25
EDA estimated it would need over $26 million
disbursed in the next 3 years (an increase from $3.6 million to approximately $8.83
million, or about 2.5 times more, to the bureaus average annual IT budget) to fund its
recovery efforts. However, EDAs intended recovery efforts
Had a fundamental flaw in acquiring funding. EDA leadership did not understand that
the funds it requested to reprogramover $17 million originally designated for
public works and disaster recoverywould actually need to be repurposed.
26
Had an unrealistic time frame for acquiring requested funding. The requests time
frame would have required EDA to gain approval by October 2012 in order to
maintain the intended schedule. This was an extremely aggressive time frame,
given the process (in which CITRB approval was the first step) and time necessary
to attain proper clearance to use the funds.
Would leave EDA reliant on a less effective grants management process. EDA users
would only have limited access to critical business applications. EDA was not
scheduled to complete development of replacement applications until the end of
FY 2014 (more than 2 years after isolation).
Conflicted with the Departments ongoing development of a grants management shared
service. EDAs request for funding to redesign its business applications overlapped
with the Departments development of a grants management shared service.
Further, the following contradicted the direction of EDAs recovery efforts:
25
The CITRB provides oversight, review, and advice to the Secretary and Deputy Secretary on both IT and non-IT
investments that meet certain criteria. This advice includes recommendations for approval or disapproval of
funding for new systems and investments, as well as major modifications to existing systems and investments.
26
According to EDA, it would have needed Departmental and OMB approval of its request to fund its recovery
efforts before presenting the request to Congress. EDA would also have needed to request that Congress change
the law dictating the original purpose and use of the funds requested.
FINAL REPORT NO. OIG-13-027-A 15

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
External incident responders identified only common malware that could be easily
mitigated. As a result, there was no need for EDA to destroy or replace existing IT
components.
NSA found no malware infection in the servers hosting EDAs primary business
application. Additionally, there was no evidence to suggest that EDAs primary
business application had been targeted by a cyber attack or maliciously altered
thus, EDA could have put the application back into full operation.
Conclusion
Although EDA intended to use federal government shared services or outsourced
commercial services during its recovery efforts, EDA had not finalized a recovery solution.
Further, the Department had existing shared IT services (e.g., image for rebuilding infected
components, enterprise e-mail, and help desk services) that were readily available to EDA.
However, only after OIG informed the Department and EDA that there was no widespread
malware infection, and therefore no significant incident, did the Department and EDA enact
a swift recovery of EDAs IT systems using the Departments shared services.
Once it started recovery efforts in February 2013, the Department needed only a little
longer than 5 weeks to restore EDAs former operational capabilities.
27
By comparison,
EDAs incomplete efforts spanned almost a year. Specifically, the Department provided EDA
with enterprise e-mail, account management services, help desk support services, and a
securely configured and uniform image for its laptops. Additionally, the Department
restored EDA users access to critical business applications.
For the time being, EDA will retain responsibility for maintaining its business applications;
however, it may in the future use the Departments grants management services. With the
Department developing and maintaining the IT systems, there is a greater likelihood that the
Department will appropriately implement the required security measures (e.g., secure
configurations, auditing and monitoring, and patch management) that EDA struggled to
implement. Fortunately, for EDA, its involvement in the Departments shared services
initiatives not only restored its critical IT systems and business applications, but should also
reduce its IT budgetary requirements.
27
EDAs previous access to NOAAs financial system has yet to be restored.
FINAL REPORT NO. OIG-13-027-A 16

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
Recommendations
We recommend that the Deputy Assistant Secretary for EDA:
1. Identify EDAs areas of IT responsibility and ensure the implementation of required
security measures.
2. Determine whether EDA can reduce its IT budget and staff expenditures, through the
increased efficiencies of EDAs involvement in the Departments shared services.
3. Ensure that EDA does not destroy additional IT inventory that was taken out of
service as a result of this cyber incident.
We recommend that the Departments Chief Information Officer:
1. Ensure DOC CIRT can appropriately and effectively respond to future cyber incidents.
2. Ensure incident response procedures clearly define DOC CIRT as the incident
response coordinator for the bureaus relying on DOC CIRTs incident response
services.
3. Ensure that DOC CIRT management has proper oversight and involvement in cyber
incidents to ensure that required incident response activities take place.
FINAL REPORT NO. OIG-13-027-A 17

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
Summary of Agency and Departmental
Responses and OIG Comments
The Deputy Assistant Secretary of Commerce for Economic Development and the
Departments Chief Information Officer (CIO) provided written responses to a draft of this
report (see appendixes C and D). We provide summaries of these responses and our
comments below.
EDA Response
The Deputy Assistant Secretary of Commerce for Economic Development concurred with our
recommendations and noted that EDA has begun implementation of the recommendations.
EDA also noted that (1) EDAs focus has been to fully and efficiently recover its IT systems, (2)
it has been abundantly cautious in its efforts to protect its staff, other Department systems,
grantees, clients, and other federal partners, (3) it continued to conduct and complete its
important work on time despite the interruption, and (4) it worked closely with the Census
Bureau for an interim recovery solution and, more recently, leveraged the Departments shared
services. EDAs response identified corrective actions it has taken and plans to take to
implement our recommendations.
EDA stated in its response that it appreciates the Office of Inspector Generals (OIG)
comprehensive review and continued involvement from the very early days of the incident
when EDA proactively requested OIGs review of the matter. While we initiated this audit at
the request of the former Acting Deputy Secretary of Commerce, we appreciate EDAs
cooperation throughout our audit.
In its response, EDA noted that its long-term recovery plan already included greater use of
shared services by leveraging Department-wide IT assets. However, prior to our briefing on
December 18, 2012, EDA had not finalized a recovery solution, such as using the Departments
available shared services.
Department CIO Response
The Departments CIO concurred with our recommendations related to DOC CIRT, noting
that the Department has initiated a comprehensive incident response improvement project.
The CIO further stated that the following project milestones have already been completed: (1)
conducting a third-party assessment of the DOC CIRT policies, procedures, and capabilities; (2)
hiring experienced and certified incident handlers; and (3) implementing an improved incident
tracking system.
In addition, the Departments CIO stated that, within the past 6 months, the OCIO and the
Office of the Secretary (OS) IT Operations worked closely with EDA to restore its
functionality by bringing EDAs grants management system online and bringing EDAs office
automation and IT service desk under the OS Information Technology Services.
FINAL REPORT NO. OIG-13-027-A 18

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
Appendix A: Objectives, Scope, and
Methodology
Our objective was to evaluate EDAs information security program and its recovery activities in
relation to EDAs cyber incident. We (1) assessed the effectiveness of EDAs IT security
program, (2) determined the significant factors that contributed to the incident, and (3)
evaluated both completed and planned activities to recover its information systems to support
critical operational requirements. To do so, we
Reviewed system-related artifacts, including policy and procedures, planning documents,
and other material supporting the security authorization process
Reviewed artifacts related to EDAs incident, including incident reports, forensic analysis,
logs, written communications, and other incident documentation
Interviewed operating unit and Department OCIO personnel, including system owners, IT
security officers, IT administrators, external incident responders, and organizational
directors and administrators regarding the security and operation of EDAs IT systems
and the incident
We also reviewed EDAs compliance with the following applicable provisions of law,
regulations, and mandatory guidance:
The Federal Information Security Management Act of 2002
Information Technology Security Program Policy, U.S. Department of Commerce,
introduced by the Chief Information Officer on March 9, 2009, and applicable Commerce
Information Technology Requirements
NIST Federal Information Processing Standards Publications
o 199, Standards for Security Categorization of Federal Information and Information
Systems
o 200, Minimum Security Requirements for Federal Information and Information Systems
NIST Special Publications
o 800-34, Contingency Planning Guide for Federal Information Systems
o 800-37, Guide for Applying the Risk Management Framework to Federal Information
Systems
o 800-53, Recommended Security Controls for Federal Information Systems and
Organizations
FINAL REPORT NO. OIG-13-027-A 19

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
o 800-53A, Guide for Assessing the Security Controls in Federal Information Systems
o 800-61, Computer Incident Handling Guide
o 800-70, Security Configuration Checklists Program for IT Products
We conducted our fieldwork from June 2012 to February 2013. We performed this audit
under the authority of the Inspector General Act of 1978, as amended, and Department
Organization Order 10-13, and in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings and conclusions based on
our audit objectives. We believe that the evidence obtained provides a reasonable basis for our
findings and conclusions.
FINAL REPORT NO. OIG-13-027-A 20

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
Appendix B: Detailed Timeline of EDAs Cyber
Incident Response and Recovery
Cyber Incident
12/6/2011 US-CERT notifies DOC CIRT of a cyber incident (components communicating
with fake antivirus sites).
12/7/2011 DOC CIRT sends EDA a first incident notification concerning US-CERTs alert.
The notification contains an inaccurate list of 146 potentially infected
components.
12/8/2011 DOC CIRT sends EDA a second incident notification containing completed
analysis that identified only two infected components.
12/9/2011 EDAs ITSO informs EDAs CIO that EDA experienced a potential widespread
malware infection.
12/13/2011 EDAs ITSO requests forensic assistance from DOC CIRT and on 12/14/2011
EDA provided DOC CIRT with the hard drives from two components that were
exhibiting malicious behavior.
12/15/2011 EDA asks DOC CIRT to block the malicious sites and addresses associated with
the US-CERT alert.
12/16/2011 EDAs CIO informs EDAs leadership that the malware infection is potentially
widespread.
1/18/2012 DOC CIRT notifies EDA that it identified a common malware infection on the
two components EDA provided to DOC CIRT on 12/14/2011. DOC CIRT
advises EDA to reimage the infected drives and put the remediated components
back into operation. EDA informs DOC CIRT that it cannot do this because
there are too many infected components.
1/20/2012 EDAs CIO notifies EDAs user base of the malware infection and advises that all
users follow good security practices. DOC CIRT requests US-CERTs assistance
and US-CERT arrives onsite.
1/24/2012 EDAs CIO believes that the e-mail server experienced a complete operational
failure and, upon restoration, an antivirus scan showed multiple malware
infections. EDAs CIO informed EDAs leadership (and the Departments Deputy
CIO) of the need to isolate EDA from the HCHB network. EDA takes the
following actions: disables its Microsoft Exchange e-mail server connection;
disables Internet access; disables its connection with regional offices; and
maintains local file-share service availability.
FINAL REPORT NO. OIG-13-027-A 21

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
1/27/2012 DOE and NIST incident responders assist onsite with the incident response.
1/30/2012 EDA hires a cyber security contractor to assist at the EDA CIOs discretion. US-
CERT issues a preliminary analysis report, which indicates the presence of
common malware but no nation-state activity or extremely persistent malware.
2/2/2012 The Department requests NSAs assistance to investigate the malware infection.
2/3/2012 DOE releases its report detailing assessment results from an assessment of one
component that indicated a common malware infection, but did not identify any
nation-state activity or extremely persistent malware.
2/7/2012 DHS issues a report that summarizes its findings and includes recommendations
for remediating the infection and establishing good IT security practices.
Additionally, the report used inaccurate information provided by DOC CIRT to
portray EDAs incident as widespread.
2/14/2012 NSA assists onsite with incident response activities.
2/17/2012 NSA analysis of the Linux systems finds no evidence of an intrusion or malware
infection.
5/15/2012 NSA releases a report stating that EDA had a widespread common malware
infection. NSA portrayed this information as fact, even though it did not
independently validate the information it received from DHS. However, NSA did
analyze EDAs Linux servers and found that the servers were not infected and
there was no indication of nation-state activity or extremely persistent malware.
Recovery
1/24/2012 EDA operates its existing IT infrastructure in isolation during interim recovery
activities in order to meet its deadlines for grants management.
2/6/2012 EDA begins coordination with the Census Bureau on its interim recovery
activities.
2/14/2012 EDA establishes a Web presence and makes e-mail service available to a limited
number of Blackberry users.
3/25/2012 The Census Bureau restores Blackberry service for all EDA staff and EDA
completes the distribution of laptops to all users. This provides office
automation capabilities, e-mail services, and Internet access for all users.
4/5/2012 EDA provides users access to a stand-alone implementation of its business
applications, which contains historical data necessary to complete its mission
activities.
FINAL REPORT NO. OIG-13-027-A 22

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
5/15/2012 EDA stops its forensic analysis activities and switches to full-time data cleaning,
involving the use of several antivirus products to scan data files for indications of
an infection. The cybersecurity contractor did not identify any additional
components with a malware infection (the contractor did identify the existence
of common malware contained in archived e-mail attachments and temporary
Internet browser files).
9/5/2012 EDA presents a request to the Commerce IT Review Board (CITRB) for funding
to carry out its recovery efforts. The CITRB does not approve EDAs request,
necessitating changes to the intended recovery efforts.
2/6/2013 OCIO begins restoration of EDAs IT systems.
3/15/2013 OCIO restores EDAs IT operations, including restoring access for all users to
its critical grants management applications.
FINAL REPORT NO. OIG-13-027-A 23

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
Appendix C: Agency Response
FINAL REPORT NO. OIG-13-027-A 24

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
FINAL REPORT NO. OIG-13-027-A 25

HlR ASSISliiNT GENERAL roR SYSTEM ACQIJISI110'1 MD 11 SI:.C'IIRil Y

Jun< 12. 2013
I of 2
Attachment
In response to the 20 ll cyber incident, EDA took proactive measures to increase the security and
efficiency of its Information and Technology (IT) system: ensure quality service to all its
stakeholders and clients; and enable accow1tability by requesting the Oflice of lnsp....--ctor General
(OIG) evaluate the agency's response and recovery activities.
RccQmmendation I: Identify EDA' s areas of IT responsibility and en ure the
implementation of required security measures.
EDA Response: EDA agrees with this recommendation.
Corrective Ac.:tion To Date:
l. EDA transferred a majority of its IT responsibilities to the Department of Commerce
(DOC). Oftice of Chief Infonnation Officer (OCIO). In addition. EDA sLoan Billing
and Managing System (LBMS) has been transferred to the National Oceanic and
Atmospheric Administration (NOAA). The c actions have resulted in a significantly
more eflicicnt and higher level of overall IT operations and security.
EDA has identified two remaining areas of IT responsibi lity: Operations Planning and
Control System (OPCS) and the Revolving Loan Fund Management System
(RLFYIS). EDA has maintained a high level of security with each of these systems
and will continue to do so as the agency works with OAA to transfer system
funct ionality to NOAA's secure environment.
2. EDA has implemented a program to increase the security trai ning and capabilities of
all staff. especially the agency' s IT staff. By October 2012. EDA had successfully
ensured that eight of its I I OfT employees had achieved DOC IT ecurity Role
Cenitications. With these ceni fication . EDA has achieved I 00 percent compliance
"ith the DOC IT Security Role Ceni fications. In addition. EDA has currently
completed 94 percent of the required FY 2013 IT Security Role-Based Training for
all of the agency's staff members and expects to reach 100 percent prior to the June
30 deadline.
Continued Action:
I. EDA will contiJ1UC its current work with NOAA 10 transfer its remaining two areas of
IT responsibility, OPCS and RLFMS. to NOAA's secure environment.
2. EDA and the DOC will reinstate the Oversight leering Committee to review and
provide guidance on IT security.
U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
FINAL REPORT NO. OIG-13-027-A 26

O,lf MORANDIJM fOR I OR GE ... ERAl AIJDI rAND EVALUATION

11. 2013
Page 2 ol 2
Recommendation 2: Determine whether EDA can reduce its IT budget and staff
expenditures, through the increased efficiencies of EDA ' s involvement in the Department's
shared servi ces.
El)A Response: EDA agrees with this recommendation.
Corrective Action To Date:
EDA has carefully reviewed its IT expenditures since the 20 II cyber incident: and through
it usc of shared services, has increased the efficiency and security of its IT system. EDA 's
previously established long-tenn recovery plan al ready included greater use of shared
services by leveragi ng existing or planned dcpanmcnt-wide (including other bureau) IT
assets. As cost efficiencies are generated from its implementation of shared services. EDA 's
first priority will be to continue its work with the Depanment and bureaus to ensure that all
required and recommended security processes. procedures. software and services are fully
implemented.
Continued Act ion:
EDA and the Depanment will reinstate the Oversight Steering Committee to review and
provide guidance on IT cost savings and increased efficiencies.
Recommendation 3: Ensure that EDA docs not d estroy additional IT inventory that was
taken out of scn ,icc as a result of this c:ybcr incident.
EDA EDA agrees with t his recommendation.
Corrective Action To Date:
In an effon to secure its IT system due to the cyber incident. EDA replaced its IT
components with equipment on loan from the Bureau of the Census. Upon migration of our
IT operati ons to the DOC HCHB network, EDA installed new equipment that had been
purchased Brim: to the cyber incident. Prior to the cyber incident. EDA had planned to clean
and surplus desktop computers and servers scheduled for replacement. As a result of this
repon and because very little of this equipment has been destroyed. EDA is able to continue
this planned action.
Continued Action:
EDA still posse ses 96 percent of its replaced inventory and intends to either put that
inventory back into service or surplus the items.
U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
FINAL REPORT NO. OIG-13-027-A 27

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
Appendix D: Departmental Response
011200000142
FINAL REPORT NO. OIG-13-027-A 28