CSB S BP Husky Investigation Report 1720741132

Fatal Naphtha Release and Fire at
U.S. Chemical Safety and

BP-Husky Toledo Refinery
Hazard Investigation Board Oregon, Ohio | Incident Date: September 20, 2022 | No. 2022-01-I-OH
Investigation Report
Published: June 2024
SAFETY ISSUES:
Liquid Overflow
Prevention
Abnormal Situation
Management
Alarm Flood
Learning from Incidents
U.S. Chemical Safety and Hazard Investigation Board

The mission of the U.S. Chemical Safety and Hazard Investigation Board (CSB) is to
drive chemical safety excellence through independent investigations to protect
communities, workers, and the environment.
The CSB is an independent federal agency charged with investigating, determining, and reporting to
the public in writing the facts, conditions, and circumstances and the cause or probable cause of any
accidental chemical release resulting in a fatality, serious injury, or substantial property damages.
The CSB issues safety recommendations based on data and analysis from investigations and safety studies.
The CSB advocates for these changes to prevent the likelihood or minimize the consequences of accidental
chemical releases.
More information about the CSB and CSB products can be accessed at www.csb.gov or obtained by
contacting:

1750 Pennsylvania Ave. NW, Suite 910
Washington, DC 20006
(202) 261-7600
The CSB was created by the Clean Air Act Amendments of 1990, and the CSB was first funded and
commenced operations in 1998. The CSB is not an enforcement or regulatory body. No part of the
conclusions, findings, or recommendations of the Board relating to any accidental release or the
investigation thereof shall be admitted as evidence or used in any action or suit for damages arising out of
any matter mentioned in such report. 42 U.S.C. § 7412(r)(6)(G).
1
The September 20, 2022, naphtha release and fire at the BP-Husky Toledo Refinery
fatally injured two employees, who were brothers:
Ben Morrissey
Max Morrissey
2
Contents
CONTENTS ........................................................................................................................................... 3
ABBREVIATIONS ................................................................................................................................. 10
EXECUTIVE SUMMARY ....................................................................................................................... 12
Safety Issues ........................................................................................................................................ 13
Cause ................................................................................................................................................... 14
Recommendations............................................................................................................................... 14
1 BACKGROUND ............................................................................................................................ 17
1.1 BP Toledo Refinery ................................................................................................................... 17
1.2 Owner and Operator ................................................................................................................ 19
1.3 Naphtha .................................................................................................................................... 20
1.4 Refinery Fuel Gas ...................................................................................................................... 20
1.5 Fuel Gas Mix Drum ................................................................................................................... 20
1.6 Crude Oil and Crude Slate ........................................................................................................ 22
1.7 Crude 1 Tower .......................................................................................................................... 22
1.8 Coker Gas Plant ........................................................................................................................ 24
1.9 Shift Operations Staffing .......................................................................................................... 26
1.10 Federal Safety Regulations ................................................................................................... 27
1.10.1 OSHA Process Safety Management Standard ................................................................... 27
1.10.2 EPA Risk Management Program Rule................................................................................ 28
1.11 Description of Surrounding Area .......................................................................................... 28
2 INCIDENT DESCRIPTION .............................................................................................................. 31
2.1 Events Leading Up to the Incident............................................................................................ 31
2.2 The Incident .............................................................................................................................. 33
2.3 Emergency Response ............................................................................................................... 38
3 INCIDENT ANALYSIS .................................................................................................................... 40
3.1 Incident Progression ................................................................................................................. 40
3.1.1 Naphtha Standpipe Overflow ............................................................................................ 40
3.1.2 Naphtha Hydrotreater Preheat Leak ................................................................................. 43
3
3.1.3 Coker Gas Plant Bypass ..................................................................................................... 45

3.1.4 Crude 1 Overhead Accumulator Drum Level Control......................................................... 46
3.1.5 Crude 1 Tower Instability................................................................................................... 48
3.1.6 Crude 1 Tower Crude Slate Change ................................................................................... 51
3.1.7 Crude 1 Overhead Accumulator Drum High Level ............................................................. 54
3.1.8 Absorber Stripper Tower Overflow .................................................................................... 55
3.1.9 Fuel Gas Mix Drum Overflow ............................................................................................. 58
3.2 Draining and Liquid Release ..................................................................................................... 59
3.3 Stop Work Authority................................................................................................................. 63
4 SAFETY ISSUES ............................................................................................................................ 66
4.1 Liquid Overflow Prevention ...................................................................................................... 67
4.1.1 Ineffective Safeguards ....................................................................................................... 67
4.1.2 Reliance on Human Intervention ....................................................................................... 73
4.1.3 Post-Incident Actions ......................................................................................................... 77
4.1.4 Industry Guidance ............................................................................................................. 78
4.2 Abnormal Situation Management ............................................................................................ 81
4.2.1 BP Toledo Refinery’s Abnormal Situation Management.................................................... 82
4.2.2 Use of Industry Guidance .................................................................................................. 86
4.2.3 Applying Industry Guidance to the Incident ...................................................................... 89
4.3 Alarm Flood .............................................................................................................................. 94
4.3.1 Alarm Flood Day of Incident .............................................................................................. 94
4.3.2 BP Guidance ...................................................................................................................... 98
4.3.3 Industry Guidance for Alarm Flood Performance ............................................................ 101
4.3.4 Industry Guidance for Alarm Flood Management ........................................................... 105
4.3.5 Post-Incident Alarm Flood Management ........................................................................ 108
4.4 Learning from Incidents.......................................................................................................... 109
4.4.1 Catastrophic Incident Warning Signs from a 2019 BP Toledo Refinery Incident ............. 109
4.4.2 Findings from the Fatal 2005 Explosion and Fire at the BP Texas City Refinery .............. 115
5 CONCLUSIONS .......................................................................................................................... 122
5.1 Findings .................................................................................................................................. 122
5.2 Cause ...................................................................................................................................... 126
6 RECOMMENDATIONS................................................................................................................ 127
6.1 Ohio Refining Company LLC ................................................................................................... 127
2022-01-I-OH-R1............................................................................................................................ 127
4
2022-01-I-OH-R2............................................................................................................................ 127
2022-01-I-OH-R3............................................................................................................................ 127
2022-01-I-OH-R4............................................................................................................................ 127
6.2 American Petroleum Institute (API)........................................................................................ 128
2022-01-I-OH-R5............................................................................................................................ 128
2022-01-I-OH-R6............................................................................................................................ 128
6.3 International Society of Automation (ISA) .............................................................................. 129
2022-01-I-OH-R7............................................................................................................................ 129
7 KEY LESSONS FOR THE INDUSTRY .............................................................................................. 130
8 REFERENCES ............................................................................................................................. 132
APPENDIX A: TIMELINES ................................................................................................................... 135
A.1 Naphtha Hydrotreater Release............................................................................................... 135
A.1.1 2018 .................................................................................................................................... 135
A.1.1.1 FEBRUARY 16, 2018 ........................................................................................................ 135
A.1.1.2 SEPTEMBER 6, 2018 ........................................................................................................ 136
A.1.2 2019 .................................................................................................................................... 137
A.1.2.1 NOVEMBER 2, 2019......................................................................................................... 137
A.1.2.2 NOVEMBER 8, 2019......................................................................................................... 137
A.1.3 2021 .................................................................................................................................... 138
A.1.3.1 MAY 2021 ..................................................................................................................... 138
A.1.3.2 JUNE 2021 ..................................................................................................................... 138
A.1.3.3 JULY 2021 ...................................................................................................................... 139
A.1.3.4 AUGUST 2021................................................................................................................. 139
A.1.3.5 NOVEMBER 2021 ............................................................................................................ 139
A.1.3.6 DECEMBER 2021 ............................................................................................................. 140
A.1.4 2022 .................................................................................................................................... 140
A.1.4.1 APRIL 2022 .................................................................................................................... 140
A.1.4.2 JULY 26, 2022 ................................................................................................................ 140
A.1.4.3 JULY 28, 2022 ................................................................................................................ 140
A.1.4.4 AUGUST 27, 2022 ........................................................................................................... 140
A.1.4.5 SEPTEMBER 19, 2022 ...................................................................................................... 141
A.1.4.5.1 7:11 P.M.................................................................................................................... 141
A.1.4.5.2 10:40 P.M. ................................................................................................................. 141
A.1.4.5.3 11:45 P.M. ................................................................................................................. 141
A.1.4.6 SEPTEMBER 20, 2022 (DAY OF INCIDENT) ............................................................................ 141
5
A.1.4.6.1 2:12 A.M.................................................................................................................... 141

A.1.4.6.2 2:45 TO 3:00 A.M........................................................................................................ 142
A.1.4.6.3 APPROXIMATELY 3:20 A.M. ............................................................................................ 142
A.1.4.6.5 APPROXIMATELY 4:30 A.M. TO 5:00 A.M.......................................................................... 142
A.1.4.6.7 6:26 A.M.................................................................................................................... 142
A.1.4.6.8 6:33 A.M. TO 6:57 A.M. ............................................................................................... 142
A.1.4.6.9 6:50 A.M. TO 6:59 A.M. ............................................................................................... 142
A.1.4.6.10 6:57 A.M. TO 7:12 A.M. ............................................................................................... 143
A.1.4.6.11 APPROXIMATELY 7:13 A.M. TO 7:28 A.M.......................................................................... 143
A.1.4.6.12 7:31 A.M.................................................................................................................... 143
A.1.4.6.17 8:06 A.M.................................................................................................................... 144
A.1.4.6.18 8:08 A.M.................................................................................................................... 144
A.1.4.6.19 8:12 A.M.................................................................................................................... 144
A.1.4.6.20 8:13 A.M.................................................................................................................... 145
A.1.4.6.21 8:15 A.M. TO 9:30 A.M. ............................................................................................... 145
A.1.4.6.22 10:27 A.M.................................................................................................................. 145
A.1.4.6.23 11:00 A.M.................................................................................................................. 145
A.2 Fuel Gas Mix Drum Release .................................................................................................... 146
A.2.1 2007 .................................................................................................................................... 146
A.2.2 2011.................................................................................................................................... 146
A.2.3 2015.................................................................................................................................... 147
A.2.3.1 MAY 2015 ..................................................................................................................... 147
A.2.3.2 JUNE 2015 ..................................................................................................................... 147
A.2.4 2016 .................................................................................................................................... 147
A.2.4.1 JANUARY 2016................................................................................................................ 147
A.2.4.2 JUNE 2016 ..................................................................................................................... 147
A.2.4.3 SEPTEMBER 2016 ............................................................................................................ 147
A.2.5 2018 .................................................................................................................................... 147
A.2.5.1 FEBRUARY 2018 .............................................................................................................. 147
A.2.5.2 APPROXIMATELY SEPTEMBER 2018 ..................................................................................... 148
6
A.2.5.3 OCTOBER 2018 ............................................................................................................... 148

A.2.6 2019.................................................................................................................................... 148
A.2.6.1 MARCH 2019 ................................................................................................................. 148
A.2.6.2 MAY 2019 ..................................................................................................................... 148
A.2.6.3 NOVEMBER 2019 ............................................................................................................ 148
A.2.7 2020.................................................................................................................................... 149
A.2.7.1 FEBRUARY 2020 .............................................................................................................. 149
A.2.8 2021 .................................................................................................................................... 149
A.2.8.1 JUNE 2021 ..................................................................................................................... 149
A.2.8.2 AUGUST 19, 2021........................................................................................................... 149
A.2.9 2022 .................................................................................................................................... 149
A.2.9.1 APRIL 20, 2022 .............................................................................................................. 149
A.2.9.2 JULY 2022 ...................................................................................................................... 149
A.2.9.3 AUGUST 2022................................................................................................................. 150
A.2.9.4 SEPTEMBER 20, 2022 (DAY OF INCIDENT) ............................................................................ 150
A.2.9.4.1 3:11 A.M.................................................................................................................... 150
A.2.9.4.2 7:26 A.M.................................................................................................................... 150
A.2.9.4.3 8:09 A.M.................................................................................................................... 150
A.2.9.4.4 8:12 A.M.................................................................................................................... 150
A.2.9.4.5 8:42 A.M.................................................................................................................... 150
A.2.9.4.6 9:17 A.M.................................................................................................................... 150
A.2.9.4.7 9:47 A.M.................................................................................................................... 150
A.2.9.4.8 10:20 A.M.................................................................................................................. 150
A.2.9.4.9 1:26 P.M.................................................................................................................... 151
A.2.9.4.10 1:30 P.M.................................................................................................................... 151
A.2.9.4.11 1:37 P.M.................................................................................................................... 151
A.2.9.4.12 3:58 P.M.................................................................................................................... 151
A.2.9.4.13 APPROXIMATELY 4:15 P.M. ............................................................................................ 151
A.2.9.4.14 4:56 P.M.................................................................................................................... 151
A.2.9.4.15 5:39 P.M.................................................................................................................... 152
A.2.9.4.16 5:41 P.M.................................................................................................................... 152
A.2.9.4.17 5:42 P.M.................................................................................................................... 152
A.2.9.4.18 5:47 P.M.................................................................................................................... 152
A.2.9.4.19 5:53 P.M.................................................................................................................... 152
A.2.9.4.20 6:06 P.M.................................................................................................................... 152
A.2.9.4.21 6:09 P.M.................................................................................................................... 153
A.2.9.4.22 6:10 P.M.................................................................................................................... 153
7
A.2.9.4.23 6:12 P.M.................................................................................................................... 153

A.2.9.4.24 6:14 P.M.................................................................................................................... 153
A.2.9.4.25 6:15 P.M.................................................................................................................... 153
A.2.9.4.26 6:16 P.M.................................................................................................................... 153
A.2.9.4.27 6:17 P.M.................................................................................................................... 153
A.2.9.4.29 6:20 P.M.................................................................................................................... 154
A.2.9.4.30 6:21 P.M.................................................................................................................... 154
A.2.9.4.31 6:22 P.M.................................................................................................................... 154
A.2.9.4.32 6:23 P.M.................................................................................................................... 154
A.2.9.4.33 6:24 P.M.................................................................................................................... 154
A.2.9.4.34 6:25 P.M.................................................................................................................... 154
A.2.9.4.35 6:26 P.M.................................................................................................................... 155
A.2.9.4.37 6:29 P.M.................................................................................................................... 155
A.2.9.4.38 6:30 P.M.................................................................................................................... 155
A.2.9.4.40 6:32 P.M.................................................................................................................... 156
A.2.9.4.42 6:35 P.M.................................................................................................................... 156
A.2.9.4.43 6:36 P.M.................................................................................................................... 156
A.2.9.4.45 6:39 P.M.................................................................................................................... 156
A.2.9.4.46 6:40 P.M.................................................................................................................... 156
A.2.9.4.47 6:43 P.M.................................................................................................................... 156
A.2.9.4.49 6:46 P.M. (TIME OF IGNITION)........................................................................................ 156
A.2.9.4.53 6:56 P.M.................................................................................................................... 157
A.2.9.4.54 7:04 P.M.................................................................................................................... 157
A.2.9.4.55 7:15 P.M.................................................................................................................... 157
A.2.9.4.56 7:21 P.M.................................................................................................................... 157
A.2.9.4.57 8:31 P.M.................................................................................................................... 157
A.2.9.4.58 8:51 P.M.................................................................................................................... 157
A.2.9.4.59 9:18 P.M.................................................................................................................... 157
A.2.9.4.60 9:26 P.M.................................................................................................................... 158
8
A.2.9.4.61 9:44 P.M.................................................................................................................... 158

A.2.9.4.62 10:10 P.M.................................................................................................................. 158
A.2.9.5 SEPTEMBER 21, 2022 ...................................................................................................... 158
A.2.9.5.1 12:18 A.M.................................................................................................................. 158
A.2.9.5.2 1:57 A.M.................................................................................................................... 158
APPENDIX B: SIMPLIFIED CAUSAL ANALYSIS (ACCIMAP).................................................................... 159
APPENDIX C: DESCRIPTION OF SURROUNDING AREA ........................................................................ 160
APPENDIX D: OSHA HAZARD ALERT LETTER ...................................................................................... 163
APPENDIX E: OSHA CITATIONS.......................................................................................................... 164
9
Abbreviations
°F degrees Fahrenheit
ACM Alarm Configuration Manager
ANSI American National Standards Institute
API American Petroleum Institute
ASAP as soon as possible
ASM Abnormal Situation Management
ASM® Abnormal Situation Management® [Consortium]
ASME American Society of Mechanical Engineers
AST Absorber Stripper Tower
BPCS Basic Process Control System
BP BP Products North America Inc.
CCPS Center for Chemical Process Safety
CFR Code of Federal Regulations
CGP Coker Gas Plant
CO carbon monoxide
CSB U.S. Chemical Safety and Hazard Investigation Board
DCS Distributed Control System
EEMUA Engineering Equipment and Materials Users Association
EMS Emergency Medical Services
EPA U.S. Environmental Protection Agency
ERT Emergency Response Team
FCC Fluid Catalytic Cracker
FGMD Fuel Gas Mix Drum
GWR Guided Wave Radar
HAZOP Hazard and Operability Study
HCS Hazard Communication Standard
HE heat exchanger
HSSE Health, Safety, Security & Environmental
HVGO Heavy Vacuum Gas Oil
IPL Independent Protection Layer
ISA International Society of Automation
LLC Limited Liability Company
LEL lower explosive limit
LFL lower flammability limit
LOPA Layer of Protection Analysis
10
LPG Liquefied Petroleum Gas

LVN Light Virgin Naphtha
MAWP maximum allowable working pressure
MAWT maximum allowable working temperature
MOC Management of Change
NHT Naphtha Hydrotreater
NIOSH National Institute for Occupational Safety and Health
ORC Ohio Refining Company LLC
ORR Operational Readiness Review
OSHA U.S. Occupational Safety and Health Administration
OWS Oily Water Sewer
PHA Process Hazard Analysis
PSIG pounds per square inch (gauge)
PSM Process Safety Management
PSV pressure safety valve
RMP Risk Management Program
ROEIV Remotely Operated Emergency Isolation Valve
SCBA Self-Contained Breathing Apparatus
SDS Safety Data Sheet
SIF safety instrumented function
SIS safety instrumented system
TAR turnaround
TFO Toledo Fuels Optimization
TIU Toledo Integrated Unit
USW United Steelworkers
WWTU Wastewater Treatment Unit
11
Executive Summary
On September 20, 2022, at approximately 6:46 p.m., a vapor cloud ignited causing a flash fire a at the BP-Husky
Refining LLC (“BP-Husky”) refinery in Oregon, Ohio. The vapor cloud formed when two BP Products North
America Inc. (“BP”) employees released flammable liquid naphtha from a pressurized vessel to the ground.
As a result of the fire, both BP employees, who were brothers, were fatally injured. In addition, the events of the
day caused approximately $597 million in property damage including loss of use. b BP estimated over 23,000
pounds of naphtha were released during the event. No off-site impacts were reported. To date, this is the largest
fatal incident at a BP operated petroleum refinery since the fatal accident at the BP Texas City Refinery in 2005,
which resulted in the deaths of 15 workers and injured 180 other people. c
The vessel typically contained only vapor (fuel gas for furnaces and boilers). However, during the incident, the
vessel filled with liquid naphtha when an upstream tower overflowed naphtha into a vapor bypass line directly
to the vessel. The upstream tower overflowed liquid naphtha through the vapor bypass line after a board
operator opened a closed valve sending liquid naphtha to the tower operating in a vapor-only mode. Other
refinery units had been shut down due to a loss of containment incident that occurred earlier that morning.
The initial process upset, the subsequent events and operational decisions made on September 20, 2022, led to
liquid naphtha filling the vessel, which normally contained fuel gas. The vessel then overflowed into vapor
piping feeding downstream furnaces and boilers. While draining the overflowing vessel as fast as they could
pursuant to the board operator’s directive communicated via radio, the BP employees opened the vessel and
released liquid naphtha to the ground.
The refinery is located in Oregon, Ohio east of the city of Toledo and was operated by BP at the time of the
incident. However, it is now owned and operated by Ohio Refining Company LLC (“ORC”) an ultimate
subsidiary of Cenovus Energy Inc. (“Cenovus”). d This report will refer to the refinery as the “BP Toledo
Refinery”.
a
A flash fire “spreads by means of a flame front rapidly through a diffuse fuel, such as […] the vapors of an ignitable liquid, without the
production of damaging pressure” [55].
b
Property damage means damage to or the destruction of tangible public or private property, including loss of use of that property.
See 40 CFR Part 1604 - Reporting of Accidental Releases.
c
BP Products North America Inc. (“BP”) also operates refineries in Whiting, Indiana, and Cherry Point, Washington, in the United
States. See Refineries | What we do (bp.com).
d
In 2021, Cenovus Energy Inc. (“Cenovus”) merged with Husky Energy Inc. (“Husky”) and became the indirect owner of the Husky Oil
Toledo Company. On August 6, 2022, Husky Oil Toledo Company announced an agreement to purchase BP’s ownership interest in the
BP-Husky Refining joint venture. The transaction closed on February 28, 2023, making Husky Oil Toledo Company the sole owner of
the refinery. After the closing, BP-Husky Refining LLC was renamed Ohio Refining Company LLC (“Ohio Refining Company”),
which became both the owner and operator of the Toledo Refinery.
12
Safety Issues
The CSB’s investigation identified the safety issues below.
Liquid Overflow Prevention. Although the BP Toledo Refinery conducted Hazard and Operability
Studies, a process hazard analysis (PHA) methodology, to assess the risk of liquid overflow events and
identify safeguards, the refinery did not have sufficient safeguards to prevent the initiating event. In
some cases, the BP Toledo Refinery relied on human intervention to respond to process upsets and
deviations. Despite the BP Toledo Refinery’s reliance on human intervention as an identified safeguard
for overflow of the Absorber Stripper Tower to the Fuel Gas Mix Drum, the refinery did not adequately
consider potential hazards that could exist if the drum contained high levels of flammable liquid (such
as naphtha) and needed to be drained. Nor did the BP Toledo Refinery have procedures, written
instructions, or documented corrective actions for operators to respond to or troubleshoot a high liquid
level in the Fuel Gas Mix Drum during either normal operations or process upsets if liquid entered the
drum. Furnace safety instrumented systems and emergency pressure-relief valves were also identified as
safeguards in the BP Toledo Refinery’s PHAs for overflow of the Absorber Stripper Tower to the Fuel
Gas Mix Drum, but neither were effective in preventing liquid overflow to the fuel gas system
Additionally, the industry lacks sufficient guidance on protective systems for a Fuel Gas Mix Drum
despite it being an integral part of a refinery’s fuel gas system. (See Section 4.1)
Abnormal Situation Management. An abnormal situation is a process disturbance with which the
basic process control system cannot cope. Abnormal situations can create a stressful environment for the
operators. If abnormal situations are not effectively managed, they can escalate into a more serious
incident. In its book, Guidelines for Managing Abnormal Situations, the Center for Chemical Process
Safety (CCPS) states: “[s]udden, potentially dangerous situations can affect human performance (the
“startle” factor), leading to a “fight or flight” response that can lead to inappropriate action being taken”
[1, pp. 87-88]. In the 24 hours leading up to the incident, the BP Toledo Refinery experienced a number
of abnormal situations across several units, escalating to overfilling the Fuel Gas Mix Drum. This
prompted two BP employees to release the Fuel Gas Mix Drum contents to the ground, ultimately
cascading to the vapor cloud, fire, and fatal injuries. (See Section 4.2)
Alarm Flood. Board operators at the BP Toledo Refinery were receiving far more than 10 alarms in 10
minutes on average, a situation in which more alarms were annunciating than a human can effectively
respond to, for nearly 12 hours preceding the incident. Between 6:50 a.m. and 6:49 p.m. September 20,
2022, a total of 3,712 alarms were recorded. Continued operation in an alarm flood state contributed to
the incident by causing delays and errors in responding to critical alarms and shift-to-shift
communications. Had the Tuesday, September 20, 2022, night shift board operators been less
overloaded with alarms, they might have identified that the Coker Gas Plant Absorber Stripper Tower
was overflowing naphtha through the Coker Gas Plant bypass piping directly to the Fuel Gas Mix Drum
and stopped liquid flow to the Fuel Gas Mix Drum, preventing or mitigating the incident. (See Section
4.3)
Learning from Incidents. The final recommendation of the report of the BP U.S. Refineries
Independent Safety Review Panel in 2007 (“the Baker Panel Report”) stated: “BP should use the lessons
learned from the Texas City tragedy and from the Panel’s report to transform the company into a
13
recognized industry leader in process safety management” [2, p. 257]. a In its investigation of the
September 20, 2022, naphtha release and fire, the CSB found similarities between the overflow events
of the BP Toledo Refinery incident and the findings from the fatal explosions in 2005 at the BP refinery
in Texas City, Texas. Catastrophic incident warning signs existed prior to the September 20, 2022,
incident at the BP Toledo Refinery, and had BP effectively recognized and acted upon the warning signs
following a 2019 incident, the company could have provided more effective safeguards to prevent the
overflow of multiple vessels during a refinery upset such as the September 20, 2022, incident. (See
Section 4.4)
See Appendix B for the accident map (AcciMap), which provides a graphical analysis of this incident.
Cause
The CSB determined the cause of the incident was operators opening valves and removing a flange on the
pressurized Fuel Gas Mix Drum to release a flammable liquid, naphtha, directly to the ground. After being
released to the ground, the flammable liquid formed a vapor cloud that reached a nearby ignition source
resulting in a flash fire.
Contributing to the incident were 1) the refinery’s failure to implement effective preventive safeguards for the
overflow of towers and vessels in various pieces of equipment which led to an over-reliance on human
intervention to prevent incidents; 2) the refinery’s failure to implement a shutdown or hot circulation through the
use of Stop Work Authority or otherwise; 3) the refinery’s ineffective policies, procedures, and practices to
avoid and control abnormal situations; 4) the refinery’s alarm system which flooded operators with alarms
throughout the day resulting in poor decision making; and 5) the refinery’s failure to learn from previous
incidents.
Recommendations
To Ohio Refining Company LLC
2022-01-I-OH-R1
Revise the safeguards used in the refinery’s process hazard analyses high level and overflow scenarios. At a
minimum, establish effective preventive safeguards that use engineered controls to prevent liquid overfill and do
not rely solely on human intervention.
2022-01-I-OH-R2
Revise the Abnormal Situation Management policy to incorporate guidance provided by the ASM Consortium
and the Center for Chemical Process Safety (CCPS). The revised policy should include, at a minimum:
a
In 2005, the CSB issued an urgent safety recommendation to the BP Group Executive Board of Directors that it convene an independent
panel of experts to examine BP’s corporate safety management systems, safety culture, and oversight of the North American refineries.
BP accepted the recommendation and commissioned the BP U.S. Refineries Independent Safety Review Panel [31, pp. 27-28]. BP
Group means BP p.l.c. and its subsidiaries and affiliates [56].
14
a) A broader definition of abnormal situations, such as that defined by the CCPS,
b) Additional predictable abnormal situations and their associated corrective procedures. At a minimum
include the following abnormal situations:
1) unplanned crude slate changes,
2) continued operation of the Crude 1 unit with the naphtha hydrotreater unit shut down, and
3) an emergency pressure-relief valve opening.
c) Guidance to determine when an abnormal situation is becoming too difficult to manage and the
appropriate actions to take, such as shutting down a process, putting it into a circulation mode, or
implementing proper procedures for bringing it to a safe state.
2022-01-I-OH-R3
Develop and implement a policy or revise existing policy that clearly provides employees with the authority to
stop work that is perceived to be unsafe until the employer can resolve the matter. This should include detailed
procedures and regular training on how employees would exercise their stop work authority. Emphasis should
be placed on exercising this authority during abnormal situations, including alarm floods.
2022-01-I-OH-R4
Revise the ‘Toledo Alarm Philosophy’ by incorporating the Engineering Equipment and Manufacturers Users
Association (EEMUA) guidance for alarm rate following an upset and not limiting alarm performance to a
single metric averaged over a month. In addition to including analyzing individual alarm flood events, the
revised philosophy document should improve refinery alarm performance to reduce alarm flood duration and
peak rate for events similar to the September 20, 2022, incident. Consult EEMUA Publication 191, Chapter
6.5.1, for guidance regarding abnormal condition performance levels. Apply the improved performance levels
where applicable, but specifically to the Crude 1 control board alarm performance.
To American Petroleum Institute (API)
2022-01-I-OH-R5
Develop a new publication or revise an existing publication, such as API Recommended Practice 556
Instrumentation, Control, and Protective Systems for Gas Fired Heaters, to incorporate the process hazards
associated with Fuel Gas Mix Drum overflow. The publication should include the following at a minimum:
a) Description of the process hazards associated with Fuel Gas Mix Drum overflow and the consequential
impacts on equipment using fuel gas,
b) Guidance for Fuel Gas Mix Drum design and sizing criteria which includes consideration of
condensation, entrainment, overflow, and draining,
c) Guidance for instrumentation to detect high level to prevent overfilling of Fuel Gas Mix Drums, and
d) Recommended practices for selecting preventive safeguards to prevent overfilling of Fuel Gas Mix
Drums.
15
2022-01-I-OH-R6
Develop a publication that addresses preventing the overflow of pressure vessels such as towers and drums. The
publication should be applicable to both new and existing pressure vessels. Include the following at a minimum:
a) Description of typical overflow events that could result during normal, upset, or transient operations
(startup, shutdown, standby) including the formation of a vapor cloud,
b) Recommended practices for instrumentation to monitor and detect a pressure vessel overflow,
c) Process hazard analysis guidance for pressure vessel overflow scenarios,
d) Recommended practices for safeguards to prevent a pressure vessel overflow,
e) Recommended field and board operator process safety training topics and methods to prevent a pressure
vessel overflow,
f) Guidelines for process safety assessments to prevent a pressure vessel overflow, and
g) Incorporate lessons learned from this CSB investigation and the CSB’s BP Texas City Refinery
investigation throughout the document.
To International Society of Automation (ISA)
2022-01-I-OH-R7
Revise American National Standard ANSI/ISA 18.2-2016, Management of Alarm Systems for the Process
Industries, to include performance targets for short-term alarm flood analysis so that users can evaluate alarm
flood performance for a single alarm flood event. The performance targets should include:
a) number of alarm floods,
b) duration of each flood,
c) alarm count in each flood, and
d) peak alarm rate for each flood.
At a minimum, a target peak alarm flood rate should be defined, such as in the guidance provided by the ASM
Consortium or Engineering Equipment and Materials Users Association (EEMUA), to establish trigger points
that require alarm performance improvement actions.
16
1 Background
1.1 BP Toledo Refinery

The BP Toledo Refinery is located east of the city of Toledo, in Oregon, Ohio. The refinery sits on 586 acres
and has operated since 1919 [3]. The refinery can process approximately 160,000 barrels of crude oil per day,
producing gasoline, diesel, jet fuel, propane, asphalt, and other products [4]. As of September 2022, the refinery
employed 588 people.a
Figure 1 shows the various units of the BP Toledo Refinery divided into six zones. The star in the red outlined
Crude 1 unit shows the approximate location of the September 20, 2022, naphtha release and fire from the Fuel
Gas Mix Drum.
Figure 1. BP Toledo Refinery Process Block Map. (Credit: BP)
A simplified schematic of the relevant portion of the BP Toledo Refinery is shown in Figure 2. Crude oil is
pumped from storage tanks to the Crude 1 Tower (See Section 1.6). After the Crude 1 Tower overhead is
a
The United Steelworkers (USW) represented approximately 355 of the BP Toledo Refinery workers as of September 2022.
17
cooled, the liquid naphtha from the Crude 1 Overhead Accumulator Drum (See Figure 4) can be sent to three
places:
1. to the Naphtha Hydrotreater unit (“the NHT unit”),
2. to the Coker Gas Plant to treat wet coker gas a (See Section 1.8), and
3. to Light Virgin Naphtha Storage.
Naphtha from the Coker Gas Plant goes to the Naphtha Hydrotreater Feed Surge Drum (“NHT Feed Surge
Drum”). The stream from the NHT Feed Surge Drum combines with the naphtha from the Crude 1
Overhead Accumulator Drum to enter NHT Preheat.b
Wet coker gas from the Coker Gas Plant combines with various other refinery fuel gas streams in the Toledo
Integrated Unit Fuel Gas Mix Drum (“the Fuel Gas Mix Drum”) (See Section 1.5). The fuel gas is burned in
various refinery boilers and furnaces.
a
Coker wet gas is from the refinery Coker units. Coking is a physical process that occurs at pressures slightly higher than atmospheric
and at temperatures greater than 900 °F that thermally crack the feedstock into products such as naphtha and distillate, leaving behind
petroleum coke. See Coking is a refinery process - U.S. Energy Information Administration (EIA).
b
“NHT Preheat” is used in this report to describe a series of seven shell and tube heat exchangers used to heat naphtha from the Crude 1
unit prior to entering the NHT unit for processing.
18
Figure 2. BP Toledo Refinery. Simplified schematic of the parts of the refinery involved in the September 20,
2022, incident. (Credit: CSB)
1.2 Owner and Operator

In 2008, BP-Husky Refining LLC (“BP-Husky”) acquired the BP Toledo Refinery from BP Products North
America Inc. (“BP”). a BP-Husky Refining LLC was a 50/50 joint venture formed by the Toledo Refinery
Holding Company, a subsidiary of BP and the Husky Oil Toledo Company, an indirect subsidiary of Husky
Energy Inc. (“Husky”) [5]. BP operated the BP Toledo Refinery for the joint venture, and refinery personnel
were BP employees operating under established BP policies, practices, and procedures.
In 2021, Cenovus Energy Inc. (“Cenovus”) merged with Husky and became the indirect owner of the Husky Oil
Toledo Company [6]. On August 6, 2022, Husky Oil Toledo Company announced an agreement to purchase
BP’s ownership interest in the BP-Husky Refining joint venture. The transaction closed on February 28, 2023,
making Husky Oil Toledo Company the sole owner of the refinery [7]. After the closing, BP-Husky Refining
LLC was renamed Ohio Refining Company LLC (“Ohio Refining Company”), which became both the owner
and operator of the Ohio Refining Company Toledo Refinery.
a
BP acquired Sohio and the Toledo Refinery in 1987 [46].
19
1.3 Naphtha
Naphtha is a fraction of crude oil that includes hydrocarbons ranging from C5 to C12 and comprises
approximately 15-30 weight percent of raw crude oil [8, p. 2]. The vapor density of naphtha is three to four
times heavier than air. Naphtha can contain hydrogen sulfide (H2S), a toxic and flammable gas.
Naphtha is a flammable liquid.a The BP Safety Data Sheet (SDS) for naphtha lists the boiling point range as 20
degrees Fahrenheit (°F) to 450 °F and the flash point as 45 °F. b Liquids with a flash point of less than 23 °F
and initial boiling point less than 95 °F fall into the highest flammable liquids hazard category of the U.S.
Occupational Safety and Health Administration (OSHA) Hazard Communication Standard (HCS). c
1.4 Refinery Fuel Gas

Fuel gas is typically produced as a vapor by-product in various refinery units, including catalytic reforming,
hydrotreating and hydrocracking, catalytic cracking, and coking [9]. According to the U.S. Environmental
Protection Agency (EPA) National Emission Standards for Hazardous Air Pollutants from Petroleum Refineries,
a refinery fuel gas system includes the off-site and on-site piping and control system that gathers gaseous
streams generated by refinery operations, may blend them with sources of gas, if available, and transports the
blended gaseous fuel at suitable pressures for use as fuel in heaters, furnaces, boilers, incinerators, gas turbines,
and other combustion devices located within or outside of the refinery [10]. d Fuel gas is typically piped directly
to each individual combustion device, and the fuel gas system typically operates above atmospheric pressure.
These gaseous hydrocarbon streams commonly contain a mixture of methane, light hydrocarbons, hydrogen, and
other miscellaneous species.
1.5 Fuel Gas Mix Drum

A Fuel Gas Mix Drum is typically used to prevent operational problems in fuel-using systems, such as boilers or
furnaces, and mixes fuel gas streams from various refinery units. Fuel-using systems are typically designed to
accept a certain degree of change in the fuel gas supply; however, their burners can be sensitive to the rate of
change. The mixing of fuel gas from various refinery units limits the effects of changes in fuel gas composition,
properties, or pressure in fuel-using systems [11].
Figure 3 is a simplified drawing of the Fuel Gas Mix Drum and associated level instrumentation at the BP
Toledo Refinery. The fuel gases from multiple refinery units mix in the drum before going to refinery boilers
a
The U.S. Occupational Safety and Health Administration (OSHA) states that a flammable liquid means any liquid having a flash point
at or below199.4 °F (93 degrees Celsius [°C]). See 29 C.F.R. § 1910.106(a)(19).
b
Flash point means the minimum temperature at which a liquid gives off vapor within a test vessel in sufficient concentration to form an
ignitable mixture with air near the surface of the liquid. See 29 CFR § 1910.106(a)(14).
c
See OSHA Hazard Communication Standard (HCS) at Hazard Communication - Appendix B | (osha.gov).
d
At the BP Toledo Refinery, butane is vaporized as a source of gas to blend with the refinery fuel gas.
20
and furnaces. Any liquid from entrainment or condensation is detected by level instrumentation and a sight
glass. a Liquid is manually drained to a Flare Knockout Drum or an Oily Water Sewer. b
Figure 3. Simplified drawing of the BP Toledo Refinery Fuel Gas Mix Drum and associated level
instrumentation.c (Credit: CSB)
Condensed liquids must be removed from fuel gas streams. Proper removal and disposal of fuel gas condensate
from a fuel gas system is an important consideration for safe operation [9]. BP’s Process Safety Series Safe
Furnace and Boiler Firing explains:
a
Condensation in piping, or carryover or overflow from towers, can be responsible for the presence of liquid hydrocarbons in fuel-gas
systems [12, p. 32]. Fuel gas can become saturated with water when processed in an amine treating unit [9]. High gas velocity through
an absorbing tower can entrain liquid and cause carryover, or a faulty bottom level controller can permit tower overflow [12, p. 32].
Additional condensation can occur when the fuel gases are used in cold climates [9]. Condensate can be a mixture of water,
hydrocarbons, or amine. Hydrogen sulfide can be present in condensate.
b
The Oily Water Sewer System is a regulated system in the refinery designed to collect process wastewater, cooling tower blowdown,
stormwater runoff, storage tank water drawdowns, and other process waste streams. These wastewaters are treated at the refinery
Wastewater Treatment Unit (WWTU) prior to passing through a permitted outfall. All refinery process areas drain by gravity directly to
the main 84-inch Oily Water Sewer to the WWTU. The BP Toledo Refinery Health, Safety, Security & Environmental (HSSE)
Handbook Revision #7 issued February 1, 2020, states that regulations that apply to the sewer system or materials that are drained to the
sewer system include 40 CFR Part 60 Subpart QQQ - Standards of Performance for VOC Emissions From Petroleum Refinery
Wastewater Systems and 40 CFR Part 61 Subpart FF - National Emission Standard for Benzene Waste Operations.
c
The Fuel Gas Mix Drum has two liquid level instrument measurement devices. One was a guided wave radar level measurement which
is “independent of most liquid properties, especially density” [63, p. 133]. The other was a differential pressure meter since “liquid level
can be measured (inferred) by measuring a differential pressure caused by the weight of a fluid column in a vessel balanced against a
reference” [64, p. 454].
21
liquid in the fuel-gas system may enter [a] burner, put out the
fire and create a severe explosion hazard in [a] furnace [12, p.
52]. a
BP’s Process Safety Series Hazards of Oil Refining Distillation Units describes the hazard of an unscheduled or
emergency shutdown and the draining of a fuel gas system. This booklet describes a 1984 incident at a
Singapore refinery where a major process upset resulted in liquid entering the fuel gas system. An explosion
occurred when workers attempted to drain the fuel gas system [13, p. 23 and 78]. b
1.6 Crude Oil and Crude Slate

Crude oil is a mixture of many hydrocarbon compounds and impurities that must be processed and purified in
order to make useful products such as gasoline and diesel fuel [14, pp. 13, 15], [15, p. 13]. There are many
grades of crude oil, but for the purposes of this report, they can be divided into two: light and heavy. “Light
crude” oil is crude oil that contains more low molecular weight components, is more volatile and less viscous,
and flows more easily. “Heavy crude” oil, by contrast, contains higher molecular weight components, and is less
volatile and more viscous than light crude oil [14, pp. 13, 15], [15, p. 13].
A refinery’s crude slate is defined as “the mix of crude oils used as inputs” [15, p. 36]. In this report, the “crude
slate” refers to the crude oil feed composition to the Crude 1 unit. The refinery processed various compositions
of crude oil, but the crude slate feeding the Crude 1 unit typically included a percentage of light crude at any
given time. The light crude, having lighter hydrocarbon components, tended to exit the Crude 1 Tower at or near
the top, and the heavy crude components tended to exit the tower from the lower sections.
1.7 Crude 1 Tower

Crude oil is delivered to the BP Toledo Refinery via pipelines and is stored in tanks. The crude oil is pumped
from storage tanks, preheated in a series of heat exchangers, and heated in a furnace before flowing to the Crude
1 Tower. The Crude 1 Tower separates the crude oil into six basic product boiling ranges. A simplified
schematic of the Crude 1 Tower is shown in Figure 4.
a
The BP Process Safety Series is a collection of booklets describing hazards and how to manage them.
b
Incidents list | Serial number 113.
22
Figure 4. Crude 1 Tower. This simplified schematic shows the crude tower BP used to separate crude oil into
various products. (Credit: CSB)
The lower boiling point materials travel to the top of the Crude 1 Tower as vapor, and the higher boiling point
materials travel to the bottom of the tower. The top, middle, and bottom pumparounds remove liquid products
and heat from the Crude 1 Tower. The pumparounds return cooled liquid, which condenses some of the rising
vapors and provides reflux, which compensates for removing product streams. The Stripper Tower has four side
strippers inside that use steam to strip lighter hydrocarbons from the heavy naphtha, diesel, light gas oil, and
heavy gas oil product draws.
The lightest material, naphtha, is partially condensed and separated in the Crude 1 Overhead Accumulator
Drum. The heaviest of the components drop to the bottom of the Crude 1 Tower and are pumped as feed to a
vacuum distillation unit.
23
The naphtha in the Crude 1 Overhead Accumulator Drum is routed most commonly to the NHT unit and the
Coker Gas Plant. The streams to the Light Virgin Naphtha Storage and Crude 1 Tower reflux are normally
closed but available for use.
Figure 5 is a simplified schematic of where naphtha in the Crude 1 Overhead Accumulator Drum could be
sent. a
Figure 5. BP Toledo Refinery Crude 1 Overhead Accumulator Drum. This simplified schematic shows the
various places BP could send liquid naphtha. (Credit: CSB)
1.8 Coker Gas Plant

To reduce sulfur emissions from furnaces at the BP Toledo Refinery, BP designed and built a Coker Gas Plant,
which was integrated into the refinery in 2018. b The Coker Gas Plant removes mercaptans and other sulfides
a
Naphtha can be sent to directly to NHT preheat on the front end or preheated in another series of heat exchangers and combined with
the naphtha exiting NHT preheat on the back end.
b
In order to meet U.S. Environmental Protection Agency’s (EPA) Operating Permits issued under Title V of the Clean Air Act, BP
developed a processing scheme to segregate and treat coker generated gas to reduce overall BP Toledo Refinery fuel gas total sulfur
levels to 120 ppm by volume or less on an annual average basis.
24
from coker wet gas, a major sulfur contributor to refinery gas.a Removal of the mercaptans and other sulfides is
accomplished using naphtha from the Crude 1 Overhead Accumulator as an absorbing medium in the Coker Gas
Plant Absorber Stripper Tower. Entrained water in the coker wet gas feed is removed from the Absorber
Stripper Tower using a Foul Condensate Draw Off Drum. Figure 6 shows a simplified schematic of the Coker
Gas Plant.
Figure 6. Coker Gas Plant. A simplified schematic of BP’s process to remove sulfur from coker wet gas. (Credit:
CSB)
A portion of the Absorber Stripper Tower bottoms liquid is sent to the Lean Oil Stripper as feed. The Lean Oil
Stripper Tower removes mercaptans from the Absorber Stripper Tower bottoms, producing mercaptan free lean
oil (naphtha). The lean oil is returned to the Absorber Stripper Tower to further reduce the mercaptans and other
sulfides in the coker wet gas.
The remaining Absorber Stripper Tower bottoms liquid is sent to the NHT Feed Surge Drum. The NHT Feed
Surge Drum accumulates naphtha from multiple sources and sends it to the NHT unit.
a
Coker wet gas from the coker wet gas compressor is water washed and amine treated to remove the majority of hydrogen sulfide (H2S)
before entering the Coker Gas Plant. The water wash prevents salt build up.
25
After being processed in the Absorber Stripper Tower, treated coker wet gas is sent to the Sour Gas Knockout
Drum to remove any entrained liquid. Next it is treated in the Polishing Amine Contactor to further remove
hydrogen sulfide before flowing to the Fuel Gas Mix Drum. The Fuel Gas Mix Drum receives gas from multiple
refinery sources and provides fuel gas to boilers and furnaces throughout the refinery.
The coker wet gas can also bypass the Coker Gas Plant and flow directly to the Fuel Gas Mix Drum as shown in
Figure 6 in orange. The bypass line allows untreated coker wet gas to be directly sent to the Fuel Gas Mix Drum
when the refinery needs to perform maintenance on the Coker Gas Plant. a
1.9 Shift Operations Staffing

Operations personnel normally worked on one of four rotating 12-hour shifts. Operators rotated through
multiple jobs within their assigned areas. Typically, only one board operator b was assigned to monitor and
control the control room consoles for the Crude 1 unit, the Vacuum 1 unit, the NHT unit, the Saturated Gas
Plant, the Coker Gas Plant, and the West Flare. c
During the night shift of September 20, 2022, two board operators were assigned to operate these consoles in the
control room. The regularly scheduled board operator had been only recently qualified as a board operator and
had worked in the control room alone for less than a month. The second board operator told the U.S. Chemical
Safety and Hazard Investigation Board (CSB) that they had received a call earlier in the day to come in and
help. The second board operator had worked six years as a board operator. As always, a refinery coordinator
was working with all the board operators on the night shift. d
An operator trainee, who had started working at the BP Toledo Refinery in March 2022, stayed over from the
day shift to the night shift to assist three regularly scheduled outside operators, e one of whom was his brother
who had been working at the BP Toledo Refinery since 2020. In this report, an outside operator and outside shift
supervisor refer to BP employees who primarily work outdoors (or in the field) at the BP Toledo Refinery.
These three outside operators were covering the Crude 1 unit, where the Fuel Gas Mix Drum was located, and
surrounding areas. The outside operators reported to an outside shift supervisor.
a
After the Coker Gas Plant start-up in 2018, a control valve was added to the Coker Gas Plant bypass line to prevent foaming in and
amine carryover from the Polishing Amine Contactor (See A.2.6.2).
b
Situation awareness is “[a]n important responsibility of console [board] operators is to prevent and respond to abnormal situations. The
nature of the abnormal situation may be of minimal or of catastrophic consequence; it is the job of the operations team to identify the
cause of the situation and execute compensatory or corrective action in a timely and efficient manner. Abnormal situations extend,
develop, and change over time in the dynamic process control environments increasing the complexity of the intervention requirements.
Proactively maintaining their situation awareness of the process, where it is, where it is going, and how quickly it is going there, is what
is required of both console [board] and field [outside] operators to effectively prevent and respond to abnormal situations when they
arise. Successfully responding to abnormal situations depends upon both console [board] and field [outside] operators knowing not only
what tasks to perform and how to perform them, but also when to perform them. To become proficient, an operator must know what to
watch, how frequently to watch, what to do, how to do it, and when to do it—all components of knowledge that contribute to effective
situation awareness” [66]. (See Section 4.2).
c
The Fuel Gas Mix Drum, located in the Crude 1 unit, is monitored by the board operator.
d
A refinery coordinator is in charge of the refinery control room and the feeds in and out of the refinery.
e
Outside operators, also known as field operators, are BP employees who perform manual tasks on unit equipment.
26
The refinery coordinator and outside shift supervisor reported directly to a shift superintendent. The BP Toledo
Refinery shift operations staffing for September 20, 2022, is shown in Figure 7 below.
Figure 7. BP Toledo Refinery shift operations staffing. The BP Toledo Refinery shift operations personnel
assigned to operate the Crude 1 unit, the Vacuum 1 unit, the NHT unit, the Saturated Gas Plant, the Coker Gas
Plant, and the West Flare are shown in blue. (Credit: CSB).
1.10 Federal Safety Regulations

BP records show that the BP Toledo Refinery was covered by both the OSHA Process Safety Management
(PSM) standard and the EPA Risk Management Program (RMP) rule [16].
1.10.1 OSHA Process Safety Management Standard

OSHA’s PSM standard establishes procedures for process safety management to “protect employees by
preventing or minimizing the consequences of chemical accidents involving highly hazardous chemicals” [17].
BP considered all the units within the refinery to be covered by the OSHA PSM standard. a
a
The PSM standard covers chemicals deemed hazardous because of their chemical composition and quantity or because of their
flammability characteristics. See Appendix A of 29 C.F.R. § 1910.119(a)(1)(ii).
27
On March 15, 2023, OSHA issued a Hazard Alert Letter to BP concerning the practice of job rotation, which is
utilized at the refinery. The OSHA Hazard Alert Letter stated that: “rotating process operators among multiple
positions, instead of a single position, can reduce the level of expertise and knowledge of operators on the unit
for which they are initially qualified. In the event of a process upset condition or catastrophic incident, this
decrease in expertise can negatively affect incident response efforts, posing a higher likelihood of exposure to
toxic vapor/gas, fire and explosion hazards.” (See Appendix D.)
The Hazard Alert Letter also recommended that BP obtain input from employees and employee representatives
on the effectiveness of the job rotation staffing pattern in place at the refinery, including determining the impact
that the job rotation policy has on operator morale and employees’ ability to respond to process safety incidents.
On March 13, 2023, OSHA issued a number of citations to BP and its successors related to the September 20,
2022, incident with a proposed penalty of $156,250. The OSHA citations are summarized in Appendix E.
1.10.2 EPA Risk Management Program Rule

The EPA’s RMP rule requires facilities using extremely hazardous substances to develop a risk management
plan [18]. According to the EPA, these plans:
identify the potential effects of a chemical accident,
show the steps the facility is taking to prevent accidents, and
outline emergency response procedures [19].
The RMP rule defines three Program levels (Program 1, 2, or 3) based on the potential consequences to the
public and the effort needed to prevent accidents [20, p. 1]. Of these three Program levels, Program 1 is the least
stringent, and Program 3 is the most rigorous.
BP filed a risk management plan with the EPA on July 17, 2018 [16]. BP included flammable mixtures in its
risk management plan, and the company identified its Crude 1 unit as being a Program Level 3 [16, p. 11]. BP’s
risk management plan submission also demonstrates that the BP Toledo’s Refinery Crude 1 unit was regulated
by both the EPA RMP rule and the OSHA PSM standard [16, p. 2].
BP combined the Crude 1 unit and the NHT unit in the risk management plan because the location of the units is
shared meaning “that an event [in] either of these processes could involve the other; therefore, they are
considered one covered process.” The risk management plan identified overfilling as one of the major hazards
identified in the Crude 1 unit and Coker Gas Plant.
1.11 Description of Surrounding Area

Figure 8 shows the area surrounding the BP Toledo Refinery. a The circle diameters are set at one (blue), three
(orange), and five (red) miles from the Fuel Gas Mix Drum. Summarized demographic data for the seven census
a
No off-site impacts were identified by monitoring conducted by BP after the September 20, 2022, incident.
28
blocks within a three-mile vicinity are shown in Table 1. a Census data showed there were about 18,750 people
residing in about 8,635 housing units within this area. In general, the local population was predominantly white,
living in single-unit housing, with 23 percent below the poverty level. Detailed demographic data are included in
Appendix C.
Figure 8. Overhead satellite image of the BP Toledo Refinery and the surrounding area.
(Credit: Google with annotations by CSB)
a
This information was compiled using 2021 Census data from the United States Census Bureau [45].
29
Table 1. Summarized demographic data.
Per
Number of
Race and Ethnicity Capita Poverty Types of Housing Units
Population Housing
(%) Income (%) (%)
Units
($)
White 71 Single Unit 75

Black 16 Multi-Unit 24
Native 0 Mobile Home 1
Asian 0 Boat, RV, Van, etc. 0
18,755 47,628 23 8,635
Islander 0
Other 4
Two+ 9
Hispanic 12
30
2 Incident Description
On September 20, 2022, at approximately 6:46 p.m., a vapor cloud ignited a fire in the Crude 1 unit. The vapor
cloud formed when two BP employees, an outside operator and operator trainee, who were brothers, opened the
pressurized Fuel Gas Mix Drum releasing flammable liquid naphtha to the ground. The vapor cloud ignited
when it found an ignition source, resulting in the deaths of the two BP employees.
2.1 Events Leading Up to the Incident

The CSB found that events preceding the incident led to an abnormal refinery status when the Tuesday night
shift arrived on September 20, 2022, and contributed to the incident. As described below, the incident was the
last in a series of cascading events that started roughly 24 hours before, beginning with a relatively minor
process upset during the previous night’s shift. These events eventually led to the incident.
The sequence of process events that led up to the vapor cloud ignition are introduced below and are described in
greater detail in Section 3.1.
Water Overflow into Crude 1 Naphtha

On the Monday night shift of September 19, 2022, shortly after 7:00 p.m., water began to accumulate in the
Crude 1 Overhead Accumulator Drum, which, several hours later, began to overflow into the naphtha stream
that normally exited the drum. Excess water in the naphtha stream then began to accumulate downstream in the
Coker Gas Plant Foul Condensate Draw-Off Drum. This drum began to overflow water into the Coker Gas Plant
Absorber Stripper Tower. The water overflow and resulting liquid flow increase out of the Absorber Stripper
Tower bottoms and into downstream equipment led to a downstream pressure increase in NHT Preheat.
Naphtha Hydrotreater Preheat Leak

The pressure increase in NHT Preheat was enough to open two emergency pressure-relief valves shortly after
7:00 a.m. on the Tuesday day shift of September 20, 2022. a b A severe piping vibration began as a result of one
of the emergency pressure-relief valves opening. A ¾-inch drain line c broke off the main naphtha piping, which
led to a liquid naphtha loss of containment. The naphtha did not ignite but resulted in an emergency shutdown of
the NHT unit and bypass of the Coker Gas Plant. The Crude 1 unit continued to operate.
Crude 1 Tower Process Upsets
The NHT unit emergency shutdown led to a Crude 1 Tower upset throughout much of the rest of the Tuesday
day shift. With the NHT unit shut down and the Coker Gas Plant bypassed, naphtha from the Crude 1 Overhead
a
b
This report uses the term emergency pressure-relief valve, however the terms pressure relief valve, safety relief valve, pressure safety
valve (PSV), relief valve, or safety valve can be used interchangeably. For a specific application, however, readers should know that
these other names can reflect different operating characteristics and using precise terminology for a specific application may be
appropriate.
c
The ¾-inch drain line was a short branch pipe off the bottom of the six-inch main naphtha piping, typically only used in preparation for
maintenance.
31
Accumulator Drum could be sent only to Light Virgin Naphtha Storage. The Crude 1 Tower experienced several
losses of pumparound cooling, a Crude 1 Overhead Accumulator Drum level upset, and 11 instances of Crude 1
Tower overpressure on day shift after the NHT shutdown.
Crude 1 Tower Feed Changes
As operations personnel worked throughout the day to stabilize the Crude 1 Tower, they made several crude
slate adjustments to the tower feed. At 4:56 p.m., the oncoming Tuesday night shift made another crude slate
change, which removed all light crude oil from the Crude 1 Tower feed.
High Level in Crude 1 Overhead Accumulator Drum
During the Tuesday night shift, another Crude 1 Tower process upset began due to the rapid and complete loss
of light crude oil feed. The Crude 1 Tower upset caused a high level of liquid in the Crude 1 Overhead
Accumulator Drum. To address the rapidly increasing level of liquid in the drum, board operators began
transferring the excess liquid to the Coker Gas Plant Absorber Stripper Tower.
Absorber Stripper Tower Overflow to Fuel Gas Mix Drum, Furnaces, and Boilers
Once the board operator intentionally opened the flow control valve from the Crude 1 Overhead Accumulator
Drum to the Coker Gas Plant (the “naphtha flow control valve to the Coker Gas Plant”), a liquid naphtha flowed
from the drum to the Absorber Stripper Tower. With the naphtha flow control valve to the Coker Gas Plant
open, naphtha began to fill the Coker Gas Plant Absorber Stripper Tower, and eventually overflowed through
the Coker Gas Plant bypass line to the Fuel Gas Mix Drum. Once the Fuel Gas Mix Drum was liquid full, the
naphtha flowed to the downstream furnaces and boilers.
Figure 9 shows the liquid naphtha overflow from the Crude 1 Overhead Accumulator Drum to the Absorber
Stripper Tower, the overflow from the Absorber Stripper Tower to the Fuel Gas Mix Drum, and the overflow to
the furnaces and boilers downstream of the Fuel Gas Mix Drum.
a
This report uses “naphtha flow control valve to the Coker Gas Plant” to describe the naphtha flow control valve from the Crude 1
Overhead Accumulator Drum to the Coker Gas Plant Absorber Stripper Tower.
32
Figure 9: Summary of overflow from the Crude 1 Overhead Accumulator Drum to the Absorber Stripper Tower
in the Coker Gas Plant, and on to the Fuel Gas Mix Drum and downstream furnaces and boilers. (Credit: CSB)
2.2 The Incident

By 6:09 p.m., the Fuel Gas Mix Drum level had begun to increase. The board operator noticed the Fuel Gas Mix
Drum level alarm on the Distributed Control System (DCS) alarm screen and radioed the outside operators to
check the level in the Fuel Gas Mix Drum at 6:16 p.m. a Four outside operations personnel arrived at the Fuel
Gas Mix Drum. Another outside operator began draining the Fuel Gas Mix Drum to the Flare Knockout Drum
a
See A.2.9.4.22 to A.2.9.4.25.
33
and Oily Water Sewer. a The radio traffic captured the following conversation between outside operators and
board operators regarding level in the Fuel Gas Mix Drum:
[Board] Hey, level in the Mix Drum. You’re going to want to check that ASAP.
[Outside] Got it.
[Outside] I’m not even in the sight glass on the Mix Drum.
[Board] Did you say the Mix Drum level is above the sight glass?
[Outside] That is correct.
[Board] Copy that, just drain it as fast as you guys can.
[Outside] We are.
The outside operators attempted to empty the Fuel Gas Mix Drum through the following flow paths:
With four operations personnel present, one of them fully opened the valve on the two-inch line to the
Flare Knockout Drum (a closed system) at approximately 6:17 p.m.;
The same four personnel were present while one of them opened the Fuel Gas Mix Drum two-inch drain
to the Oily Water Sewer, designed for this purpose, at approximately 6:17 p.m. At a later unknown time,
a second one-inch drain line from the Fuel Gas Mix Drum guided wave radar level transmitter to the
Oily Water Sewer was opened;
a
Although the BP Toledo Refinery had a refinery-wide procedure for “Draining of Process Equipment and Lines”, the refinery did not
have any procedures, written instructions, or documented corrective actions for board operators or outside operators to respond to or
troubleshoot a high liquid level in the Fuel Gas Mix Drum, during either normal operations or process upsets, if liquid entered the drum.
The draining procedure in place at the time of the incident required a number of steps to be taken before draining could be done and was
not followed in this emergency situation. The procedure stated:
“Before releasing or draining a material, an evaluation must be made of the following:
Potential environmental impact of the material,
Is or will the material be below the environmental targeted value?
Is there a better way to prevent the release or draining of the material?
Once it has been determined draining or releasing the material is the best way, follow the steps below”.
1.0 Assess the Material to be Drained.
Which included determining if hydrogen sulfide is present, reviewing the Safety Data Sheet and a Caution which stated, “Many
materials are reportable if released or drained. Minimize all material drained to the sewer or purged to atmosphere and never
drain material to the ground […]”.
2.0 Isolate the equipment or Line to Be Drained.
Use BP Toledo Refinery lock-out and tag-out procedure “to properly isolate the line or equipment to be drained”.
3.0 Determine How & Where to Drain the Product.
Which included having a discussion “with the supervisor the location that he/she would like the material drained, the method to
use, and the level of PPE you both think is necessary to prevent personal exposure…”.
3.3 Communicate the “intention to drain material to the Refinery Coordinator. Any material more than two [barrels] to the Oily
Water Sewer requires notification of the BP Toledo Refinery Waste Water Treatment Unit. Once proper notifications to affected
areas are given, the Refinery Coordinator shall grant permission to commence”.
Procedure Deviations “Any deviations, omissions, or additions to this operating procedure as written (including steps that may
not be applicable), and have been reviewed for safety and health considerations, must require supervisor approval by signature or
initials on this procedure”.
34
Two of the operations personnel left the area around the Fuel Gas Mix Drum to check on the Sweet Gas
Knock Out Pot in the Coker Gas Plant, leaving two BP employees, one outside operator and an operator
trainee who were brothers, at the Fuel Gas Mix Drum to finish draining it. These BP employees opened
two ¾-inch bleed valves to the ground at approximately 6:32 p.m.: one at the Fuel Gas Mix Drum
differential pressure level transmitter, a and one at the Fuel Gas Mix Drum sight glass; and
Unbeknownst to anyone, the outside operator and operator trainee began releasing liquid from the Fuel
Gas Mix Drum directly to the ground (while wearing a Self-Contained Breathing Apparatus [SCBA]
and a hydrogen sulfide [H2S] gas detector) b from a two-inch valve on the side of the Fuel Gas Mix
Drum just before 6:39 p.m. This valve normally had a blind flange bolted on the discharge end during
refinery operation. c These BP employees removed the blind flange from the two-inch valve in order to
release liquid from the Fuel Gas Mix Drum.
Figure 10 below shows where the Fuel Gas Mix Drum was drained to the Flare Knockout Drum and Oily Water
Sewer, and where the valves and flange were opened to release material to the ground. Despite all the openings
to drain and release liquid from the Fuel Gas Mix Drum, liquid continued to overflow the Fuel Gas Mix Drum
into downstream furnaces and boilers.
a
When outside operators opened the bleed at the differential pressure level transmitter, the opened bleed altered the pressure differential,
causing the level measurement to indicate zero percent. This gave the false impression that the Fuel Gas Mix Drum was emptying, when
in fact it was still full, despite the draining attempts.
b
One outside operator told the outside operator draining the Fuel Gas Mix Drum to the Oily Water Sewer to put on an SCBA to mitigate
any hydrogen sulfide (H2S) inhalation hazards. SCBA was additional PPE, beyond the norm. Hydrogen sulfide (H2S) monitors were
standard PPE in the refinery process areas (See A.2.9.4.36).
c
This valve on the side of the Fuel Gas Mix Drum was only intended for access by maintenance as part of turnaround or maintenance
activities.
35
Figure 10: Fuel Gas Mix Drum during overfill and attempted emptying. Liquid exit locations are shown in blue.
(Credit: CSB)
At 6:39 p.m., likely as a result of naphtha releasing to the ground and vaporizing, a flammable gas detector near
the Fuel Gas Mix Drum indicated 100 percent of the lower flammability limit (LFL). a This detector only
alarmed locally, not in the control room. The CSB determined that the alarm had been inadvertently disabled b to
the DCS, and therefore only the local alarm horn and lights were functional at the time of the incident, although
the DCS console did provide an analog reading of the percent of LFL. The alarm horn for the detector was
audible in the background of a radio transmission at 6:40 p.m., but there was no evidence that anyone inside the
control room was aware that the two BP employees were releasing liquid from the Fuel Gas Mix Drum to the
ground.
A worker standing nearby saw the two workers near the Fuel Gas Mix Drum along with a visible vapor cloud,
stating to the CSB in an interview after the incident:
a
Most flammable gas detectors “give a reading of the %LEL (or %LFL)” [49, p. 29]. The LFL is defined as the “lowest concentration of
a flammable gas in air capable of being ignited by a spark or flame” [49, p. 35].
b
There was no LFL alarm in the control room as the alarm was set to “Disabled” due to an Alarm Configuration Manager (ACM)
enforcement configuration error that was enforcing the alarm into the “disabled” state. Although the audible and visual board alarming
was disabled, the LFL detector readings would have still shown in the control room.
36
It looks like someone is draining product from the mix drum. And I seen water
being sprayed on it. […] I saw product coming out of the drain. It was like
somebody was draining it to the sewer. They laid something on…to kind of
deflect the stuff spraying out, so it’ll stay […] going into the sewer, instead of
spraying everywhere. So at first, originally, I thought it was a flange or something
had let loose. But, no, it was somebody was draining it. And strong, strong smell.
And I saw that vapor cloud coming from it […] I decided to back up and I […]
stepped back about 20 feet and then it went boom.
An approaching rainstorm shifted the wind, which likely directed the vapor cloud toward the nearby Crude 1
Furnace, the likely ignition source. The vapor cloud ignited at 6:46 p.m., as shown in Figure 11. Figure 12
shows the Crude 1 Furnace’s proximity to the Fuel Gas Mix Drum, the area of the naphtha release, and the
eyewitness’s location.
Figure 11: Vapor cloud ignition and the ensuing six seconds. Ignition at Crude Furnace (circled in red, upper left
photo). (Credit: BP with annotations by CSB)
37
Figure 12: Overhead view of naphtha release, likely ignition source, and eyewitness location. (Credit: BP with
annotations by CSB)
2.3 Emergency Response

After the vapor cloud ignited at 6:46 p.m., the BP Toledo Refinery Emergency Response Team (ERT) reported
to the scene at 6:54 p.m. At 7:04 p.m., a board operator realized that naphtha was flowing to the Fuel Gas Mix
Drum from the Crude 1 Overhead Accumulator Drum through the Coker Gas Plant. The board operator closed
the naphtha flow control valve to the Coker Gas Plant, which was allowing naphtha to flow to the Coker Gas
Plant Absorber Stripper Tower and Fuel Gas Mix Drum. This was the same flow control valve that had been
opened earlier in the attempt to alleviate the high level in the Crude 1 Overhead Accumulator Drum. The ERT
assembled three separate teams to approach the Fuel Gas Mix Drum and close the open drain points where fire
was emanating from the drum. To extinguish the fires, emergency responders had to close all the Fuel Gas Mix
Drum valves that had been opened by the two BP employees:
to the flare system,
to the Oily Water Sewer,
at the sight glass,
at the differential pressure level measurement,
at the guided wave radar device to the Oily Water Sewer, and
the two-inch valve on the side of the Fuel Gas Mix Drum that had had the blind flange removed.
Figure 13 shows the Fuel Gas Mix Drum after the incident.
38
Figure 13: Fuel Gas Mix Drum post-incident. (Credit: BP with annotations by CSB)
The fire was extinguished by 10:10 p.m. BP estimated that 23,502 pounds of naphtha and 66,889 pounds of fuel
gas were released during the incident.
The two BP employees who were draining the Fuel Gas Mix Drum were fatally injured from burns they
received in the fire.
39
3 Incident Analysis
3.1 Incident Progression

As described in Section 2, the CSB investigated the events leading up to the incident. The CSB found that the
events, actions, and decisions preceding the incident led to an abnormal refinery status when the night shift
arrived and contributed to the incident. The sequence of events that led up to draining the Fuel Gas Mix Drum
are described in detail in this section.
3.1.1 Naphtha Standpipe Overflow

The Crude 1 Overhead Accumulator Drum included multiple level measurement devices. Among these was a
measurement for the water phase in the bottom of the drum, which was separate from the other devices
measuring total level in the drum. During a turnaround at the refinery in 2022,a the water phase level
measurement technology was changed to a guided wave radar. Although the change used the same vessel
connections, the new device was calibrated differently from the previous one. This change meant that water
would overflow the naphtha standpipe at 69 percent indicated level, rather than at an indicated 100 percent level,
with the original level device. This change in level indication had not been communicated to operators, b and had
been in service for approximately six weeks before the incident occurred. Consequently, operations personnel
were likely unaware that the 69 percent water phase level meant that water could carry over into the naphtha
stream.
On the evening of September 19, 2022, at approximately 7:10 p.m., the Crude 1 Overhead Accumulator Drum
water phase level began to increase because a board operator had partially closed the Crude 1 Overhead
Accumulator Drum water level control valve in order to increase water flow to the Crude 2 c unit. With the water
level control valve in manual mode, the water phase level in the Crude 1 Overhead Accumulator Drum steadily
increased. The water phase level indicator plateaued at 69 percent at approximately 11:45 p.m. Although the
Crude 1 Overhead Accumulator Drum normally contained some water, the water phase reached the height of the
naphtha standpipe inside the Crude 1 Overhead Accumulator Drum at 69 percent water phase level. As a result,
water carried over to downstream naphtha users, as shown in Figure 14 below.
a
See A.2.9 for a timeline of these events. The level measurement technology was changed from a displacer to a guided wave radar as part
of a piping change during a four-month maintenance outage, called the “2022 Turnaround,” which lasted from April to August 2022.
b
While the Management of Change (MOC) itself was communicated to operators before starting up after the change, the communication
did not include that this level would read differently than it did before.
c
The BP Toledo Refinery operates two crude units. This report refers to them as Crude 1 and Crude 2.
40
Figure 14: Crude 1 Overhead Accumulator Drum water phase level before water control valve adjustment (top)
and during water carryover into naphtha stream (bottom). (Credit: CSB)
Downstream of the Crude 1 Overhead Accumulator Drum, the flow of water-laden naphtha to the Coker Gas
Plant Absorber Stripper Tower resulted in water collecting in the Coker Gas Plant Foul Condensate Draw Off
Drum, which plateaued at 100 percent at approximately 2:12 a.m. on September 20, 2022. Once the drum was
full, the water began to backflow and return to the Absorber Stripper Tower. The water in the Absorber Stripper
Tower reduced the reboiler exit temperature, which decreased the vapor flow up the tower. The lower vapor
flow allowed the liquid on the Absorber Stripper Tower trays to de-inventory and drop liquid down to the
bottom of the tower, increasing the bottoms level. The Absorber Stripper Tower bottoms level control valve
41
opened to remove the excess level, causing increased liquid level in the NHT Feed Surge Drum. This NHT Feed
Surge Drum level increase resulted in higher flow and pressure to NHT Preheat when the NHT Feed Surge
Drum level control valve in turn opened further. a Figure 15 shows this water overflow and resulting liquid flow
increase out of the Absorber Stripper Tower bottoms and into the NHT Feed Surge Drum.
Figure 15: Water overfill in Coker Gas Plant Absorber Stripper Tower and flow increase to NHT Feed Surge
Drum (green arrows). (Credit: CSB)
The increased liquid flow into NHT Preheat resulted in a downstream pressure increase to the series of heat
exchangers (HE) downstream of the NHT Feed Surge Drum. As a result, at approximately 3:20 a.m. on
Tuesday, September 20, 2022, emergency pressure relief-device PSV-D, which had the lowest set pressure in
the heat exchanger train, b opened (See Figure 16). The night shift c operations personnel were able to isolate the
emergency pressure-relief valve from the NHT Preheat process and reseat the emergency pressure-relief valve.
a
The NHT Feed Surge Drum level control valve, which had been operating at approximately 30-40 percent open at steady state, peaked
at 56 percent open at 3:27 a.m.
b
A timeline of this event can be found beginning at A.1.4.6.3.
c
Night shift ended and day shift began at approximately 4:30 a.m.
42
Figure 16: Location of first emergency pressure-relief valve (labeled as PSV-D) opening on NHT Preheat heat
exchangers (each labeled as HE). a (Credit: CSB)
The CSB concludes that water carryover from the Crude 1 Overhead Accumulator Drum initiated a cascade of
events that caused the emergency pressure-relief valve in NHT Preheat to open.
3.1.2 Naphtha Hydrotreater Preheat Leak

On day shift, at approximately 7:13 a.m., outside operators noted PSV-D opening due to a second liquid flow
and pressure increase into NHT Preheat, similar to what had occurred on the previous shift. The day shift was
aware of PSV-D opening on the previous night shift. Another valve, PSV-B, upstream of PSV-D, also opened at
approximately this time and chattered.b The inlet piping to PSV-B began vibrating severely. One outside
operator later described it to the CSB:
I’ve seen a lot of PSVs lift. I’ve never seen a PSV lift like this. The design of that
PSV, there’s something the matter, you know. So […] the closest I can explain
is if you’ve ever seen the…put a jackhammer on the front of a Bobcat to break
up concrete. The force that that has, that’s the force that this PSV was lifting at.
It was just shaking and rattling and everything. Insulation was falling off. Valve
handles are rattled off. Valves are opening and closing. It was…it was incredibly
bad.
The vibration was strong enough that the bypass valve around the heat exchangers and PSV-B inlet in NHT
Preheat vibrated open at least twice. The emergency pressure-relief valve chatter continued as operations
personnel attempted to troubleshoot and reduce pressure in the system. At approximately 7:54 a.m., a leak
developed near a ¾-inch drain valve on the vibrating piping. One outside operator charged c a nearby fire
monitor and started the monitor water flow on the leak. Another outside operator later told the CSB about the
situation:
a
All emergency pressure-relief valves (shown as PSVs) in NHT Preheat discharged to the West Flare (not shown).
b
Chattering is “the rapid opening and closing of a pressure-relief valve. The resulting vibration may cause misalignment, valve seat
damage, and if prolonged, mechanical failure of valve internals and associated piping” [48].
c
Fire monitors are devices used for manual firefighting or in automatic fire protection systems to “discharge large volumes of water and
have good straight stream range. Discharge can be controlled by the type and size of adjustable nozzle or diameter of straight stream
nozzle” [17, p. 369]. The process of “charging” a fire monitor involves filling piping from a nearby water source and placing it under
pressure so that the water can be directed, through a nozzle, toward the fire site when needed.
43
We can’t… stay like this. We can’t leave this thing lifting. If I can’t block it in,
we’re going to tear something up. That’s when I told the operators I was with,
I’m like, “We got to go charge the fire monitors.” Because we… You just knew
that something was going to fail as hard as everything was shaking.
The refinery coordinator attempted to reduce the NHT Preheat pressure by maximizing cooling in NHT Preheat,
among other things, stating, “we’re forcing more feed through the [NHT Preheat] exchangers, carrying more
heat away” to outside operators over the radio. Outside operators requested to bypass the chattering PSV-B and
its associated heat exchanger at 7:48 a.m., but the refinery coordinator did not approve at first. By the time the
refinery coordinator in the control room authorized the bypass at 7:55 a.m., outside operators could not access
the bypass valve. The leak was too severe and too close to the bypass valve to allow outside operators access.
The shift superintendent left the control room and went outside to NHT Preheat to assist. This superintendent
observed the vibration along with outside operators, and alerted the BP Toledo Refinery ERT, who reported to
the scene. a
The continuing piping vibration then caused the branch connection to the drain valve to fail completely, and the
¾-inch bleed broke off at approximately 8:12 a.m., 18 minutes after the first smaller leak began. With this loss
of primary containment, the naphtha leak was significantly larger. Operations personnel decided to implement
an emergency depressurization and shut down of the NHT unit. Figure 17 shows the locations of the ¾-inch
bleed failure where the loss of containment occurred, and both emergency pressure-relief valves that opened.
Figure 17: NHT Preheat loss of containment and emergency pressure-relief valve (each labeled as PSV)
openings at NHT Preheat heat exchangers (each labeled as HE). (Credit: CSB)
At this point, outside operators directed significant amounts of water onto the release. Their quick action likely
prevented ignition of the naphtha. b As a result of the incident and response, the containment area around NHT
Preheat filled with a water-naphtha mixture. Four operations personnel received first aid for skin irritation due to
a
The loss of containment was contained with water spray and did not ignite.
b
See A.1.4.6.15 to A.1.4.6.19.
44
contact with this water-naphtha mixture as they worked to isolate the leak. BP later estimated that the failure
released an estimated 63,625 pounds of naphtha.
The CSB concludes that vibration from the NHT Preheat emergency pressure-relief valve chatter caused the
leak in NHT Preheat.
The CSB also concludes that had the BP Toledo Refinery bypassed the affected heat exchangers in NHT Preheat
in response to the initial leak, it might have avoided the pipe failure. The pipe failure caused an emergency
shutdown of the NHT unit.
3.1.3 Coker Gas Plant Bypass

Operations personnel successfully isolated the naphtha leak, shut down the NHT unit, and bypassed the Coker
Gas Plant since naphtha could no longer flow to the NHT unit. To bypass the Coker Gas Plant, operators fully
opened the control valve in the coker wet gas bypass line in manual mode, but the Absorber Stripper Tower
coker wet gas inlet was also still open. a Both the gas flow path entering the Absorber Stripper Tower and the gas
bypass flow path remained open, meaning that the Coker Gas Plant was not fully isolated (or bypassed). The
naphtha flow control valve to the Coker Gas Plant was closed to stop the liquid naphtha flow into the Absorber
Stripper Tower. Figure 18 below shows this “bypass mode” configuration. b The Crude 1 unit continued to
operate while refinery teams began to evaluate a repair plan for the failed branch connection.
a
A control valve was installed in the Coker Gas Plant bypass line in 2019 to control the pressure differential of the Coker Gas Plant
Polishing Amine Contactor to prevent Coker Gas Plant process upsets. (See A.2.6.2)
b
While the bypass mode isolated liquid naphtha flow from the Coker Gas Plant, it did not isolate gas flow into the Coker Gas Plant. Gas
flow continued through both the bypass line and through the Coker Gas Plant simultaneously.
45
Figure 18: Coker Gas Plant bypass configuration on the day of the incident. (Credit: CSB)
The CSB concludes that the NHT unit emergency shutdown necessitated bypassing the Coker Gas Plant, but the
Coker Gas Plant was not fully isolated, even though its operational state was considered “in bypass.” Bypassing
the Coker Gas Plant left an open flow path from the Absorber Stripper Tower to the Fuel Gas Mix Drum.
3.1.4 Crude 1 Overhead Accumulator Drum Level Control

With the NHT unit shut down and the Coker Gas Plant bypassed, naphtha from the Crude 1 Overhead
Accumulator Drum could be sent only to Light Virgin Naphtha Storage, as shown below in Figure 19.
However, the liquid level in the Accumulator Drum could not be maintained with the Light Virgin Naphtha flow
control valve solely. For the majority of time between 1:30 p.m. and 3:30 p.m., while the flow control valve was
near or at its fully open position, the accumulator drum level exceeded its high-high alarm setpoint, and the flow
reading to the Light Virgin Naphtha Storage was above the meter’s range, as shown below in Figure 20.
46
Figure 19: Liquid naphtha flow paths out of the Crude 1 Overhead Accumulator Drum. (Credit: CSB)
47
Figure 20: Crude 1 Overhead Accumulator Drum level control and flow to Light Virgin Naphtha. The yellow
shaded areas indicate periods of accumulator level (gray) over the high-high alarm setpoint, while the flow
meter to Light Virgin Naphtha (blue) was out of range and the flow control valve (orange) nearly or fully
opened. (Credit: CSB)
The CSB concludes that there was only one destination available for Crude 1 Overhead flow after the NHT unit
was shut down and the Coker Gas Plant was bypassed.
The CSB concludes that the Crude 1 Overhead Accumulator Drum level could be managed under normal
conditions with only one destination available by sending it to Light Virgin Naphtha Storage, but the upset or
excess flow conditions in the Crude 1 Tower in this incident exceeded the control valve capacity to Light Virgin
Naphtha, meaning that the Crude 1 Overhead Accumulator Drum level could not be well controlled.
3.1.5 Crude 1 Tower Instability

After the NHT unit shutdown, throughout the late morning and afternoon, workers continued to attempt to
stabilize the Crude 1 Tower. While they did so, top pumparounda and middle pumparound flows dropped to zero
a
The Crude 1 Tower and pumparounds are described above in Section 1.7.
48
several times. a Figure 21 shows the Crude 1 Tower overpressures b and loss of pumparounds that day, beginning
shortly after the NHT unit shutdown. Operations personnel worked through the afternoon to stabilize the Crude
1 Tower overhead pressure, temperature profile, and pumparound flows. By 4:30 p.m., as night shift began
reporting to the refinery for shift turnover, the Crude 1 Tower was beginning to stabilize, although the tower
pressure was still much closer to the emergency pressure-relief valve set pressures than it had been that morning,
before the NHT unit emergency shutdown: 25 to 28 pounds per square inch gauge (psig) before compared with
over 36 psig at shift change. The last Crude 1 Tower overpressure event to occur on day shift ended at
approximately 4:17 p.m. The Crude Tower overhead pressure reached the high alarm point, set at 32 psig, at
approximately 10:00 a.m., and remained in alarm until approximately 6:00 p.m. In other words, the alarm was
sounding continuously for roughly eight hours.
a
A complete loss of all three pumparounds leads to a loss of cooling resulting in high Crude 1 Tower temperatures and high Crude 1
Tower vapor velocities making it difficult to re-establish the pumparounds.
b
The Crude 1 Tower overhead line contained five emergency pressure-relief valves, set at staggered pressures ranging from 38 to 40
pounds per square inch gauge (psig).
49
Figure 21: Crude 1 Tower overhead pressure. Red arrows indicate likely emergency pressure-relief valve(s)
(PSV) opening. Crude 1 Tower pressure (blue trend) was above the high alarm point (32 psig) for much of the
day. (Credit: CSB)
The CSB concludes that as a result of the NHT unit emergency shutdown, the Crude 1 Tower operation was
unstable throughout the day of the incident. This was demonstrated by high Crude 1 Tower overhead pressure
with Crude 1 Tower emergency pressure-relief valves opening multiple times, multiple losses of pumparound
cooling, and inability to control Crude 1 Overhead Accumulator Drum level.
In the mid-afternoon, between approximately 3:00 p.m. and 4:00 p.m., refinery leadership and process
engineering personnel a met to address Crude 1 Tower instability and to discuss plans for oncoming night shift
support, among other things. During the meeting, these personnel discussed shutting down Crude 1 or putting
Crude 1 in “circulation,” b among other options, in an effort to stabilize Crude 1 Tower operation with the NHT
a
The Interim Asset Superintendent, the Process Engineering Superintendent, and the Production Planning Superintendent told CSB
investigators they attended the meeting.
b
The BP Toledo Refinery defined “hot circulation,” or simply “circulation” as a process configuration, typically for startups, in which
feed was turned off to the Crude 1 Tower, but oil was recirculated, at least partial reflux was maintained, and the Crude 1 Furnace
remained on but at temperature lower than normal. Although located in a startup procedure, written guidelines were available for this
process configuration on the day of the incident.
50
unit shut down and the Coker Gas Plant bypassed. Ultimately, it was decided that neither shutdown nor
circulation would be done at that time, which allowed the crude oil feed to Crude 1 to continue, even though the
Crude 1 Tower had overpressured eight times (Figure 21), and the Crude 1 Overhead Accumulator Drum level
had been above the high-high alarm point for approximately two hours (Figure 20), by 3:00 p.m.
Moreover, although one of the participants in the meeting summarized the meeting’s outcome in an email, the
email was never sent. Further, the CSB found no evidence that the actions agreed upon in the meeting were ever
conveyed to the night shift refinery coordinator or board operators, either verbally or in writing. Although
reducing feed temperature to the Crude 1 Tower and putting the tower on circulation were discussed in the
meeting as possible actions, the night shift operations did not have these items in the turnover notes, and the
night shift operators did not indicate to the CSB that they were aware of these options as possible corrective
actions. The night shift refinery coordinator told the CSB that with “the crude tower on, sat gas plant, coker gas
plant, NHT down,” he had “never seen it operate like that” and “had no technical guidance on how to operate
it.”
The CSB concludes that on the day of the incident, BP Toledo Refinery personnel involved in the afternoon
meeting regarding Crude 1 Tower instability did not adequately communicate the guidance to safely operate the
Crude 1 Tower from that meeting to the oncoming night shift personnel. This left night shift board operators to
decide how to operate the tower under the given conditions. Had the BP Toledo Refinery considered this process
instability that occurred throughout the day, and had there been better communications through shift change,
there could have been safeguards put in place before or early in the night shift.
3.1.6 Crude 1 Tower Crude Slate Change

In response to the NHT unit shutdown and Coker Gas Plant bypass earlier that day, the crude slate to the Crude
1 Tower had been adjusted several times throughout the day as operations personnel worked to stabilize the
tower. Night shift personnel arrived at approximately 4:30 p.m. on September 20, 2022, and shortly afterward, a
significant crude slate change to the Crude 1 Tower occurred. Light crude oil flow was 26,000 barrels per day,
or approximately 32 volume percent of the total feed, at 4:56 p.m., but it was eliminated entirely by 5:10 p.m. A
board operator cut the light crude feed because he thought the light crude oil flow was too high given that the
NHT unit was down. Another online crude oil pump, connected to heavy crude oil storage, automatically
increased speed to maintain the total feed flow rate to the Crude 1 Tower. Abruptly replacing light crude oil feed
with heavy crude oil created a large and rapid change in the crude oil composition, which reached the Crude 1
Tower approximately 40 minutes later. Figure 22 shows the crude oil compositions and flow rates feeding the
Crude 1 Tower before and after the light crude oil flow stopped.
51
Figure 22: Crude 1 Tower Feed rate (blue) and percent light feed composition (orange) day of incident. The
yellow bars indicate shift change turnover. (Credit: CSB)
At approximately 5:30 p.m., Crude 1 Tower temperatures began to increase rapidly, initiating the Crude 1
Tower upset, as shown in Figure 23 below.
52
Figure 23: Crude 1 Tower temperature profile. Before the upset, temperatures are hottest at the bottom of
the tower and decrease going up the tower, normal for any distillation tower. (Credit: CSB)
As tower temperatures increased, there was not enough liquid volume in the Crude 1 Tower pumparound
circuits to maintain the vapor-liquid interface in each section of the Tower and to ensure sufficient cooling. As
the pumparound flows successively dropped to zero (bottom at 5:41 p.m., middle at 5:47 p.m., and top at 6:13
p.m.), the Crude 1 Tower temperature profile further increased. Volatile materials in the tower flashed due to the
increased temperatures, which led to high level in the Crude 1 Overhead Accumulator Drum as the increased
vapor flow condensed in the overhead coolers.
The CSB concludes that the night shift board operator removed all light crude feed from the Crude 1 Tower in
an effort to reduce the overhead naphtha flow, in response to the limited destinations available for naphtha.
The CSB concludes that rapidly eliminating all the light crude oil feed to the Crude 1 Tower initiated another
process upset during the night of the incident. This change created a rapid increase of vapor flow up the tower
and led to 1) high level in the Crude 1 Overhead Accumulator Drum, 2) loss of pumparound cooling, and 3)
increased temperatures throughout the tower.
53
3.1.7 Crude 1 Overhead Accumulator Drum High Level

The Crude 1 Tower upset created a tower overhead flow surge a few minutes later, which in turn filled the
Crude 1 Overhead Accumulator Drum. Within a five-minute period, the Crude 1 Overhead Accumulator Drum
level rose from a steady state of 47 percent to 68 percent (the high level alarm), to 70 percent (the high-high
level alarm), and finally to 89 percent level at 5:40 p.m. The Crude 1 Overhead Accumulator Drum level
increased so rapidly that the level indication went from 65 percent to 89 percent in less than two minutes. In
response to this rapid level increase, the board operators searched for ways to reduce level in the drum quickly.
With other liquid naphtha flow paths out of the Crude 1 Overhead Accumulator Drum already either at full
capacity or shut down, a a board operator opened the naphtha flow control valve to the Coker Gas Plant in an
effort to reduce the Crude 1 Overhead Accumulator Drum level even though the Coker Gas Plant was intended
to be bypassed at the time. Figure 24 below shows this valve opening (in red) and the Crude 1 Overhead
Accumulator Drum’s rapid level increase and resulting decrease (in blue) after the naphtha flow control valve to
the Coker Gas Plant was opened.
Figure 24: Crude 1 Overhead Accumulator Drum level (blue) and naphtha flow control valve to the Coker Gas
Plant opening (red) in response to the high level. (Credit: CSB)
a
The control valve to Light Virgin Naphtha Storage did have a bypass around it that could only be opened in the field. However, this
manual bypass valve was valve had a sticker on it that read “Open with Caution” and the refinery coordinator considered the flow to
Light Virgin Naphtha Storage to be “maxed out” while the bypass remained closed. Since the bypass could only be opened in the field,
there may have been insufficient time available to open the bypass in this case. The CSB could find no operating procedure referencing
opening of this bypass. Therefore, the CSB did not consider this manual bypass to be a viable option.
54
One board operator later explained to CSB investigators that overfilling the Crude 1 Overhead Accumulator
Drum could send liquid to a downstream compressor, which would shut down another refinery unit:
…our [Crude 1 Tower] overhead was filling up and, you know… The [Light
Virgin Naphtha] valve […] was wide open. So that was, like, the only place we
were going with the overhead, or could go, and it was filling up. […] And I don’t
know what percentage we were at [on the Crude 1 Overhead Accumulator Drum
level], but all I know is we were pushing 80 percent, or maybe we were over,
but… We looked at each other and we were like, you know, what do we do? And
so I started looking at other units and I said, well, the gas plant, the Coker Gas
Plant is open. I mean, the [NHT] feed drum a is only at, like, 36 percent. I was
like, let’s send it to the Coker Gas Plant. Even though it’s down, we can store the
feed in there, you know? And that way we don’t trip the compressor off.
The CSB concludes that the board operators did not have clear instructions about how to manage Crude 1
Overhead Accumulator Drum high level. With only one destination available for naphtha, and the control valve
to Light Virgin Naphtha at maximum capacity, the improvised solution by the board operators was to transfer
excess Crude 1 Overhead Accumulator Drum level to the bypassed Coker Gas Plant.
The CSB concludes that opening the naphtha flow control valve to the Coker Gas Plant while the bypass valves
were open allowed liquid naphtha to flow into the Coker Gas Plant and then overflow into the Fuel Gas Mix
Drum and proceed to furnaces and boilers.
3.1.8 Absorber Stripper Tower Overflow

The night shift board operators were unaware that earlier that day, the day shift had closed a valve on the Coker
Gas Plant Absorber Stripper Tower bottoms piping, known as a Remotely Operated Emergency Isolation Valve
(ROEIV). This closed valve prevented flow through the Absorber Stripper Tower bottoms into the NHT Feed
Surge Drum, so the liquid naphtha began to fill the Coker Gas Plant Absorber Stripper Tower more quickly than
if the ROEIV had remained open. Figure 25 illustrates the flow path from the Crude 1 Overhead Accumulator
Drum to the Coker Gas Plant Absorber Stripper Tower.
a
The NHT Feed Surge Drum and the Crude 1 Overhead Accumulator Drum were of similar volumes, each approximately 14,000–16,000
gallons.
55
Figure 25: Flow path from Crude 1 Overhead Accumulator Drum to Coker Gas Plant. (Credit: CSB)
Once the Absorber Stripper Tower filled with liquid naphtha up to the elevation of the coker wet gas inlet
piping, a line normally for gas flow only, the liquid naphtha overflowed into the Coker Gas Plant bypass piping
and into the Fuel Gas Mix Drum. Figure 26 below shows the overflow path from the Coker Gas Plant Absorber
Stripper Tower to the Fuel Gas Mix Drum.
56
Figure 26: Overflow path from Coker Gas Plant to Fuel Gas Mix Drum (orange arrows). a (Credit: CSB)
The closed ROEIV hastened the overflow into the Fuel Gas Mix Drum through the open bypass line, but the
overflow likely would have occurred even if the ROEIV had been open. This was demonstrated in a 2019 near
miss,b similar to the September 20, 2022, incident, in which the ROEIV was open, but the overflow from the
Absorber Stripper Tower, through the Coker Gas Plant bypass, to the Fuel Gas Mix Drum still occurred. This
indicates that the inflow to the Absorber Stripper Tower was greater than the outflow from it even with the
ROEIV open.
The CSB concludes that the closed ROEIV on the Absorber Stripper Tower bottoms caused the overflow
through the Coker Gas Plant bypass line to the Fuel Gas Mix Drum to occur more quickly than it otherwise
would have, giving board operators less time to troubleshoot and respond.
a
The Coker Gas Plant bypass piping elevation was not above the top of the Absorber Stripper Tower. The drawing shows the process
connectivity only and is not intended to be an elevation drawing.
b
See Section 4.4.1.
57
3.1.9 Fuel Gas Mix Drum Overflow

The Absorber Stripper Tower began overflowing into the Fuel Gas Mix Drum at approximately 6:09 p.m. While
the outside operations personnel were draining the Fuel Gas Mix Drum (See Section 2.2), refinery workers in
various other units communicated by radio that they observed smoke coming from furnace and boiler stacks.
The Crude 1 Furnace started emitting black smoke from its stack first, at 6:20 p.m. A boiler stack emitted green
smoke at 6:22 p.m. Smoke from several other boilers and furnaces followed over the next 30 minutes
(approximately). The smoke indicated that the Fuel Gas Mix Drum was overflowing liquid hydrocarbons
(naphtha) to the furnaces and boilers by 6:20 p.m. Approximately 11 minutes elapsed from the time that liquid
naphtha began to enter the Fuel Gas Mix Drum to the time that the naphtha reached the Crude 1 Furnace (at 6:20
p.m.).
When the naphtha entered the furnaces, the night refinery coordinator announced on the radio, “We got liquid in
the fuel gas system. Keep away from those furnaces.” The smoke coming from the boiler and furnace stacks
caused the night shift refinery coordinator to assert on the radio to shut down the refinery, including stopping
fuel flow to all furnaces and boilers, at 6:27 p.m. The refinery coordinator later told the CSB that he took these
actions due to the potential for “an explosion in the furnace.” By this time, however, it was too late to prevent
the consequences from the excess liquid that already had flowed into the Fuel Gas Mix Drum.
A furnace fire was reported at approximately 6:24 p.m. under the Coker 2 furnace. This indicates that not only
was liquid in the fuel gas system, but also that all furnaces and boilers were fire hazards with liquid overflowing
to the fuel gas system.
The night shift supervisor and other operations personnel later told the CSB that while draining material to the
Flare Knockout Drum and Oily Water Sewer, they believed the material to be an amine-water solution, not
naphtha. The night shift supervisor stated that he mentioned to the outside operator and operator trainee to get
SCBAs nearby out of concern they could need supplied air due to the potential hazard of hydrogen sulfide (H2S)
in the amine-water solution. The two BP employees who released the Fuel Gas Mix Drum to the ground may
have believed that the material was amine-water solution, rather than naphtha, just as other operations personnel
did. The outside operations personnel were not aware of the actions by the board operator that led to naphtha
overflowing from the Absorber Stripper Tower to the Fuel Gas Mix Drum.
The CSB concludes that the BP Toledo Refinery recognized the potential for furnace fires or explosions if liquid
entered the fuel gas systems.
The CSB concludes that the two BP employees who released naphtha from the Fuel Gas Mix Drum to the
ground may have believed that the material was an amine-water solution just as other operations personnel did.
Because the BP Toledo Refinery lacked established corrective actions for the situation, the two BP employees
may have improvised a solution in real time without being aware of the consequences of releasing to the ground.
Figure 27 shows a summary timeline of some of the key events in the hour preceding the incident, showing
limited time available to refinery personnel for decision-making. Appendix A.2.9.4 contains a detailed timeline
of the event.
58
Figure 27: Summary timeline of some key events for the hour preceding the incident. (Credit: CSB)
3.2 Draining and Liquid Release

As discussed in Section 1.5, the BP Toledo Refinery Fuel Gas Mix Drum collects entrained liquids from the
refinery fuel gas system. The Fuel Gas Mix Drum has a two-inch line to dispose of those liquids to the Flare
Knockout Drum as shown below in Figure 28. A two-inch line connected to the line to the Flare Knockout
Drum allowed direct draining to the refinery Oily Water Sewer. A guided wave radar level instrument had a
hard piped drain to the Oily Water Sewer. However, on the day of the incident, there were additional valves on
the Fuel Gas Mix Drum opened and used by two BP employees to release liquid naphtha to the ground including
a ¾-inch tap from the differential pressure level instrument, a ¾-inch drain from the sight glass, and a two-inch
gate valve with a bolted closed blind flange on the side of the Fuel Gas Mix Drum which was located
approximately seven feet from the ground. a
a
This two-inch gate valve with a bolted closed blind flange was to be used for steaming out the vessel for maintenance activities or
turnaround.
59
Figure 28. BP Toledo Refinery Fuel Gas Mix Drum. Valves circled in green were opened to drain and release
naphtha. (Credit: CSB)
The Fuel Gas Mix Drum at the BP Toledo Refinery had a diameter of seven feet and eleven inches and a seam-
to-seam height of nearly fourteen feet and nine inches. Consistent with industry standards, a the liquid level
measurement from the sight glass, differential pressure level, and guided wave radar all measured only the
bottom six-foot span of the Fuel Gas Mix Drum and could not indicate a liquid height greater than that.
a
API Recommended Practice 551 Second Edition (API RP 551) Process Measurement, Section 3.4 Instrument Selection provides
guidance for instrument selection and identifying expected operating cases such as normal flow as well as emergencies and upsets [63,
p. 18]. Section 7.2.2 of API RP 551 provides guidance for range selection and states to “determine the maximum process liquid level,
for most services, a liquid holdup time of between 5 to 10 minutes is used to ensure controllability and safety. Consequently, there is
between 2 ½ and 5 minutes between the normal mid-rage set point and loss of measurement” [63, p. 112].
60
Figure 29 shows the rapid progression of overflow events after the naphtha flow control valve to the Coker Gas
Plant was opened by a board operator. In approximately 14 minutes the Coker Gas Plant Absorber Stripper
Tower reached 100 percent level and approached overflow through open bypass piping. In approximately seven
minutes, naphtha from the overflowing Coker Gas Plant Absorber Stripper Tower filled the six-foot span in the
Fuel Gas Mix Drum.
Figure 29. Overflow progression. Decreasing level in the Crude 1 Overhead Accumulator Drum and increasing
levels in the Coker Gas Plant Absorber Stripper Tower and Fuel Gas Mix Drum.a (Credit: CSB)
At this point, the guided wave radar and sight glass indicated the liquid level in the Fuel Gas Mix Drum had
reached the top of the measured six-foot level span around 6:15 p.m. b As the liquid continued to fill the Fuel
Gas Mix Drum, these instruments could not provide any indication that the drum was overflowing liquid into the
fuel gas piping.
a
The Fuel Gas Mix Drum differential pressure (dP) level is shown. The Fuel Gas Mix Drum guided wave radar level was working during
the September 20,2022, incident, however, the data points were not recorded by the BP Toledo Refinery process data historian. (See
A.2.8.2)
b
The Fuel Gas Mix Drum differential pressure level indication plateaued at around 67 percent of the six-foot span even though the
guided wave radar level instrument and sight glass showed the Fuel Gas Mix Drum at capacity. This incorrect indication was due to the
density of naphtha being less than the density of the typical amine-water solution used to calibrate the differential pressure level meter.
The BP SDS for naphtha lists the relative density of straight run naphtha at 0.72 relative to water. The differential pressure level
measurement was calibrated with a density of 0.998. Consequently, the Fuel Gas Mix Drum differential pressure level measurement
never reached the Fuel Gas Mix Drum high-high level alarm point or indicated liquid was overflowing to the fuel gas piping.
61
The measured naphtha flow rate to the Coker Gas Plant exceeded 18,700 barrels per day after the naphtha flow
control valve to the Coker Gas Plant was opened to lower the liquid level in the drum. This flow rate into the
Fuel Gas Mix Drum was more than the 9,800 barrels per day estimated to be draining from the Fuel Gas Mix
Drum to the Flare Knockout Drum. a
The volume of the naphtha flow into the Fuel Gas Mix Drum exceeded the capacity out of the Fuel Gas Mix
Drum to the Flare Knockout Drum, a closed system. b The Fuel Gas Mix Drum two-inch drain to the Oily Water
Sewer provided an additional estimated draining capacity of 15,600 barrels a day.c,d,e
The CSB concludes that the Fuel Gas Mix Drum drain piping did not have enough capacity to drain naphtha
overflowing from the open Coker Gas Plant in a closed system to the Flare Knockout Drum.
The CSB concludes that limiting or stopping the flow of naphtha to the Coker Gas Plant would have been
required to prevent an overflow of the Fuel Gas Mix Drum since more naphtha could flow into the Fuel Gas Mix
Drum through the Coker Gas Plant bypass than could be removed to a closed system.
a
BP estimated the rate of naphtha being drained from a fully open two-inch valve to the flare at 9,792 barrels per day.
b
A closed system consists of piping and vessels connected to selected hydrocarbon drains for the containment, recovery, or safe disposal
of collected liquids, which would otherwise cause hazardous releases of hydrocarbon or toxic vapors such as hydrogen sulfide to the
atmosphere or to an oily water drainage system [57, p. 190].
c
BP estimated the rate of naphtha being drained from a fully open two-inch valve to the Oily Water Sewer at 15,624 barrels per day.
d
The bottom drain out of the Coker Gas Plant Absorber Stripper Tower was a ten-inch nozzle which reduced to a four-inch line to the
NHT Feed Drum.
e
After the September 20, 2022, incident, Ohio Refining Company LLC estimated the flow control valve to be able to be open enough to
flow 30,000 barrels a day.
62
3.3 Stop Work Authority

The Center for Chemical Process Safety (CCPS) notes that “Leaders should
make it clear that any employee can stop work or shut down the process if KEY LESSON
they perceive a potentially unsafe situation” [21, p. 87]. This is commonly
referred to as “Stop Work Authority.” a At the time of the incident, the BP Companies must ensure
Toledo Refinery had a procedure called “Handling Employee Health/Safety (through training, clearly
Concerns of Assigned Work,” which stated: written procedures, and
other means) that
All workers at BP Toledo Refinery have the right to a employees not only are
safe work environment. clearly empowered to
All BP refinery employees and all contractor employees exercise Stop Work
have the right and the responsibility to STOP any work Authority, but that
that may be UNSAFE. employees also clearly
understand they are
Additionally, some BP Toledo Refinery operating procedures stated (in their expected to do so.
introductions): “All qualified unit operators have the full authority to take However, companies
action (including shutting the unit down) when conditions are unsafe to should not rely on Stop
continue to operate.” b Work Authority
programs alone to
Some BP Toledo Refinery operations personnel told the CSB that they felt
prevent a catastrophic
they could invoke Stop Work Authority, and some stated that they had
process incident since
already done so in the past. On the morning of the incident, the NHT unit
they require humans to
emergency shutdown was initiated (Section 3.1.2). That same evening, as
take action to shut down
soon as liquid was known to be in the fuel gas piping and inside furnaces, the
a job or a process. Stop
night shift refinery coordinator initiated a shutdown of the refinery, although
Work Authority is not a
by this time, it was too late to prevent the consequences from the excess
substitute for effective
liquid that already had flowed into the Fuel Gas Mix Drum. The refinery
process safety
coordinator explained to CSB investigators:
management systems.
We obviously had liquid in the fuel gas. But at that point,
it was a very easy decision for me because reports were
coming that the furnaces were…were getting rich. Starting
with the crude furnace. And that’s when I was like, “Pull
the fuel gas.” We’re…just pull the fuel gas out of the
furnace. We’re going to lose [shut down] this refinery.
a
The CSB has addressed Stop Work Authority previously in its 2015 investigation of the Chevron Richmond Refinery Fire and its 2001
investigation of the Tosco Avon Refinery Petroleum Naphtha Fire. Additionally, as outlined in the CSB’s October 26, 2022, comments
submitted to the EPA on the EPA’s then-proposed revisions to the RMP rule: “The CSB has always stated that facilities must also have
effective measures in place for incident prevention that will foster a “culture of safety” wherein workers are encouraged and empowered
to advocate for their safety on the job. The CSB believes that any program that does not appropriately enable workers to feel free to
exercise stop work authority in necessary circumstances would allow risks to occur and accumulate.” See Notice of Proposed
Rulemaking Status Change Letter (csb.gov)
b
Operators must take the written qualification test to achieve qualification, and requalification training is provided at least every three
years through the administration of a field test to ensure the operator understands and adheres to the current operating procedures of the
process they are assigned.
63
Moreover, some operators had communicated concerns to refinery management about operations on the day of
the incident. For example, on the morning of the incident, before the NHT Preheat loss of containment, an
outside operator communicated several statements of serious concern on the radio, all within a two-minute
window, approximately 25 minutes before the ¾-inch bleed severed from the naphtha piping in NHT Preheat
(Section 3.1.2):
“I don’t know how much longer we can let this go…it’s going to tear something
up…Somebody needs to make a call out here ASAP.”
“We're going to hose some stuff up for real here if we let this go much longer.”
“I really need somebody to ok me to block that PSV in.”
“I don’t know how much longer this pipe can take this. It’s getting beat up really
bad.”
Although no operators appear to have explicitly invoked Stop Work Authority for the NHT Preheat leak the
morning of the incident, the radio traffic above indicates that operations personnel did communicate concerns
about continuing to operate the NHT unit consistent with the purpose of Stop Work Authority. Although an
emergency shutdown of the NHT unit was conducted, it occurred too late to prevent the loss of containment in
NHT Preheat and led to the subsequent cascading upsets. Additionally, as noted, although the night shift
refinery coordinator initiated a shutdown as soon as liquid was known to be in the fuel gas piping and inside
furnaces, by the time the night shift refinery coordinator acted, it was too late to prevent the consequences from
the excess liquid that already had flowed into the Fuel Gas Mix Drum.
If BP had implemented actions to stabilize operations, the Crude 1 unit could have already been in a safe state
by the time night shift operators arrived, and the Crude 1 Tower upset on the night shift could have been
avoided. If the Crude 1 Tower upset on night shift had been avoided, the excess liquid in the Crude 1 Overhead
Accumulator Drum, the Absorber Stripper Tower, and the Fuel Gas Mix Drum would have also been avoided,
and the fatal incident would not have occurred.
Critically, no action, following the Crude 1 Tower upset, was taken to prevent naphtha from overflowing into
the fuel gas piping. Instead, as one board operator told CSB investigators, it was viewed as just a “normal
upset”, despite the cascading process upsets and rapidly deteriorating conditions, and he thought at the time that
“we can get this.”
The board operator’s statement illustrates that key personnel at the BP Toledo Refinery did not understand the
dire nature of the events that were rapidly unfolding, and, as such, no one invoked Stop Work Authority in time
to prevent the occurrence of the catastrophic incident on September 20, 2022. In this instance, Stop Work
Authority would have been not only appropriate, but absolutely necessary in light of the unsafe conditions that
existed at the time and would continue to worsen thereafter.
While Stop Work Authority is an extremely important program that companies should have in place to stop an
unsafe event, it is important to recognize that it is fundamentally a “last resort” type human-initiated action,
prone to failure in correcting broader process-related hazards, as illustrated by the incident at the BP Toledo
Refinery. By design, Stop Work Authority is a decision process embedded into the chaos of a process safety
64
event, often in a stressful atmosphere in which an individual employee must assert a dissenting viewpoint
against a group. Therefore, reliable systems, including automated preventive overfill safeguards, must be
implemented and maintained. A robust risk-based process safety program must also be in place to identify and
prevent hazards before reaching the point of relying on Stop Work Authority to prevent a catastrophic process
event.
Regardless of whether the BP Toledo Refinery had adequate Stop Work Authority procedures in place that
might have provided a means for personnel to take action to prevent the situation from worsening, the fact is that
no effective steps were taken at any point to prevent the fatal incident.
65
4 Safety Issues
The following sections discuss the safety issues contributing to the incident, which include:
Liquid Overflow Prevention
Abnormal Situation Management
Alarm Flood
66
4.1 Liquid Overflow Prevention

Vessel overflow events can harm workers, communities, and the environment, damage equipment, and lead to
catastrophic events.
Many companies, including BP, use hazard analysis techniques, including process hazard analyses (PHAs) to:
identify hazards,
evaluate worst-case/high-severity scenarios, and
select necessary prevention and mitigation measures for the risks.
The hierarchy of controls a and layer of protection analysis (LOPA) b methodologies are used to determine the
adequacy of available safeguards and select prevention and mitigation strategies when the current safeguards
are deemed insufficient. Prevention and mitigation measures include but are not limited to instrumentation,
process controls, and safety instrumented systems (SISs).
Although the BP Toledo Refinery conducted PHAs to analyze liquid overflow events and LOPAs c to determine
the effectiveness of identified safeguards, d the refinery’s safeguards were not effective in preventing liquid
naphtha from overflowing the Coker Gas Plant Absorber Stripper Tower to the Fuel Gas Mix Drum and into the
fuel gas piping, as described in Section 3.1.8. In addition, the BP Toledo Refinery did not have effective
administrative controls such as written procedures and adequate training for operations personnel to recognize
and prevent liquid overflow events.
4.1.1 Ineffective Safeguards

Prior to the incident, the BP Toledo Refinery had identified the liquid overflow of process vessels, including
liquid overflow from the Fuel Gas Mix Drum to the fuel gas system, as a process deviation that could lead to a
a
The National Institute of Occupational Safety and Health (NIOSH) defines the hierarchy of controls as “a way of determining which
actions will best control exposures [to hazards in the workplace]” [47].
b
A layer of protection analysis (LOPA) is “an approach that analyzes one incident scenario (cause-consequence pair) at a time, using
predefined values for the initiating event frequency, independent protection layer failure probabilities, and consequence severity, in
order to compare a scenario risk estimate to risk criteria for determining where additional risk reduction or more detailed analysis is
needed. Scenarios are identified elsewhere, typically using a scenario-based hazard evaluation procedure such as a HAZOP Study” [68].
LOPAs evaluate risks and the sufficiency or effectiveness of protection layers in reducing the frequency and/or consequence severity of
hazardous events.
c
The BP Toledo Refinery completed its Coker Gas Plant LOPA of Record Report in February 2020. “Applicable Generic Suite LOPAs
were reviewed by the [BP Toledo Refinery] team and applied, localized to reflect actual site conditions.”
d
A safeguard is “any device, system, or action that interrupts the chain of events following an initiating event or that mitigates the
consequences” [54].
67
safety hazard in its PHAs. The refinery conducted multiple PHAs a on its Crude 1 unit and Coker Gas Plant
between 2014 and 2020. b
BP Guidance
The BP Safety and Operational Risk Group provided written liquid overfill hazard management guidance in its
publication, Overfill of Process Vessels and Columns Risk Assessment and Mitigations for Downstream. The
guidance lists acceptable Independent Protection Layers (IPLs) that can be used for liquid overfill hazards.
Among these are:
1. An independent high level alarm as part of:
a. Operator response to a safety related alarm, if sufficient operator response time would be
available in the scenario;
b. A basic process control system IPL “that stops or redirects flow into the vessel or that provides
a secondary outflow path can be utilized with the independent high level alarm;”
2. A pressure relief system (PSV and associated piping, structure, etc.) designed for liquid overfill and that
discharges to a safe location;
3. A safety instrumented function (SIF) c that shuts down flow, “that stops or re-directs all feeds away from
the vessel, or that provides a secondary path for flow out of the vessel/column.”
For the scenario of Absorber Stripper Tower overflow to the Fuel Gas Mix Drum, the BP Toledo Refinery
implemented high level alarms coupled with operator response and emergency pressure-relief system valves as
safeguards. The BP Toledo Refinery, however, did not implement automatic or engineering controls to stop flow
into the Absorber Stripper Tower or Fuel Gas Mix Drum.
4.1.1.1 PREVENTIVE VERSUS MITIGATIVE SAFEGUARDS

Safeguards are used to reduce the risk from potential hazardous scenarios, either by preventing the initiating event
from occurring or mitigating the consequences of the process deviation. Preventive safeguards keep loss events,
a
The BP Toledo Refinery generally used the Hazard and Operability Study (HAZOP) technique to conduct its PHAs. HAZOP is a
“systemic qualitative technique that identifies process hazards and potential operating problems using a series of guidewords to study
process deviations. A HAZOP is used to question every part of a process to discover what deviations from the intention of the design
can occur and what their causes and consequences may be” [71].
b
The BP Toledo Refinery conducted PHAs on its Crude 1 unit in 2014 and 2019. The BP Toledo Refinery conducted PHAs on the Coker
Gas Plant in 2015, 2018, and 2020.
c
A safety instrumented function (SIF) is a “system composed of servers, logic servers, and final control elements for the purpose of
taking the process to a safe state when predetermined conditions are violated” [70].
68
such as fires, explosions, and toxic releases, from happening “when an initiating
event causes a process to deviate from normal operation.” On the other hand, KEY LESSON
mitigative safeguards “can reduce the impact of loss events,” even though the
event itself still may occur. PHAs are performed to determine the preventive and PHA scenarios should
mitigative safeguards needed for the identified hazardous scenarios and consider both
consequences. While safeguards are used to prevent and mitigate loss events, preventive and
they may not necessarily prevent the initiating event, as was the case at the BP mitigative safeguards
Toledo Refinery for the overflow of vessels. and not unrealistically
rely on human
The safeguards identified in the BP Toledo Refinery’s PHAs for Absorber intervention.
Stripper Tower overflow to the Fuel Gas Mix Drum were not effective
safeguards to prevent overflow. One type of identified safeguard included a
furnace SIS that aligns with the American Petroleum Institute Recommended
Practice 556 (API RP 556) Instrumentation, Control, and Protective Systems for
Gas Fired Heaters guidance. SISs take “automated action to keep a plant in a
safe state, or to put it into a safe state when abnormal conditions are present”
[22]. a However, the SISs identified in the 2020 Coker Gas Plant PHA to prevent
the consequences of liquid overflow b into furnaces were all downstream of the
Fuel Gas Mix Drum and could not take action until liquid was already inside the
fuel gas piping downstream of the Fuel Gas Mix Drum, as shown in Figure 30.
No SIS prevented the Absorber Stripper Tower overflow to the Fuel Gas Mix
Drum by stopping naphtha flow into the tower once a high level of liquid in the
tower was detected.
Emergency pressure-relief valves are a second type of safeguard the PHA identified. Emergency pressure-relief
valves are safeguards designed to protect process equipment during a process overpressure. The identified
emergency pressure-relief valve safeguards at the BP Toledo Refinery were intended to protect the Absorber
Stripper Tower against potential overpressure. These emergency pressure-relief valves were installed on the
overhead line exiting the top of the Absorber Stripper Tower. As such, the Absorber Stripper Tower emergency
pressure-relief valves were not designed to prevent the process overflow into the bypass line, just the hazard of
potential overpressure arising from an overflow scenario.
A third type of safeguard identified during the PHAs relied on human intervention response to address alarms
from the Absorber Stripper Tower level and differential pressure instrumentation to mitigate the potential for
naphtha overflow of the Absorber Stripper Tower into the Fuel Gas Mix Drum and furnaces. As discussed
further in Section 4.3, human intervention can be unreliable depending on the workload of the operator. As
such, in certain situations, human intervention as a safeguard can prove to be ineffective.
a
The International Society of Automation (ISA) Standard 61511 Functional Safety – Safety Instrumented Systems for the Process
Industry Sector contains “requirements for the specification, design, installation, operation and maintenance of a safety instrumented
system (SIS)” to achieve and maintain a safe state of the process [58, p. 9]. Its SIS Design and Engineering section states, “Where the
SIS operator interface is via the BPCS [Basic Process Control System] operator interface, account shall be taken of credible failures that
may occur in the BPCS operator interface” [58, p. 59].
b
The consequence of liquid overfill in furnaces is further discussed in Section 4.1.4.
69
Figure 30 illustrates the Absorber Stripper Tower safeguards identified in the 2020 Coker Gas Plant PHA in
red, along with the unit or safeguard changes noted above.
Figure 30: Absorber Stripper Tower safeguards identified in the 2020 Coker Gas Plant PHA, shown in red.
(Credit: CSB)
The CSB concludes that the Absorber Stripper Tower emergency pressure-relief valves and the refinery
furnaces’ safety instrumented systems would not prevent a vessel overflow event. Instead, they just protected
equipment after an overflow has already occurred.
The CSB also concludes that had the BP Toledo Refinery recognized the likelihood of liquid overflow to the
Fuel Gas Mix Drum, it could have implemented more effective preventive safeguards, such as a high level
interlock to close the naphtha feed valve to the Absorber Stripper Tower. a Such an interlock would have
automatically stopped the identified liquid overflow events instead of relying on alarms that require human
intervention, emergency pressure-relief valves, and downstream safety instrumented systems.
a
An interlock is a protective response that is initiated by an out-of-limit process condition [67].
70
4.1.1.2 MISSED OPPORTUNITIES TO PREVENT OVERFLOW

The BP Toledo Refinery completed at least two plant modifications and investigated at least one incident of
liquid naphtha overflow into the Fuel Gas Mix Drum, prior to the September 2022 incident, in which the
refinery could have identified high liquid level scenarios and implemented effective safeguards to prevent
overflow of naphtha into the fuel gas system.
Initial Coker Gas Plant Project PHAs

Before starting up the new Coker Gas Plant in 2018, the BP Toledo Refinery KEY LESSON
performed a series of PHAs on the new unit. These PHAs identified scenarios in
PHAs should evaluate
which the Absorber Stripper Tower reached high level from a variety of causes,
overfill hazards and
and they listed the consequences as “potential overpressure, damage, leak,
consider scenarios in
hydrogen sulfide (H2S) exposure, fire, injury, fatalities.” However, these PHAs
which a vessel may not
did not identify any potential scenario in which liquid naphtha in the Absorber
overfill to the top but
Stripper Tower would overflow into the Fuel Gas Mix Drum. Since this scenario
may instead overflow or
was not considered, the 2016 design PHA team may have actually deleted an
backflow through other
identified preventive safeguard before it was ever implemented—an interlock
piping connections.
that would have closed the naphtha flow control valve to the Coker Gas Plant
when the liquid level in the Absorber Stripper Tower was high. A comment
included in Coker Gas Plant project documentation stated:
Per Generic LOPA, only PSV needed to mitigate scenario.
BPCS [Basic Process Control System] closure of [naphtha
flow control valve to the Coker Gas Plant] is good design,
not IPL. No SIF [safety instrumented function] needed
here. a
Operational Changes to Coker Gas Plant Bypass

After the initial Coker Gas Plant startup and commissioning, the BP Toledo Refinery found that the Coker Gas
Plant Polishing Amine Contactor experienced upsets when treating high flow rates of coker wet gas. To resolve
this and prevent Coker Gas Plant upsets, the BP Toledo Refinery began to operate the Coker Gas Plant with the
bypass partially open, to allow a portion of the coker wet gas flow to bypass the Coker Gas Plant to achieve stable
operation. However, operating the Coker Gas Plant with the bypass continually open created two flow paths, and
the safeguards identified in the Coker Gas Plant PHAs were not effective in preventing overflow in this new
operating configuration.
In 2019, the BP Toledo Refinery replaced a manual gate valve installed in the Coker Gas Plant bypass piping
with an automatic valve that would control the Polishing Amine Contactor differential pressure by bypassing
gas around the unit. This new control valve fundamentally changed how the Coker Gas Plant had previously
operated. Previously, with the manual valve, only outside operators (and not board operators) could respond,
a
A Basic Process Control System (BPCS) is a “system that responds to input signals from the process and its associated equipment, other
programmable systems, and/or from an operator, and generates output signals causing the process and its associated equipment to
operate in the desired manner and within normal production limits” [69].
71
and they had to manually open or close the bypass valve during a Coker Gas Plant upset. The replacement valve
enabled automatic control and allowed for a quicker response by board operators.
The MOC documentation for implementing the new automatic control valve identified coker wet gas as the
normal process material and stated that an “Evaluation Team” had confirmed that neither “flows” nor
“backflow” would be a concern or problem with the change to the automatic valve. a It did not specify or
reference any concerns with liquid naphtha overflow through piping connections or valve failure. However, the
BP Toledo Refinery’s 2019 Crude 1 PHA did not evaluate potential liquid overflow scenarios into the Fuel Gas
Mix Drum b related to the change. Additionally, during the CSB’s investigation, the CSB was provided with
documentation that included a “PHA Team Review Comment,” stating, “No new or increase in existing hazards
resulted from this change.” During the evaluation of the change to the automatic control valve, the BP Toledo
Refinery missed an opportunity to thoroughly analyze and address the backflow scenario through the bypass
line.
2019 Absorber Stripper Tower Incident

In November 2019, a Fuel Gas Mix Drum high level incident occurred following a refinery-wide loss of steam
header pressure c (See Section 4.4.1 below). In this incident, a board operator opened the naphtha flow control
valve to the Coker Gas Plant while responding to high level in the Crude 1 Overhead Accumulator Drum.
Liquid naphtha overflowed into the Coker Gas Plant bypass piping, ultimately reaching the Fuel Gas Mix Drum.
In contrast to the September 2022 incident, operators troubleshooting the 2019 overflow at the time closed the
naphtha flow control valve to the Coker Gas Plant before liquid naphtha reached the downstream furnaces and
boilers. After the 2019 incident, the BP Toledo Refinery did not identify or implement any new engineering
controls as preventive safeguards for a process upset in which naphtha overflowed into the Fuel Gas Mix Drum.
The CSB concludes that had the BP Toledo Refinery implemented additional preventive safeguards to prevent
liquid overflow from the Coker Gas Plant to the fuel gas system, the incident in September 2022 may not have
happened.
The CSB recommends that Ohio Refining Company LLC revise the safeguards used in the refinery’s process
hazard analyses high level and overflow scenarios. At a minimum, establish effective preventive safeguards that
use engineered controls to prevent liquid overfill and do not rely solely on human intervention. (See
Recommendation 2022-01-I-OH-R1).
a
The MOC document stated, “[t]he process is not changing.”
b
The Fuel Gas Mix Drum is included in the Crude 1 PHA scope.
c
The 2019 Fuel Gas Mix Drum incident occurred after the 2019 Crude 1 PHA had been completed.
72
4.1.2 Reliance on Human Intervention

The Challenge with Relying on Human Intervention
People may make poor choices that can lead to accidents in part because of flaws in human decision-making.
Instituting good process and job design can help reduce the consequences of such flaws and “can eliminate the
root cause of accidents and occupational exposure to hazards in many different ways” [23, p. 709]. As discussed
in the Handbook of Human Factors and Ergonomics, “models of human information processing” suggest that
time pressure or lack of awareness or knowledge, among other things, can cause errors.a Abnormal situations
can also contribute to human error, as further discussed in Section 4.2.
At the time of the 2022 incident at the BP Toledo Refinery, human action was required to drain liquid from the
Fuel Gas Mix Drum, regardless of the liquid’s destination, and human errors contributed to the magnitude and
severity of the incident. In the book, An Introduction to System Safety Engineering (2023), Nancy Leveson
explains:
Different problems occur when humans are part of the system. One common
complication is that assumptions may be made that the [operators] will not only
recognize the failure (or hazard) but will also respond appropriately. Ironically
accidents are often blamed on inadequate […] operator behavior while at the
same time assuming they behave correctly in the hazard or risk assessment.
Clearly, there are many cases where this assumption that the human operator will
“save the day” does not hold. The mental model of the system operator plays an
important role in accidents [24, p. 410].
Leading up to the incident, the BP Toledo Refinery relied on safeguards that required operator intervention to
respond to process upsets and deviations. In some cases, the refinery depended on the board operator to
acknowledge an alarm and radio outside operators to verify and address the issue, as was done, for the high level
in the Fuel Gas Mix Drum during the 2022 incident. Draining the Fuel Gas Mix Drum and attempting to address
the high level through manual draining possibly led the refinery employees to release liquid from the Fuel Gas
Mix Drum to the ground, which created a vapor cloud that ignited and caused the fire that resulted in the BP
employees’ deaths.
Operators Manually Draining Vessels

On the evening of the incident at 6:10 p.m., less than 30 minutes after the board operator opened the naphtha
flow control valve to the Coker Gas Plant (Section 2.2), liquid naphtha filled the Fuel Gas Mix Drum and
activated a high level alarm on the drum. The board operator noticed a high-level alarm and radioed the outside
operators to check the level in the drum. The outside operators arrived at the Fuel Gas Mix Drum, checked the
sight glass, and began draining the drum manually. Draining vessels was a common task for outside operators.
An operator explained to CSB investigators that before attempting to correct high level in the Fuel Gas Mix
Drum,
a
Errors can also be caused by lapses in attention, distractions, forgetting, or information overload and can be shown to differ depending
on the level of task performance [23, p. 708].
73
[…] We [outside operators] got a call from the inside board to check the level in
the Fuel Gas Mix Drum. So, we immediately responded. Upon arriving at the
drum, [ …] the sight glass was completely clear with no definitive line or level
in it. So, my first response was to…drain or to check the level in the sweet gas
knockout, and then I opened the flare valve on the fuel gas mix drum 100 percent.
And then, I opened the drain valve on the Fuel Gas Mix Drum approximately
maybe two to three turns. And my reasoning for opening it to the sewer a couple
turns was due to the sight glass being clear. We were checking to see if there was,
in fact, a liquid in there or if it was lying to us, you know if we could get bubbles
in there.
To address high level in the Coker Gas Plant Sweet Gas Knockout Drum, an operator explained to CSB
investigators,
When I [an outside operator] got to the sweet knockout pot, it was full. So, I
grabbed the dead man valve and pulled it and held it and it took a while for the
level to come down in the sight glass and to drop out of it. But then, once I got it
to where the level was out of the sight glass, I released the dead man valve, and
it just started filling right back up.a
These tasks were not new or unfamiliar to the outside operators. Board operators at the BP Toledo Refinery had
to call outside operators on the radio in order to address overflow events, such as high levels in the Fuel Gas
Mix Drum and other Coker Gas Plant downstream equipment. A board operator described his normal interaction
with the outside operators whenever he noticed a high level on the DCS screen, as follows:
[…] We just call over the radio [to the outside operators], Hey, I have a high
level. Can you help me? … and they’ll [the outside operators] be like, okay, let
me go look. And then, you know, you’ll see the level go away, and you’ll know
they helped you.
The CSB concludes that manually draining vessels was a common task for outside operators at the BP Toledo
Refinery.
a
A dead man valve, also known as a spring closing lever valve, is a manual valve that a human has to hold open by use of the lever
handle, which will automatically close when the lever is released. Typically used to prevent leaving a valve unattended, such as during
manual draining, a dead man valve can be defeated by securing the handle in the open position [52, p. 25]. (See A.2.9.4.30).
74
2019 Crude 1 PHA

In 2019 the BP Toledo Refinery performed a cyclic PHA of the Crude 1 unit.a
The PHA methodology used by the BP Toledo Refinery was a Hazard and
Operability Study (HAZOP). HAZOPs may not evaluate the hazards of job KEY LESSON
tasks to determine safe work practices, such as how to drain the Fuel Gas Mix
Companies should
Drum. Consequently, the PHA did not identify the hazards associated with
evaluate their PHAs for
manually draining the Fuel Gas Mix Drum to the Oily Water Sewer or the Flare
opportunities to
Knockout Drum, whether the liquid was hydrocarbon or an amine-water
implement additional
mixture.
safeguards to prevent
The PHA did include a high level deviation caused by failure to drain the Fuel initiating events that
Gas Mix Drum while the drum contained non-flammable amine-water solution reduce the reliance on
and determined that it was “reasonable to expect that the situation will be human intervention.
detected before the hazard is manifested.” b However, the 2019 PHA did not
consider potential hazards that could exist if the drum contained flammable
liquid (such as naphtha) and did not include any details or remarks regarding
the refinery’s policy or expectation of how or to where operators should drain
the Fuel Gas Mix Drum or whether this task was even safe to perform.
Procedures
The BP Toledo Refinery had procedures for preparing equipment for maintenance and returning it to service. In
addition, there was a procedure for draining process equipment and lines, which stated: “Many materials are
reportable if released or drained. Minimize all material drained to the sewer or purged to atmosphere and never
drain material to the ground.” c These procedures instructed outside operators to use lock-out and tag-out
procedures to properly isolate equipment before draining it. However, the BP Toledo Refinery did not have any
procedures, written instructions, or documented corrective actions for board operators or outside operators to
respond to or troubleshoot a high liquid level in the Fuel Gas Mix Drum, during either normal operations or
process upsets, if liquid entered the drum. Had there been training on how to troubleshoot and address high level
in the Fuel Gas Mix Drum, such as identifying the source of the liquid and stopping the flow, board operators
may have taken different actions instead of calling the outside operators to respond to a high-level alarm in the
Fuel Gas Mix Drum. Outside operators did not have adequate procedures for how to address a high liquid level
in the Fuel Gas Mix Drum. High level in the Fuel Gas Mix Drum, especially above the top of the sight glass, is
not normal operation.
The BP Toledo Refinery could have provided operators with instructions, such as instructing board operators to
confirm the material in the vessel based on the differential pressure level indication and the guided wave radar
level indication and to determine the source of the liquid causing the high liquid level before asking outside
operators to check the level in the pressurized vessel. d Once the board operators identified the source of liquid
and stopped the liquid from flowing to the Fuel Gas Mix Drum, outside operators could have then been
a
As an OSHA PSM-covered process, the Crude 1 unit was required to undergo a PHA every five years.
b
The Crude 1 PHA also evaluated the consequences of “failure to drain liquid to amine sump at EPA Sweet Gas Knockout drum.”
c
See footnote a on page 34 for additional details regarding this procedure.
d
See 4.1.3 Post-Incident Actions for the deviation alarm between the level measurements.
75
instructed to drain to either the Flare Knockout Drum or the Oily Water Sewer. If instructions such as these
existed at the BP Toledo Refinery on September 20, 2022, board operators and outside operators may not have
made their own decisions in real time about how to drain the high level from the Fuel Gas Mix Drum.
The 2023 CCPS Monograph, Human Factors Primer for Front Line Leaders, provides guidance to help
managers and supervisors enable workers to make good decisions and perform work successfully when draining
liquid from a pressurized vessel as shown in Figure 31 below [25, p. 2].
Figure 31: Draining unwanted liquid from a pressurized system. (Credit: CCPS)
In Figure 31 above, the CCPS states that the task of draining a pressurized vessel should be adequately risk-
assessed by considering the frequency of the task, the complexity of the system including interconnections, and
the levels of required task verification. Had the BP Toledo Refinery conducted a similar assessment of the task
of manually draining the Fuel Gas Mix Drum, it may have identified the potential for naphtha filling the drum
76
and provided written procedures for determining the cause of the liquid level and how to drain it. While the
CCPS Monograph, Human Factors for Frontline Leaders, was published in 2023, after the September 20, 2022,
incident, the concept of conducting risk assessments on operational tasks is not new, and guidance for
completing a Safety Critical Task Analysis is described in the 2020 Energy Institute publication Guidance on
human factors safety critical task analysis (second edition) [26].
The CSB concludes that had the BP Toledo Refinery 1) conducted a thorough risk assessment of the operational
task of draining or addressing high level in the Fuel Gas Mix Drum, 2) provided its operators with the necessary
written instructions and consistent training, and 3) ensured the competency of operations personnel to perform
the task safely, BP employees may have made different decisions on September 20, 2022.
4.1.3 Post-Incident Actions

The Ohio Refining Company LLC told the CSB that post-incident the company “reviewed scenarios to ensure
equipment is adequately sized and designed to handle upset conditions” and made “equipment modifications and
updates to provide inherently safer designs.” Below are some of the changes that have been implemented at the
refinery since the September 20, 2022, incident:
1. An automated system a for draining the Fuel Gas Mix Drum to the Flare Knockout Drum was installed.
2. A spectacle blind was added to the branch to the Oily Water Sewer to prevent draining to the sewer
during normal operations.
3. A level deviation alarm b was added to the Fuel Gas Mix Drum to provide a warning that the liquid level
is likely to be hydrocarbon.
4. New Fuel Gas Mix Drum procedures were established, including draining procedures.
5. Installed a control-based maximum stop of 41 percent open, which corresponds to a flow rate of
approximately 10,000 barrels per day c, on the naphtha flow control valve to the Coker Gas Plant to
prevent Absorber Stripper Tower overfill.
The modified Fuel Gas Mix Drum is shown below in Figure 32.
a
An automated full port ball valve activates when the Fuel Gas Mix Drum reaches five percent level indication. The valve remains open
based on a timer function that closes the valve once the timer expires. In the event that the drum is not draining below five percent, the
valve will remain open and not close. Operations has the ability to open or close the valve, but this ability will be interlocked out if the
situation stated above is occurring. Operations can bypass this automatic draining by activating a bypass controller.
b
The differential pressure (dP) level instrument is calibrated to read the level based on the specific gravity of an amine-water solution.
The guided wave radar (GWR) level instrument will read the liquid level regardless of the specific gravity or type of material. A liquid
level of hydrocarbon will result in the level indications not matching. When the corresponding alarm to a level deviation of greater than
10 percent activates there is a strong indication of hydrocarbon being present in the Fuel Gas Mix Drum.
c
The flow control valve is estimated to be able to open enough to flow 30,000 barrels a day.
77
Figure 32. Toledo Refinery Fuel Gas Mix Drum with modifications made after the September 20, 2022,
incident. (Credit: CSB)
4.1.4 Industry Guidance

The BP Toledo Refinery had protective systems for its downstream furnaces in accordance with API
Recommended Practice 556 (API RP 556) Instrumentation, Control, and Protective Systems for Gas Fired
Heaters. However, the BP Toledo Refinery did not have sufficient preventive safeguards to prevent overflowing
the Fuel Gas Mix Drum into the downstream fuel gas system.
In addition to API RP 556, the API has published other industry documents that provide general instrumentation
guidance and specific design considerations to mitigate hazards arising from overfill events for certain systems,
including the following:
1. API Standard 521 Seventh Edition (API 521) Pressure-relieving and Depressuring Systems in Section
4.4.7, a
2. API Standard 2350 Fifth Edition (API 2350) Overfill Prevention for Atmospheric Storage Tanks in
Petroleum Facilities, b
a
The API 521 standard provides mitigation guidance for liquid overfilling scenarios related to pressure-relieving and vapor depressuring
systems [43]. Although the standard is intended for designing overpressure protection systems, it includes general design
considerations and mitigation measures such as 1) installing a safety instrumented system (SIS) to prevent liquid overfill, 2) operator
training and procedures, and 3) level instrumentation and alarms [43, pp. 22 - 23].
b
API Standard 2350 Fifth Edition, September 2020, Errata 1, April 2021
78
3. API Recommended Practice 14C Eighth Edition (API RP 14C) Recommended Practice for Analysis,
Design, Installation, and Testing of Basic Surface Safety Systems for Offshore Production Platforms in
Section 6.2.2.4, and
4. API Recommended Practice 551 Second Edition (API RP 551) Process Measurement in Section 7.
However, the API does not have guidelines for protective systems to prevent Fuel Gas Mix Drum liquid overfill
that can directly impact downstream fuel gas systems.
The CSB reviewed API RP 556 a as part of this investigation to determine its effectiveness in providing guidance
for process hazards from liquid overflow of fuel gas mix drums to downstream gas fired heaters, causing a
potential flameout condition. A flameout condition is present when the flame on a gas fired heater “burner goes
out while fuel is still being charged to the firebox” [27]. A flameout can be a dangerous situation, and “when not
noticed and left unattended, flameout [may] result in the explosion of the fired heater” [27].
API RP 556 addresses “primary measuring and actuating instruments, controls, alarms, and protective systems
as they apply to fired heaters.” In the General section, API RP 556 states,
Instrumentation and control applications incorporate systems and devices to
satisfy equipment specific requirements […] [including] safety, process control,
data collection, […].
The recommended practice also lists design considerations and protective functions for process hazards, such as
the accumulation of combustibles within the firebox potentially due to a loss of flame, that could lead to “an
explosion which may result in the partial or total destruction of the fired heater and which may be hazardous to
personnel in the operating area” [28, p. 29]. It explains,
Process deviations that precede flameout are typically associated with
operational limits. Approaching or exceeding operational limits can lead to rapid
accumulation of combustibles within the firebox. For example, loss of flame may
result in the rapid accumulation of combustibles to an unacceptable hazard level
in less than 10 seconds. Process deviations that precede flame out include […]
slug of liquid in fuel gas system that causes loss of flame [28, pp. 29 - 30].
[…]
Alarms may be set to alert operators of abnormal process conditions that are
approaching operational limits which may lead to flameout and the rapid
accumulation of combustibles within the firebox. The alarms may be triggered
by […] high liquid level in an upstream fuel gas drum [28, p. 31].
and
a
API RP 556 provides guidance for instrumentation and protective systems for gas-fired heaters in petroleum production, refineries, and
petrochemical and chemical plants [28, p. 1].
79
Once a rapid accumulation event is initiated, it may be challenging to achieve

safe state within the process safety time [28, p. 32]. a
However, despite this level of detail relating to flameout-related concerns in gas-fired heaters, API RP 556 does
not provide any guidance for instrumentation, preventive safeguards, and recommendations on preventing liquid
from overfilling a fuel gas mix drum into the fuel gas system, which could ultimately result in an accumulation
of combustibles.
The CSB concludes that although the API Recommended Practice 556 Instrumentation, Control, and Protective
Systems for Gas Fired Heaters provides industry guidance for alarms and protective functions to address
process hazards associated with the accumulation of combustibles in gas fired heaters, API RP 556 lacks
guidance to implement preventive safeguards for liquid overflow from a fuel gas mix drum which may lead to a
flameout and rapid accumulation of combustibles in gas fired heaters.
The CSB concludes that had industry guidance for preventive safeguards, such as safety instrumented systems
and controls, been available to prevent liquid overflow from the Fuel Gas Mix Drum, and had the BP Toledo
Refinery incorporated such guidance, the BP Toledo Refinery could have eliminated reliance on human
intervention to drain liquid from the Fuel Gas Mix Drum.
The CSB recommends that the American Petroleum Institute develop a new publication or revise an existing
publication, such as API Recommended Practice 556 Instrumentation, Control, and Protective Systems for Gas
Fired Heaters, to incorporate the process hazards associated with Fuel Gas Mix Drum overflow. The publication
should include the following at a minimum:
Drums. (See Recommendation 2022-01-I-OH-5).
a
Process safety time is the time interval between the initialing event leading to an unacceptable process deviation and the hazardous
event [28, p. 26].
80
4.2 Abnormal Situation Management

The Abnormal Situation Management® Consortium (ASM Consortium) is a group of companies and universities
“that have jointly invested in research and development to create knowledge, tools, and products designed to
prevent, detect, and mitigate abnormal situations that affect process safety in the control operations
environment” [29, p. iv]. a BP has been a member of the ASM Consortium since 1994.
In its guideline publication, Effective Operations Practices (2019), the ASM Consortium defines abnormal
situations as “undesired plant disturbances or incidents with which the control system is not able to cope,
requiring a human to intervene to supplement the actions of the control system” [29, p. iv]. Accurately
identifying when abnormal situations are occurring and appropriately responding to them are key to ensuring the
abnormal situations are mitigated and not exacerbated. In its book, Guidelines for Managing Abnormal
Situations (2023) the CCPS states:
Process operation during abnormal situations can create a high-pressure

environment for the operators. Efficient management and the correct handling of
the situation are key in preventing its escalation into a more serious incident.
Sudden, potentially dangerous situations can affect human performance (the
“startle” factor), leading to a “fight or flight” response, that can lead to
inappropriate action being taken [1, pp. 87-88].
In the 24 hours leading up to the incident, the BP Toledo Refinery experienced a large number of abnormal
situations across several units, eventually leading to the Fuel Gas Mix Drum overfilling, which in turn resulted
in two refinery employees releasing flammable liquid from the Fuel Gas Mix Drum to the ground, ultimately
cascading to the vapor cloud and fatal fire at 6:46 p.m. on September 20, 2022. As discussed above in Section 2
and Section 3, among these abnormal situations were:
1. Water accumulation in the Crude 1 Overhead Accumulator Drum.
2. Coker Gas Plant Foul Condensate Draw Off Drum overfilling into the Absorber Stripper Tower, which
led to tower level control instability.
3. Emergency pressure-relief valves opened in NHT Preheat, one of which chattered, causing a loss of
containment.
4. The loss of containment in the morning led to an NHT unit shutdown.
5. Operating Crude 1 while the NHT unit was shut down and Coker Gas Plant was bypassed left the
Crude 1 Tower with abnormally limited destination options and flow capability available for the
overhead naphtha stream.
6. Multiple instabilities in Crude 1 Tower as a result of the NHT unit emergency shutdown, such as loss
of cooling (pumparounds) and overpressure, including Crude 1 Tower emergency pressure-relief valves
opening as late as approximately 4:17 p.m. that day.
a
ASM and Abnormal Situation Management are U.S. registered trademarks of Honeywell International, Inc.
81
7. Rapidly eliminating the light crude feed to the Crude 1 Tower caused the final Crude 1 Tower upset, as
described in Section 3.1.6. This upset resulted in high vapor flow up the Crude 1 Tower, which caused
high level in the Crude 1 Overhead Accumulator Drum.
8. A board operator opened the naphtha flow control valve to the Coker Gas Plant in response to the high
level in the Crude 1 Overhead Accumulator Drum, even though the Coker Gas Plant was bypassed at
the time.
9. The Coker Gas Plant overflowed liquid naphtha through the Coker Gas Plant bypass line to the Fuel
Gas Mix Drum. This was abnormal both because liquid in the Fuel Gas Mix Drum was unusual and
because previously when liquid had gotten into the Fuel Gas Mix Drum, it was typically a
nonflammable amine-water solution, with a significantly different specific gravity compared with
naphtha.
10. The Fuel Gas Mix Drum overflowed liquid naphtha to furnaces, causing excessive smoke to exit
multiple furnace and boiler stacks and shutting them down.
11. The two BP Toledo Refinery employees opened valves on the pressurized Fuel Gas Mix Drum to
release the drum’s contents to the ground.
Given the interconnectivity of equipment in the refinery, other, additional abnormal situations likely occurred
elsewhere in the refinery, some of which required site personnel to manage. Thus, the abnormal situation
management workload was likely unusually high that day.
As described below, the abnormal situations began as relatively minor issues or process upsets, but ultimately
progressed to two significant liquid naphtha releases and a fatal fire. So many abnormal situations occurred
during a single shift that day that a shift operator with over 18 years of experience told the CSB:
Even before the explosion when we lost our guys, […] That was the worst day
of my life. Inside or outside, anywhere […] it was bad.”
4.2.1 BP Toledo Refinery’s Abnormal Situation Management

At the time of the incident, the BP Toledo Refinery had an Abnormal Situation Management (ASM) policy,
created in 2015. According to the policy, its purpose was to describe “the process for managing abnormal
operations” and provide “a template for assessing, mitigating potential risks, and determining the overall risk
level after safeguards are put in place.”
The refinery’s ASM policy indicated that for certain conditions or situations, an ASM form was to be completed
and that “[o]nce documentation is complete, personnel can continue operation in a mode which was not
anticipated.” The form included prompts for abnormal situation description, analysis, and risk ranking, and
required varying levels of approval signatures based on risk ranking.
However, the BP Toledo Refinery ASM policy narrowly defined abnormal operations as the following
situations:
Losses of instruments that impact a safety system;

82
Losses of field actuators (control valves, dampers, etc.) that impact a

safety system;
“Abnormal” line-ups in the [tankage] area that are not on the “normal
line-up” list and not covered in a procedure;
Bypassing equipment that would normally be in service, without an

associated procedure;
Changes to the normal operation of refinery wide systems; a
Continuing operation with a valve leaking through when the valve is part
of a safety system; and
Bypassing Independent Protection Layers (IPLs).
In effect, the BP Toledo Refinery’s ASM policy was more of an emergency MOC procedure than instructions
for truly handling abnormal situations as the ASM® Consortium defines them.b This is shown in the BP Toledo
Refinery ASM policy itself, in Figure 33 below, where it is indicated as a type of MOC.
Figure 33: Excerpt of the BP Toledo Refinery ASM procedure, indicating its relation to MOC. (Credit: BP)
Because the refinery had a procedure to bypass the Coker Gas Plant and the NHT unit emergency shutdown
procedure included switching the Crude 1 Tower overhead flow to Light Virgin Naphtha, these situations did
not constitute “abnormal situations” by the policy’s definition. Moreover, other abnormal situations that
occurred that day, such as those listed above, were not recognized as such by refinery personnel during the day
of the incident.
a
“Refinery wide systems” refer to systems that were not contained in a single processing unit, but connected to multiple processes across
the refinery, such as the fuel gas system, steam systems, or nitrogen supply system.
b
The ASM Consortium definition of abnormal situations, as described above in Section 4.2, is “undesired plant disturbances or incidents
with which the control system is not able to cope, requiring a human to intervene to supplement the actions of the control system” [29,
p. iv].
83
The CSB concludes that the BP Toledo Refinery Abnormal Situation Management policy was not effective for
proactive recognition of abnormal situations. The policy narrowly defined abnormal situations such that process
disturbances that occurred before the incident did not fit the policy’s definition, even though the basic process
control system was unable to cope with these situations.
On the day of the incident, the Crude 1 Tower and downstream equipment experienced several abnormal
situations with which the basic process control system was unable to cope. For example:
The Crude 1 Tower overhead became overpressured at least 11 times, opening at least one of the five
emergency pressure-relief valves each time (See Section 3.1.5 above). Additionally, the Crude 1 Tower
overhead pressure was over the high alarm point continuously for roughly eight hours, from
approximately 10:00 a.m. to 6:00 p.m., with alarms sounding throughout this time. This indicated that
the Crude 1 Tower pressure was unstable for several hours that day.
The Crude 1 Tower overhead naphtha flow rate to Light Virgin Naphtha was at maximum for over an
hour that afternoon on day shift (See Section 3.1.4 above), while the Crude 1 Overhead Accumulator
Drum level increased. This indicated that excess overhead flow could not flow through the Light Virgin
Naphtha control valve fast enough to prevent overfilling the Crude 1 Overhead Accumulator Drum.
Extremely high alarm rates occurred on day shift for extended periods (Section 4.3.1 below), making
other abnormal situations difficult for board operators to manage manually.
The BP Toledo Refinery’s operations management and process engineering personnel who participated in the
afternoon review meeting, discussed above in Section 3.1.5, determined that putting the Crude 1 Tower on
circulation or shutting Crude 1 down was unnecessary, indicating that they did not fully comprehend the
magnitude of the cascading abnormal situations occurring. While the meeting apparently identified some
potential action items to possibly stabilize the Crude 1 Tower, it is the CSB’s understanding that this
information was not communicated to the night shift.
Abnormal situations often begin with little warning and thus often do not allow for planning, training, and
developing procedures as they are occurring. Consequently, procedures and policies to manage abnormal
situations should be established before an abnormal situation occurs. Doing so would help operating personnel
recognize abnormal situations, identify the source of the deviation, and either safely restore normal operating
conditions, or determine that this is not possible and stabilize the process. Refinery leadership can then capture
lessons learned to prevent or mitigate similar abnormal situations in future [1, pp. 3-4]. While not every
abnormal situation may be predicted such that formal written procedures exist well in advance, well thought out
same-day guidance can help fill the need if properly communicated. For example, the afternoon meeting notes
included instructions to reduce Crude 1 Furnace outlet temperature “to stabilize crude tower pressure,” which
would reduce heat input to, and therefore pressure in, the Crude 1 Tower. Instead, in contrast, a night shift board
operator’s shift turnover instructions were to re-establish Crude 1 Tower pumparounds. The board operator later
told the CSB “the next thing I needed to do was to get the pumparounds going again on the crude tower.”
The ASM Consortium provides the following example:

While it is not practically feasible to provide written instructions for all potential
limit excursions, one site has adopted the practice of identifying the most critical
84
process deviations and providing guidance on appropriate responses within the

high complexity, high-risk operating procedures [29, p. 88].
The CSB concludes that on the day of the incident, the BP Toledo Refinery did not provide effective guidance
for managing abnormal situations. The lack of effective guidance required operations personnel to make
improvised real-time decisions. Had effective guidance been communicated to the night shift, the incident could
have been avoided.
After the incident, the Ohio Refining Company LLC created a new “Loss of Pumparound Response Procedure”
for the Crude 1 Tower. a The new procedure provides guidance for mitigating the consequences of a Crude 1
Tower upset, as well as steps to re-establish pumparound cooling. The procedure includes general safety steps,
such as evacuating all non-essential personnel, and specific process stabilizing steps, such as minimizing energy
input to the tower by reducing furnace outlet temperature. Crucially, the procedure also defines several
abnormal situations and their proper responses, as shown in Figure 34.
Figure 34: Excerpt from new “Loss of Pumparound Response” procedure. (Credit: Ohio Refining Company LLC)
This new “Loss of Pumparound Response” procedure provides clear and actionable guidance for when human
intervention is required to shut down the Crude 1 Tower or put the Crude 1 unit in circulation mode.
Importantly, this guidance would have caused the refinery to shut down the Crude 1 Tower or place the Crude 1
unit in circulation before noon on the day of the incident. The Crude 1 Tower pressure exceeded 38 psig 11
times during the day shift, and twice before 11:00 a.m., as described in Section 3.1.5. Thus, by the time that
night shift operators arrived, the Crude 1 unit could have already been in a safe state, and the Crude 1 Tower
upset on night shift could have been avoided. If the Crude 1 Tower upset on night shift had been avoided, the
excess liquid in the Crude 1 Overhead Accumulator, the Absorber Stripper Tower, and the Fuel Gas Mix Drum
would have also been avoided, and the fatal incident would not have occurred. This is one example of
a
After February 28, 2023, the Ohio Refining Company LLC is the owner and operator of the Toledo Refinery. See Section 1.2
85
effectively providing guidance for managing an abnormal situation. Section 4.2.3 below further discusses tools
to mitigate or prevent abnormal events, using the day of the incident as an example.
4.2.2 Use of Industry Guidance

The ASM Consortium has published guidance documents for alarm management practices, Human-Machine
Interface design and development, procedural practices, and operations practices [30]. The ASM Consortium
Guidelines Effective Operations Practices (2019) state [29, pp. 3-4]: a
The effective operations practices guidelines are organized under seven
categories: KEY LESSON
Understanding Abnormal Situations - addresses measuring,
Companies should define
reporting, analyzing, and communicating the causes and effects
operating limits beyond
associated with abnormal situations in the plant.
which Abnormal
Situation Management
Organization Roles, Responsibilities, and Work Processes -
procedures should be
addresses the influence of work culture through definition of work
followed and clearly
processes, staff roles and responsibilities, and valued behaviors.
define those corrective
Knowledge and Skill Development - addresses a competent work actions to be followed, in
force through use of comprehensive training fundamentals and the order to stop a chain of
creation of a continuous learning environment. abnormal events.
Communications - addresses effective daily and situational

dialog between functional groups and within operations.
Procedures - addresses key challenges associated with procedural operations such

as accessibility, accuracy, clarity, policy compliance, and feasibility.
Operations Work Environment - addresses factors associated with 24/7 operations

that impact operator performance.
Process Monitoring, Control, and Support Applications - addresses software and

hardware platforms deployed for the operator are appropriate and maintained over
their lifecycles [29, pp. 3-4].
Based on the BP Toledo Refinery’s ASM policy discussed in Section 4.2.1, there is no evidence that the BP
Toledo Refinery utilized this ASM Consortium guidance in the refinery’s ASM policy, since the site policy does
not incorporate guidelines or categories from the ASM Consortium for effective operations practices and does
not describe a structure or documents where the guidelines above are used.
The CSB concludes that the BP Toledo Refinery did not effectively use previously existing industry guidance,
such as that available from the ASM Consortium, to develop its ASM policy. Had the refinery done so prior to
a
The initial release of the Effective Operations Practices guideline document for members was in 2001. The content above is from the
revised 2019 edition [29, p. v].
86
the incident, it could have had a framework to provide effective guidance to the night shift to safely operate the
refinery after the NHT shutdown.
Although BP had access to the ASM Consortium guidance for at least a decade before the incident, the CCPS
more recently provided a comparable list of ASM elements. In its book, Guidelines for Managing Abnormal
Situations (2023), the CCPS provided eight tools and methods for effective management of abnormal situations,
including:
Predictive Hazard Identification. Potential abnormal situations should be evaluated to identify and
document hazards and their consequences for all operating phases and include analysis of process
design, the process control strategy for responding to process upset conditions, and inherently safer
design features [1, p. 113]. Predictive hazard identification of abnormal situations is typically performed
as part of the PHA [1, pp. 113-114]. To be effective, the PHA must include reviews of abnormal
situations affecting the process control system (See Section 4.1.1) and historical abnormal scenarios
(See Section 4.4). This may require a supplemental abnormal situation review, tabletop exercises, and
simulations to see how the operating team would respond in response to upset conditions [1, pp. 114-
115].
Process Control System. The control system should provide an interface so that a board operator can
observe trends of multiple critical parameters simultaneously. Displays providing an overview of key
parameters and a “big picture” view of the process are critical to a well-designed control panel; for
example, a display showing flow into and flow out of a tower. To be effective, system design should
enable instruments to stay in a normal range, even during transient operations [1, p. 117]. The system
should also allow for effective alarm management, such that operators are not overloaded with low
priority and nuisance alarms [1, p. 119]. Alarm flood is further discussed in Section 4.3.
Policies and Administrative Procedures. Formal, written policies must be clear to personnel regarding
their authority to make timely decisions, including those which allow any personnel to halt operations
over safety concerns [1, p. 126]. It is also critical to establish effective and structured communication
between teams and shifts, such as the use of standardized log sheets, checklists, and a record of issues,
which summarize both normal and abnormal conditions encountered during the shift [1, pp. 126-128].
Operating Procedures. Procedures are the first tier of human response safeguards to unexpected
situations. Operating procedures should include the safe operating conditions, provide warnings against
deviations from safe limits, and include steps on how to re-establish safe status after a deviation occurs
[1, pp. 130-131]. Often, operating procedures consider only routine situations and fail to provide written
instructions on how to handle abnormal situations [1, p. 131]. To ensure procedures are properly written
to handle abnormal situations, companies should perform an observational audit of procedures to note
missed steps and confusing items [1, p. 132]. In addition, companies should perform a Procedural PHAa
a
Several methods are available for execution of a Procedural PHA, such as the Transient Operations Hazard and Operability Study
(TOH) [1, pp. 133-134] or by applying more traditional PHA methods such as Hazard and Operability Study (HAZOP) or Structured
“What-If” Technique (SWIFT) [59, pp. 37-41, 46-48].
87
which focuses on operational tasks, timely identification of hazards,

and procedural controls in response to abnormal situation [1, pp. 133- KEY LESSON
134].
“Abnormal situations
Training and Drills. Training for board and outside operators should introduce stress, and
include methods for managing abnormal situations, such as tabletop operators under stress
exercises covering desired responses to situations, emergency response can make poor decisions,
drills, alarm response training, and process simulation [1, p. 137]. which then exacerbate
the situation. How
Ergonomics and Other Human Factors. Control room layout,
companies prepare and
environmental conditions, graphics and displays, and human-machine
equip their operators to
interfaces should all be considered to ensure optimal operator
deal with these
performance and response to abnormal situations [1, p. 139].
problematic and stressful
Learning from Previous Abnormal Situation Incidents. While most situations is critical to
companies investigate incidents, abnormal situations may have ensuring the return of the
occurred without becoming a major event. In addition to investigating unit to a safe state.
incidents, companies should identify and investigate near-misses and Often, process safety
address the cause through changes to the process, software controls, incidents are a result of
and procedural changes [1, pp. 148-149]. Section 4.4 further discusses organizations failing in
learning from previous incidents. this area.” — CCPS,
Guidelines for Managing
Management of Change. Companies should ensure changes to process Abnormal Situations [1,
control systems, such as logic changes, software revisions, tuning of p. 24].
controllers, process control alterations, alarm setpoints, and interlock
setpoints are evaluated as part of an established and effective
Management of Change (MOC) process [1, p. 151].
While this CCPS guidance was published after the BP Toledo Refinery incident, the refinery can still use the
methods above to improve its outcomes when future abnormal situations occur. In addition, since the ASM
Consortium guidance above was available as early as 2001, the refinery could have implemented policies and
procedures more aligned with the ASM Consortium’s Effective Operations Practices before the incident
occurred.
The CSB recommends that Ohio Refining Company LLC revise the Abnormal Situation Management policy to
incorporate guidance provided by the ASM Consortium and the Center for Chemical Process Safety (CCPS). The
revised policy should include, at a minimum:
a) A broader definition of abnormal situations, such as that defined by the CCPS,
b) Additional predictable abnormal situations and their associated corrective procedures. At a minimum
88
c) Guidance to determine when an abnormal situation is becoming too difficult to manage and the
implementing proper procedures for bringing it to a safe state. (See Recommendation 2022-01-I-OH-R2).
4.2.3 Applying Industry Guidance to the Incident

Many incidents are preventable if abnormal situations are recognized promptly,
diagnosed correctly, and corrected in time, breaking the chain of events [1, p.
22]. For example, the new procedure issued by the new ownership of the Toledo KEY LESSON
Refinery after the incident, as shown in Section 4.2.1 above, clearly instructs Thinking through
operations to shut down the Crude 1 Tower if prolonged, excessive tower abnormal situations
pressure is detected. before they occur, having
plans in place, and
As an illustration of how some abnormal situations can cascade to escalating practicing those plans
consequences, Figure 35 shows several abnormal situations that occurred on can greatly improve
the day of the incident, their consequences, and how those consequences and operator and manager
other decisions led to the next abnormal situation, escalating throughout the confidence and decision-
day. making during an
abnormal situation.
Throughout the day, abnormal situations continued to escalate, culminating in
Simulators, desktop
the release of naphtha from the Fuel Gas Mix Drum to the ground, resulting in a
drills, incident reviews,
fatal fire. None of the decisions made that day were made in a vacuum; instead,
or field walkthroughs
they were frequently made in response to other previous abnormal situations and
can improve abnormal
their consequences (shown in red in Figure 35 below). These are examples of
situation management
situations that can affect the performance of board operators, outside operators,
skills.
or refinery management performance [1, pp. 87-88].
89
Figure 35: Examples of abnormal situations the day of the incident (in blue), and the consequences of them (in
orange). Note that in some cases, consequences and decisions contributed to the next abnormal situation (in
red). (Credit: CSB)
Table 2 below shows how using the CCPS abnormal situation management tools and methods described above
in Section 4.2.2 can be used to mitigate or prevent abnormal situations, using the abnormal situations the refinery
encountered the day of the incident as examples.
90
Table 2: How CCPS tools and methods could apply to manage abnormal situations encountered leading up to
the incident. (Credit: CSB)
CCPS ASM Tools and Examples from the BP Toledo Refinery Incident Report
Methods to Manage Section
Predictive Hazard In an attempt to manage high level in the Crude 1 Overhead 4.1.1
Identification Accumulator Drum, the naphtha flow control valve to the Coker Gas
Plant was opened, which overfilled the Absorber Stripper Tower to
Potential abnormal
the Fuel Gas Mix Drum and the downstream furnaces. While the
situations should be
PHA identified a similar scenario, it used ineffective preventive
evaluated to identify
safeguards.
hazards and
consequences, usually Crude slate changes to the Crude 1 Tower are a routine part of
in a PHA. PHAs should refinery operation. However, large and rapid changes can cause
include reviews of major process upsets. Such process upsets could be foreseen and 3.1.6
abnormal situations prevented or mitigated through automatic controls to assist or
affecting the process minimize operator intervention when pumparound flow is lost, or
control system. to only allow crude slate changes in small step changes or within
certain ranges (outside of emergencies) to prevent input errors.
Process Control Instruments should be calibrated to read the full range of possible 3.1.5
System values, but the Crude 1 Tower Overhead naphtha to Light Virgin
Naphtha flow meter was out of range, displaying misleading
The control system
information to the day shift board operator and masking the
should enable
inability to reliably control Crude 1 Overhead Accumulator Drum
instruments to stay in a
level.
normal range, even
during transient A DCS overview graphic showing a material balance a could allow 4.4.2
operations, provide an board operators to see from where the Absorber Stripper Tower
interface so board overflow was coming and could have prompted board operators to
operators can observe take appropriate corrective action. Adding this type of graphic was a
multiple critical CSB recommendation in its BP Texas City incident investigation [31,
parameters p. 215].
simultaneously, and
allow for effective The control system should allow for effective alarm filtering and
4.3
alarm management. suppression so as not to overwhelm board operators while still
highlighting important alarms, such as high Fuel Gas Mix Drum level,
in an abnormal situation.
a
A material balance calculation can be used to determine how much total liquid is in a given unit; it is determined by comparing the
amount of incoming feed to the amount of outgoing product [31, p. 83].
91
Policies and The BP Toledo Refinery’s ASM policy narrowly defined an abnormal 4.2.1
Administrative situation and did not consider other abnormal situations outside
Procedures that definition ahead of time, overloading the employees in an
urgent or emergent situation. At least some abnormal situations can
Formal, written policies
be pre-defined, with a predetermined, written, and communicated
must be clear to
plan in place, to reduce the mental load during abnormal situations.
personnel regarding
their authority to make
timely decisions.
Operating Procedures The NHT unit emergency shutdown procedure did instruct 3.1.5
operations to divert Crude 1 overhead naphtha to the Light Virgin
Should include safe Naphtha Storage but did not anticipate that the Light Virgin
operating conditions, Naphtha Storage may not be capable of receiving all the flow in a
provide warnings Crude 1 Tower upset condition.
against deviations from
safe limits, and include Although the Crude 1 Tower was in an upset condition for hours,
steps on how to re- and loss of pumparound cooling was not uncommon, there was no 3.1.5
establish safe status procedure for managing the loss of all pumparound cooling, and no
after a deviation guidance for which abnormal conditions should trigger putting the
occurs. Crude 1 Tower on circulation or shutting it down.
Loss of pumparound cooling in the Crude 1 Tower was a predictable

abnormal situation, and the refinery should have developed a 4.2.1
written procedure to manage it in anticipation of such an event
(and did so after the incident occurred).
92
Training and Drills Board operators were often qualified on the Crude 1 job with 3.1.6
minimal, if any, experience in managing process upsets. This
Training for board and experience was learned on the job, in real time. A simulator or mock
outside operators exercises/drills would allow training to manage abnormal situations
should include without potentially hazardous consequences, and in a lower stress
methods for managing environment. While at least one other unit at the refinery did have
abnormal situations, a simulator, there was no simulator for the Crude 1 Tower. Drastic
such as tabletop changes to Crude 1 Tower crude slate, and how to control tower
exercises, emergency operation safely if such an upset did occur, could be discussed and
response drills, alarm practiced outside the operating refinery environment, rather than
response training, and in real time on an operating process.
process simulation.
As with board operators, outside operators could have been trained
3.1.9,
using mock exercises/drills for abnormal situations such as liquid in
3.2
the fuel gas system, which could have helped outside operators
avoid making high-risk decisions.
Learning from The Coker Gas Plant bypass had overfilled the Fuel Gas Mix Drum 4.4.1
Previous Abnormal with liquid naphtha in a 2019 incident, but the 2019 investigation
Situation Incidents did not result in any engineering controls being implemented to
prevent overfill recurrence and the company did not effectively
In addition to
communicate this incident and its consequences to board
investigating incidents,
operators.
companies should
identify and investigate The 2019 incident could have been effectively communicated to 3.1.9,
near-misses and outside operators to ensure they understood the incident and could 3.2
address causes through take appropriate action should it happen again. As it was, outside
process, software, or operators with limited knowledge of the prior incident were left to
procedural changes. improvise a solution during a stressful situation.
93
Management of For the NHT Preheat piping, the pressure relief system could have 3.1.2
Change been designed to avoid chatter, and piping with a higher pressure
rating could have been installed to avoid operating so close to the
Ensure changes to
emergency pressure-relief valve set pressure.
process control
systems (logic changes, The water phase level indication on the Crude 1 Overhead
software revisions, 3.1.1
Accumulator Drum was changed as part of an MOC. Because this
controller tuning, change was an unintended consequence of the MOC, it could not be
alarm and interlock communicated to operations. Operations did not know that water
setpoints) are would overflow the naphtha standpipe at 69 percent water phase
effectively evaluated. level, rather than the previous 100 percent level indicated before
the 2022 Turnaround.
The CSB concludes that had the BP Toledo Refinery more effectively used
Abnormal Situation Management tools and methods, such as Predictive Hazard KEY LESSON
Identification, Process Control Systems, Policies and Administrative
Procedures, Operating Procedures, Training and Drills, Learning from Previous Managing abnormal
Abnormal Situation Incidents, and Management of Change, the large number of situations goes beyond
cascading abnormal situations might have been stopped and the fatal incident PSM compliance alone.
could have been prevented. PSM tools can be used,
but for abnormal
situations, the tools
4.3 Alarm Flood must be applied with
abnormal situations
The ASM Consortium, described above in Section 4.2.2, defines alarm flood as
specifically in mind,
“[t]he situation where more alarms are received than can be physically addressed
particularly cascading
by a single process control operator.” The specific quantity of alarms that
abnormal situations.
constitute a “flood” can vary by process, site, position, and individual, but a
commonly accepted performance target is that more than 10 alarms in 10
minutes (Section 4.3.3 below) is alarm flood. According to the ASM
Consortium in its Guidelines for Effective Alarm Management:
[P]eople can analyze and react only to a maximum of about 10 alarms in any 10-
minute period. If more than 10 alarms occur in 10 minutes, operator performance
is significantly impacted and the risk level in the plant will rise dramatically.
4.3.1 Alarm Flood Day of Incident

On September 20, 2022, the board operator DCS console that included the Crude 1 unit, the Fuel Gas Mix
Drum, the Vacuum 1 unit, the NHT unit, the Saturated Gas Plant, Coker Gas Plant, and the West Flare received
94
a very large number of alarms, both on day shift and night shift.a,b The alarm rate was at or above 10 alarms in
10 minutes starting at approximately 7:00 a.m., and continuing throughout the day, up to and after the incident at
6:46 p.m., with the exception of two 10-minute periods (although alarms sounded during these two periods as
well). c Figure 36 shows this alarm rate, in 10-minute blocks throughout the day.
Figure 36: DCS console alarm rate on September 20, 2022, with some process events (in red). The gray line
indicates alarm flood rate (10 alarms in 10 minutes). (Credit: CSB)
Figure 36 illustrates the alarm rate that board operators faced throughout the day, with many alarms
annunciating at a rapid pace, sometimes for hours together, after the NHT Preheat emergency pressure-relief
valve chatter and emergency shutdown that morning and leading up to the incident that evening. Such a situation
is virtually impossible to manage for extended periods without missing critical alarms or errors occurring
(Section 4.3.3 below).
a
See A.1.4.6.9 for details from BP’s control system event log.142
b
Throughout this section of the report, the “DCS Console” refers to the board position that includes these same areas of the refinery:
Crude 1, the Vacuum 1 unit, the NHT unit, the Saturated Gas Plant, Coker Gas Plant, the Fuel Gas Mix Drum, and the West Flare.
c
There were five alarms between 7:20 a.m. and 7:30 a.m. and three alarms between 4:20 p.m. and 4:30 p.m. All other 10-minute periods
between 7:00 a.m. and 7:00 p.m. were at or over the alarm threshold of 10 alarms in 10 minutes.
95
For example, the Fuel Gas Mix Drum high level alarm went off at the DCS console at 6:10 p.m., but a board
operator did not radio to the outside operators until 6:17 p.m., just after the Fuel Gas Mix Drum high-high level
alarm annunciated. As discussed above in Section 4.1.1, the Fuel Gas Mix Drum high level alarms coupled with
operator response was a safeguard against a hazardous condition. For alarms that require operator response to
prevent hazardous conditions, a rapid response is essential for the safeguard to be effective. The high level alarm
was a high priority alarm and was color coded to indicate its high priority status, but alarm prioritization can be
ineffective if the board operator is simply overwhelmed in an extreme alarm flood situation.
From 6:10 p.m. through 6:17 p.m., the DCS console had 55 alarms, the equivalent of approximately one alarm
every nine seconds. Managing the alarm flood during this time likely created a delayed response to the Fuel Gas
Mix Drum high level. One board operator described his perspective of the process upset before the incident to
the CSB:
Meanwhile, the [Fuel Gas Mix Drum] is high level. The alarms are coming in.
And at this time, feeling very overwhelmed, I can remember. […] At this point,
almost everything’s in manual and alarms are still just pouring in.
As further discussed in Section 4.3.3 below, board operators are very likely to be overwhelmed in alarm flood
with so many alarms for such extended periods, such as on the day of the incident, when between 6:50 a.m. and
6:49 p.m., a total of 3,712 alarms were recorded.
Moreover, between the beginning of the night shift at approximately 4:50 p.m. and the incident at 6:46 p.m., the
DCS console received between 24 and 281 alarms every 10 minutes, which is 2.4 to 28 times the alarm flood
threshold.a The night shift alarm rates are shown in detail in Figure 37.
a
Throughout this section, the alarm performance targets are based on one board operator. This position was normally staffed by a single
board operator.
96
Figure 37: DCS console alarm rate on night shift, with some process events (in red). The gray line indicates
alarm flood rate. (Credit: CSB)
Another major problem for the night shift board operators was that once the Crude 1 Overhead Accumulator
Drum began to overfill, and a board operator opened the naphtha flow control valve to the Coker Gas Plant at
5:41 p.m., alarms were sounding at a rate of 48 per 10 minutes (Figure 37 above) or higher until after the
incident occurred at 6:46 p.m. The board operators were potentially so overwhelmed that they did not have time
to process the source of overflow to the Fuel Gas Mix Drum and close the naphtha flow control valve to the
Coker Gas Plant in time to prevent overflow to the furnaces and boilers. As shown in Table 3 below, the BP
Toledo Refinery averaged 254.50 alarms per hour on the day of the fatal incident. Further, as noted above,
between 6:50 a.m. and 6:49 p.m. that day a total of 3,712 alarms were recorded.
The CSB concludes that the board operators experienced an alarm flood condition for nearly 12 hours preceding
the incident, experiencing more than 3,700 alarms during the 12-hour period. Alarm flood contributed to the
incident by overwhelming and distracting the board operators, causing delays and errors in responding to critical
alarms. Had the night shift board operators been able to recognize the source of liquid in the fuel gas system
during the alarm flood and closed the naphtha flow control valve to the Coker Gas Plant, they could have
stopped liquid flow to the Fuel Gas Mix Drum, preventing or mitigating the fatal incident.
97
Invocation of Stop Work Authority (as discussed in Section 3.3) would have been especially important given
the severe alarm flood occurring on September 20, 2022. As previously stated, board operators were in an alarm
flood condition for nearly 12 hours preceding the incident, experiencing more than 3,700 alarms between 6:50
a.m. and 6:49 p.m. Alarm flood contributed to the incident by overwhelming and distracting the board operators,
causing delays and errors in responding to critical alarms. A shutdown, or putting the unit in circulation mode,
likely would have enabled operators and other refinery personnel to cope with the alarm flood and identify
critical actions to be taken, including closing the naphtha flow control valve to the Coker Gas Plant and stopping
liquid flow to the Fuel Gas Mix Drum.
The Toledo Refinery must ensure (through training, clearly written procedures, and other means) not only that
the refinery’s employees are clearly empowered to invoke Stop Work Authority but also that the refinery’s
employees clearly understand that they are expected to do so, especially during an abnormal situation, including
an alarm flood, such as existed during the day of the fatal incident.
The CSB recommends that the Ohio Refining Company LLC develop and implement a policy or revise existing
policy that clearly provides employees with the authority to stop work that is perceived to be unsafe until the
employer can resolve the matter. This should include detailed procedures and regular training on how employees
would exercise their stop work authority. Emphasis should be placed on exercising this authority during
abnormal situations, including alarm floods. (See Recommendation 2022-01-I-OH-R3).
4.3.2 BP Guidance
While the BP Toledo Refinery did not have site-specific guidance for managing or preventing alarm flooding at
the time of the incident, BP’s guidance for alarm management included two generalized documents:
Downstream Alarm Philosophy and Alarm System Design and Management. BP’s guidance defined alarm flood,
or alarm overload, as a “situation in which more alarms are received than can be processed by a single console
operator.” This guidance also stated that “an alarm rate greater than 10 alarms in 10 minutes represents an alarm
flood condition.”
BP’s alarm management guidance defined several key process indicators to track alarm system performance.
Among these were alarm rate and alarm flooding. The guidance defined alarm rate, or average dynamic load, as
the number of alarms going off per hour per board operator position over the time span of interest. BP’s
performance targets for alarm rate and flood frequency are summarized in Figure 38.
98
Figure 38: BP alarm systems metrics and performance levels for alarm rate and flooding. (Credit: BP)
The BP Toledo Refinery tracked alarm system performance according to the BP guidelines. For primary
performance metrics, the “challenging” performance category required weekly reviews, reporting the
performance gap to entity leadership and adding the item to a site risk register, with a resourced action plan. The
CSB did not determine whether the BP Toledo Refinery was within or out of compliance with the guidance long
term, but on September 20, 2022, the DCS console alarms were well above the challenging threshold due to the
process upsets that day. Table 3 shows the alarm performance against primary metrics in 2022 for various
available time periods. It should be noted that the refinery was in a planned outage starting in April 2022, started
up from the outage in July 2022, and there was an unplanned partial Crude 1 shut down followed by a restart in
August 2022. Data for these months could be skewed low due to the outage, or high due to refinery or
equipment startup or shutdown.
99
Table 3: 2022 alarm performance data. Performance levels are listed according to BP guidance as in Figure 38
above, to indicate target, improving, and challenging performance. (Credit: CSB)
Time Period Average Alarms Percentage of 10-minute Performance Level per
per Hour Time Slots > 10 Alarms BP Guidance
Jan-2022 4.85 0.18 Target
Feb-2022 5.65 0.30 Target
Mar-2022 8.13 1.32 Improving
Apr-2022 (shutdown) a 7.20 1.10 Improving
May-2022 (down) 2.72 0.60 Target
Jun-2022 (down) 2.44 0.53 Target
July-2022 (startup) 31.17 13.13 Challenging
Aug-2022 (startup) 15.39 3.94 Improving
19-Sep-2022 9.46 0.69 Improving / Target
20-Sep-2022 254.50 68.75 Challenging
The BP guidance specifically pointed out that process upsets were vulnerable periods for errors to occur, and
should be reviewed for alarm system improvements where possible:
Typically, the biggest alarm load and potential for an alarm flood is after a major
plant upset. Such disturbances are often particularly stressful for the operator and
can be relatively hazardous periods of operation. Therefore, it is particularly
important to improve alarm performance during this period.
The secondary metric for flooding in Figure 38, the number of alarms in any 10-minute period, shows the extent
of alarm flood the day of the incident. The number of alarms in any 10-minute period was far over the
challenging level of performance, or greater than 50 alarms in 10 minutes, for four hours that day between
approximately 8:00 a.m. and 7:00 p.m., as shown above in Figure 36.
The CSB concludes that on September 20, 2022, the BP Toledo Refinery alarm performance was classified as
“challenging,” according to BP’s guidance and the high extent and duration of alarm flood likely contributed to
the incident by overloading the board operators, contributing to miscommunications, errors, and missed alarms.
a
Data for April through August 2022 could be skewed low due to the outage or high due to startup or shutdown of the BP Toledo
Refinery.
100
4.3.3 Industry Guidance for Alarm Flood Performance

Several sources provide industry guidance for alarm management in general and handling alarm flooding in
particular. The CCPS guidance in Guidelines for Safe Automation of Chemical Processes is consistent with BP’s
guidance and other industry guidance, defining alarm flood as follows:
Alarm flood—The presentation of more alarms in a given period of time than an
operator can effectively respond to […] (typically >10 alarms in ten minutes
following an upset event). Alarm flooding is one of the most dangerous problems
with alarm systems and potentially the most complex to solve. […] These alarm
floods overwhelm the operator, which make it difficult to process the alarms,
determine the cause and priority of the event, and to respond to new alarms due
to the developing event or resulting cascade events [32, p. 427].
The ASM Consortium also recognizes the importance of preventing or minimizing alarm floods, and defines the
human limits for handling floods: “If more than 10 alarms occur in 10 minutes, operator performance is
significantly impacted and the risk level in the plant will rise dramatically” [33, p. 83].
The CCPS, the ASM Consortium, API, and other industry guidance organizations often simply reference the
International Society of Automation (ISA) Standard 18.2, Management of Alarm Systems for the Process
Industries (ISA 18.2), a commonly used industry guidance document for alarm systems [34, p. 1]. This
standard’s guidance for alarm flood is as follows:
Alarm floods are variable-duration periods of alarm activity with annunciation
rates likely to exceed the operator response capability. […] As a recommended
target, an alarm system should be in flood for less than ~1 % of the time.
Improvements to the alarm system and process operation may be indicated by
the analysis of alarm floods. No targets are provided for these metrics [emphasis
added]. Alarm flood analysis should include:
b) duration of each alarm flood,
c) alarm count in each alarm flood, and
d) peak alarm rate for each alarm flood [34, pp. 75-76].
To summarize, multiple industry guidance sources, including the ISA, the CCPS, the ASM Consortium, and the
BP guidance above in Section 4.3.2, use 10 alarms in a 10-minute time block to define alarm flood. The ISA,
among other sources, also identifies that alarm flood condition should be present less than one percent of the
time (but must include at least 30 days of data), as a guideline. Figure 39 below summarizes the alarm rate and
alarm flood guidelines in ISA 18.2.
101
Figure 39: ISA Standard 18.2 summary of select recommended alarm performance metrics, based upon at least
30 days of data [34, p. 78]. (Credit: ISA)
The CSB concludes that on September 20, 2022, the BP Toledo Refinery alarm performance did not meet
industry guidance, exceeding 10 alarms in 10 minutes for hours at a time. The high extent and duration of alarm
flood contributed to incident by overloading the board operators, contributing to miscommunication, errors, and
missed alarms, ultimately leading to the fatal incident. The high alarm rate was also indicative of ongoing
abnormal situations.
As shown in Figure 39 above, ISA 18.2 specifies that performance metrics should be “based upon at least 30
days of data” [34, p. 78]. It is important to note, however, when calculated for a month long period, an alarm
flood condition that occurs one percent of the time would equate to approximately 7.2 hours. If one process
upset occurs each month that is, for example, seven hours long but occurs on a single shift, the operator working
that shift can easily be overloaded even if the site meets the one percent monthly target under ISA 18.2. As
shown above in Section 4.3.1 and in industry guidance, multiple hours of alarm flood clustered together can
overwhelm board operators, contributing to errors and catastrophic incidents. As noted above, ISA 18.2
recommends that the number of alarm floods, duration of each alarm flood, alarm count in each flood, and peak
alarm rate for each flood should be analyzed, but it does not provide any performance targets for such analyses.
The Engineering Equipment and Materials Users Association (EEMUA) provides short-term alarm guidance in
its publication 191, Alarm Systems: Guide to design, management and procurement [35]. For abnormal
condition performance levels such as after a major process upset, the EEMUA states that 20 to 100 alarms in 10
minutes is “hard to cope with,” and over 100 alarms in 10 minutes is “definitely excessive and very likely to
lead to the operator abandoning use of the system” [35, p. 215]. The ASM Consortium guidance references the
EEMUA guidance, and adds that an alarm impact assessment is warranted whenever the peak alarm rate
exceeds 50 alarms in 10 minutes [33, p. 23]. Figure 40 summarizes the available guidance for alarm rate
following an upset, compared with the BP Toledo Refinery alarm performance preceding the incident.
102
Figure 40: Alarm rate guidance and actual performance following a process upset [35, p. 103]. (Credit: CSB)
Had the BP Toledo Refinery evaluated its alarm performance following process upsets, such as, for example, the
amount of time a board operator was exposed to a peak alarm rate greater than 50 alarms in 10 minutes, alarm
floods in 24 hours might have been more obvious than evaluating alarm performance based solely on the
percentage of time the alarm system is in a flood condition based upon at least 30 days of data could have been
more obvious than evaluating performance based solely on monthly average data. On September 20, 2022, the
board operators experienced alarm peak rates over 50 alarms in 10 minutes for four hours before the incident
occurred. a Figure 41 shows the ISA 18.2 alarm targets compared with the DCS console monthly performance,
and also compared with the September 20, 2022, data (shown by a red dot). Although the two months (July and
August 2022) in which refinery or unit startups occurred were outside ISA 18.2 targets, most of the months in
2022 were within the ISA 18.2 targets (in green). The September 20, 2022, data are shown compared with
monthly average targets because ISA 18.2 currently does not provide any short-term targets.
a
This is not including any alarms after 7:00 p.m., so that alarms triggered in response to the fatal incident itself would not be included in
the calculation.
103
Figure 41: BP Toledo Refinery DCS console alarm performance compared with ISA 18.2 targets (green dotted
lines), both monthly (blue dots) and for the day of the incident (red dot) [34, p. 78]. The green shaded area in
the lower left corner shows ISA 18.2 target performance. (Credit: CSB)
The CSB concludes that while ISA 18.2 provides guidance and a performance target for an alarm flood over a
period of at least 30 days, no additional targets are provided for items such as the number of alarm floods in a
month, duration of each flood, alarm count in each flood, or peak alarm rate for each flood. Had such targets
been established in industry guidance, the BP Toledo Refinery could have analyzed and improved alarm flood
performance following a process upset, such as that occurred in the hours preceding the incident.
The CSB recommends that the International Society of Automation revise American National Standard
ANSI/ISA 18.2-2016, Management of Alarm Systems for the Process Industries, to include performance targets
for short-term alarm flood analysis so that users can evaluate alarm flood performance for a single alarm flood
event. The performance targets should include:
104

that require alarm performance improvement actions. (See Recommendation 2022-01-I-OH-R7).
4.3.4 Industry Guidance for Alarm Flood Management

ISA 18.2 identifies several basic alarm design elements that can minimize alarm load on operators, including:
Alarm justification first should ensure that all alarms in a control system are necessary, will not become
a nuisance, and are not duplicative. For example, any alarm for which the operator action is simply to
communicate information to another person or group could be moved to another system so as not to
burden the operator or the alarm system [34, p. 48].
Alarm prioritization allows the critical alarms, such as safety limits, to be more visible to an operator,
typically by identifying high priorities by color or alarm tone, for example [34, p. 49].
Alarm deadband minimizes the number of alarm annunciations for a given abnormal condition by
preventing an alarm from returning to normal state until the alarm condition clears the deadband, which
is a defined range around the alarm setpoint [34, p. 52].
Alarm on-delay or off-delay, also known as a de-bounce timer, can avoid unnecessary alarms when a
signal temporarily overshoots the alarm setpoint by annunciating or returning to normal only after a set
time delay. This reduces or eliminates multiple alarms for the same parameter when the process value is
hovering near the alarm setpoint [34, pp. 52-53].
Shelving allows an operator to temporarily suppress a an alarm they already know about and do not need
repeated alarms for, with automatic controls that reinstate the alarm, usually after a set period of time.b
Some high-priority or safety alarms may be designed to disallow shelving, however [34, p. 24].
a
Alarm suppression is to “prevent the annunciation of the alarm to the operator when the alarm is active” [34, p. 24].
b
Alarms were shelved on September 20, 2022. See A.2.9.4.20
105
In the event that alarm flood is not at target performance with only the basic
alarm design elements above, ISA 18.2 also identifies “enhanced and advanced
KEY LESSON
alarm methods” to mitigate alarm floods [34, p. 76]. These include additional
layers of logic, programming, or modelling to modify alarms, and are used “to Companies should
guide operator action during abnormal process conditions” [34, p. 64]. Specific ensure that alarms are
advanced and enhanced alarm methods identified by ISA 18.2 include: well justified. While
DCS technology allows
Alarm attribute modification alters some alarm attributes such as
alarms to be created
setpoints or priority based on a defined set of conditions using logic such
easily, it also can cause
as decision trees.
board operators to be
Logical alarm suppression uses the states of some alarms to modify the inundated with low
attributes of other alarms, such that under certain conditions, alarms that priority or irrelevant
are not useful or are redundant can be suppressed automatically. information during an
abnormal situation if
State-based alarming modifies alarm attributes such as setpoint, priority,
alarms are not properly
or suppression status based on operating states for equipment or
designed. This also
processes. For example, while equipment or a process unit is in a
places additional stress
shutdown mode, most of the equipment’s alarms may not be necessary
on board operators,
[34, p. 65].
reducing their
The BP Toledo Refinery used at least some of the techniques above, such as effectiveness when it is
shelving, a prioritization, and state-based alarming. Shortly after the Crude 1 needed most.
Tower crude slate change at 4:56 p.m., the board operators received many
alarms in a short time. As the Absorber Stripper Tower filled with naphtha, a
board operator temporarily shelved b the two Absorber Stripper Tower high
level alarms. Shelving the high level alarms meant that the Absorber Stripper
Tower level alarms would no longer appear on the DCS active alarm screens the board operators were
monitoring for a period of time.
However, despite using some tools, alarm flood occurred during the September 20, 2022, process upsets as
shown in Figure 37 above. For example, from 5:30 p.m. to 6:45 p.m. (approximately from the time the Crude 1
Tower upset began to the time that the fatal incident occurred), the DCS console received 765 annunciated
alarms. Out of the 203 different tags that created alarms during this time, the six most frequent alarms accounted
for nearly half of the alarm annunciations (47.2 percent). The most frequent alarm went off 22 times in one
minute at 6:30 p.m., or on average once every 2.7 seconds, when board operators were already overwhelmed
with alarms. Figure 42 below shows the top six most annunciated alarms during this time.
a
Alarms were shelved on September 20, 2022. See A.2.9.4.20.
b
Alarm shelving is defined by the Engineered Equipment and Materials Users Association (EEMUA) as “a facility where the operator is
able to temporarily prevent an alarm from being displayed when it is causing a nuisance. A shelved alarm will be removed from the list
and will not re-annunciate until un-shelved.” At the BP Toledo Refinery, alarms could be shelved by operators for a maximum of 12
hours. Not all alarms could be shelved at all, and some could only be shelved for shorter periods. The EEMUA is “an Association
established by the owners and operators of industrial assets” [35, pp. 156, ii].
106
Figure 42: Most frequent alarms annunciated at DCS console between 5:30 p.m. and 6:45 p.m., the hour and
15 minutes preceding the incident. The blue bars show the number of annunciations for each tag, and the
orange line shows the cumulative percentage of alarms in that time. (Credit: CSB).
The most frequent alarm, “High Flow from Coker Gas Plant to Fuel Gas Mix Drum,” might have afforded board
operators a clue as to the source of liquid entering the Fuel Gas Mix Drum. This flow measurement was
normally in gas flow only, not liquid flow, as was occurring before the incident. Significantly, however, the
CSB found no evidence that anyone in the control room acted on this alarm. Board operators may have been too
overwhelmed with alarm flood to understand why the alarm was annunciating so frequently. In any case, board
operators did not take corrective action until after the fatal incident occurred, as the naphtha flow control valve
to the Coker Gas Plant remained open until 7:04 p.m., nearly 20 minutes after the incident.
To better reduce or eliminate alarm floods, the BP Toledo Refinery could have used alarm deadband or alarm
on-delay/off-delay to reduce the number of repeat incoming alarms to the board. This would allow board
operators to focus on the process upset itself, without having to sort out all the repeat alarms that they were
already aware of.
The CSB concludes that had the BP Toledo Refinery more fully utilized some of the available alarm flood
management techniques in ISA 18.2 prior to the fatal incident, such as deadband and on-delay/off-delay, the
107
alarm flood on the day of the incident would have been more manageable and board operators could have
prevented or stopped the flow of liquid naphtha to the Fuel Gas Mix Drum.
4.3.5 Post-Incident Alarm Flood Management

In June 2023, the Ohio Refining Company LLC issued a new Toledo Alarm Philosophy, which includes alarm
flood management strategies. The document described several forms and examples of alarm flood management
techniques, including state-based alarming and logical alarm suppression such as group suppression, where
“alarms can be grouped using logic to present the operator with a single common alarm, such as to indicate the
alarm condition of several related field sensors.” The new document specifically addresses alarm suppression
following a major event or trip:
Alarm systems can be difficult to manage following a trip or major event (such
as the trip of a compressor) as operators may be subjected to alarm floods. Such
disturbances are particularly stressful and can be considered as relatively
hazardous periods of operation. During an alarm flood, the operator’s
effectiveness is diminished because they can be overwhelmed and miss important
information. In order to minimize the number of alarms following the trip or
event, alarm flood (dynamic) suppression may be required.
Alarm flood suppression is the dynamic management of pre-defined groups of

alarms based on detection of equipment state and triggering events. In this
technique alarms are suppressed following an event when they are not relevant
or meaningful to the operator and when suppressing them cannot result in a
hazardous situation.
The new policy indicated that “flood events should be analyzed for total percentage of time each month that the
alarm system spends in a flood condition.” a While this is consistent with ISA 18.2, the flood performance metric
does not necessarily focus the analysis on plant upsets, which may cause alarm flooding over a period of hours,
as occurred the day of the incident (Figure 36 above).
The CSB concludes that the new ‘Toledo Alarm Philosophy’ follows ISA Standard 18.2 guidance but contains
the same gap as the guidance: it does not include short-term targets for items such as number of alarm floods,
duration of each flood, alarm count in each flood, or peak alarm rate for each flood. Such targets could provide
the refinery with more appropriate tools to analyze and improve alarm performance in an alarm flood that lasts
several hours, such as what occurred in the hours preceding the incident.
The CSB recommends that Ohio Refining Company LLC revise the ‘Toledo Alarm Philosophy’ by
incorporating the Engineering Equipment and Manufacturers Users Association (EEMUA) guidance for alarm
rate following an upset and not limiting alarm performance to a single metric averaged over a month. In addition
to including analyzing individual alarm flood events, the revised philosophy document should improve refinery
alarm performance to reduce alarm flood duration and peak rate for events similar to the September 20, 2022,
a
According to the Ohio Refining Company LLC, the Toledo Refinery alarm rationalization program utilizing the Toledo Alarm
Philosophy, evaluating if the appropriate alarms are in place, setpoints, and associated response actions for several systems, including
those involved in the incident is ongoing.
108
incident. Consult EEMUA Publication 191, Chapter 6.5.1, for guidance regarding abnormal condition
performance levels. Apply the improved performance levels where applicable, but specifically to the Crude 1
control board alarm performance. (See Recommendation 2022-01-I-OH-4).
4.4 Learning from Incidents

Accident investigations sometimes seek to identify lessons, so as to ensure that an accident like the one under
investigation never happens again. Nevertheless, accidents may repeat themselves [36, p. 65].
In the introduction to the BP Process Safety Series Hazards of Oil Refining Distillation Units, Jesse C.
Ducommun, Vice-President, Manufacturing and Director of American Oil Company in 1961 and Vice -
President American Petroleum Institute in 1964 stated the following:
It should not be necessary for each generation to rediscover

principles of process safety which the generation before
discovered. We must learn from the experience of others rather
than learn the hard way. We must pass on to the next generation
a record of what we have learned [13].
4.4.1 Catastrophic Incident Warning Signs from a 2019 BP Toledo

Refinery Incident
The CSB found the BP Toledo Refinery had investigated a previous incident of naphtha back flowing through a
Coker Gas Plant bypass line in 2019. Figure 43 shows the Initial Incident Report of high level in the Fuel Gas
Mix Drum. a
a
The common name at the BP Toledo Refinery for the Fuel Gas Mix Drum was the TIU mix drum (Toledo Integrated Unit).
109
Figure 43. BP Initial Incident Report. High level in the Fuel Gas Mix Drum during a refinery upset. (Credit: BP
with annotations by CSB)
The challenge presented by a first known instance like this 2019 incident is described by the CCPS:
In the case of setbacks, we need to know what went wrong and

how we can avoid repeating the same errors. This may be the
most difficult challenge. The biggest error that an organization
can make is in missing these opportunities [37, p. 149].
In 2012, the CCPS published Recognizing Catastrophic Incident Warning Signs in the Process Industries, a
book focused on recognizing when “something is wrong or about to go wrong” and then taking action to prevent
a major incident [37, pp. 1-3]. A warning sign is defined as a subtle indicator of a problem that could lead to an
incident [37, p. 1].
110
Some of the warning signs identified by the CCPS are:

failure to learn from previous incidents,
frequent process upsets or off-specification product,
abnormal instrument readings not recorded or investigated,
failure to report near misses and substandard conditions,
superficial incident investigations resulting in improper findings,
incident reports downplay impact, and
environmental performance does not meet regulations or company targets [37, p. 151].
The CCPS states that:
There is one common characteristic shared by the incident

warning signs presented here: The organization does not
perceive or recognize them [37, p. 1].
The BP Toledo Refinery had an opportunity to prevent the multiple vessel overflows that occurred in the
September 20, 2022, incident following the investigation of a 2019 refinery-wide upset. The investigation of the
2019 event determined that the Coker Gas Plant had been operating with the bypass line open when the entire
refinery unexpectedly lost steam. A boiler tripped as a cold front arrived in Toledo and quickly set off a series of
abnormal situations. Without steam on the Coker Gas Plant Absorber Stripper Tower and Lean Oil Stripper
reboilers and with all the Lean Oil Stripper overhead fans continuing to run, the temperature on the Coker Gas
Plant Lean Oil Stripper overhead system became low enough to plug the Lean Oil Stripper overhead system
with hydrates. a
During the refinery-wide upset, the Crude 1 Overhead Accumulator Drum developed a high level as the only
pump available at the time to pump liquid out of the drum was a steam-driven pump, which was inadequate. The
investigation found that the same naphtha flow control valve to the Coker Gas Plant as was involved in the
September 20, 2022, incident (See Section 3.1.7) was opened 80 to 100 percent during the 2019 incident in an
attempt to unload the high drum level to the Coker Gas Plant.
As in the September 20, 2022, incident, during the 2019 incident, naphtha began to fill the Coker Gas Plant
Absorber Stripper Tower and, according to the 2019 incident investigation, the naphtha backed up through the
Coker Gas Plant bypass due to high pressure that had developed in the Lean Oil Stripper from the hydrate
formation. As shown in orange in Figure 44, naphtha flowed from the Absorber Stripper Tower through the
Coker Gas Plant bypass line. A high liquid level developed in the Fuel Gas Mix Drum and the Polishing Amine
Contactor Sweet Gas Knockout Drum. In the 2019 incident, the high liquid level in the Fuel Gas Mix Drum was
drained to the Flare Knockout Drum and the Oily Water Sewer. Operations changed the crude slate being fed to
the Crude 1 unit to reduce naphtha production and reduced the opening of the naphtha flow control valve to the
a
Hydrates are solid compounds that can be formed during oil and gas production, causing interruptions in the flow of the produced
fluids.
111
Coker Gas Plant. a
Figure 44. Coker Gas Plant. Configuration of the Coker Gas Plant during a 2019 refinery upset. (Credit: CSB)
The CSB concludes that the facts, conditions, and circumstances of the 2019 incident show that while operating
the Coker Gas Plant with the bypass open, the BP Toledo Refinery did not have adequate safeguards to prevent
overflow of naphtha from the Coker Gas Plant to the Fuel Gas Mix Drum.
The Initial Incident Report description in Figure 43 above shows how in 2019 the BP Toledo Refinery
identified a warning sign when abnormal instrument readings on the Fuel Gas Mix Drum level instrumentation
were observed. The guided wave radar level instrument indicated 100 percent level when the liquid was above
the six-foot measured span near the bottom of the Fuel Gas Mix Drum while at the same time the specific
gravity-dependent differential pressure (dP) level instrument flatlined at roughly 65 percent level. This
discrepancy in the Fuel Gas Mix Drum level measurements is an example of a warning sign known as an
abnormal instrument reading. Abnormal instrument readings can indicate that a serious problem is occurring
that can lead to incorrectly analyzing an impending critical situation, as well as human error and potential
a
There was no release of naphtha to the ground or vapor cloud formation during this incident.
112
disaster [37, p. 154]. The 2019 Fuel Gas Mix Drum abnormal instrument readings are the same as described in
Section 3.2 for the September 20, 2022, incident.
As shown in Figure 45 below, in 2019 the BP Toledo Refinery investigated the high level in the Fuel Gas Mix
Drum using a Five Whys investigation methodology and developed actions to prevent future recurrence.
Figure 45. BP Process Incident Investigation. The preventive measures, lessons learned, and actions taken from
the 2019 incident of high level in the Fuel Gas Mix Drum. (Credit: BP, truncated with annotations by CSB)
Five Whys investigations are described in BP Toledo Refinery incident investigation procedures, but in the book
An Introduction to System Safety Engineering (2023), Nancy Leveson explains that this technique for incident
analysis “can lead an investigation team to omit important systemic causes” [24, p. 575]. The precise lineup of
causes determines the outcome, and unless the causes are systematically analyzed and addressed with follow-up
actions, there is a probability that the incident will recur. In fact, under slightly different circumstances the same
incident may have consequences that are more serious [37, p. 151], which is what happened at the BP Toledo
Refinery in 2022.
Figure 46 is a graphical representation of the 2019 Five Whys Investigation shown in Figure 45. The red flags
indicate causes of high level in the Fuel Gas Mix Drum such as high level in Crude 1 Overhead Accumulator
Drum and the open Coker Gas Plant bypass that could have prevented the September 20, 2022, incident, but
were not addressed with long term actions to prevent future reoccurrence.
113
Figure 46. Graphical representation of the Five Whys investigation of high level in the Fuel Gas Mix Drum.
(Credit: CSB)
Another warning sign described by the CCPS is superficial investigations. When action items are applied only to
the specific equipment affected, a superficial investigation can result in improper findings and not learning from
experience [37, p. 156].
Two action items were issued by the 2019 incident investigation that addressed a specific piece of equipment,
the Coker Gas Plant Lean Oil Stripper overhead, which formed hydrates and plugged after the cold front arrived
at the refinery. The 2019 incident investigation did not recommend any long-term actions related to the overflow
of naphtha from the Coker Gas Plant to the Fuel Gas Mix Drum, however.
114
The CSB concludes the BP Toledo Refinery 2019 Five Whys incident
investigation focused only on action items to prevent plugging in the Lean Oil
Stripper overhead system, failing to learn important safety lessons from the 2019
KEY LESSON
incident. The BP Toledo Refinery did not perceive the need to issue Accident investigation
recommendations related to overflow of naphtha through the Coker Gas Plant techniques, such as Five
bypass to the Fuel Gas Mix Drum or the abnormal Fuel Gas Mix Drum level Whys, suggest that there
instrument readings identified in the 2019 Initial Incident Report, which could is only one root cause
have prevented the September 20, 2022, incident. and one linear path to
an accident [24, p. 291].
To reduce the risk of a catastrophic incident, the CCPS stated that recognizing Using such approaches
and responding to warning signs is an important first step. The CCPS presented for process safety
a call to action for companies to embrace the use of warning signs as predictors incidents, even if
of increased danger and include warning signs within their process safety recommended in
management systems. To integrate warning sign detection and prevention company safety
methods into process safety management systems, the CCPS recommended that procedures, can lead an
companies: investigation team to a
perform an initial survey of warning signs, superficial analysis that
does not prevent an
build warning sign analysis into the safety management system, accident from recurring.
use the new system and track related action items,
evaluate effectiveness in the next safety management system
assessment, and
maintain vigilance against recurring warning signs [37, pp. 175-179].
4.4.2 Findings from the Fatal 2005 Explosion and Fire at the BP Texas
City Refinery
In his 1993 book Lessons From Disaster: How Organizations Have No Memory and Accidents Recur, process
safety expert Trevor Kletz stated:
It might seem to an outsider that industrial accidents occur

because we do not know how to prevent them. In fact, they occur
because we do not use the knowledge that is available [38, p. 1].
Ensuring that towers and process vessels such as the Absorber Stripper Tower at the BP Toledo Refinery do not
overflow is not a new safety lesson nor a novel process safety concept. Like many others, this safety lesson
could have and should have been learned from previous chemical disasters. Analyzing past disasters is important
to show how we can learn from accidents and apply safety lessons to help prevent similar chemical disasters in
the future [39, p. 1]. There are safety findings from the 2005 BP Texas City Refinery explosion and fire,
115
investigated by the CSB, which fatally injured 15 workers and injured 180 others that BP could have applied at
the BP Toledo Refinery to prevent the September 20, 2022, incident [31, p. 1]. Unfortunately, BP did not.a
The fatal 2005 BP Texas City incident occurred during the restart of a
hydrocarbon isomerization unit after a maintenance shutdown. At the time of the
incident, procedures not reflecting actual practice were used as a Raffinate
KEY LESSON
Splitter Tower filled with flammable hydrocarbons and overflowed [31, pp. 2- Organizations should
3]. A CSB animation describes the incident, including the overflow of the develop systems to
Raffinate Splitter Tower. ensure that learnings
from internal and
Even with numerous alarms, refinery operators were unaware of the overflow of external incidents are
the BP Toledo Refinery Coker Gas Plant Absorber Stripper Tower through the incorporated
vapor bypass piping and the 2005 overflow of the Texas City Raffinate Splitter throughout the
into the overhead line. The BP Texas City Raffinate Splitter Tower overflowed organization to prevent
and subsequently overfilled a blowdown drum, which discharged into the recurring failures, such
atmosphere. Flammable hydrocarbons flowed out of the Texas City blowdown as overflow of process
drum and formed a vapor cloud that ignited [31, p. 11]. The BP Toledo Refinery vessels, that can lead to
Coker Gas Plant Absorber Stripper Tower overflowed in 2022 through the vapor a catastrophic incident.
bypass piping and subsequently overfilled the Fuel Gas Mix Drum. Two BP
employees releasing flammable naphtha from the overflowing pressurized Fuel
Gas Mix Drum directly to the ground formed a vapor cloud that ignited, just like
the vapor cloud in the BP Texas City incident.
After the fatal Texas City Refinery explosion and fire, BP commissioned a report to look into managements’
accountability for the “Texas City Isomerization Explosion.” b This internal BP report stated:
Finally, Texas City Refinery either did not learn or did

not apply the lessons from prior incidents at Texas
City and other BP refineries … [40, p. 9].
Similarly, the CSB found that the BP Toledo Refinery did not effectively learn, apply lessons, or institutionalize
the knowledge of overflow scenarios from the fatal Texas City incident, despite the warning from the final report
of BP’s Management Accountability Project.c
Table 4 lists findings from the overflow event that were either not learned or applied from the prior Texas City
incident that could have prevented the September 20, 2022, incident.
a
The BP Texas City Refinery and the BP Global Executive Directors closed all recommendations issued by the CSB from the BP Texas
City Refinery Explosion and Fire Investigation.
b
The Management Accountability Project Texas City Isomerization Explosion Final Report is also known as the “Bonse Report”. The
Bonse Report is a separate report from the Baker Panel Report. Unlike the Baker Panel Report which examined BP's safety culture
across all of BP’s refineries in the U.S., the task of this BP team for the Bonse Report was to examine specific individual management
accountability within BP.
c
See Section 1.5 for a refinery incident of an explosion while draining a fuel gas system.
116
Table 4. BP Refinery incident findings comparison. Comparison of the BP Texas City Refinery and the BP Toledo
Refinery overflow events.
BP Texas City BP Texas City Refinery BP Toledo Refinery
Incident Finding Raffinate Splitter Blowdown Drum Absorber Fuel Gas
[31, pp. 22-26] Tower Stripper Tower Mix Drum
Level measurement No Not Applicable Yes Yes
calibrated for correct fluid Calibrated with (High Level Switch, Guided Wave Radar
during start-up/shutdown/ incorrect fluid [31, pp. 311,324- level measurement
upset conditions specific gravity 325]) is independent of
[31, pp. 131, 327] specific gravity
No
dP level calibrated
for amine-water
solution and not for
naphtha or other
hydrocarbons in an
upset condition,
plateaued around 67
percent while
overflowing
Overflow condition detected No No No No
on a control room console Overflow from Overflow to Sewer Overflow from Overflow
display designed to provide tower overhead line and Atmosphere Absorber Stripper determined after
adequate information of to emergency [31, p. 40] Tower to the Fuel multiple high level
flows in and out to alert pressure-relief Gas Mix Drum alarms and not from
operators a valves discharging a mass balance
(mass balance graphic) to the Blowdown graphic designed to
Drum [31, p. 23] provide information
of flows in and out
Engineered Control provided No No No No
(Control system to prevent A safeguard to Manual draining
overfilling) b close the naphtha required,
flow control valve closed system
to the Coker Gas draining capacity
Plant on high level inadequate for
identified but not Coker Gas Plant
implemented overflow
a
Prior CSB Recommendation 2005-4-I-TX-14 to the BP Texas City Refinery.
b
Prior CSB Recommendation 2005-4-I-TX-14 to the BP Texas City Refinery.
117
BP Texas City BP Texas City Refinery BP Toledo Refinery

Incident Finding Raffinate Splitter Blowdown Drum Absorber Fuel Gas
[31, pp. 22-26] Tower Stripper Tower Mix Drum
Level high alarm provoked No No No Yes
an appropriate response Alarm remained Alarm did not Board operator Manual draining
active for the entire activate [31, p. 324] shelved tower started to Flare
period of the startup active high level Knockout Drum
[31, p. 82] alarms after
intentionally
opening the
naphtha flow
control valve to the
Coker Gas Plant
Filling vessel with bottom No Yes No No
outlet valve open Raffinate Splitter Manual drain valve Absorber Stripper Fuel Gas Mix Drum
base level control to sewer was Tower bottoms manual valves to
valve had been chained open ROEIV closed by Flare Knockout
closed and was [31, pp. 38-40] prior shift Drum and Oily
later opened by a Water Sewer were
board operator closed.
[31, pp. 22-23] These valves were
later opened by
outside operators in
response to high
level in the Fuel Gas
Mix Drum.
Use of Approved Operating No No No No
Procedures Lack of procedure Lack of procedure Lack of a Lack of an approved
(For example, using informal to document filling for chaining manual procedure for procedure for
procedures for temporary the bottom of the valve to sewer correcting high draining high level
hold-up of liquid level or raffinate splitter open a [31, p. 332] Crude 1 Overhead in Fuel Gas Mix
draining)
above the range of Accumulator Drum
the level transmitter Drum level
(Informal practice
not unusual for
start-up) [31, p. 73]
Rigorously documented No No
handover Evidenced by events on March 23, 2005 Missing Board Operator shift log on
[40, p. 16] [31, p. 23], [40, p. 16] September 20, 2022
a
In 1998, PHA action item recommended chaining open the discharge valve to the sewer on the blowdown drum to prevent a high liquid
level from increasing the backpressure on the relief valve headers. This action item was addressed by chaining open the valve without
assessing its potential impacts on health and safety [31, p. 332]. The CCPS notes, in “Guidelines for Design Solutions for Process
Equipment Failures,” that locking open a valve is “not merely a common sense decision; rather at an operating facility it is a design
change. It is a procedural design solution that requires a documented design basis and a subsequent safety review” [73, p. 27].
118
The table above outlines similarities between the multiple vessel overflow events in the 2005 BP Texas City
incident and the 2022 BP Toledo Refinery incident, including inadequate overflow detection, lack of engineered
controls to prevent overflow, lack of adequate operating procedures, and inadequate shift handover. The final
recommendation of the report of the BP U.S. Refineries Independent Safety Review Panel in 2007 (“the Baker
Panel Report”) stated: “BP should use the lessons learned from the Texas City tragedy and from the Panel’s
report to transform the company into a recognized industry leader in process safety management” [2, p. 257]. a
The occurrence of the 2022 incident at the BP Toledo Refinery is evidence that BP failed to implement vessel
overflow findings from the BP Texas City incident.
Among the recommendations issued from the Baker Panel Report was a recommendation that BP should
involve the relevant stakeholders to develop a positive, trusting, and open process safety culture within each
U.S. refinery and “measure the effectiveness of this effort to “improve process safety culture by conducting
periodically an anonymous process safety culture survey among the U.S. refineries” [2, pp. 249-250]. b The
Baker Panel Report found that “significant safety issues” existed at all five of BP’s U.S. refineries, not just the
Texas City refinery. The report indicated that significant portions of the Toledo workforce at that time did not
believe that process safety was a core value and that BP had a weak process safety culture at Toledo.
Specifically, the report stated: “Toledo has a weak safety culture, largely because of chronic morale problems
and a history of poor relations between refinery management and the unionized workforce” [2, p. 118]. The
report also stated: “At Toledo, higher levels of management typically stated that decisions regarding production
and cost savings did not override process safety concerns, but that belief tended to change in the middle and
lower ranks of the Toledo organization. Many lower and middle managers interviewed expressed skepticism
about whether process safety concerns came first. Toledo hourly workers interviewed widely believed that
production was a higher priority than process safety” [2, pp. 61-62].
During the investigation of the September 20, 2022, incident, BP told the CSB it was unaware of any process
safety culture assessment completed at the BP Toledo Refinery since the Baker Panel Report. c
The BP Texas City incident is one of several publicly investigated incidents that are so iconic and impactful that
they should become part of the basic knowledge of everyone across the chemical industry [41, p. 193].d In its
2020 book, Driving Continuous Process Safety Improvement from Investigated Incidents, the CCPS explains for
incidents like BP Texas City:
Even if a person cannot recite the details, if they are in the industry, they should
be able to share the collective sense of vulnerability. They should then use that
sense of vulnerability to motivate safe work and dedication to completing all
tasks with professionalism. […] If any of the gaps that led to these incidents
a
See The Report of the BP U.S. Refineries Independent Safety Review Panel.
b
See The Report of the BP U.S. Refineries Independent Safety Review Panel.
c
The CSB did not issue a recommendation to develop, implement, and maintain and effective process safety culture assessment program,
such as prescribed by the California Code of Regulations, Title 8, Section 5189.1. Process Safety Management for Petroleum
Refineries, due to the change of owner and operator from BP to Ohio Refining Company LLC effective February 28, 2023.
d
In addition to the fatal 2015 explosion and fire at the BP Texas City Refinery, the CSB has investigated other incidents involving BP,
including a fatal March 2001 incident at the BP Amoco Polymers plant in Augusta, Georgia and the fatal April 2010 blowout and
explosion at the Macondo off-shore well in the Gulf of Mexico (where BP was the main operator/lease holder responsible for the well
design). In calling on the CSB to investigate the Macondo incident, the U.S. House Committee on Energy and Commerce stated that the
“CSB’s past work on BP puts it in a unique position to address questions about BP’s safety culture and practices.” (Letter from US
Representatives Henry A. Waxman and Bart Stupak to the CSB Chairman, dated June 8, 2010) [65, p. 11].
119
currently exist within our operations, we should make it a priority to eliminate

them [41, p. 193].
After BP Texas City, ExxonMobil sensed that “vulnerability” and concluded that there was no clear guidance
available to the industry to evaluate the potential risk associated with a liquid overfill of process and storage
vessels [42, p. 1]. a Consequently, ExxonMobil developed a Liquid Overfill Risk Assessment Tool to analyze
and prioritize liquid overfill risks using various probability and consequence factors. The tool included
evaluating the probability of effective operator intervention. ExxonMobil analyzed approximately 500 pressure
vessels and found that 30 percent required some type of mitigation to reduce risk to an acceptable level [42, pp.
8-9].
One reason that BP did not have effective safeguards in place at the BP Toledo Refinery to prevent overflowing
multiple pressure vessels may be that there is a lack of good industry guidance, which will always supersede any
corporate guidance or companies “forgetting” and becoming complacent again. b
After the BP Texas City incident, the API updated API Recommended Practice 521, Guide for Pressure
Relieving and Depressuring Systems to ensure the guidance:
a. identifies overfilling vessels as a potential hazard for evaluation in selecting
and designing pressure relief and disposal systems, and
b. addresses the need to adequately size disposal drums for credible worst-case
liquid relief scenarios, based on accurate relief valve and disposal collection
piping studies [43, p. 247].
The API Recommended Practice 521 guidance, however, is limited to preventing potential hazards of overfilling
vessels when selecting and designing emergency-pressure relief and disposal systems. The API does not have
any other guidance to prevent the potential hazards of overfilling pressure vessels. c
The CSB concludes that a publicly available industry publication outlining recognized and generally accepted
good engineering practices to prevent the overflow of pressure vessels is needed to help drive safety
improvements in chemical processing units and refineries across the United States. Had such good practice
guidance been in place during the design and operation of the Coker Gas Plant, the September 20, 2022, fatal
incident may have been prevented.
a
An attribute of a committed process safety culture is “maintaining a sense of vulnerability” [21, p. 4]. (See Section 3.4)
b
After the 2005 raffinate splitter tower overflow incident, BP experienced another distillation tower high level incident at its Whiting,
Indiana refinery. On December 13, 2005, during unit startup, a distillation tower overfilled, leading to the filling of a flare knockout
drum and liquid flow into the fuel gas system, causing flames to shoot out of two furnace fire boxes. A newly installed level transmitter
was found to have failed, leading to overfilling the tower while the tower outlet valve was closed. Other pressure indicators on the tower
that could have provided additional information on the high-level condition were not working. The BP investigation team
recommendations included installing an additional level indicator and repairing the two malfunctioning pressure indicators [31, p. 107].
c
The API does have (1) overfill prevention guidance for storage tanks. See ANSI/API Standard 2350, Overfill Prevention for Storage
Tanks in Petroleum Refineries | Fifth Edition, September 2020, Errata 1, April 2021 and (2) overfill prevention guidance for Offshore
Production Platforms. See API Recommended Practice 14C, Analysis, Design, Installation, and Testing of Basic Surface Safety Systems
for Offshore Production Platforms | Eighth Edition, February 2017, Errata 1, May 2018.
120
The CSB recommends the American Petroleum Institute (API) develop a publication that addresses preventing
the overflow of pressure vessels such as towers and drums. The publication should be applicable to both new
and existing pressure vessels. Include the following at a minimum:
vessel overflow,
investigation throughout the document. (See Recommendation 2022-01-I-OH-6).
121
5 Conclusions
5.1 Findings
Incident Progression
1. Water carryover from the Crude 1 Overhead Accumulator Drum initiated a cascade of events that caused the
emergency pressure-relief valve in NHT Preheat to open.
2. Vibration from the NHT Preheat emergency pressure-relief valve chatter caused the leak in NHT Preheat.
3. Had the BP Toledo Refinery bypassed the affected heat exchangers in NHT Preheat in response to the initial
leak, it might have avoided the pipe failure. The pipe failure caused an emergency shutdown of the NHT
unit.
4. The NHT unit emergency shutdown necessitated bypassing the Coker Gas Plant, but the Coker Gas Plant
was not fully isolated, even though its operational state was considered “in bypass.” Bypassing the Coker
Gas Plant left an open flow path from the Absorber Stripper Tower to the Fuel Gas Mix Drum.
5. There was only one destination available for Crude 1 Overhead flow after the NHT unit was shut down and
the Coker Gas Plant was bypassed.
6. The Crude 1 Overhead Accumulator Drum level could be managed under normal conditions with only one
destination available by sending it to Light Virgin Naphtha Storage, but the upset or excess flow conditions
in the Crude 1 Tower in this incident exceeded the control valve capacity to Light Virgin Naphtha, meaning
that the Crude 1 Overhead Accumulator Drum level could not be well controlled.
7. As a result of the NHT unit emergency shutdown, the Crude 1 Tower operation was unstable throughout the
day of the incident. This was demonstrated by high Crude 1 Tower overhead pressure with Crude 1 Tower
emergency pressure-relief valves opening multiple times, multiple losses of pumparound cooling, and
inability to control Crude 1 Overhead Accumulator Drum level.
8. On the day of the incident, BP Toledo Refinery personnel involved in the afternoon meeting regarding
Crude 1 Tower instability did not adequately communicate the guidance to safely operate the Crude 1
Tower from that meeting to the oncoming night shift personnel. This left night shift board operators to
decide how to operate the tower under the given conditions. Had the BP Toledo Refinery considered this
process instability that occurred throughout the day, and had there been better communications through shift
change, there could have been safeguards put in place before or early in the night shift.
9. The night shift board operator removed all light crude feed from the Crude 1 Tower in an effort to reduce
the overhead naphtha flow, in response to the limited destinations available for naphtha.
10. Rapidly eliminating all the light crude oil feed to the Crude 1 Tower initiated another process upset during
the night of the incident. This change created a rapid increase of vapor flow up the tower and led to 1) high
level in the Crude 1 Overhead Accumulator Drum, 2) loss of pumparound cooling, and 3) increased
temperatures throughout the tower.
122
11. The board operators did not have clear instructions about how to manage Crude 1 Overhead Accumulator
Drum high level. With only one destination available for naphtha, and the control valve to Light Virgin
Naphtha at maximum capacity, the improvised solution by the board operators was to transfer excess Crude
1 Overhead Accumulator Drum level to the bypassed Coker Gas Plant.
12. Opening the naphtha flow control valve to the Coker Gas Plant while the bypass valves were open allowed
liquid naphtha to flow into the Coker Gas Plant and then overflow into the Fuel Gas Mix Drum and proceed
to furnaces and boilers.
13. The closed ROEIV on the Absorber Stripper Tower bottoms caused the overflow through the Coker Gas
Plant bypass line to the Fuel Gas Mix Drum to occur more quickly than it otherwise would have, giving
board operators less time to troubleshoot and respond.
14. The BP Toledo Refinery recognized the potential for furnace fires or explosions if liquid entered the fuel gas
systems.
15. The two BP employees who released naphtha from the Fuel Gas Mix Drum to the ground may have
believed that the material was an amine-water solution, just as other operations personnel did.
Draining and Liquid Release
16. The Fuel Gas Mix Drum drain piping did not have enough capacity to drain naphtha overflowing from the
open Coker Gas Plant in a closed system to the Flare Knockout Drum.
17. Limiting or stopping the flow of naphtha to the Coker Gas Plant would have been required to prevent an
overflow of the Fuel Gas Mix Drum since more naphtha could flow into the Fuel Gas Mix Drum through the
Coker Gas Plant bypass than could be removed to a closed system.
18. The Absorber Stripper Tower emergency pressure-relief valves and the refinery furnaces’ safety
instrumented systems would not prevent a vessel overflow event. Instead, they just protected equipment
after an overflow has already occurred.
19. Had the BP Toledo Refinery recognized the likelihood of liquid overflow to the Fuel Gas Mix Drum, it
could have implemented more effective preventive safeguards, such as a high level interlock to close the
naphtha feed valve to the Absorber Stripper Tower. Such an interlock would have automatically stopped the
identified liquid overflow events instead of relying on alarms that require human intervention, emergency
pressure-relief valves, and downstream safety instrumented systems.
20. Had the BP Toledo Refinery implemented additional preventive safeguards to prevent liquid overflow from
the Coker Gas Plant to the fuel gas system, the incident in September 2022 may not have happened.
21. Manually draining vessels was a common task for outside operators at the BP Toledo Refinery.
22. Had the BP Toledo Refinery 1) conducted a thorough risk assessment of the operational task of draining or
addressing high level in the Fuel Gas Mix Drum, 2) provided its operators with the necessary written
123
instructions and consistent training, and 3) ensured the competency of operations personnel to perform the
task safely, BP employees may have made different decisions on September 20, 2022.
23. Although the API Recommended Practice 556 Instrumentation, Control, and Protective Systems for Gas
Fired Heaters provides industry guidance for alarms and protective functions to address process hazards
associated with the accumulation of combustibles in gas fired heaters, API RP 556 lacks guidance to
implement preventive safeguards for liquid overflow from a fuel gas mix drum which may lead to a
flameout and rapid accumulation of combustibles in gas fired heaters.
24. Had industry guidance for preventive safeguards, such as safety instrumented systems and controls, been
available to prevent liquid overflow from the Fuel Gas Mix Drum, and had the BP Toledo Refinery
incorporated such guidance, the BP Toledo Refinery could have eliminated reliance on human intervention
to drain liquid from the Fuel Gas Mix Drum.
25. The BP Toledo Refinery Abnormal Situation Management policy was not effective for proactive recognition
of abnormal situations. The policy narrowly defined abnormal situations such that process disturbances that
occurred before the incident did not fit the policy’s definition, even though the basic process control system
was unable to cope with these situations.
26. On the day of the incident, the BP Toledo Refinery did not provide effective guidance for managing
abnormal situations. The lack of effective guidance required operations personnel to make improvised real-
time decisions. Had effective guidance been communicated to the night shift, the incident could have been
avoided.
27. The BP Toledo Refinery did not effectively use previously existing industry guidance, such as that available
from the ASM Consortium, to develop its ASM policy. Had the refinery done so prior to the incident, it
could have had a framework to provide effective guidance to the night shift to safely operate the refinery
after the NHT shutdown.
28. Had the BP Toledo Refinery more effectively used Abnormal Situation Management tools and methods,
such as Predictive Hazard Identification, Process Control Systems, Policies and Administrative Procedures,
Operating Procedures, Training and Drills, Learning from Previous Abnormal Situation Incidents, and
Management of Change, the large number of cascading abnormal situations might have been stopped and
the fatal incident could have been prevented.
124
Alarm Flood
29. The board operators experienced an alarm flood condition for nearly 12 hours preceding the incident,
experiencing more than 3,700 alarms during the 12-hour period. Alarm flood contributed to the incident by
overwhelming and distracting the board operators, causing delays and errors in responding to critical alarms.
Had the night shift board operators been able to recognize the source of liquid in the fuel gas system during
the alarm flood and closed the naphtha flow control valve to the Coker Gas Plant, they could have stopped
liquid flow to the Fuel Gas Mix Drum, preventing or mitigating the fatal incident.
30. On September 20, 2022, the BP Toledo Refinery alarm performance was classified as “challenging,”
according to BP’s guidance and the high extent and duration of alarm flood likely contributed to the incident
by overloading the board operators, contributing to miscommunications, errors, and missed alarms.
31. On September 20, 2022, the BP Toledo Refinery alarm performance did not meet industry guidance,
exceeding 10 alarms in 10 minutes for hours at a time. The high extent and duration of alarm flood
contributed to the incident by overloading the board operators, contributing to miscommunication, errors,
and missed alarms, ultimately leading to the fatal incident. The high alarm rate was also indicative of
ongoing abnormal situations.
32. While ISA 18.2 provides guidance and a performance target for an alarm flood over a period of at least 30
days, no additional targets are provided for items such as the number of alarm floods in a month, duration of
each flood, alarm count in each flood, or peak alarm rate for each flood. Had such targets been established
in industry guidance, the BP Toledo Refinery could have analyzed and improved alarm flood performance
following a process upset, such as that occurred in the hours preceding the incident.
33. Had the BP Toledo Refinery more fully utilized some of the available alarm flood management techniques
in ISA 18.2 prior to the fatal incident, such as deadband and on-delay/off-delay, the alarm flood on the day
of the incident would have been more manageable and board operators could have prevented or stopped the
flow of liquid naphtha to the Fuel Gas Mix Drum.
34. The new ‘Toledo Alarm Philosophy’ follows ISA Standard 18.2 guidance but contains the same gap as the
guidance: it does not include short-term targets for items such as number of alarm floods, duration of each
flood, alarm count in each flood, or peak alarm rate for each flood. Such targets could provide the refinery
with more appropriate tools to analyze and improve alarm performance in an alarm flood that lasts several
hours, such as what occurred in the hours preceding the incident.
35. The facts, conditions, and circumstances of the 2019 incident show that while operating the Coker Gas Plant
with the bypass open, the BP Toledo Refinery did not have adequate safeguards to prevent overflow of
naphtha from the Coker Gas Plant to the Fuel Gas Mix Drum.
36. The BP Toledo Refinery 2019 Five Whys incident investigation focused only on action items to prevent
plugging in the Lean Oil Stripper overhead system, failing to learn important safety lessons from the 2019
incident. The BP Toledo Refinery did not perceive the need to issue recommendations related to overflow of
naphtha through the Coker Gas Plant bypass to the Fuel Gas Mix Drum or the abnormal Fuel Gas Mix Drum
125
level instrument readings identified in the 2019 Initial Incident Report, which could have prevented the
September 20, 2022, incident.
37. A publicly available industry publication outlining recognized and generally accepted good engineering
practices to prevent the overflow of pressure vessels is needed to help drive safety improvements in
chemical processing units and refineries across the United States. Had such good practice guidance been in
place during the design and operation of the Coker Gas Plant, the September 20, 2022, fatal incident may
have been prevented.
5.2 Cause
The CSB determined the cause of the incident was operators opening valves and removing a flange on the
pressurized Fuel Gas Mix Drum to release a flammable liquid, naphtha, directly to the ground. After being
released to the ground, the flammable liquid formed a vapor cloud that reached a nearby ignition source
resulting in a flash fire.
Contributing to the incident were 1) the refinery’s failure to implement effective preventive safeguards for the
overflow of towers and vessels in various pieces of equipment which led to an over-reliance on human
intervention to prevent incidents; 2) the refinery’s failure to implement a shutdown or hot circulation through the
use of Stop Work Authority or otherwise; 3) the refinery’s ineffective policies, procedures, and practices to
avoid and control abnormal situations; 4) the refinery’s alarm system which flooded operators with alarms
throughout the day resulting in poor decision making; and 5) the refinery’s failure to learn from previous
incidents.
126
6 Recommendations
To prevent future chemical incidents, and in the interest of driving chemical safety excellence to protect
communities, workers, and the environment, the CSB makes the following safety recommendations:
6.1 Ohio Refining Company LLC

2022-01-I-OH-R1
Revise the safeguards used in the refinery’s process hazard analyses high level and overflow scenarios. At a
minimum, establish effective preventive safeguards that use engineered controls to prevent liquid overfill and do
not rely solely on human intervention.
2022-01-I-OH-R2
Revise the Abnormal Situation Management policy to incorporate guidance provided by the ASM Consortium
and the Center for Chemical Process Safety (CCPS). The revised policy should include, at a minimum:
d) A broader definition of abnormal situations, such as that defined by the CCPS,
e) Additional predictable abnormal situations and their associated corrective procedures. At a minimum
f) Guidance to determine when an abnormal situation is becoming too difficult to manage and the
implementing proper procedures for bringing it to a safe state.
2022-01-I-OH-R3
Develop and implement a policy or revise existing policy that clearly provides employees with the authority to
stop work that is perceived to be unsafe until the employer can resolve the matter. This should include detailed
procedures and regular training on how employees would exercise their stop work authority. Emphasis should
be placed on exercising this authority during abnormal situations, including alarm floods.
2022-01-I-OH-R4
Revise the ‘Toledo Alarm Philosophy’ by incorporating the Engineering Equipment and Manufacturers Users
Association (EEMUA) guidance for alarm rate following an upset and not limiting alarm performance to a
127
single metric averaged over a month. In addition to including analyzing individual alarm flood events, the
revised philosophy document should improve refinery alarm performance to reduce alarm flood duration and
peak rate for events similar to the September 20, 2022, incident. Consult EEMUA Publication 191, Chapter
6.5.1, for guidance regarding abnormal condition performance levels. Apply the improved performance levels
where applicable, but specifically to the Crude 1 control board alarm performance.
6.2 American Petroleum Institute (API)

2022-01-I-OH-R5
Develop a new publication or revise an existing publication, such as API Recommended Practice 556
Instrumentation, Control, and Protective Systems for Gas Fired Heaters, to incorporate the process hazards
associated with Fuel Gas Mix Drum overflow. The publication should include the following at a minimum:
Drums.
2022-01-I-OH-R6
Develop a publication that addresses preventing the overflow of pressure vessels such as towers and drums. The
publication should be applicable to both new and existing pressure vessels. Include the following at a minimum:
vessel overflow,
investigation throughout the document.
128
6.3 International Society of Automation (ISA)

2022-01-I-OH-R7
Revise American National Standard ANSI/ISA 18.2-2016, Management of Alarm Systems for the Process
Industries, to include performance targets for short-term alarm flood analysis so that users can evaluate alarm
flood performance for a single alarm flood event. The performance targets should include:
that require alarm performance improvement actions.
129
7 Key Lessons for the Industry

To prevent future chemical incidents, and in the interest of driving chemical safety excellence to protect
communities, workers, and the environment, the CSB urges companies to review these key lessons:
Stop Work Authority
1. Companies must ensure (through training, clearly written procedures, and other means) that employees not
only are clearly empowered to exercise Stop Work Authority, but that employees also clearly understand
they are expected to do so. However, companies should not rely on Stop Work Authority programs alone to
prevent a catastrophic process incident since they require humans to take action to shut down a job or a
process. Stop Work Authority is not a substitute for effective process safety management systems.
2. PHA scenarios should consider both preventive and mitigative safeguards and not unrealistically rely on
human intervention.
3. PHAs should evaluate overfill hazards and consider scenarios in which a vessel may not overfill to the top
but may instead overflow or backflow though other piping connections.
4. Companies should evaluate their PHAs for opportunities to implement additional safeguards to prevent
initiating events that reduce the reliance on human intervention.
5. Companies should define operating limits beyond which Abnormal Situation Management procedures
should be followed and clearly define those corrective actions to be followed, in order to stop a chain of
abnormal events.
6. “Abnormal situations introduce stress, and operators under stress can make poor decisions, which then
exacerbate the situation. How companies prepare and equip their operators to deal with these problematic
and stressful situations is critical to ensuring the return of the unit to a safe state. Often, process safety
incidents are a result of organizations failing in this area.” — CCPS, Guidelines for Managing Abnormal
Situations [1, p. 24].
7. Thinking through abnormal situations before they occur, having plans in place, and practicing those plans
can greatly improve operator and manager confidence and decision-making during an abnormal situation.
Simulators, desktop drills, incident reviews, or field walkthroughs can improve abnormal situation
management skills.
8. Managing abnormal situations goes beyond PSM compliance alone. PSM tools can be used, but for
abnormal situations, the tools must be applied with abnormal situations specifically in mind, particularly
cascading abnormal situations.
Alarm Flood
130
9. Companies should ensure alarms are well justified. While DCS technology allows alarms to be created
easily, it also can cause board operators to be inundated with low priority or irrelevant information during an
abnormal situation if alarms are not properly designed. This also places additional stress on board operators,
reducing their effectiveness when it is needed most.
10. Accident investigation techniques, such as Five Whys, suggest that there is only one root cause and one
linear path to an accident [24, p. 291]. Using such approaches for process safety incidents, even if
recommended in company safety procedures, can lead an investigation team to a superficial analysis that
does not prevent an accident from recurring.
11. Organizations should develop systems to ensure that learnings from internal and external incidents are
incorporated throughout the organization to prevent recurring failures, such as overflow of process vessels,
that can lead to a catastrophic incident.
131
8 References
[1] Center for Chemical Process Safety (CCPS), Guidelines for Managing Abnormal Situations, New York: John Wiley & Sons, Inc.,
2023.
[2] The BP U.S. Refineries Independent Safety Review Panel, "The Report of the BP U.S. Refineries Independent Safety Review
Panel," January 2007. [Online]. Available: http://sunnyday.mit.edu/Baker-panel-report.pdf. [Accessed 5 October 2022].
[3] BP, "Welcome to BP-Husky Refining LLC!," [Online]. Available:
https://web.archive.org/web/20230613223044/http://toledobp.com/pages/history.aspx. [Accessed 13 June 2023].
[4] BP, "BP United States Where We Operate Ohio," [Online]. Available:
https://web.archive.org/web/20220923204045/https://www.bp.com/en_us/united-states/home/where-we-operate/ohio.html.
[Accessed 23 September 2022].
[5] Husky Energy Inc., "Husky Energy," 31 March 2008. [Online]. Available:
https://huskyenergy.com/downloads/newsreleases/2008/HSE_033108_Husky_Completes_JV_Agreement.pdf. [Accessed 12
January 2023].
[6] Cenovus Energy Inc., "News releases," 4 January 2021. [Online]. Available: https://www.cenovus.com/News-and-Stories/News-
releases/2021/2152436. [Accessed 28 February 2023].
[7] Cenovus Energy Inc., "News releases," 28 February 2023. [Online]. Available: https://www.cenovus.com/News-and-Stories/News-
releases/2023/2617620. [Accessed 28 February 2023].
[8] R. Prestvik, K. Moljord, K. Grande and A. Holmen, "Compositional Analysis of Naphtha and Reformate," in Catalytic Naphtha
Reforming, Second Edition Revised and Expanded, New York, NY, Marcel Dekker, Inc., 2004, p. 2.
[9] B. Grover and P. Di Zanno, "Study examines use of refinery fuel gas for hydrogen production," Oil & Gas Journal, vol. 105, no.
24, 25 June 2007.
[10] Code of Federal Regulations, "40 CFR 63.641 Definitions; National Emission Standards for Hazardous Air Pollutants: Petroleum
Refineries," 18 August 1995. [Online]. Available: https://www.ecfr.gov/current/title-40/chapter-I/subchapter-C/part-63/subpart-
CC/section-63.641. [Accessed 15 February 2023].
[11] H. Pandya and A. M. Fantolini, "Limit the rate of change of fuel gas properties," Hydrocarbon Processing, pp. 67-70, March 2016.
[12] BP, BP Process Safety Series - Safe Furnace and Boiler Firing, (Fifth Edition), Institute of Chemical Engineers (IChemE), 2012.
[13] BP, BP Process Safety Series - Hazards of Oil Refining Distillation Units, Institution of Chemical Engineers (IChemE), 2014.
[14] W. L. Leffler, Petroleum Refining in Nontechnical Language, (Fifth Edition), Nashville: PennWell, 2020.
[15] The Brattle Group, "Understanding Crude Oil and Product Markets," Americal Petroleum Institute (API), 2014.
[16] BP-Husky, "Risk Management Plan Submission for BP-Husky | EPA Facility Identifier 1000 0008 8683," 2018.
[17] U.S. Department of Labor (USDOL), "Final Rule on Process Safety Management of Highly Hazardous Chemicals; Explosives and
Blasting Agents," 26 May 1992. [Online]. Available: https://www.osha.gov/laws-regs/federalregister/1992-02-24. [Accessed 14
August 2019].
[18] U.S. Environmental Protection Agency (EPA), "Risk Management Program (RMP) Rule," [Online]. Available:
https://www.epa.gov/rmp. [Accessed 14 August 2019].
[19] U.S. Environmental Protection Agency (EPA), "Risk Management Program (RMP) Rule Overview," [Online]. Available:
https://www.epa.gov/rmp/risk-management-plan-rmp-rule-overview. [Accessed 14 August 2019].
[20] U.S. Environmental Protection Agency (EPA), "General RMP Guidance - Chapter 2: Applicability of Program Levels," [Online].
Available: https://www.epa.gov/sites/production/files/2013-10/documents/chap-02-final.pdf. [Accessed 14 August 2019].
[21] Center for Chemical Process Safety (CCPS), Essential Practices for Creating, Strengthening, and Sustaining Process Safety
Culture, New York: John Wiley and Sons, Inc., 2018.
[22] American Institute of Chemical Engineers (AIChE), "Safety Instrumentation Systems," [Online]. Available:
https://www.aiche.org/topics/process-safety-technical-areas/safety-instrumentation-
systems#:~:text=A%20safety%20instrumented%20system%20(SIS,process%20hazards%20in%20your%20plant. [Accessed 22
February 2024].
[23] G. Salvendy, Handbook of Human Factors and Ergonomics, (Fourth Edition), New York: John Wiley & Sons, Inc., 2012.
[24] N. G. Leveson, An Introduction to System Safety Engineering, Cambridge: The MIT Press, 2023.
[25] Center for Chemical Process Safety (CCPS), CCPS Monograph: Human Factors Primer for Front Line Leaders, New York: John
Wiley and Sons, Inc., 2023.
[26] Energy Institute, Guidance on human factors safety critical task analysis (Second Edition), London: Energy Institute, 2020.
132
[27] The Process Technology and Operator Academy, "Flame Management 101 - Flameout," 3 September 2015. [Online]. Available:
https://www.processtechacademy.com/flame-management-
101/#:~:text=A%20flameout%20condition%20is%20present,explosion%20of%20the%20fired%20heater. [Accessed 27 March
2024].
[28] American Petroleum Institute (API), "Instrumentation, Control, and Protective Systems for Gas Fired Heaters | API Recommended
Practice 556 | Second Edition, April 2011, Reaffirmed, April 2019," 2019.
[29] P. T. Bullemer, ASM Consortium Guidelines: Effective Operations Practices, Houston: ASM Consortium, 2019.
[30] Honeywell International, Inc., "ASM Guidelines," 2024. [Online]. Available: https://process.honeywell.com/us/en/site/asm-
consortium/get-the-guidelines. [Accessed 3 April 2024].
[31] U.S. Chemical Safety Board (CSB), "Investigation Report, Refinery Explosion and Fire, BP, Texas City, TX," 2007.
[32] Center for Chemical Process Safety (CCPS), Guidelines for the Safe Automation of Chemical Processes, (Second Edition), New
York: John Wiley & Sons, Inc., 2017.
[33] J. Errington, D. V. Reising and C. Burns, ASM Consortium Guidelines: Effective Alarm Management Practices, ASM Consortium,
2009.
[34] International Society of Automation (ISA), ANSI/ISA 18.2-2016 Management of Alarm Systems for the Process Industries,
International Society of Automation, 2016.
[35] Engineering Equipment and Materials Users Association (EEMUA), "Alarm Systems: Guide to design, management and
procurement. Publication 191, Edition 3," EEMUA, London, 2013.
[36] A. Hopkins, Failure to learn: the BP Texas City refinery disaster, Sydney: CCH Australia, 2008.
[37] Center for Chemical Process Safety (CCPS), Recognizing Catastrophic Incident Warning Signs in the Process Industries, New
[38] T. Kletz, Lessons From Disaster: How Organizations Have No Memory and Accidents Recur, Houston: Gulf Publishing Company,
1993.
[39] T. Kletz, Learning from Accidents, (Third Edition), Woburn, Massachusetts: Butterworth-Heinemann, 2001.
[40] Brent Coon & Associates, "The Evidence," Febuary 2007. [Online]. Available:
http://www.texascityexplosion.com/uploads/1/1/6/4/116433877/bonse_main_report.pdf. [Accessed 22 June 2023].
[41] Center for Chemical Process Safety (CCPS), Driving Continuous Process Safety Improvement from Investigated Incidents, New
[42] R. L. Wong, "Risk Assessment Tool for Liquid Overfill of Process and Storage Vessels," in 18th Annual International Symposium,
Mary Kay O’Connor Safety & Risk Conference, College Station, 2015.
[43] American Petroleum Institute (API), "Pressure-Relieving and Depressuring Systems | API Standard 521 | Seventh Edition, June
2020," 2020.
[44] "Census Reporter," [Online]. Available: https://censusreporter.org/. [Accessed 11 August 2023].
[45] United States Census Bureau, "Explore Census Data," U.S. Department of Commerce, 3 August 2023. [Online]. Available:
https://data.census.gov/. [Accessed 18 August 2023].
[46] BP, "Heritage Brands," [Online]. Available: https://www.bp.com/en/global/corporate/who-we-are/our-history/heritage-
brands.html#accordion_History%20of%20Sohio. [Accessed 26 April 2023].
[47] National Institute for Occupational Safety and Health (NIOSH), "Hierarchy of Controls," 17 January 2023. [Online]. Available:
https://www.cdc.gov/niosh/topics/hierarchy/default.html. [Accessed 6 September 2023].
[48] Center for Chemcial Process Safety (CCPS), "Process Safety Beacon," April 2013. [Online]. Available:
https://www.aiche.org/sites/default/files/cep/20130424.pdf. [Accessed 10 October 2023].
[49] BP, BP Process Safety Series - Hazardous Substances in Refineries, Institution of Chemical Engineers (IChemE), 2005.
[50] D. A. Crowl and S. A. Tipler, "Sizing Pressure Relief-Devices," Chemical Engineering Progress, pp. 68-76, October 2013.
[51] A. Ehtesham, "What is the difference between Class 150, 300 & 600 flange," 18 April 2017. [Online]. Available:
https://www.trupply.com/blogs/news/109162438-what-is-the-difference-between-class-150-300-600-flange. [Accessed 18 April
2024].
[52] S. Bridges, "Abnormal Modes of Operation – A Risk-Based Focus," in Sixteenth Global Congress of Process Safety, Houston,
2020.
[53] M. Hellemans, Safety Relief Valve Handbook - Design and Use of Process Safety Valves to ASME and International Codes and
Standards, Oxford: Elsevier Ltd, 2009.
[54] American Institute of Chemical Engineers (AIChE), "Center for Chemical Process Safety (CCPS) Glossary - Safeguard," [Online].
Available: https://www.aiche.org/ccps/resources/glossary/process-safety-glossary/safeguard. [Accessed 4 March 2024].
[55] American Institute of Chemical Engineers (AIChE), "Center for Chemical Process Safety (CCPS) Glossary - Flash Fire," [Online].
Available: https://www.aiche.org/ccps/resources/glossary/process-safety-glossary/flash-fire. [Accessed 15 March 2024].
133
[56] BP p.l.c., "BP.com Legal Notice," 2022. [Online]. Available: https://www.bp.com/en/global/corporate/legal-notice.html. [Accessed
4 April 2024].
[57] D. P. Nolan, Handbook of Fire and Explosion Protection Engineering Principles for Oil, Gas, Chemical and Related Facilities, Gulf
Professional Publishing, 2019.
[58] International Society of Automation (ISA),
Sector, Washington: American National Standards Institute (ANSI), 2018.
[59] Americal Society of Safety Engineers, "American National Standard - Risk Assessment Techniques," Americal Society of Safety
Engineers, Des Plains, IL, 2011.
[60] C. Mullins-Jaime, S. Case and J. K. Wachter, "An Integrated Approach for Understanding, Correcting & Preventing Unsafe Acts,"
Professional Safety Journal, pp. 21-27, February 2021.
[61] Marsh & McLennan, "Remotely Operated Emergency Isolation Valves (ROEIVS)," September 2020. [Online]. Available:
https://www.marsh.com/content/dam/marsh/Documents/PDF/UK-en/remotely-operated-emergency-isolation-valves.pdf. [Accessed
28 September 2023].
[62] I. Sutton, Plant Design and Operations, (Second Edition), Cambridge: Elsevier, 2017.
[63] American Petroleum Institute (API), "Process Measurement | API Recommended Practice 551 | Second Edition, February 2016,"
2016.
[64] B. G. Liptak, Ed., Instrument Engineers' Handbook, (Fouth Edition), Process Measurement and Analysis, Volume I, Boca Raton:
CRC Press, 2003.
[65] U.S. Chemical Safety Board (CSB), "FINAL REPORT: Executive Summary of Explosion and Fire at the Macondo Well," 2016.
[66] American Fuel & Petrochemical Manufacturers (AFPM), "AM-10-118 Situation Awareness for Refining and Petrochemical
Process Operators – Not by Technology Alone," Human Centered Solutions, LLC , [Online]. Available:
https://www2.afpm.org/forms/store/ProductFormPublic/am-10-118-situation-awareness-for-refining-and-petrochemical-process-
operators-not-by-technology-alone. [Accessed 24 May 2024].
[67] American Institute of Chemical Engineers (AIChE), "Center for Chemical Process Safety (CCPS) Glossary - Interlock," [Online].
Available: https://www.aiche.org/ccps/resources/glossary/process-safety-glossary/interlock. [Accessed 4 May 2024].
[68] American Institute of Chemical Engineers (AIChE), "Center for Chemical Process Safety (CCPS) Glossary- Layer of Protection
Analysis (LOPA)," [Online]. Available: https://www.aiche.org/ccps/resources/glossary/process-safety-glossary/layer-protection-
analysis-lopa. [Accessed 18 February 2024].
[69] American Institute of Chemical Engineers (AIChE), "Center for Chemical Process Safety (CCPS) Glossary - BPCS," [Online].
Available: https://www.aiche.org/ccps/resources/glossary/process-safety-glossary/basic-process-control-system-bpcs. [Accessed 22
March 2024].
[70] American Institute of Chemical Engineers (AIChE), "Center for Chemical Process Safety (CCPS) Glossary - Safety Instrumented
Functions (SIF)," [Online]. Available: https://www.aiche.org/ccps/resources/glossary/process-safety-glossary/safety-instrumented-
functions-sif. [Accessed 20 November 2022].
[71] American Institute of Chemical Engineers (AIChE), "Center for Chemical Process Safety (CCPS) Glossary - HAZOP," [Online].
Available: https://www.aiche.org/ccps/resources/glossary/process-safety-glossary/hazard-and-operability-study-hazop. [Accessed
15 August 2023].
[72] D. C. Hendershot, "Inherently Safer Design: An Overview of Key Elements," American Society of Safety Professionals, 2011.
[Online]. Available: https://aeasseincludes.assp.org/professionalsafety/pastissues/056/02/048_055_F2Hendershot_0211Z.pdf.
[Accessed 7 October 2023].
[73] Center for Chemical Process Safety (CCPS), Guidelines for Design Solutions for Process Equipment Failures, New York: John
Wiley & Sons, Inc., 1998.
134
Appendix A: Timelinesa
A.1 Naphtha Hydrotreater Release
A.1.1 2018
A.1.1.1 FEBRUARY 16, 2018
A pipe failure occurred at a process unit at BP’s Whiting Refinery located in Indiana. The failure occurred on a
branch fitting that connected an emergency pressure-relief valve inlet line with the main process, in liquid
service, running between two heat exchangers (See Figure A1-1).b The emergency pressure-relief valve
experienced instability (e.g., chattering, cycling, and/or fluttering) during relief demands. The failure resulted in
the release of flammable hydrocarbons. The unit was shut down and de-pressured to enable the leak to be
isolated. There were no injuries or damage to other equipment.
Figure A1-1. Failed branch fitting from the BP Whiting Refinery. (Credit: BP)
a
Various sources of information were relied upon to construct this incident timeline. These sources include control system information,
radio logs and event logs. All times are approximate to actual time vs. source time.
b
This report uses the term “emergency pressure-relief valve”; however, the terms pressure relief valve, safety relief valve, pressure
safety valve (PSV), relief valve, or safety valve can be used interchangeably. For a specific application, however, readers should know
that these other names can reflect different operating characteristics and using precise terminology for a specific application may be
appropriate.
135
A.1.1.2 SEPTEMBER 6, 2018
BP’s Safety and Operational Risk group issued a High Value Learning entitled “Effect of excessive forces on
mechanical integrity of relief systems subject to pulsation” to all BP downstream refineries and petrochemical
plants to share key findings of the investigation of the February 16, 2018, Whiting refinery incident. The key
findings include:
The system as designed led to frequent excursions above 90 percent of emergency pressure-relief valve
set pressure and numerous relief valve demands.
The emergency pressure-relief valve experienced instability (e.g., chattering, cycling, and/or fluttering)
during relief demands.
During the incident, the emergency pressure-relief valve was relieving the full liquid flow due to a
blocked-in pump, creating large reaction forces/stresses due to fluid momentum while cycling (opening
and closing).
Other downstream refineries were given action items to identify the emergency pressure-relief valves covered
by the High Value Learning and develop a mitigation plan. A screening process was prescribed with the
objective of identifying emergency pressure-relief valves in pumped liquid service that had the potential for
failure due to dynamic loading from instability during a relief event. For these emergency pressure-relief valves,
further integrity assessment requirements and appropriate mitigations were specified. Each refinery also had an
action item intended to help identify and avoid the design of future installations for this failure scenario.
136
A.1.2 2019
A.1.2.1 NOVEMBER 2, 2019
A reduction in the feed rate to the NHT unit at the BP Toledo Refinery resulted in a temperature rise exceeding
the maximum allowable working temperature (MAWT) of two heat exchangers, HE-6 and HE-7, a in NHT
Preheat b (See Figure A1-2). The temperature excursions lasted approximately four hours.
Over the previous four years, the exchanger HE-7 had multiple temperature excursions above the MAWT. c
Figure A1-2. Heat exchangers that experienced temperature excursions. (Credit: BP with annotations by CSB) d
A.1.2.2 NOVEMBER 8, 2019
A fitness for service e evaluation was performed on the two NHT preheat exchangers that experienced
temperature excursions on November 8, 2019. The fitness for service evaluation determined that the equipment
was not damaged by the previous temperature excursions, even though exchangers 6 and 7 exceeded their
MAWT by 29 °F and 34 °F, respectively.
a
The shell side design conditions for HE-6 were 680 psig at 532 °F. The shell side operating conditions during the upset were 540 psig at
561 °F. The shell side design conditions for HE-7 were 680 psig at 550 °F. The shell side operating conditions during the upset were
540 psig at 584 °F.
b
c
If the MAWT was 580 °F, all but five temperature excursions would have been eliminated and if the MAWT was 610 °F all but one
temperature excursion would have been eliminated.
d
In the NHT Preheat diagram, emergency pressure-relief valves are represented with PSV, heat exchangers with HE, and pressure
transmitters with PI.
e
Fitness for service is a best practice and standard (API RP 579-1/ASME FFS-1) used by the oil & gas and chemical process industries
for in-service equipment to determine its fitness for continued service.
137
A.1.3 2021
A.1.3.1 MAY 2021
BP identified piping issues while reviewing the scope of work around PSV-E, which protected the shell side of
the HE-6 and associated piping (See Figure A1-2). The set pressure of PSV-E was higher than the existing
piping components’ maximum design pressure.
The PSV-E emergency pressure-relief valve set pressure was 680 psig, while the piping design pressure was 635
psig at 550 °F. To achieve a pressure of 680 psig for the existing piping the design temperature would have to be
lowered to 463 °F. This was not feasible due to the average inlet operating temperature of 491°F.
BP identified that the operating pressure was too close to the 680 psig emergency pressure-relief valve setpoints
of PSV-D and PSV-E since the operating pressure at times would be above 600 psig. For example, the operating
pressure margin for the shell side outlet was 91.2 percent (in this instance, an operating pressure of 620 psig) of
the PSV-E set pressure. a The emergency pressure-relief valve manufacturer recommends the margin should be
below 90 percent (in this instance, 612 psig) to “minimize leakage and spurious opening” of the conventional
emergency pressure-relief valves.
BP determined that re-rating b the piping was not feasible during the 2022 turnaround (TAR) because the
schedule was set and material would not be available. Instead, the project team identified a different option to
replace PSV-D and PSV-E with soft seat emergency pressure-relief valves.
A MOC is approved to add a block valve and bleed valve on both the inlet and outlet of the PSV-E to be
installed off the outlet of HE-6.
A.1.3.2 JUNE 2021
A project was submitted on June 8, 2021, to address fitness for service of HE-6 and HE-7 that had previously
experienced temperature excursions above the MAWT. The project scope is to rerate the HE-6 shell to 630 °F
and add a backing ring, and to replace HE-7 with a shell rated at 750 °F.
a
The operating margin is the difference between the set pressure and the maximum operating pressure. The required operating margin
depends on the type of relief device and the pressure control capability of the process [50, p. 69].
b
An inherently safer design approach of re-rating the piping components to a higher temperature and pressure is a robust and reliable
method of containment [72].
138
A.1.3.3 JULY 2021
A project was approved to add a second block valve and bleed valve on the inlet and outlet of conventional
emergency pressure-relief valve PSV-B. a The project was included in the 2022 TAR scope.
A.1.3.4 AUGUST 2021
A project was approved to modify PSV-B by replacing the 2.5-inch conventional emergency pressure-relief
valve with a larger three-inch conventional emergency pressure-relief valve, and to lower the set point from 681
to 675 psig. This project was included in the 2022 TAR scope.
On August 20, 2021, a scope of work for the 2022 TAR was issued to develop a MOC to eliminate threaded
piping connections on the Crude 1 Overhead Accumulator Drum in order to comply with “BP’s Safety
Requirements for LPG [Liquefied Petroleum Gas] Processing, Storage & Handling”. The project included
replacing the existing threaded level instrument with a new flanged guided wave radar level instrument to
measure the water level in the Crude 1 Overhead Accumulator Drum.
On August 21, 2021, BP determined that a “checklist style Risk Evaluation” was appropriate for the MOC for
this project. Physical equipment changes such as the piping thread elimination were included in the MOC
checklist risk evaluation. Changes to the way the level of water in the Crude 1 Overhead Accumulator Drum
will be reported to the operators (as a result of the new guided wave radar level instrument) were not included in
the MOC checklist risk evaluation.
A.1.3.5 NOVEMBER 2021
On November 16, 2021, BP approved a project scope to change PSV-D and PSV-E to a pilot-operated
emergency pressure-relief valve b and lower their setpoints to 635 psig to protect the heat exchangers’ piping.
The project scope stated: “The final resolution of this problem will require the replacement/upgrading of the
existing NHT Feed piping system from [class] c 300 [ …] flanges to [class] 600 […] flanges (approximately 600
[linear feet] of 8” piping). If needed, the next opportunity for this will be during the 2027 TIU TAR by a
separate capital project.”
a
A conventional emergency pressure-relief valve is a direct spring-loaded emergency pressure-relief valve that is held closed by a spring
force that can be adjusted within a certain range and whose operational characteristics are directly affected by changes in the
backpressure which is exercised at the outlet of the valve [53, p. 30].
b
PSV-D and PSV-E were changed from conventional emergency pressure-relief valves to pilot-operated emergency pressure-relief
valves “in order to allow the operating pressure to be within 98 percent of the new PSV setpoint (635 psig).” A pilot-operated
emergency pressure-relief valve has the inherent ability to maintain premium tightness close to the set pressure, allowing a higher
normal system-operating pressure than with a conventional emergency pressure-relief valve [53, p. 112].
c
The pressure rating of a flange ranges from 150# to 2500#. The term "lb," "class," and "#" are used interchangeably to designate the
pressure rating of the flange. The fact is that 150 “lb” has no relation to 150 pounds per square inch (psi) and so 300 or 600 “lb” does
not correlate to a 300 or 600 psi pressure rating. Pressure rating of the flange depends on the material, the heat treat condition and
pressure "class." The term "class" is used here to not confuse the designation with pressure "rating" [51].
139
A.1.3.6 DECEMBER 2021
A PHA was completed for the NHT unit. Previous PHAs were conducted on this unit in March 2017 and April
2012.
A.1.4 2022
A.1.4.1 APRIL 2022
The 2022 TAR began on April 20, 2022, with the Crude 1 unit and the NHT unit shutting down.
Figure A1-3 shows the 2022 TAR work to NHT Preheat which made modifications to heat exchanger HE-6,
replaced heat exchanger HE-7, replaced emergency pressure-relief valves PSV-D and PSV-E, and modified
PSV-B.
Figure A1-3. NHT Preheat changes made during 2022 TAR. (Credit: BP with annotations by CSB)
A.1.4.2 JULY 26, 2022
The 2022 TAR ended and the modified NHT Preheat started back up.
A.1.4.3 JULY 28, 2022
The Crude 1 unit started up.
A.1.4.4 AUGUST 27, 2022
PSV-D opened to the flare. Operators blocked in this emergency pressure-relief valve, allowed it to cool, and
then returned it to service.a The BP Toledo Refinery incident report description states: “PSV lifted while stable
operations were below lift set pressure.” b
a
PSV-D was replaced during the 2022 TAR (See Section A.1.4.1).
b
Lift is the actual travel away from the closed position when an emergency pressure-relief valve is relieving [53, p. 32].
140
A.1.4.5 SEPTEMBER 19, 2022
A.1.4.5.1 7:11 P.M.
Board Operator adjusted the position of a control valve operating in manual and regulating the water leaving the
Crude 1 Overhead Accumulator Drum from 54 percent open to 36 percent open.
A.1.4.5.2 10:40 P.M.
High level alarm for water in the Crude 1 Overhead Accumulator Drum sounded at 60 percent and
acknowledged by the board operator.
A.1.4.5.3 11:45 P.M.
The water level in the Crude 1 Overhead Accumulator Drum continued to increase and reached the top of the
internal naphtha standpipe, a resulting in water overflowing into the Crude 1 Overhead Accumulator Drum’s
naphtha stream and flowing to the naphtha preheat train early the next morning.
A.1.4.6 SEPTEMBER 20, 2022 (DAY OF INCIDENT)
A.1.4.6.1 2:12 A.M.
Naphtha and water from the Crude 1 Overhead Accumulator Drum flowed to the Coker Gas Plant Absorber
Stripper Tower resulting in water collecting in the Coker Gas Plant Absorber Stripper Tower Foul Condensate
Draw Off Drum (See Figure 6).b Once full, the Foul Condensate Draw Off Drum began to backflow and
returned water back to the Absorber Stripper Tower.
a
The indicated liquid level from the new guided wave radar instrument plateaued at 69 percent level, and water overflowed into the
standpipe, whereas before changes made during the recent 2022 TAR, water did not overflow into the standpipe until the level indicated
100 percent level. (See Section A.1.3.4.)
b
The Coker Gas Plant Absorber Stripper Tower Foul Condensate Draw Off Drum is shown in Figure 6.
141
A.1.4.6.2 2:45 TO 3:00 A.M.
Increasing amounts of water in the Absorber Stripper Tower caused the reboiler exit temperature to drop,
decreasing the vapor flow up the Absorber Stripper Tower. The lower vapor flow allowed the liquid on the
Absorber Stripper Tower trays to de-inventory and increased the bottoms level, which in turn increased the
liquid level in the NHT Feed Surge Drum. The level increase in the NHT Feed Surge Drum level resulted in a
flow surge to NHT Preheat.
A.1.4.6.3 APPROXIMATELY 3:20 A.M.
PSV-D opened.
The shift supervisor created an incident report of PSV-D opening in the BP Toledo Refinery incident report
database. a The report states “Operating pressure of the NHT feed system was allowed to get up to about 621
psig. The [PSV-D] set pressure is 635 [psig].”
A.1.4.6.5 APPROXIMATELY 4:30 A.M. TO 5:00 A.M.
Shift change occurred.
Outside operators began manually draining the water from the Crude 1 Overhead Accumulator Drum to the Oily
Water Sewer.
A.1.4.6.7 6:26 A.M.
PSV-D opened again. b
A.1.4.6.8 6:33 A.M. TO 6:57 A.M.
Feed rate to NHT Preheat was increased from approximately 7,900 barrels per day to about 13,200 barrels per
day, and the feed pressure increased from approximately 570 psig to 684 psig (See PI-1in Figure A1-3).
PSV-B was set to provide emergency pressure-relief at 675 psig.
A.1.4.6.9 6:50 A.M. TO 6:59 A.M.
BP’s control system event log recorded an alarm flood state greater than 10 alarms for every 10-minute period
between 6:50 a.m. and 6:49 p.m. except for five alarms from 7:20-7:29 a.m., 10 alarms from 2:00-2:09 p.m., 10
alarms from 2:10-2:19 p.m., and three alarms from 4:20-4:29 p.m. Between 6:50 a.m. and 6:49 p.m. a total of
3,712 alarms were recorded.
a
PSV-D had lifted previously on August 27, 2022 (See A.1.4.4) and had been replaced during the 2022 TAR (See A.1.4.1).
b
This was the second lifting of PSV-D that morning. The earlier lifting (See A.1.4.6.3) occurred at approximately 3:20 a.m.
142
A.1.4.6.10 6:57 A.M. TO 7:12 A.M.
The pressure of the fluid being sent to NHT Preheat began to cycle between 660 and 675 psig, likely the result
of PSV-B, opening and closing.
A.1.4.6.11 APPROXIMATELY 7:13 A.M. TO 7:28 A.M.
Outside operators radioed that an emergency pressure-relief valve was opening.
Outside operators observed PSV-B opening and chattering. a They attempted to manually reseat PSV-B, but it
continued to open and close. Outside operators described the emergency pressure-relief valve opening as
“incredibly bad” and “extremely loud” consistent with chattering.
A.1.4.6.12 7:31 A.M.
The flow rate to NHT Preheat reached 21,000 barrels per day.
PSV-D was manually re-seated by the outside operators and stopped opening.
An outside operator charged a fire monitor in preparation to quickly respond in case of an incident, as a result of
the issues they were experiencing. b
Outside operator found a leak on the ¾-inch bleed valve located at the low point on the tube side inlet for HE-2
as shown in Figure A1-4. PSV-B continued to chatter.
Outside operator started water flow from the previously charged fire monitor.
a
Chattering “is the rapid opening and closing of a pressure-relief valve. The resulting vibration may cause misalignment, valve seat
damage, and, if prolonged, mechanical failure of valve internals and associated piping” [48].
b
Fire monitors are devices used for manual firefighting or in automatic fire protection systems to “discharge large volumes of water and
have good straight stream range. Discharge can be controlled by the type and size of adjustable nozzle or diameter of straight stream
nozzle” [62, p. 369]. The process of “charging” a fire monitor involves filling piping from a nearby water source and placing it under
pressure so that the water can be directed, through a nozzle, toward the fire site when needed.
143
Figure A1-4. Location of loss of containment. (Credit: BP with annotations by CSB)
Additional personnel were sent to NHT Preheat to troubleshoot and help.
Operations requested the ERT to be on standby due to the emergency pressure-relief valve chattering.
A.1.4.6.17 8:06 A.M.
Shift superintendent made a radio call stating, “we’re going to have to shut the gas plant [Sat Gas Plant] down
… unless we can get all the feed out of it” in order to by-pass the piping section that was leaking.
A.1.4.6.18 8:08 A.M.
The feed rate to the NHT unit was 16,200 barrels per day.
A.1.4.6.19 8:12 A.M.
Outside operators reported a significant loss of containment when the ¾-inch bleed valve severed from the
piping near the PSV-B inlet (See Figure A1-5).
144
Figure A1-5. Failed ¾-inch branch fitting from the BP Toledo Refinery. (Credit: CSB)
A.1.4.6.20 8:13 A.M.
Outside operations called on radio requesting an emergency shutdown and ERT assistance. The ERT arrived
shortly after requested and remained available to respond.
A.1.4.6.21 8:15 A.M. TO 9:30 A.M.
At approximately 8:15 a.m., the NHT shutdown was initiated by shutting fuel gas to the NHT feed furnace, and
at approximately 8:17 a.m., the NHT unit started to be de-pressured to the flare. By approximately 9:30 a.m., the
feed to the NHT unit had stopped.
A.1.4.6.22 10:27 A.M.
The ERT was released from the unit.
A.1.4.6.23 11:00 A.M.
The final entry for the Incident Activity Log states: “Radio All Call—Muster All Clear no work in process
block.”
145
A.2 Fuel Gas Mix Drum Release
A.2.1 2007
The Report of the BP U.S. Refineries Independent Safety Review Panel (“the Baker Panel Report”) was issued
in 2007. The Baker Panel Report indicated that significant portions of the Toledo workforce did not believe that
process safety was a core value and found BP had a weak process safety culture at Toledo. Specifically, the
report stated: “At Toledo, higher levels of management typically stated that decisions regarding production and
cost savings did not override process safety concerns, but that belief tended to change in the middle and lower
ranks of the Toledo organization. Many lower and middle managers interviewed expressed skepticism about
whether process safety concerns came first. Toledo hourly workers interviewed widely believed that production
was a higher priority than process safety” [2, pp. 61-62]. The report also stated: “Toledo has a weak safety
culture, largely because of chronic morale problems and a history of poor relations between refinery
management and the unionized workforce” [2, p. 118].
A.2.2 2011
Figure A2-1 shows how high liquid level in the Crude 1 Overhead Accumulator Drum from process upsets is
routed to the Flare Knockout Drum after the Fluid Catalytic Cracker (FCC) unit wet gas compressor is isolated
from the Crude 1 Overhead Accumulator Drum overhead flow.
Figure A2-1. Crude 1 Overhead Accumulator Drum prior to the installation of the Coker Gas Plant. This
simplified schematic shows how high level of liquid in the Crude 1 Overhead Accumulator Drum liquid can
overflow to the Flare Knockout Drum. (Credit: CSB)
146
A.2.3 2015
A.2.3.1 MAY 2015
A Relief & Overpressure Inherently Safer Design Principles Toledo Fuels Optimization (TFO) Sulfur Reduction
project specific document that outlines the fundamental overpressure scenarios and develops an inherently safer
design approach to a relief and overpressure protection design for the TFO Sulfur Reduction project was issued
for design. A Basic Process Control System (BPCS) was determined not to be needed for liquid overfill of the
Coker Gas Plant Absorber Stripper Tower even though hydrocarbon would flow into the Fuel Gas System.
A.2.3.2 JUNE 2015
A project design PHA was completed for the new Coker Gas Plant.
A.2.4 2016
A.2.4.1 JANUARY 2016
BP committed the capital to build the Coker Gas Plant.
A.2.4.2 JUNE 2016
The Fuel Gas Mix Drum was replaced. a The new Fuel Gas Mix Drum is approximately the same size as the
previous Fuel Gas Mix Drum with a Maximum Allowable Working Pressure (MAWP) of 150 psig and an
emergency pressure-relief valve set pressure of 90 psig. The emergency pressure-relief valve was “installed to
adequately relieve any overpressure scenario that will be appliable from [the Coker Gas Plant].”
A.2.4.3 SEPTEMBER 2016
During the September 2016 Coker Gas Plant LOPA, a decision was made to delete the following safeguard: “the
DCS to command FV3816 to close, based on a high level in the CGP Absorber Stripper using independent
transmitter.” An identified consequence for the deviation of high level is a “higher level resulting in potential
overfill, overpressure, damage, leak, H2S exposure, fire, injury/fatalities.”
A.2.5 2018
A.2.5.1 FEBRUARY 2018
The 2015 Coker Gas Plant project PHA was updated to include changes in design since the previous review.
The BP PHA team met the regulatory requirements of 29 C.F.R. § 1910.119(e)(4). There was no process
operator in attendance.
a
The prior Fuel Gas Mix Drum had an MAWP of 64 psig and an emergency pressure-relief valve set pressure of 60 psig. Multiple over
pressure scenarios had been identified and the prior fuel gas mix drum could no longer be equipped with an emergency pressure-relief
valve to provide adequate relief.
147
A.2.5.2 APPROXIMATELY SEPTEMBER 2018
Training for operations was conducted before the Coker Gas Plant started up. Training consisted of one day of
classroom training, pairs of operators being assigned specific activities to perform on their own to familiarize
themselves with the new Coker Gas Plant, and qualification by their supervisor. Operators were able to observe
construction activities.
A.2.5.3 OCTOBER 2018
An Operational Readiness Review (ORR) for the Coker Gas Plant was completed, and then the unit started up
and was brought online soon afterward on October 22.
A.2.6 2019
A.2.6.1 MARCH 2019
A PHA revalidation was completed for Crude 1, which included the Fuel Gas Mix Drum. The previous PHA for
Crude 1 was performed in March 2014.
A.2.6.2 MAY 2019
A six-inch butterfly control valve replaced a six-inch manual gate valve in the Coker Gas Plant bypass line. The
control valve controls the Coker Gas Plant Polishing Amine Contactor differential pressure by bypassing coker
wet gas around the Coker Gas Plant. The Coker Gas Plant Polishing Amine Contactor had seen foaming, the
onset of jet flooding, and amine carryover at differential pressure greater than eight pounds per square inch.
A.2.6.3 NOVEMBER 2019
On November 13, 2019, a boiler trip occurred during a cold front which resulted in a refinery-wide loss of steam
and the Coker Gas Plant being bypassed. A high level occurs in the Crude 1 Overhead Accumulator Drum when
the only pump to remove liquid from the drum available at the time was a steam-driven pump whose
performance was compromised by the loss of steam.
To lower the level in the Crude 1 Overhead Accumulator Drum the naphtha flow control valve to the Coker Gas
Plant was manually opened to 80–100 percent range. Liquid flowed through the Coker Gas Plant but was
blocked when high pressure developed due to the Lean Oil Stripper overhead system getting too cold and
forming a hydrate.
Using a Five Whys investigation methodology, the investigation team determined that “naphtha carried over
through the [Coker Gas Plant] unit bypass a filled up the KO drum and the [Fuel Gas Mix Drum].” a The Fuel
Gas Mix Drum’s liquid level indication reached 100 percent on the guided wave radar level instrument, and the
a
When investigating unsafe acts or resulting incidents, companies will often use root-cause analysis aiming, to focus their attention on
fixing whatever is determined to be the singular root cause of the unsafe act or incident. Top industry choices for investigation techniques
are cause-and-effect models, such as fishbone diagrams and Five Whys methods. Cause-and-effect models that only seek a single root
cause can be problematic. While the intent is positive and admirable, this approach often fails to recognize that human decision-making is
complex and diverse [60, p. 25].
148
differential pressure (dP) liquid level instrument indication rose and flatlined at 65 percent. Outside operators
manually drained the Fuel Gas Mix Drum to the Flare Knockout Drum and Oily Water Sewer. The naphtha flow
to the Coker Gas Plant was stopped, and the Fuel Gas Mix Drum was eventually emptied of liquid naphtha.
Recommendations were made to prevent low temperature on the Lean Oil Stripper overhead system.
A.2.7 2020
A.2.7.1 FEBRUARY 2020
A PHA revalidation was completed for the Coker Gas Plant. The PHA included a review of prior Coker Gas
Plant incidents. The PHA team did not review the November 2019 incident of naphtha overflow through the
Coker Gas Plant bypass directly to the Fuel Gas Mix Drum since the Fuel Gas Mix Drum is part of the Crude 1
PHA (See A.2.6.3).
A.2.8 2021
A.2.8.1 JUNE 2021
A new trainer was assigned to the Crude 1 unit. The trainer had previous experience as an operator, supervisor,
and refinery coordinator a but had never worked as an outside operator in the Crude 1 unit.
A.2.8.2 AUGUST 19, 2021
The guided wave radar level indicator on the Fuel Gas Mix Drum stopped reporting readings to the refinery’s
computerized process data historian.b The Fuel Gas Mix Drum guided wave radar level indicator continued to
read the actual measured level in the control room DCS.
A.2.9 2022
A.2.9.1 APRIL 20, 2022
The 2022 TAR began with the Crude 1 and the NHT units shutting down.
A.2.9.2 JULY 2022
The 2022 TAR ended and the modified NHT Preheat train (as described in Appendix A.1) started up on July
26, 2022.
Crude 1 started up on July 28, 2022.
a
A refinery coordinator is required to be qualified on all board operator jobs, including the Crude 1 unit. Additional detail on the refinery
coordinator role can be found in Section 1.9.
b
The configuration for recording the history of this process variable was not restored until after the incident.
149
A.2.9.3 AUGUST 2022
Operational issues resulted in losing all three pumparounds on the Crude 1 Tower (about one or two weeks after
starting up) after a furnace tripped. Crude 1 Tower overhead fin fans were cleaned.
A.2.9.4 SEPTEMBER 20, 2022 (DAY OF INCIDENT)
A.2.9.4.1 3:11 A.M.
The level in Crude 1 Tower Overhead Accumulator Drum began to increase from 55 percent and reached 73
percent by 4:30 a.m. a
A.2.9.4.2 7:26 A.M.
The control valve in the Coker Gas Plant bypass line was placed in manual mode and began to be opened by the
board operator (See A.2.6.2).
A.2.9.4.3 8:09 A.M.
The control valve in the Coker Gas Plant bypass line reached 100 percent open, where it remained.
A.2.9.4.4 8:12 A.M.
The differential pressure level indicator on the Fuel Gas Mix Drum began increasing (starting at a reported level
of zero percent full) and reached a maximum level of 23 percent at approximately 8:18 a.m.
Crude 1 naphtha flow control valve to the NHT unit was closed, indicating the start of the NHT unit shutdown.
A.2.9.4.5 8:42 A.M.
Board operator began to close the naphtha flow control valve to the Coker Gas Plant.
A.2.9.4.6 9:17 A.M.
The Crude 1 Overhead Accumulator Drum naphtha flow to the Coker Gas Plant stopped, after the naphtha flow
control valve to the Coker Gas Plant was fully closed.
A.2.9.4.7 9:47 A.M.
The differential pressure level indicator on the Fuel Gas Mix Drum read zero percent.
A.2.9.4.8 10:20 A.M.
The Crude 1 Tower overhead pressure exceeded the lowest emergency pressure-relief valve set pressure (38
psig) for the first time. The overhead pressure exceeded 38 psig 11 times altogether on day shift, with the last
a
The level fluctuated throughout the day shift with levels varying between 6.9 percent and 87.8 percent.
150
overpressure occurring at approximately 4:15 p.m. The Crude 1 Tower overhead line includes five emergency
pressure-relief valves, with staggered set pressures between 38 and 40 psig.
A.2.9.4.9 1:26 P.M.
The outside operator was directed to close the Coker Gas Plant Absorber Stripper Tower bottoms outlet
ROEIV. a,b
A.2.9.4.10 1:30 P.M.
The Coker Gas Plant Absorber Stripper Tower bottoms outlet ROEIV was closed by the outside operator.
A.2.9.4.11 1:37 P.M.
The two-inch flow control valve from the Crude 1 Overhead Accumulator Drum to the Light Virgin Naphtha
Storage was fully opened and remained fully opened until 7:04 p.m. A four-inch manual bypass valve around
the two-inch flow control valve remained closed.c
The flow meter from the Crude 1 Overhead Accumulator Drum to the Light Virgin Naphtha Storage reached the
instrument top of range of 10,000 barrels per day and remained there until 2:01 p.m. At 2:23 p.m., the flow
meter returned to the instrument top of range of 10,000 barrels per day and remained there until 3:32 p.m.
A.2.9.4.12 3:58 P.M.
The Crude 1 Tower Overhead pressure exceeded the emergency pressure-relief valve set pressure for the last
time on day shift. The pressure remained above the emergency pressure-relief valve set pressure until 4:17 p.m.
A.2.9.4.13 APPROXIMATELY 4:15 P.M.
Approximate time the shift change period started for turnover from day shift to shift personnel. Crude 1 night
shift personnel arrived anywhere from 4:14 p.m. to 4:49 p.m.
A.2.9.4.14 4:56 P.M.
The light crude oil flow to Crude 1 was reduced from 26,000 barrels per day, eventually reaching a flow of 0
barrels per day by 5:11 p.m. Even though the light crude oil flow feeding Crude 1 went to zero, the pump
remained turned on.
a
In recent reports, the CSB has referred to equipment needing to isolate a flammable or toxic release from a safe (remote) location as a
remotely operated emergency isolation valve (ROEIV). Other industry standards, good practice guidance, or earlier CSB reports have
described similar isolation equipment using different names, including emergency block valve (EBV), emergency isolation valve (EIV),
remotely operated block valve (RBV), emergency shutdown valve (ESDV), or a remotely operated shutoff valve (ROSOV).
b
While a ROEIV will not prevent a loss of containment event, proper application of remote isolation equipment could mitigate the
severity of a hazardous chemical release [61].
c
The four-inch manual bypass valve had a sticker on it that read “Open with Caution.”
151
A.2.9.4.15 5:39 P.M.
DCS high level alarm at 68 percent and high-high level alarm at 70 percent for the Crude 1 Overhead
Accumulator Drum sounded and was displayed on the DCS alarm summary screen.
A.2.9.4.16 5:41 P.M.
The Crude 1 Tower lost the flow in the bottom pumparound.

At the same time, the level in the Crude 1 Overhead Accumulator Drum reached 89 percent. One board operator
told the other board operator to open the naphtha flow control valve to the Coker Gas Plant to use the NHT Feed
Surge Drum for naphtha storage to reduce the high level in the Crude 1 Overhead Accumulator Drum. The
naphtha flow control valve to the Coker Gas Plant was fully opened and then settled in at 79 percent open at
5:45 p.m. until closed at 7:05 p.m.
Once the naphtha flow control valve to the Coker Gas Plant was opened, a the measured flow of the liquid
naphtha rate went above the instrument’s top range of 18,700 barrels per day. The flow remained above the
instrument’s top range until 6:42 p.m. and then reduced from 17,800 barrels per day down to 0 at 6:55 p.m.
Since the Coker Gas Plant Absorber Stripper Tower bottoms outlet ROEIV was closed (See A.2.9.4.10), the
Absorber Stripper Tower level began to increase at 5:45 p.m.
The flow meter, from the Crude 1 Overhead Accumulator Drum to the Light Virgin Naphtha Storage, returned
to the instrument’s top of range of 10,000 barrels per day. The flow rate generally remained over 9,900 barrels
per day until 6:42 p.m.
A.2.9.4.17 5:42 P.M.
A crude oil pump seal was reported to be leaking crude oil (See A.2.9.4.14). The pump, lined up to light crude
oil storage, was running with no flow beginning at 5:11 p.m.
The crude oil pump was turned off after the crude oil leak was discovered.
A.2.9.4.18 5:47 P.M.
The Crude 1 Tower lost flow in the middle pumparound.
A.2.9.4.19 5:53 P.M.
The alarms for high level from the Coker Gas Plant Absorber Stripper Tower level controller and independent
level transmitter sounded.
A.2.9.4.20 6:06 P.M.
A board operator shelved b Coker Gas Plant Absorber Stripper Tower high level alarms for the level controller
and independent level transmitter.
a
The BP Toledo Refinery process data historian shows the control valve output went directly from minus five percent fully closed to 105
percent fully open, then to 85 percent open for two minutes, before settling out at 79 percent open.
b
Shelving alarms caused the alarms to no longer appear on the DCS active alarm page.
152
A.2.9.4.21 6:09 P.M.
Liquid naphtha began to flow into and quickly accumulated inside the Fuel Gas Mix Drum. The naphtha was
flowing through the Coker Gas Plant bypass line, after the level of naphtha reached tray 37 in the Coker Gas
Plant Absorber Stripper Tower (See Section 1.8).
A.2.9.4.22 6:10 P.M.
The high level alarm for the guided wave radar level transmitter sounded at 10 percent and the high level alarm
for the Fuel Gas Mix Drum differential pressure level transmitter sounded at a level of six percent.
A.2.9.4.23 6:12 P.M.
The Crude 1 Tower lost the flow in the top pumparound.
A.2.9.4.24 6:14 P.M.
The high differential pressure alarm for the Coker Gas Plant Absorber Stripper Tower sounded at 15 psi.
A.2.9.4.25 6:15 P.M.
The high-high level alarm for the Fuel Gas Mix Drum guided wave radar level transmitter sounded at 85
percent.
A.2.9.4.26 6:16 P.M.
A board operator radioed to outside operators to check the level in the Fuel Gas Mix Drum, “Check ASAP!”
The Fuel Gas Mix Drum differential pressure level indicator reached approximately 67 percent and plateaued. a
The differential pressure indicator for the liquid level gave a false indication that the liquid level is below the
six-foot level measurement span of the differential pressure indicator and deviated from the guided wave radar
indicator, which was already alarming at a high-high level. The differential pressure indicator never alarmed at
the high-high liquid level (See A.2.9.4.25).
A.2.9.4.27 6:17 P.M.
Three outside operators quickly responded to the report of liquid level in the Fuel Gas Mix Drum. They first
confirmed that the high liquid level in the Fuel Gas Mix Drum existed. Then one operator went immediately to
“the valve going to the flare and opened it all the way.” Next, they opened the valve to drain liquid to the Oily
Water Sewer.
An outside operator radioed that the Fuel Gas Mix Drum was draining.
a
The liquid (naphtha) in the Fuel Gas Mix Drum had a specific gravity of approximately 0.67. However, the Fuel Gas Mix Drum
differential pressure level indicator had been calibrated using a specific gravity of 0.998 (at 60 °F).
153
The shift supervisor mentioned to the outside operator to “get air by.” a The shift supervisor also told the outside
operator to keep draining to the sewer and not let the liquid splash out, and to follow the drained liquid with
water to dilute the liquid “and to keep vapors down too.”
A.2.9.4.29 6:20 P.M.
Crude 1 Furnace started smoking black smoke.
A.2.9.4.30 6:21 P.M.
The level indicator in the Coker Gas Plant Sweet Gas Knockout Drum reported 100 percent. An outside operator
went to this drum and manually held open the drum’s “dead man valve” b and drained the liquid to a spent amine
line.
An outside operator reported on the radio that the Fuel Gas Mix Drum “level is above the sight glass.” A board
operator responded with “copy that, just drain it [the Fuel Gas Mix Drum] as fast as you guys can.” The outside
operator responded, “we are.”
A board operator acknowledged the high-high Fuel Gas Mix Drum guided wave radar level transmitter level
alarm.
A.2.9.4.31 6:22 P.M.
Black smoke from the Crude 1 Furnace was reported on the radio. The CO Boiler started smoking green smoke.
Fuel gas was cut from the Crude 1 Furnace.
A.2.9.4.32 6:23 P.M.
Black and yellow smoke was observed on camera “coming from the units.”
A.2.9.4.33 6:24 P.M.
Crude 1 Furnace stopped smoking.
A.2.9.4.34 6:25 P.M.
The shift superintendent issued an “All call” for the ERT, a full mobilization of the refinery’s emergency
responders.
a
To “get air by” was later explained to CSB investigators to mean having air (an SCBA) located “close to them if they needed it” (not
necessarily over the head and hanging around the neck or actually donning the SCBA mask), because “the rich amine product is pretty
heavy H2S.”
b
A “dead man valve,” also known as a spring closing lever valve, is a manual valve that a human has to hold open by use of the lever
handle, which will automatically close when the lever is released. Typically used to prevent leaving a valve unattended, such as during
manual draining. Can be defeated by securing the handle in the open position [52, p. 25].
154
A.2.9.4.35 6:26 P.M.
A furnace in the diesel hydrotreater unit started smoking black smoke.
Two BP employees, an outside operator and operator trainee who were brothers, drained the Fuel Gas Mix
Drum to the Flare Knockout Drum and the Oily Water Sewer. They were both observed to be wearing an SCBA
prior to the incident. One SCBA was provided by another outside operator who yelled to “put the SCBA on”.
The shift supervisor later explained to CSB investigators that the SCBAs were out of a concern that hydrogen
sulfide could be present in any amine-water solution drained from the Fuel Gas Mix Drum.
A.2.9.4.37 6:29 P.M.
The area LFL detector a reached 2.8 percent LFL, which was the first reading above zero.
The Crude 1 Furnace started smoking again and the CO Boiler stopped smoking.
A.2.9.4.38 6:30 P.M.
The area LFL detector reached 11.6 percent LFL, which exceeded the detector’s high alarm setpoint of 10
percent LFL.
An audible horn alarm triggered by the LFL detector could be heard in the radio traffic. There was no LFL
alarm in the control room as the alarm was set to “Disabled” due to an Alarm Configuration Manager (ACM)
enforcement configuration error that was enforcing the alarm into the “disabled” state. Although the
audible and visual board alarming was disabled, the LFL detector readings would have still shown in the
control room.
A ¾-inch bleed valve on the Fuel Gas Mix Drum sight glass piping is opened by the operator and operator
trainee releasing liquid naphtha directly to the ground.
a
Most flammable gas detectors “give a reading of the %LEL (or %LFL)” [49, p. 29]. The lower flammability limit (LFL) is defined as
the “lowest concentration of a flammable gas in air capable of being ignited by a spark or flame” [49, p. 35]. This LFL detector was
located approximately 19 feet southwest of the Fuel Gas Mix Drum.
155
A.2.9.4.40 6:32 P.M.
Fuel Gas Mix Drum differential pressure level indication decreased immediately to zero percent when a ¾-inch
valve was opened by the outside operator and operator trainee releasing naphtha directly to the ground.
The outside operator and operator trainee used water from a hose to direct the liquid draining to the ground from
the Fuel Gas Mix Drum to a drain.
A.2.9.4.42 6:35 P.M.
A board operator radioed that it looks like the liquid “level might have broke” in the Fuel Gas Mix Drum.
However, this was a false understanding because the differential pressure level instrument indicated zero percent
because it had been opened and was releasing naphtha directly to the ground (See A.2.9.4.39).
A.2.9.4.43 6:36 P.M.
After a request for Fuel Gas Mix Drum status, an outside operator reported that a high liquid level in the Fuel
Gas Mix Drum was observed in the sight glass.
A blind flanged two-inch maintenance valve on the side of the Fuel Gas Mix Drum was opened and released
naphtha directly to the ground.
A.2.9.4.45 6:39 P.M.
An area LFL detector reached 100 percent LFL, then dropped down to 21 percent LFL.
A.2.9.4.46 6:40 P.M.
The area LFL detector’s audible alarm can be heard in the background during a radio call from an outside
operator.
A.2.9.4.47 6:43 P.M.
The area LFL detector reached 100 percent LFL and remained at 100 percent LFL.
An outside operator detected a “strong, strong smell” like a “distillate type naphtha or something” and observed
vapors being released from the Fuel Gas Mix Drum, much “like someone is draining product from the mix drum
[Fuel Gas Mix Drum], and I seen water being sprayed on it.”
A.2.9.4.49 6:46 P.M. (TIME OF IGNITION)
Vapor cloud ignited.
156
The outside operator, wearing his SCBA, was found on fire near the base of the Crude 1 Tower. A fellow
outside operator opened a nearby fire monitor, hosed water directly onto him, which quickly put out the fire on
the outside operator.
Flames on the operator trainee were also extinguished.
Since the operator trainee could still walk, he was led to the North pod a where colleagues administered first aid
until the trainee was evacuated from the BP Toledo Refinery by ambulance.
A board operator requested Emergency Medical Services (EMS) at the North pod. Additional EMS were
requested for the South pod.
A.2.9.4.53 6:56 P.M.
A blocked-in pipe in a nearby pipe rack ruptured, which provided additional fuel to the fire.
A.2.9.4.54 7:04 P.M.
A board operator closed the naphtha flow control valve to the Coker Gas Plant and the flow control valve from
the Crude 1 Overhead Accumulator Drum to Light Virgin Naphtha Storage.
A.2.9.4.55 7:15 P.M.
Oregon EMS transported one of the burned operators to a nearby hospital.
A.2.9.4.56 7:21 P.M.
A second ambulance transported the other burned operator to the same hospital.
A.2.9.4.57 8:31 P.M.
The fire was still burning on the south side of the Fuel Gas Mix Drum.
A.2.9.4.58 8:51 P.M.
The BP Toledo Refinery ERT accepted mutual aid from a nearby refinery, which sends one “rig with six to eight
firefighters.”
A.2.9.4.59 9:18 P.M.
Incident Activity Log entry states “Fire appears to be out […]”
a
Pods were blast resistant modular buildings located in the BP Toledo Refinery.
157
A.2.9.4.60 9:26 P.M.
Incident Activity Log entry states “Fire is confirmed out. Investigating for additional leaks.”
A.2.9.4.61 9:44 P.M.
Incident Activity Log entry states “investigating the new [fire] in the [north/south] pipe alley.”
A.2.9.4.62 10:10 P.M.
Incident Activity Log entry states “Command reports small fires are out.”
A.2.9.5 SEPTEMBER 21, 2022
A.2.9.5.1 12:18 A.M.
All ERT equipment and personnel were released from the area near the fires and returned to headquarters.
A.2.9.5.2 1:57 A.M.
All clear sent out and the refinery lock down was lifted.
158
Appendix B: Simplified Causal Analysis (AcciMap)
Figure B-1. AcciMap (Credit: CSB)
159
Appendix C: Description of Surrounding Area

Figure C-1 shows the seven census blocks within approximately five miles of the BP Toledo Refinery that the
CSB reviewed [44].
Figure C-1. Census blocks near the BP Toledo Refinery. (Credit: Census Reporter with annotations by CSB)
160
Table C-1 contains demographic data for these census blocks. In general, the population is predominantly white
with 23 percent of the population below the poverty level.
Table C-1. Tabulation of demographic data.
Number
Median
Block Median Race and Ethnicity of Types of Housing Units
Population Income Poverty
Number Age (%) Housing (%)
($) (%)
Units
68 White 83 Single Unit

14 Black 17 Multi-Unit
0 Native 0 Mobile Home
0 Asian 0 Boat, RV, van, etc.
1 2,169 37 36,534 25 1,044
0 Islander
7 Other
10 Two+
17 Hispanic
2 1,096 44 60,844 12 466
0 Islander
2 Other
7 Two+
7 Hispanic
3 3,871 45 66,887 3 1,770
0 Islander
3 Other
9 Two+
12 Hispanic
4 3,502 33 36,534 43 1,860
0 Islander
4 Other
13 Two+
17 Hispanic
161
Number
Median
Block Median Race and Ethnicity of Types of Housing Units
Population Income Poverty
Number Age (%) Housing (%)
($) (%)
Units

5 2,825 31 23,500 37 997
0 Islander
4 Other
7 Two+
9 Hispanic
6 2,247 40 57,976 16 945
0 Islander
3 Other
10 Two+
10 Hispanic
7 3,045 44 53,798 20 1,553
0 Islander
2 Other
7 Two+
8 Hispanic
162
Appendix D: OSHA Hazard Alert Letter
On March 15, 2023, OSHA issued a voluntary recommendation to BP stemming from the September 20, 2022,
incident. OSHA disclosed the following hazard(s) at the refinery:
The site utilizes a staffing pattern commonly referred to as job rotation at the refinery. The
policy of rotating process operators among multiple positions, instead of a single position,
can reduce the level of expertise and knowledge of operators on the unit for which they are
initially qualified. In the event of a process upset condition or catastrophic incident, this
decrease in expertise can negatively affect incident response efforts, posing a higher
likelihood of exposure to toxic vapor/gas, fire and explosion hazards.
In the interest of workplace safety and health and the lack of an applicable OSHA standard, OSHA recommended the
refinery voluntarily take steps to eliminate or materially reduce employee exposure to such hazards. OSHA stated
that feasible methods of control could include:
Conduct a feasibility study, including obtaining input from employees and employee
representatives, on the effectiveness of the job rotation staffing pattern in place at the
refinery, including determining the impact that such a job rotation policy has on operator
morale and ability to respond to process safety incidents.
163
Appendix E: OSHA Citations
Citation
Standard Cited Summary of Citation
Number
Citation 1 29 C.F.R. § a) the employer failed to ensure that it documented that equipment in the process
Item 1 1910.119(d)(3)(ii) complied with the employer's chosen recognized and generally accepted good
engineering practices such as but not limited to API 520 (2020) Section 5, when the
employer did not include an evaluation of excessive built-up backpressure for PSV-
1457 located on the tube side of the HVGO/Crude 1 Naphtha Intermediate Reflux
Shell & Tube heat exchangers (PR-544011/12). Failure to evaluate excessive built-up
back pressure resulted in the exposure of employees to fire and explosion hazards
from the release of flammable liquids or gasses.
b) the employer failed to ensure that it documented that equipment in the process
complied with the employer's chosen recognized and generally accepted good
engineering practices such as but not limited to API 520 (2020) and API 521 (2015),
when the employer did not include an evaluation of two phase flow and all potential
relief scenarios for the TIU Fuel Gas Mix Drum (PR-510253) PSV-1464 including
but not limited to overfilling of the drum. Failure to evaluate for the potential of two-
phase flow and all potential relief scenarios including but not limited to overfilling of
the drum, exposed employees to fire and explosion hazards from potential releases of
flammable liquids or gasses.
c) the employer failed to document that the level indicator instrumentation used in
conjunction with the TIU Fuel Gas Mix Drum (PR510253) complied with recognized
and generally accepted good engineering practices in that level instrumentation relied
on for controlling liquid level accumulation in the drum was not designed and utilized
for determining liquid levels of naphtha.
Citation 1 29 C.F.R. § the employer failed to identify, evaluate, and control the hazard of high liquid level
Item 2 1910.119(e)(1) resulting from all potential flammable liquid overfill scenarios including but not
limited to detecting flammables and stopping the flow of liquid naphtha to the TIU
Fuel Gas Mix Drum. The Crude 1 PHA did not include the Coker Gas Plant as a
source of liquid naphtha during an overfill scenario involving the Absorber Stripper
Tower (PR550025), Sour Gas KO Drum (PR510286), Polishing Amine Contactor
(PR550032), Sweet Gas KO Drum (PR510283) to the fuel gas header and the TIU
Fuel Gas Mix Drum. Failure to control the high level in the TIU Fuel Gas Mix Drum
resulted in a release of liquid naphtha, exposing employees to flammable vapor, fire,
hydrogen sulfide, and explosion hazards.
Citation 1 29 C.F.R. § the employer failed to evaluate the TIU Fuel Gas Mix Drum in the Crude 1 Unit for
Item 3 1910.119(e)(3)(iii) engineering or administrative controls needed to maintain drainage to a closed system
and prevent the manual draining of liquid in the drum to the sewer. Open draining of
liquid from the mix drum can expose employees to hydrogen sulfide, explosion and
fire hazards.
164
Citation
Number
Citation 1 29 C.F.R. § a) the employer failed to ensure that the PHA addressed the consequences of failure
Item 4 1910.119(e)(3)(iv) of administrative controls by not following the South/Coker Gas Plant: Bypassing and
Returning to Service Coker Gas Plant Procedure (CGP 02.004). The procedure
required closing FV-3816 while the Coker Gas Plant was in bypass mode. During the
incident, FV-3816 was opened causing the Absorber Stripper Tower (PR-550025)
and the downstream equipment including the TIU Fuel Gas Mix Drum (PR-510253)
to overfill with liquid naphtha. This led to a direct path of liquid naphtha, causing an
uncontrollable high level in the TIU Fuel Gas Mix Drum, exposing employees to fire,
explosion hazards, and toxic gases from potential releases of fuel gas, flammable
liquids, and hydrogen sulfide.
b) the employer failed to ensure that the PHA addressed the consequences of failure
of administrative controls by not following Coker Gas Plant and NHT Feed Surge
Drum Safe Operating and Design Limits which required steps to avoid an overfill
scenario of the Absorber Stripper Tower (PR-550025) to include verifying that the
Absorber Stripper Tower bottom valve (XV3821) was open. Keeping this valve
closed led to overfilling of the Absorber Stripper Tower (PR-550025) and the
downstream equipment including the TIU Fuel Gas Mix Drum (PR-510253). This led
to a direct path of liquid naphtha, causing an uncontrollable high level in the TIU
Fuel Gas Mix Drum, exposing employees to fire, explosion hazards, and toxic gases
from potential releases of fuel gas, flammable liquids, and hydrogen sulfide.
c) the employer failed to ensure that the PHA addressed the consequences of failure
of engineering controls when the high-level switch (LSH-805) was not available to
detect high level of flammables in the TIU Fuel Gas Mix Drum during an overfill
scenario of the Absorber Stripper Tower (PR-550025) and downstream vessels.
Failure to evaluate loss of engineering controls resulted in an uncontrollable high
level in the TIU Fuel Gas Mix Drum, exposing employees to fire, explosion hazards,
and toxic gases from potential releases of fuel gas, flammable liquids, and hydrogen
sulfide.
Citation 1 29 C.F.R. § the employer failed to ensure that the most recent PHA revalidation reflected current
Item 5 1910.119(e)(6) process equipment and conditions, in that the high level switch (LSH-805) was not
available to detect high level of flammables in the TIU Fuel Gas Mix Drum during a
high liquid level event. Failure to ensure that engineering control safeguard taken
credit for in PHA revalidations are in place and operational, can contribute to an
uncontrollable high level in the TIU Fuel Gas Mix Drum, exposing employees to fire,
explosion hazards, and toxic gases from potential releases of fuel gas, flammable
liquids, and hydrogen sulfide.
165
Citation
Number
Citation 1 29 C.F.R. § a) the employer failed to ensure that temporary operating procedures were developed
Item 6 1910.119(f)(1)(i)(C) and implemented for operation of the Crude 1 Unit while the Coker Gas Plant and
SatGas/NHT Units were intended to be in a bypass condition. The lack of operating
procedures for this transient operating condition contributed to carry-over of naphtha
to the TIU Fuel Gas Mix Drum and led to liquid in the fuel gas system, exposing
employees to fire and explosion hazards.
b) the employer failed to ensure that operating procedures developed for bypassing
the Coker Gas Plant were implemented. Failure to implement this established
procedure (CGP 02.004 - Bypassing and Returning to Service Coker Gas Plant)
contributed to FCV3816 being opened, allowing naphtha to overfill downstream
process vessels and flow into the refinery's fuel gas system, exposing employees to
fire and explosion hazards.
Citation 1 29 C.F.R. § a) the employer failed to ensure that emergency shutdown of process equipment in
Item 7 1910.119(f)(1)(i)(D) the NHT/Sat Gas units occurred when requested by outside operators. Upset
conditions involving the lifting and reseating of PSV-1457 and PSV-1462 caused
process equipment vibration and instability. A release of naphtha during this upset
condition exposed employees to explosion and fire hazards.
b) the employer failed to ensure that emergency operating procedures for the Crude 1
Unit Crude Tower (PR556936) included the scenario involving the loss of all three
process pumparounds on the tower simultaneously, which can inhibit process
temperature control and stable operation of the tower. Failure to shutdown the Crude
Unit during this process upset condition can expose employees to explosion and fire
hazards.
Citation 1 29 C.F.R. § a) the employer failed to ensure that procedure PSM 025 was implemented for the
Item 8a 1910.119(f)(1)(i)(E) closing of PSV-1457 in the SatGas/NHT Unit, for the purpose of reseating the relief
valve, exposing employees to explosion and fire hazards.
b) the employer failed to ensure that emergency operating procedures were developed
and implemented for the safe draining of liquid from the TIU Fuel Gas Mix Drum in
the Crude 1 Unit during process upset conditions. The lack of emergency operating
procedures for this condition exposed employees to hydrogen sulfide, explosion and
fire hazards.
Citation 1 29 C.F.R. § the employer failed to ensure that safe work practices were developed and
Item 8b 1910.119(f)(4) implemented for the safe draining of liquid from the TIU Fuel Gas Mix Drum in the
Crude 1 Unit during process upset conditions. The lack of safe work practices
developed and implemented for this work activity exposed employees to hydrogen
sulfide, explosion and fire hazards.
166
Citation
Number
Citation 1 29 C.F.R. § the employer failed to ensure that the procedure for Normal Operation of Crude 1
Item 9 1910.119(f)(1)(ii) (CRD1 02.012) included steps to avoid or correct deviations in process parameters,
including for liquid level in the TIU Fuel Gas Mix Drum. The lack of detailing steps
to avoid or correct deviations for all process parameters such as liquid level, exposed
employees to hydrogen sulfide, explosion and fire hazards.
Citation 1 29 C.F.R. § a) the employer failed to ensure that operators in the Crude Unit were trained to
Item 10 1910.119(g)(1)(i) respond to rising liquid levels in the TIU Fuel Gas Mix Drum located in the Crude 1
Unit, in that there was no prohibition against draining the liquid to the oily water
sewer. This lack of training resulted in employees being exposed to fire, explosion
and hydrogen sulfide hazards.
b) the employer failed to ensure that inside and outside operators were trained to
evaluate and identify the presence of naphtha in the TIU Fuel Gas Mix Drum located
in the Crude 1 Unit, during transient, temporary operating conditions, including the
NHT/SatGas and Coker Gas Plants being outside of normal operating conditions. The
lack of training resulted in employees being exposed to fire, explosion and hydrogen
sulfide hazards.
c) the employer failed to ensure that inside and outside operators were trained on
operating limits of the TIU Fuel Gas Mix Drum, related to liquid level, including the
consequences of deviation and steps to avoid and correct liquid level outside of
acceptable limits. The lack of training resulted in employees being exposed to fire,
explosion and hydrogen sulfide hazards.
Citation 2 29 C.F.R. § the employer failed to address human factors in the process hazard analysis to ensure
Item 1 1910.119(e)(3)(vi) that delays in screen loading on the South A Board in the Control Room were
corrected to allow for timely operator response in the event of an upset condition.
Delays in the inside board operators to access control board screens timely, can
inhibit their response in the operating units in the South Area.
167
Members of the U.S. Chemical Safety and Hazard Investigation Board:
Steve Owens
Chairperson
Sylvia E. Johnson, Ph.D.

Member
Catherine J. K. Sandoval
Member
168

CSB S BP Husky Investigation Report 1720741132

Uploaded by

Copyright:

Available Formats

CSB S BP Husky Investigation Report 1720741132

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CSB S BP Husky Investigation Report 1720741132

Uploaded by

Copyright:

Available Formats

Fatal Naphtha Release and Fire at

U.S. Chemical Safety and

U.S. Chemical Safety and Hazard Investigation Board

U.S. Chemical Safety and Hazard Investigation Board

3.1.3 Coker Gas Plant Bypass ..................................................................................................... 45

A.1.4.6.1 2:12 A.M.................................................................................................................... 141

A.2.5.3 OCTOBER 2018 ............................................................................................................... 148

A.2.9.4.23 6:12 P.M.................................................................................................................... 153

A.2.9.4.61 9:44 P.M.................................................................................................................... 158

LPG Liquefied Petroleum Gas

a) A broader definition of abnormal situations, such as that defined by the CCPS,

1) unplanned crude slate changes,

3) an emergency pressure-relief valve opening.

To American Petroleum Institute (API)

To International Society of Automation (ISA)

1.1 BP Toledo Refinery

Figure 1. BP Toledo Refinery Process Block Map. (Credit: BP)

1. to the Naphtha Hydrotreater unit (“the NHT unit”),

3. to Light Virgin Naphtha Storage.

1.2 Owner and Operator

1.4 Refinery Fuel Gas

1.5 Fuel Gas Mix Drum

1.6 Crude Oil and Crude Slate

1.7 Crude 1 Tower

1.8 Coker Gas Plant

1.9 Shift Operations Staffing

1.10 Federal Safety Regulations

1.10.1 OSHA Process Safety Management Standard

1.10.2 EPA Risk Management Program Rule

identify the potential effects of a chemical accident,

show the steps the facility is taking to prevent accidents, and

outline emergency response procedures [19].

1.11 Description of Surrounding Area

Table 1. Summarized demographic data.

White 71 Single Unit 75

2.1 Events Leading Up to the Incident

Water Overflow into Crude 1 Naphtha

Naphtha Hydrotreater Preheat Leak

Crude 1 Tower Process Upsets

Crude 1 Tower Feed Changes

High Level in Crude 1 Overhead Accumulator Drum

2.2 The Incident

[Outside] Got it.

[Outside] That is correct.

[Board] Copy that, just drain it as fast as you guys can.

2.3 Emergency Response

3.1 Incident Progression

3.1.1 Naphtha Standpipe Overflow

3.1.2 Naphtha Hydrotreater Preheat Leak

3.1.3 Coker Gas Plant Bypass

3.1.4 Crude 1 Overhead Accumulator Drum Level Control

3.1.5 Crude 1 Tower Instability

3.1.6 Crude 1 Tower Crude Slate Change

3.1.7 Crude 1 Overhead Accumulator Drum High Level

3.1.8 Absorber Stripper Tower Overflow

3.1.9 Fuel Gas Mix Drum Overflow

3.2 Draining and Liquid Release

3.3 Stop Work Authority

“I really need somebody to ok me to block that PSV in.”