6484 FutureTime JP 20110131 Web
6484 FutureTime JP 20110131 Web
6484 FutureTime JP 20110131 Web
Presented at the
13th Annual Western Power Delivery Automation Conference
Spokane, Washington
March 29–31, 2011
1
Abstract—Advancements in power system control and be argued that, if present, an independent synchrophasor-
analysis methods continue to drive requirements for increasingly based situational awareness network, by itself, may have been
precise synchronization of time clocks in control devices and sufficient to prevent this blackout.
computers. Increasing varieties of sources for precise time are
online or are planned for deployment in the near future.
In the aftermath of the Northeast Blackout, we have seen a
Activities in IEEE and IEC standards reflect these time- large increase of interest in precise globally coordinated time
synchronization requirements, with specifications and synchronization, with some of the applications and
recommendations for time-distribution mechanisms within a performance limits becoming specified by international
substation or plant. The availability of precise time sources, standards and government regulatory agencies.
distribution methods, and devices that can accurately Overall, the availability and reliable dissemination of
synchronize time has opened the door to applications that use
precise time to coordinate operations between sites for improved
precise Coordinated Universal Time (UTC) are becoming
voltage control and reduced wear on equipment. Synchronized critical parts of power grid operation.
phasor measurement and control applications also drive the need
for precise time coordination and holdover. II. TIME-SYNCHRONIZATION ACCURACY NEEDS
This paper summarizes present and future sources of precise
The needs and accuracy of time synchronization have
time, including ground-based radio stations (e.g., WWVB, CHU,
and GBZ), enhanced Long Range Navigation (LORAN), Global evolved. With new technologies and better distribution
Navigation Satellite Systems (GNSS), and hybrid land-based and methods, most modern intelligent electronic devices (IEDs)
satellite systems. include at least one form of time synchronization [1]. Many
Time-synchronization requirements are matched to time- devices offer multiple ways to synchronize device time,
distribution methods, including IRIG-B, Ethernet Network Time providing different or higher-accuracy synchronization. This
Protocol (NTP), Simple Network Time Protocol (SNTP), section discusses some of these time-synchronization needs.
IEEE 1588 Precision Time Protocol (PTP), and communications
network-based systems, such as a synchronous optical network A. IED Time-Stamp Resolution
(SONET).
It is important to understand the difference between
accuracy and resolution when considering devices with time-
I. INTRODUCTION
stamped data. “Accuracy is the degree of absolute correctness
The need for precise measurement and distribution of time of a measurement device; resolution is the smallest number
is an important part of our society, from early civilizations that the device can display or record” [2]. Thus, accuracy is a
needing to know when to plant crops, to navigating ships or measure of how close the device time stamp is to an absolute
coordinating the first train networks. In the power industry, reference (or how close the device relative time is to the true
the need to have globally synchronized time was recognized or absolute time). Resolution refers to the smallest increment
very early, with many devices capable of time-tagging events of time allowed by the time stamp.
and collecting vast amounts of power system data. The quality IED time-stamp resolution varies from manufacturer to
of time synchronization, however, was very sporadic, and the manufacturer and from device to device. Time-stamp
United States Northeast Blackout of 2003 exposed many resolution is typically available to the millisecond. This means
weaknesses of time synchronization. Following the blackout, that as events occur through inputs, outputs, and logic, they
it took several months to parse through and piece together are time-stamped to the millisecond of when the events were
numerous event records, where time-stamp quality ranged detected. Another important factor to note is the accuracy of
from “not synchronized” to “utterly wrong.” A small number the given time stamp, which is based upon the processing
of systems had access to Global Positioning System- interval of the product. Depending on if it is a relay,
synchronized (GPS-synchronized) clocks, which simplified programmable automation controller (PAC), or supervisory
the overall data analysis. control and data acquisition (SCADA) device, the accuracy of
The Northeast Blackout also exposed the value of the 1-millisecond resolution time stamp can vary by
precision time-based wide-area data collection technologies, ±1 millisecond to ±4 milliseconds. When viewing time-tagged
such as synchronized phasor measurements (also called data from multiple sources, great care should be taken to use
synchrophasors), which, at the time, were mostly deployed in the accuracy of the device and not the resolution. North
the western part of the North American electrical grid. It can
2
American Electric Reliability Corporation (NERC) protection and control devices located on the same network.
PRC-018-1 currently requires that all internal clocks in Because process bus inputs are sampled at high rates
disturbance and monitoring equipment used by transmission (typically 4 to 16 kHz) with independent digitizers distributed
and generator system owners be synchronized to within throughout the substation, time synchronization becomes
±2 milliseconds of UTC [3]. critical for all applications that require data from multiple
locations (for example, bus differential protection).
B. Power System Characteristics
Because precise time synchronization of process bus
Because ac power systems are sinusoidal and operate at a measurements is as important as the measurement values
more or less fixed frequency (typically 50 or 60 Hz), some themselves, a mechanism must be implemented to deal with
timing accuracy needs are fairly concrete. In a 60 Hz system, a system startup, network component failures, maintenance-
single power cycle is approximately 16 milliseconds in length, related shutdown, and other events that may affect data
with zero crossings occurring approximately every delivery and time synchronization.
8 milliseconds. Power system events, such as arc extinction, IEC 61850-5 recognizes this fact and defines the
are inherently discretized at the 8-millisecond level. synchronization performance classes, as listed in Table I.
Thus, time synchronization needs to be accurate to a level
TABLE I
less than 8 milliseconds. Most power system IEDs process SYNCHRONIZATION PERFORMANCE CLASSES IN IEC 61850-5
analog signals every one-quarter of a cycle or faster. This
aligns well with the typical 1-millisecond time-stamp Performance
Accuracy Application
resolution. Class
Critical process bus and
C. IEEE C37.118 Synchrophasors TS5 ±1 µs
synchrophasor applications
Synchrophasors provide a precise absolute time-based TS4 ±4 µs Process bus, synchrophasors
measurement technology that enables users to collect power
TS3 ±25 µs Miscellaneous
system state information in real time, monitor system stability,
implement system-wide control, and perform precise post- TS2 ±100 µs
Point-on-wave switching, zero
event analysis. The IEDs that produce this information are crossing, synchronism check
called phasor measurement units (PMUs). These devices take TS1 ±1 ms Event time tags (1 ms)
real-time measurements of the power system currents and TS0 ±10 ms Event time tags (10 ms)
voltages and time-synchronize these data across a wide
geographic area. Required time accuracy is normally in the IEC 61850-5 recommends that time synchronization be
order of 1 microsecond. In order to maintain this accuracy, a implemented over the same communications infrastructure
time-synchronization source must provide synchronization to used for data exchange. In practice, this means the preference
±500 nanoseconds. The most widely used technology that can is given to Ethernet-based synchronization methods, such as
provide that level of accuracy is GPS clocks that send out Simple Network Time Protocol (SNTP) for ±10 milliseconds
IRIG-B for time distribution. (class TS0) and Precision Time Protocol (PTP) for the
remaining accuracy classes. Because IEEE C37.238 Standard
D. Time-Synchronized Control
Profile for Use of IEEE Std. 1588 Precision Time Protocol in
Time-synchronized control is an emerging application Power System Applications is currently going through the
where multiple control actions are scheduled, synchronized to final approval process, practically all existing process bus
each other, and executed in a coordinated fashion [4]. applications currently use alternate time-synchronization
Controlled devices include breakers, load tap changers, and methods, such as IRIG-B or one pulse per second (PPS). This
capacitor banks. Being able to operate multiple devices at the results in the need for two physical networks (communication
same time provides better power system stability and and time distribution), making it more difficult to deploy and
minimizes disturbances to the system. This type of control maintain. The situation is expected to improve once PTP
requires a time-synchronization accuracy of 1 millisecond. equipment becomes widely available.
E. Process Bus Applications
Process bus is an emerging technology for the high-speed III. TIME-STAMP SYNCHRONIZATION ACCURACY
real-time data exchange of instantaneous voltage and current As time-stamp synchronization accuracy approaches the
measurements, using an Ethernet network. It is based on ±1-microsecond level, it becomes increasingly important to
IEC 61850-9-2 and related international standards, some of precisely define time-stamp semantics (meaning of the time
which are still under development. Once fully deployed, stamp). Pioneering work in this field was initiated by the
process bus technology promises to seamlessly deliver smart IEEE C37.118-2005 synchrophasor standard, which
instrument transformer measurements to a wide variety of recognizes the importance of measurement delays caused by
3
signal filtering and requires that all such delays be to record system analog signals and disturbances but did not
compensated at the source by applying the correct time stamp. provide time-synchronization capabilities.
This approach, known as source-based group delay
compensation, makes it possible to precisely align time-
stamped event records regardless of the type of filtering or
reporting rate used in the particular application. Fig. 1
illustrates the group delay compensation concept, showing the
PMU output for an ideal 10 percent magnitude step occurring
at a time equal to zero (shown for a P-class PMU example).
generate UTC are at different locations around the world and some provisions are made to account for delays in the
are generally at a distance from the radio stations that transmit communications channel. For example, DNP3 calculates
these signals. The UTC(NIST) timing laboratories are located message latency and accounts for this time in its time-
at a distance from the WWVB radio station that transmits the synchronization message. Use of the DNP3 protocol to time-
timing signal. The WWVB radio station has its own clock that synchronize IEDs provides accuracies to within several
is continuously disciplined to match with UTC(NIST). The seconds.
deviation between the WWVB signal and UTC(NIST) is
D. Network Time Protocol
shown in Fig. 3 for timing samples recorded over a one-year
period. This graph also illustrates the typical hardship faced Network Time Protocol (NTP) is the most widely used
daily by the national laboratories in charge of maintaining the time-synchronization protocol in the world [6]. Almost every
worldwide UTC time, which is synchronized to TAI. TAI is computer connected to the Internet is time-synchronized by
derived by calculating the weighted average of over NTP. All PCs running Windows® or Linux® come with NTP
200 atomic clocks operated by over 70 national laboratories time synchronization. Computers and other communications
around the world. Individual TAI clocks are periodically devices in a substation benefit from the ease of using NTP to
compared to each other using a variety of communications- set the local time in these devices, where accuracies of
based techniques. The main difference between UTC and TAI subsecond synchronization are acceptable. NTP is distributed
is the fact that UTC keeps track of leap-second insertion, through Ethernet-capable devices. Depending on the network
while TAI does not. topology and the number of devices, NTP accuracies vary
from 10 to 250 milliseconds.
E. IRIG-B
In 1956, the TeleCommunications Working Group
Nanoseconds
54750
54775
54800
54825
54850
54875
54900
54925
54950
54975
55000
55025
55050
55075
IRIG-B is normally distributed using coaxial cables or fiber- simple slaves by default, defines extensive remote
optic transceivers, giving it excellent shielding and wide management capability using Simple Network Management
bandwidth resources necessary for preserving timing signal Protocol (SNMP), and allows for the existence of low-cost
integrity. IEDs with limited path delay measurement capabilities.
From the user perspective, the most important
F. IEEE 1588 PTP
characteristic of the new PTP-based time-distribution system
The need for precise time synchronization is directly is the fact that high-precision PTP distribution requires that
related to the ability of the distributed systems to the network infrastructure (all Ethernet switches) support PTP.
communicate with each other. If communication fails, making Because PTP requires hardware support, this means that
it impossible to exchange data, it becomes almost irrelevant if existing LAN systems cannot be upgraded without replacing
the time remains synchronized or not. Systems without outside virtually all of their components. On the IED side, this means
communication are free to start drifting from each other and that most IEDs requiring microsecond-level precision must be
can rely on a local reference. The problem reappears once the equipped with the same hardware (and the associated PTP
data from the two systems are brought together algorithm support). Fortunately, because IEDs already use
(communication is reestablished). The communications precise time, transition to PTP support is somewhat simplified
method can be as simple as hand-carrying the recorded data to because it can be viewed as an alternate time-delivery
a common location. mechanism (in addition to IRIG-B), leaving most of the time-
It is easy to see that, in an ideal situation, the same channel synchronization function intact. At this time, PTP support is
should be used to provide both communications and time- available in grand master clocks and Ethernet switches, while
synchronization services. This approach guarantees all the availability of relays and other substation IEDs remains
communicating systems have the same notion of time, with relatively limited.
well-defined failure and recovery mechanisms. In a local-area
network-based (LAN-based) environment, this goal can be V. EXISTING TIME-SYNCHRONIZATION INFRASTRUCTURE
accomplished using IEEE 1588 PTP, which is capable of
Existing infrastructures use various time-synchronization
achieving ±100-nanosecond accuracy levels [8]. This accuracy
methods that obtain accuracies from hundreds of milliseconds
is achieved by using dedicated hardware circuitry for precision
to 1 microsecond. Each product that exists in the substation
time-stamping of the Ethernet frame arrival and departure
has different requirements for time accuracy and inherently
times, along with methods for precise measurement of
uses a suitable time-distribution method to meet that accuracy.
communications link delays, as shown in Fig. 4.
A. Communications Devices
In substations, devices such as information processors,
SCADA devices, substation computers, and remote terminal
units (RTUs) have different synchronization methods and very
loose accuracy requirements. Most of these Ethernet-capable
devices typically support time synchronization through NTP.
NTP on a segregated network can provide reasonable accuracy
to a few milliseconds. The problem with NTP is that the
accuracy is not consistent. NTP accuracy varies depending on
the network topology and use. Moreover, accuracy also
depends on how well or what type of NTP the devices
support. Devices that support NTP are much more accurate
Fig. 4. PTP hardware-based packet time-stamping example than those that support SNTP. NTP devices use more
complicated algorithms to determine variable network delay
The IEEE 1588 PTP standard also provides a well-defined statistics, whereas SNTP directly sets the time as it is
method for the election of the best-available master clock and processed.
a wealth of mechanisms capable of operating in laboratory, Most of these products also offer an IRIG-B input—either
consumer, and highly critical industrial and power system IRIG-B000/-B002 or IRIG-B120/-B122. The IRIG-B signal
networks. supplied to these devices from a GPS source is accurate to the
Unfortunately, the wealth of supported mechanisms means hundreds of nanoseconds using IRIG-B000, but the devices
that users are left with the responsibility of choosing a subset only synchronize to around 1 millisecond. For these devices,
of mechanisms that are most appropriate for their application. 1 millisecond is the highest level of accuracy needed.
The selection problem is simplified by the development of
PTP profiles, notably the IEEE C37.238 standard profile for B. Digital Fault Recorders and Relays
use of PTP in power system applications. The power system Microprocessor-based relays in substations almost all have
profile establishes default initialization variables, gives a form of IRIG-B as their time-synchronization source. The
preference to the peer delay-based path delay measurement time-synchronization requirement for relays is 1 millisecond.
mechanism, limits the number of participants in the best This accuracy is used to align sequence of events or
master clock election by requiring that IEDs be configured as
6
oscillographic events from faults on the system. An accuracy the wide-area fiber-optic communications infrastructure
of 1 millisecond is adequate to review station-wide events and (privately owned or leased) is not entirely clear, but it is
time-align oscillographic event reports from multiple relays. expected that the precise wide-area time distribution currently
IRIG-B is the only distribution method available that can needed primarily by power system applications will become
provide this level of accuracy. IRIG-B000 can provide ubiquitous and will be extended to the general population.
1-microsecond accuracy. IRIG-B120 can provide GPS-based systems are likely to persist for some time, with
10-microsecond accuracy. Either one of these sources their role slowly changing to emergency backup and/or
provides relays with the accuracy needed and required by holdover-type applications.
NERC.
VIII. VULNERABILITIES OF TIME DISTRIBUTION
C. PMUs
Given the advantages of global UTC-based power system
Synchrophasor information is a fast developing technology
applications, it is easy to see why precise time distribution is
that uses dedicated PMUs or relays and digital fault recorders
expected to become critical for reliable power system
with PMU capabilities to stream power system values accurate
operation. As time distribution becomes more critical, it will
enough to measure power system angles to one degree. In
need to become very robust with well-defined availability
order to obtain that level of accuracy, the time synchronization
requirements. Those requirements can easily be met by using
must be 1 microsecond or better. Currently, the only
hardware redundancy. In addition, time-delivery systems need
widespread time-distribution method available to give that
to be hardened against accidental and malicious cybersecurity
accuracy is IRIG-B000. The PMU must be connected directly
attacks. Such attacks include GPS signal spoofing, GPS signal
to a GPS clock source that supports IRIG-B000 to maintain
denial and/or interference, malicious PTP system hijacking
that level of accuracy.
(through misuse of the best master clock election mechanism),
PTP packet spoofing, and, in general, denial of the time-
VI. TRANSITION INFRASTRUCTURES
distribution service. While any of these attacks can be
In the near term (possibly lasting ten years), we expect to implemented locally, it is crucial that the time-distribution
see the ever-increasing need for high-precision time-based system be designed in such a way as to eliminate the
applications. One of the major drivers is expected to be possibility of a centralized wide-area attack.
system-wide deployment of synchrophasor data collection Fig. 5 illustrates a novel method for implementing secure
networks, followed by process bus applications. As the data wide-area time distribution. This method is based on the well-
collected become more critical, so does the need to provide known frequency-synchronization mechanisms commonly
reliable system-wide time synchronization. During the used in synchronous optical network (SONET)
transition phase, it is expected that time-distribution methods communications networks. The main difference is the fact
will remain mixed, with IEDs required to support multiple that, while often frequency-locked to a Stratum 1 source
time-synchronization methods, including SNTP, IRIG-B, and (1 • 10–11 often achieved with a GPS clock), conventional
PTP. LAN infrastructure (i.e., Ethernet switches and routers) SONET systems do not have any provisions for transferring
is likely to start being deployed with PTP support, with the absolute time information. Nevertheless, SONET systems
largest penetration seen in new installations. Wide-area time- provide a great platform that can be designed from the ground
distribution capability is likely to remain GPS-based, with up to serve as a secure precision time-delivery system. A
GPS clocks required to deliver time using all of the SONET system uses a self-healing ring topology, which
enumerated methods. From the user perspective, time makes the system more resilient to fiber-optic path failure. By
distribution should not be a problem, as long as IED equipping every node with a built-in GPS receiver, the time-
manufacturers continue to support multiple time- distribution system becomes resilient to multiple fiber failures.
synchronization methods. Under normal operation, multiple GPS receivers make it
possible to perform continuous signal quality monitoring,
VII. FUTURE TIME-SYNCHRONIZATION INFRASTRUCTURE virtually eliminating the threat of localized GPS signal-based
Barring the development of unforeseen power system attack. Furthermore, the system remains functional even when
applications with significantly higher time precision or the GPS system becomes 100 percent unavailable by simply
communications requirements, time-synchronization systems switching to a global (system-wide) holdover mode.
within the substation are expected to migrate towards PTP Individual devices within the network remain synchronized to
implemented over Ethernet LANs. each other and, assuming the existence of a terrestrial link to
In addition to substation LANs, we also expect an increase other rings and/or networks, remain synchronized with each
in terrestrial-based, wide-area time-distribution systems other.
implemented using long-distance optical fibers. Ownership of
7
X. REFERENCES
[1] E. O. Schweitzer, III, D. Whitehead, G. Zweigle, and K. G. Ravikumar,
GPS Clock GPS Clock “Synchrophasor-Based Power System Protection and Control
Standalone Applications,” proceedings of the 36th Annual Western Protective Relay
GPS Clock Conference, Spokane, WA, October 2009.
[2] E. O. Schweitzer, III and D. Whitehead, “Real-Time Power System
Control Using Synchrophasors,” proceedings of the 34th Annual
Western Protective Relay Conference, Spokane, WA, October 2007.
SONET OC-48 [3] NERC Standard PRC-018-1. Available: http://www.nerc.com.
Ring Network
SONET Node With
[4] H. J. Altuve Ferrer and E. O. Schweitzer, III (eds.), Modern Solutions
Integrated GPS Clock for Protection, Control, and Monitoring of Electric Power Systems.
Schweitzer Engineering Laboratories, Inc., Pullman, WA, 2010.
[5] M. Lombardi, “How Accurate Is the Radio Controlled Clock?”
Horological Journal, March 2010.
[6] RFC 1305 Network Time Protocol (Version 3). Available:
http://www.faqs.org/rfcs/rfc1305.html.
[7] IRIG Serial Time Code Formats, IRIG Standard 200-04,
Fig. 5. Wide-area time-distribution and communications system September 2004.
[8] IEEE Standard for a Precision Clock Synchronization Protocol for
Holdover time is greatly extended by using the weighted Networked Measurement and Control Systems, IEEE
average of all clock oscillators located in individual SONET Standard 1588-2008, July 2008.
nodes. Holdover time can be further extended by strategically [9] K. Fodero, C. Huntley, and D. Whitehead, “Secure, Wide-Area Time
locating a redundant set of atomic frequency standards, whose Synchronization,” proceedings of the 12th Annual Western Power
Delivery Automation Conference, Spokane, WA, April 2010.
cost becomes negligible when compared with the overall
communications system cost.
XI. BIOGRAPHIES
Wide-area time distribution among SONET nodes is
performed using a private communications channel, which Jackie Peer has a BS in electrical engineering from Washington State
University (WSU). In 1996, she joined Schweitzer Engineering Laboratories,
cannot be accessed from the outside and is not exposed to the Inc. (SEL) as an application engineer. She managed distribution engineering
general data payload. Locally, each SONET node behaves as a in research and development (R&D) and product marketing. She is presently
grand master clock with IRIG-B and, in the near future, PTP the R&D manager for precise time and communications solutions. Prior to
joining SEL, she worked for the U.S. Army Corp of Engineers at a
time output. When in PTP mode, the grand master clock is hydroelectric facility. She is a member of the American Marketing
configured not to participate in the best master clock election Association, IEEE, and IEEE Women in Engineering (WIE) and a senior
process, thus eliminating security concerns normally member of the Society of Women Engineers. She also serves on the WSU
Electrical Engineering and Computer Science Industry Advisory Board.
associated with PTP. Additional details about this system and
the methods used to seamlessly transmit Ethernet traffic can Eric Sagen received his BS in electrical engineering from Washington State
be found in [4] and [9]. University in 1997. He joined General Electric in Pennsylvania as a product
Fig. 5 shows how the wide-area communications and time- engineer. In 1999, Eric was employed by Schweitzer Engineering
Laboratories, Inc. as a distribution product engineer. Shortly after, he was
distribution system can be hardened at the basic architecture promoted to lead distribution product engineer. Eric transferred to the time
level. However, it is important to remember that sound system and communications group in 2006 and is currently a lead product engineer.
architecture represents only one of the security factors. A He is certified in Washington as an Engineer in Training (EIT).
robust system requires the establishment of layered security
Shankar Achanta received his MS in electrical engineering from Arizona
procedures and policies, known as the defense-in-depth State University in 2002. He joined Schweitzer Engineering Laboratories, Inc.
strategy, where continuous performance monitoring and (SEL) in 2002 as a hardware engineer, developing electronics for
alarming systems may be as important as the overall system communications devices, data acquisition circuits, and switch mode power
architecture. supplies. Shankar received a patent for a self-calibrating time code generator
while working at SEL. He currently holds the position of development
manager for the precise time and communications group at SEL.
IX. CONCLUSION
Precision time synchronization is gaining popularity in the Veselin Skendzic is a principal research engineer at Schweitzer Engineering
Laboratories, Inc. He earned his BS in electrical engineering from FESB,
electric power industry, with a number of new applications University of Split, Croatia; his Masters of Science from ETF, Zagreb,
promising to improve power system operation. As the number Croatia; and his PhD from Texas A&M University, College Station, Texas.
and criticality of applications increase, the same happens with He has more than 25 years of experience in electronic circuit design and
power system protection-related problems. He is a senior member of the
the importance of the substation time-synchronization IEEE, has written multiple technical papers, and is actively contributing to
infrastructure. This paper documents some of the most popular IEEE and IEC standard development. He is a member of the IEEE Power
synchronization methods and explains some of the trends seen Engineering Society (PES) and the IEEE Power System Relaying Committee
(PSRC) and a chair of the PSRC Relay Communications Subcommittee (H).
in the industry.