Synchronism-Check Application Over

a Wide-Area Network

Russell Hyde
Mercury

Brett Hampson
Schweitzer Engineering Laboratories, Inc.

Presented at the
EEA Conference
Wellington, New Zealand
June 21–23, 2017
 Synchronism-Check Application Over
a Wide-Area Network
Russell Hyde
Mercury

Brett Hampson
Schweitzer Engineering Laboratories, Inc.

Brett Hampson, Presenting Author

EEA Conference & Exhibition 2017, 21 - 23 June, Wellington

Abstract
Recent technological developments have enabled solutions to power system protection
and control issues that have previously been either too difficult or too expensive to
implement. Synchronising generators to the power system requires analogue and digital
information from both the generator and the network. Transmitting information via copper
cabling requires the power station and the point of connection to the power system to be in
close proximity, but how do you achieve this when the sites are 1 kilometre or more apart?
This paper describes the first stage of a synchronism-check upgrade to the Whakamaru
Power Station and its connection to the 220 kV transmission network. The paper describes
the design installation, commissioning, and operation of the first stage of the scheme. The
scheme has been designed to allow any of four 27 MVA hydro generators to be synchronised
from any of four 220 kV line voltage transformers. The choice of hardware and the design
process have allowed for future enhancements such as the addition of an autosynchroniser.
The scheme has been designed to protect against the following:
• Misoperation of the existing autosynchroniser equipment
• Incorrect connection of the synchronising signals to the autosynchroniser
• Manually synchronising machines out of phase
The scheme employs a real-time automation controller using IEC 61131 logic. The main
unit is located at the power house and connected via 1 kilometre of single-mode fibre to the
remote unit at the ODM. Analogue and digital information is transmitted between the ODM
and the power house using a high-speed, industrial-based communications protocol known as
EtherCAT. Synchronised sampling of the local and remote measurement units avoids the
need for an external GPS clock, reducing complexity and improving the availability of the
system.
 Synchronism-Check Application Over
a Wide-Area Network
Russell Hyde
Mercury
Brett Hampson
Schweitzer Engineering Laboratories, Inc.

I. INTRODUCTION process logic. Electromechanical devices

Whakamaru is a Mercury-owned power relied on hard-wired inputs and outputs or
station on the Waikato River in New summation CTs to transfer information. The
Zealand. The station is presently undergoing arrival of electronics allowed programmable
a generator rehabilitation project. The logic controllers (PLCs) to communicate
existing 27.7 MVA generators will be with and control many of the devices that
replaced with 35 MVA units over the next control industrial processes. Systems for
four years. This paper describes the design coordinated control of valves, switches, and
and testing of a project to upgrade the other devices typically employ a PLC over a
existing synchronism-check equipment. This distributed bus network.
work will be carried out during the individual Many industries have adopted a bus
unit outages scheduled for the rehabilitation approach to connect multiple devices with
project, and commissioning will take place as high-speed control and communication.
part of the overall unit commissioning plan. Control of the electric power system is
The solution allows any of four achieved by connecting to circuit breakers
generating units at the Whakamaru Power and switches via a remote terminal unit
Station to be synchronised with any of four (RTU) or, more recently, by connecting
220 kV lines at an outdoor switchyard 1 km directly to an electronic device (relay),
away. The system is capable of providing protection to each individual piece
simultaneously synchronising multiple of equipment. For many years, distributed
generators to independent VTs, if required. protection and control systems have used
This paper describes the scope of the serial peer-to-peer communications and
project, the choices available, and the SCADA. Adoption of Ethernet protection
selection of the final design. The design and control protocols has been slow as
process and the final design itself are also compared with other industries.
described, as are testing and a recommended In the high-voltage (HV) electricity
commissioning procedure. The paper industry, many generating stations use PLCs.
concludes with a summary of the lessons For the remainder of the HV network, cost
learned from adopting this solution. and complexity have constrained the
The scheme has been acceptance-tested development of wider-area control.
and is ready to install and commission, but Distribution and transmission systems rely
due to delays in the generator upgrade on copper or fibre-optic cabling and digital
project, at the time of this writing the I/O to implement simple schemes (e.g., on-
synchronism-check system is not in service. load tap change control and circuit breaker
failure or interlocking) within substations or
II. BACKGROUND local areas. Communications between
Historically, many schemes for the devices in HV transmission networks
control of power systems were restricted to employing dedicated communications
one or two devices and limited by their equipment depend on other technologies,
ability to communicate information or such as power line carrier or the embedding

1
of optical fibre to communicate through the simultaneous synchronisation of up to four
optical ground wire. Today, microprocessor- generators with one or more of the 220 kV
based relays employ communications lines from the outdoor switchyard. The first
protocols to share digital and analogue stage of the project provides a close-permit
information over simple media such as radio output for each of the machines to allow its
and fibre-optic cabling. own autosynchroniser to close the unit
The adoption of technology used in other breaker(s). The controller must operate in
industries to connect devices by Ethernet has fail-safe mode. A failure of any part of the
led to the development of newer protocols to system bypasses the close-permit outputs to
transfer high-speed, time-critical information allow manual synchronisation of the
for both control and protection signalling and generators.
the application of special protection schemes
in the electric utility industry. However, IV. HOW WILL THE SYSTEM COMMUNICATE?
adopting Ethernet communications requires The overarching requirement of the
additional engineering, including topology project is the reliable transmission of line
design and management and message timing voltage signals to the power station. Many
issues that bring cybersecurity concerns with devices, including PLCs, have the logic and
them. processing capability to provide the
The solution described in this paper synchronism-check function needed;
employs a high-speed, industrial Ethernet however, very few of these devices transfer
communications protocol known as analogue values reliably and
®
EtherCAT . This protocol eliminates the deterministically.
need for switches and external time The first option investigated to transmit
synchronisation, and it allows for the high- these values was the use of IEEE C37.118
speed transfer of digital and analogue synchrophasors. This solution requires the
information. The speed and simplicity of this use of phasor measurement units at both sites
protocol make it the ideal choice for this that are connected via Ethernet switches and
application. which employ high-accuracy external time
synchronisation. Although a sound solution,
III. WHAT DOES THE SYSTEM NEED? the availability of a dedicated Ethernet
The first stage of the synchronism-check channel allowed a second solution to be
upgrade was to design, install, and considered: the use of a single real-time
commission a controller to provide a close- automation controller coupled with
permit signal to the autosynchronisation EtherCAT.
equipment of each of the four generators at The hardware for this solution comprises
the Whakamaru Power Station. The existing two modular chassis with one CPU, power
unit PLCs provide contact outputs to declare supplies, I/O, and CT/PT modules. The
which unit is to be synchronised with which chassis are connected directly via the
220 kV line. There are four three-phase VTs Ethernet channel. The measurement logic
at the outdoor switchyard and each generator control and timing requirements are all met
has its own set of three-phase VTs. In by the one system. The CPU uses an internal
addition, any of the 220 kV line VTs can be clock to provide the timing for the system.
selected and switched, via a copper circuit, to Although a synchrophasor-based solution
a single-phase running bus signal for use at would have worked, the lower product count,
the power station. standalone nature, and simplicity favoured
The power station and the outdoor the EtherCAT solution.
switchyard are located 1 km apart. The
system communicates the necessary V. HIGH-SPEED COMMUNICATIONS
information over the existing fibre-optic EtherCAT is an industrial Ethernet
network. It is required to provide protocol that employs standard Ethernet

2
frames as defined by IEEE 802.3. The variations. Timing accuracies of less than
topology is an open-ended ring in which each 1 ms between the master and slave devices
slave device (whether on the same backplane are easily achievable through the distributed
as the master or in a remote device) is clock mechanism.
connected to the EtherCAT bus [1]. The
payload, or EtherCAT telegram, contains up VI. THE REAL-TIME AUTOMATION
to 1,486 bytes of process data consisting of CONTROLLER SOLUTION
individual datagrams. Each slave device To the authors’ knowledge, this
subscribes to a particular datagram and reads application is the first of its kind to use a
or writes to it as the message passes through. real-time automation controller to perform a
Unlike traditional Ethernet, the datagram is synchronism-check function. A trial project
processed in hardware and is not dependent using revenue meters was assembled, and
upon stacks, queues, or CPU burden [2]. though this preliminary project was not fully
Known as “Ethernet on the fly,” the transfer tested and was never placed in service on live
of information takes less than 1 µs. equipment, it provided a good starting point.
The last device in the chain transmits the Before commencing the development of the
updated message back to the master. A solution in earnest, equipment was sourced,
typical update time for 1,000 distributed I/O and initial testing and evaluation proved that
devices is only 30 µs, including the the concept was feasible.
processing delay of each device. The fast A single automation controller at the
throughput allows up to 60 devices to be powerhouse contains the CPU. Five CT/PT
connected to the master controller while still modules collect the voltage signals from each
guaranteeing time-stamping with better than of the generating units and the running bus
1 ms accuracy for the digital I/O devices. signal. A digital input module connects to the
Analogue quantities require high outputs of each of the four unit PLCs, and a
precision. Currents and voltages are sampled digital output card provides the close permit,
at a rate of 24 kHz. The sampling rate is synchronism-check active, and alarm signals.
synchronised to the top of the second, with The remote slave module at the outdoor
the datagrams time-stamped at each CT/PT switchyard collects the voltage data from the
module. 220 kV line VTs and publishes them as
Time synchronisation of the modules is synchrophasor data to the master via the
achieved via the distributed clock mechanism high-speed Ethernet channel connecting the
described in IEEE 1588. The reference clock two units.
in this system is the automation controller Users can interface with the automation
clock. The controller broadcasts a message to controller via a web browser or via
the address of the distributed clock receive configuration software. A web browser
time port of each CT/PT module. Each interface administers users and
module responds with a time-stamped reply communications parameters and provides
for both the forwarding path and the access to alarms, Sequence of Events (SOE)
processing path of the ring. After collecting data, and control of the HMI. The software
the time stamps from the CT/PT modules, the provides a way to programme and
controller calculates the individual offset communicate with the controller. The
times from the known topology information software completely defines the project in
and the time stamps. It then writes the offsets terms of hardware topology, logic, and
into the system time offset register of each control algorithms. Online, the software can
module [3]. Any CT/PT module can then be used to monitor all internal tags and logic
calculate the system time by adding its local points, and measure internal cycle times and
time to the offset time. This process is CPU performance. It can also be used to
repeated periodically to counter any drift assist testing by forcing the states of logic
caused by temperature or crystal oscillator

3
and digital points and by writing values to declared. Main_Sync_Check closes the unit
analogue points. close-permit and the circuit breaker close-
interlock outputs to allow the existing
VII. LOGIC autosynchronisers to finish synchronising the
The processor for the automation machines.
controller runs a Linux® operating system If the two systems lose synchronism
using CODESYS as its logic engine. The when the voltage or frequency differences
hardware was connected to secondary test exceed the predefined limits, then the
equipment and the controller configuration program removes the synchronism-check
software was used to view online data to condition and releases the outputs. A
measure behaviour and to help develop and communications or hardware alarm is
prove the scheme logic. generated for any module or communications
failure. All outputs are de-energised and
VIII. PROCESSING LOGIC
external relaying bypasses the system to
The primary components of the allow for manual synchronisation.
synchronism-check application include the
Main_Sync_Check programme and the IX. DESIGN CONSIDERATIONS
FB_Sync_Check function block. IEC 61131 logic in the automation
Main_Sync_Check monitors hardware controller can be programmed in structured
inputs, controls the output contacts, and calls text (ST), continuous function chart (CFC),
the functions and function blocks within the and ladder diagram (LD) forms. Because this
automation controller logic. was a development project, ST was chosen to
Twenty digital outputs from the four make use of cut-and-paste and other hot-key
generation unit PLCs are wired to the digital functions to edit the programme as required.
input card of the automation controller at the The CT/PT cards sample the input
power station. The inputs select which line or quantity at 24 kHz and are synchronised to
running bus signal is to be synchronised to the top of the second. Three phase-to-neutral
each generation unit. It is possible that up to (line) or phase-to-phase (unit) voltages and
four inputs can be selected at any one time. one single-phase (running bus) voltage are
Assertion of an input calling for a unit to published with the frequency at a rate of
be synchronised causes Main_Sync_Check to 120 Hz. The automation controller offers
take several actions. The relevant unit several types of analogue tags to publish,
synchronism-checks active output closes. It including fundamental, root-mean-square
also instantiates the FB_Sync_Check (rms), and synchrophasor. Synchrophasor
function block and parses the relevant tags have a smaller dataset and were chosen
parameters necessary to synchronise the unit in the interest of minimising the EtherCAT
with the synchronising voltage. Twenty bandwidth. Forty-seven percent of the
instances enable numerous synchronising EtherCAT bandwidth is used for this
combinations. The voltage select (V_SEL) application.
function comprises several CASE statements Analogue tags are published between
that parse the appropriate polarising and –179.99 degrees and 180 degrees. When
synchronising voltages and frequencies to the calculating the angle difference, attention
newly created instance of the needs to be given to whether the angles are
FB_Sync_Check function block. positive or negative. Simply subtracting the
FB_Sync_Check checks that the voltage two angles to find the difference is not
magnitudes are within 5 percent of nominal always valid. Because negative numbers are
and that the slip of the two systems does not the issue, they were eliminated by adding
exceed 0.1 Hz. If these conditions are met 180 degrees to the measured angle and then
and the normalised voltage angles are within subtracting one from the other. The
±20 degrees, a valid synchronism check is synchronism-check angle (SCA) was

4
subtracted from the remainder to normalise Tracking is maintained until all three phase
the angles to zero degrees. The upper and voltages are lost. A dead bus is declared if
lower angle limits were set to ±20 degrees. the voltages are not within 5 percent of
The SCA is defined as the number of nominal, and the synchronism-check
degrees the unit (polarising) voltage lags the function is disabled. A loss of frequency
system (synchronising) voltage. For the line tracking caused by a loss of voltages is
and running bus, SCAs of 180 and covered by the dead bus; however,
120 degrees are required, respectively. qualification of the frequency measurement
The nominal system voltages are 220 kV (i.e., the addition of the FREQOK bit) will be
and 11 kV. For simplicity, the VT ratios were considered before the scheme is
set to 1. All calculations are performed with commissioned.
secondary values (63.5 V line-to-neutral and Testing the synchronism-check window
110 V line-to-line). at different levels of slip yielded different
The minimum automation controller results. As the slip frequency increased, the
processing interval rate is 4 ms. With the ±20 degree window shifted. At the upper
amount of logic programmed and the limit of 0.1 Hz, the operation of the close-
EtherCAT bandwidth available, the cycle permit outputs is delayed by ~6.5 degrees.
rate is set to 30 ms. The CPU burden runs at Investigation revealed that the relationship is
~60 percent under operating conditions. The linear and is due to the measurement latency
40 percent buffer is more than adequate for of the M class synchrophasor messages. An
the additional overhead generated by event offset variable has been added to the
report retrieval and online interrogation, and synchronism-check window to account for
it can also accommodate additional this latency. The use of P class
functionality in the future. synchrophasors guarantees a latency of less
than 40 ms. Further testing will be necessary
X. TESTING to determine the offset required if the
Much of the testing used the automation measurement type is changed.
controller itself to prove correct operation.
Being able to monitor logic online, add XI. WHERE TO FROM HERE?
watches and break points in the It is difficult to see any benefit from
programme(s), force analogue and digital changing ST to CFC or LD programming for
tags, and trigger oscillographic event reports future projects. ST is simple to debug and
was invaluable during the design and testing develop. The synchronism-check function
stages. A separate relay was also connected block could be written in C++ and included
in parallel to validate the test procedure, as a library for use in the future, although
particularly when results were not as customising the function for future sites
expected. could be difficult. One advantage of doing so
A secondary injection test set provided would be to prevent changes to the function
the voltages and frequencies to simulate onsite.
installed conditions. Occasionally, step There are several options to consider
changes to the frequency or voltage angles regarding the development of the existing
caused the synchronism-check active outputs scheme. The RTU functionality could make
to drop out momentarily. Event reports were the controller visible to SCADA. The HMI
triggered, and analysis provided confidence function could also be enabled to improve
that the algorithms in the automation local visibility. An autosynchronising
controller were secure. function could be added to replace the aging,
Testing revealed that synchrophasor tags existing equipment.
default to the nominal frequency if tracking
is lost. Tracking uses an alpha voltage
quantity of Va + Vb/2 + Vc/2 > 20 V.

5
 XII. CONCLUSIONS
One of the key requirements of the
design was simplicity. Using EtherCAT
protocol and programming the logic using ST
supports this requirement. The automation
controller is designed for the harsh substation
environment, and the low device count (no
external time clocks or PMUs) improves the
reliability and, hence, the availability of the
system.
Testing has revealed that consideration
should be given to the use of P class
synchrophasor data and supervision of the
frequency measurement. However, testing in
conjunction with online monitoring has
provided confidence that the scheme is
robust and ready to be placed into service.
At the time of writing, delays in the
generator upgrade project have prevented the
installation and commissioning of this
scheme.
REFERENCES
[1] V. Q. Nguyen and J. W. Jeon,
“EtherCAT Network Latency Analysis,”
proceedings of the 2016 International
Conference on Computing,
Communication and Automation,
Noida, India, April 2016.
[2] M. Rostan, “High-Speed Industrial
Ethernet for Semiconductor Equipment,”
proceedings of the SEMI Technology
Symposium, San Francisco, CA,
July 2004.
[3] G. Cena, I. C. Bertolotti, S. Scanzio,
A. Valenzano, and C. Zunino, “On the
Accuracy of the Distributed Clock
Mechanism in EtherCAT,” proceedings
of the 8th IEEE International Workshop
on Factory Communication Systems,
Nancy, France, May 2010.

© 2017 by Mercury and

Schweitzer Engineering Laboratories, Inc.
All rights reserved.
20170503 • TP6794-01