Cigre 464
Cigre 464
Cigre 464
Working Group
B5.06
June 2011
Maintenance Strategies for Digital
Substation Automation Systems
Members
Volker Leitloff (FR) – (Convenor), Michael Eckl (AT), Peter Jenåker (SE), Martin Herzig (CH),
Frank Koers (AN), Steven A. Kunsman (USA), Ivan Lorencin (SI), Rannveig S. J. Løken (NO),
Alberto Lopez de Viñaspre (ES), Mika Loukkalahti (FI), Iony Patriota de Siqueira (BR),
Marcelo Paulino (BR), Massimo Petrini (IT), John Robertson (CA), Ville Tiesmäki (FI),
Peter Wittlinger (DE), John Wright (UK)
Copyright © 2011
“Ownership of a CIGRE publication, whether in paper form or on electronic support only infers right of use
for personal purposes. Are prohibited, except if explicitly agreed by CIGRE, total or partial reproduction of
the publication for use other than personal and transfer to a third party; hence circulation on any intranet or
other company network is forbidden”.
Disclaimer notice
“CIGRE gives no warranty or assurance about the contents of this publication, nor does it accept any
responsibility, as to the accuracy or exhaustiveness of the information. All implied warranties and conditions
are excluded to the maximum extent permitted by law”.
Preface
In 2007, CIGRE Study Committee B5 (Power System Protection and Local Control) instigated the
formation of the Working Group B5.06 to look into the state of the art of Maintenance Strategies for Digital
Substation Automation Systems (DSAS).
The background of the Working Group is the fact that the maintenance of Digital SAS is different from that
of conventional DSAS under different aspects :
• Digital Substation Automation Systems pose new questions on how to ensure the effectiveness of
maintenance over the whole life of the system.
• The professional profile of maintenance specialists has been changing according to the technological
evolution of substation automation devices and systems. However, there is no consensus on the
impact of maintenance on existing and future DSAS. In consequence work on maintenance
represents a strategic interest for SC B5.
• The investigation of maintenance strategies for DSAS is well within SC B5 scope. However, it is
required to clearly define the limits of this study with respect to the activities of other SC committees.
The impact of Digital SAS on maintenance strategies needs to be clarified, and there is also an impact of
maintenance constraints on the design of Digital SAS. The task of WG B5.06 is the description of these
aspects.
Scope
The scope of work was thus defined as:
The analysis and consolidation of strategies for conserving the capability of hardware and software repair,
correction and update during the lifetime of the DSAS, taking into account the items and limits listed
below:
1. Items within the scope:
• Conservation of knowledge and abilities on the systems;
• Management of spare-part stocks;
• Maintenance strategies and frequencies for DSAS
• Appropriate use of local and remote maintenance
• Cases of systems using IEC 61850 and proprietary protocols.
Deliverables: Technical brochure with the strategies for the maintenance of digital SAS.
2
Table of contents
1 Introduction ............................................................................................................................................. 5
2 Definitions and General Features ............................................................................................................ 5
2.1 Digital Substation Automation Systems (DSAS) ............................................................................ 5
2.2 Impact of substation types on DSAS and its maintenance .............................................................. 6
2.3 Substation communication architecture and -protocols................................................................... 6
2.4 Replacement, Refurbishment and Retrofit ...................................................................................... 7
2.5 Maintenance .................................................................................................................................... 7
2.5.1 Types of maintenance.............................................................................................................. 7
2.5.2 Maintenance repair levels........................................................................................................ 8
2.5.3 Maintenance service levels...................................................................................................... 9
2.6 Self-Supervision .............................................................................................................................. 9
2.7 Benefits of modern communication standards .............................................................................. 10
2.8 Reliability ...................................................................................................................................... 10
3 Life cycle strategies of DSAS ............................................................................................................... 11
3.1 General aspects regarding maintenance ........................................................................................ 11
3.2 Strategy for Design and Testing .................................................................................................... 12
4 Maintenance strategies for DSAS ......................................................................................................... 13
4.1 General considerations about Maintenance Strategies .................................................................. 14
4.2 Assessment and feedback .............................................................................................................. 15
4.3 Outsourcing and Contracting of Maintenance............................................................................... 16
4.3.1 Scope of Maintenance Contracts ........................................................................................... 16
4.3.2 Special considerations for subcontracted maintenance ......................................................... 16
4.3.3 Consideration of maintenance issues in the contract preparation .......................................... 17
4.3.4 Maintenance Contract Management ...................................................................................... 17
4.4 Special considerations for maintenance done by utility personnel ................................................ 18
4.5 Preventive maintenance................................................................................................................. 19
4.6 Corrective maintenance and self-supervision ................................................................................ 20
4.7 Use of remote maintenance ........................................................................................................... 21
4.7.1 Possibilities in the remote access .......................................................................................... 21
4.7.2 Security in the remote access to substations.......................................................................... 22
4.7.3 Access and information levels............................................................................................... 22
5 Technical Expertise and Training ......................................................................................................... 23
6 Specifications and testing requirements for DSAS maintenance .......................................................... 23
6.1 General aspects concerning specification...................................................................................... 23
6.2 Importance of documentation and tool update for testing ............................................................. 24
6.3 Requirements concerning maintenance and testability ................................................................. 25
6.4 Testing of distributed and integrated functions ............................................................................. 26
6.5 Test coverage ................................................................................................................................ 28
6.6 Maintenance tools ......................................................................................................................... 28
7 Quality Control and Management of DSAS.......................................................................................... 29
7.1 Generic process of Managing Settings change .............................................................................. 29
7.2 Impact of settings, firmware and hardware versions on maintenance ........................................... 30
8 Considerations regarding enhancement and modifications of DSAS after commissioning .................. 31
8.1 Use of maintenance to postpone refurbishment ............................................................................ 31
8.2 Considerations for upgrading the DSAS ....................................................................................... 31
9 Management of spare-part stocks .......................................................................................................... 32
9.1 Consistence and volume of the spare-part stock ........................................................................... 32
9.2 Configuration/personalisation of spare-part IEDs before insertion in Digital SAS ...................... 33
9.3 Possible strategies for spare part management .............................................................................. 34
9.4 Use of decommissioned devices to increase spare-part stock ....................................................... 35
9.5 Disposal of obsolete spare-parts .................................................................................................... 35
9.6 Management of obsolescence of components ............................................................................... 35
10 Case of systems based on different communication standards.......................................................... 36
3
11 Overview of Utility Maintenance Practices for DSAS...................................................................... 39
11.1 Maintenance Principles ................................................................................................................. 39
11.2 Maintenance Organisation............................................................................................................. 40
11.3 Maintenance Method ..................................................................................................................... 40
11.4 Training of DSAS Maintenance .................................................................................................... 41
12 Conclusion ........................................................................................................................................ 41
13 References ......................................................................................................................................... 42
14 Annexe 1 : Restitution of questionnaire ............................................................................................ 43
15 Annexe 2 : B5.06 Colloquium paper ................................................................................................. 47
4
1 Introduction
Substation Automation Systems (SAS) and associated technologies have evolved considerably over the
preceding decades, with the aim to increase their overall efficiency and to decrease the costs of
ownership. Amongst other factors, this trend is driven by the rapid evolution of computer and
telecommunication technology. The Substation Automation Systems commissioned today are digital and
more integrated. They are designed and configured with powerful engineering tools by highly trained staff.
Extension, refurbishment and replacements of the SAS or part of it have been covered by WG B5.08 [7]
and are not treated in this brochure, with the exception of cases concerning upgrading or corrective
maintenance.
The maintenance of Digital SAS (DSAS) is different from that of conventional SAS under different aspects:
• Digital Substation Automation Systems pose new questions on how to ensure the effectiveness of
maintenance over the whole life of the system.
• The professional profile of maintenance specialists has been changing according to the technological
evolution of substation automation devices and systems. This has an impact on maintenance of
existing and future DSAS.
• This new technology implies a change in the roles carried out by the traditional actors in the
maintenance of DSAS and imposes different strategies for their maintenance practice in order to take
into consideration new constraints.
The approach for the maintenance of DSAS has to take into account several factors, including system
availability, total cost of ownership and the possibility of modification, extension or partial replacement of
components. Specification, design, configuration, testing, training and documentation are key elements
which contribute to the ability of the utilities to maintain a Substation Automation Systems during its
lifetime.
Different types of protection and control systems exist, based on electromechanical, electronic(solid state)
or digital technologies. The scope of this report is limited to Digital SAS (DSAS).
This Technical Brochure intends to give some guidance for utilities, which require an understanding of
how to adapt the maintenance of their Substation Automation Systems to this new context. The topics
developed also concern the other stakeholders in DSAS maintenance, in particular the vendors.
5
systems. Data is exchanged between different devices of the DSAS through a local network based on
fibre optic cables and/or copper wires. It may be based on LAN and switches, which connect all IED’s to
the station computer, to gateways and to the time synchronising equipment.
The terms and expressions used in this report such as IED (= Intelligent Electronic Device), HMI (=
Human Machine Interface) etc. are commonly used today in literature, specifications, technical datasheets
and operation manuals and therefore not listed explicitly.
Serial communication channels are used to integrate the different IEDs based on different standards
or proprietary serial protocols. They have in general been in use for a long time (IEC 60870-5-
101/103, MODBUS, DNP, etc..) and are often proprietary.
Normally the DSAS installed will include many serial communication channels for the interconnection
between IEDs in the bays and equipment on the substation level (station computer, RTU, etc..).
In this type of architecture it is also usual to find physical media converters (e.g. RS-485 to Fiber optic
converters), multiplexers (e.g. Fiber optic multiplexers) and star couplers for fiber optics as
intermediate devices.
6
2. Ethernet Communication Architecture
In some cases IEDs are directly integrated using Ethernet based protocols, or they use a serial
connection through a serial to network converter.
These protocols are state of the art of today. In particular, IEC 61850, which allows interoperability, is
being deployed in many substations.
For both types of network communication architecture, it is necessary to consider the communication
devices (e.g. Ethernet switches) in the software maintenance program and to include them in the version
management of the DSAS. This may be a more sensitive issue for the network communication
architecture.
The use of IEC 61850, which is more than a conventional communication protocol, will push towards a
wider use of software tools. This also has an impact on maintenance.
2.5 Maintenance
7
• Corrective maintenance has to be done after a malfunction or the detection of a failure of a
system or device. Corrective maintenance is event driven. The functions of the DSAS concerned
by the correction have to be tested.
• Updating maintenance is aimed at updating the software or the firmware of a component part of
the DSAS or the information in a database without enhancing the functions. Updating
maintenance may be done in the general context of a corrective or a preventive action.
Whereas corrective maintenance is event driven, the intervention of the other types of maintenance
usually can be planned in advance. This may also be the case for corrective maintenance concerning
benign problems.
One maintenance action can cover one or more of the above mentioned types.
In addition, upgrading actions bring enhancement due to firmware or software upgrades of system
components or IED’s, exchange of equipment on bay or station level. Upgrading goes further than
maintenance but does not imply the replacement (refurbishment) of the control system. These actions can
be combined with maintenance of the DSAS and may be considered by some utilities as upgrading
maintenance. This can be motivated, eg., by new functional - or cyber security requirements (cf. § 4.7.2).
DSAS can contribute to the preventive maintenance of the grid, the HV equipments and the DSAS itself.
In this context, IED’s can provide data about the status of the circuit breakers, cables, overhead lines, etc.
The cumulated break current I2t is a measurement of the breaker’s and cable’s health condition. Weak
insulation spots on the overhead lines can be localised with the help of the distance-to-fault location.
Preventive maintenance is, as of today, not commonly applied to DSAS itself.
Maintenance of DSAS systems is normally done on-site, but remote maintenance and –setting is a new
trend which probably will become more important in the future (cf. §4.7).
Examples of maintenance actions are:
1. Monitoring – Health Check (examples : Check of DSAS status i.e. alarms, self supervision,
switches, gateways, communication quality statistics – Bit Error Rate etc., Extract information on
alarms for study, events, disturbance records, Protection communications, GPS status,
teleprotection, Check of records of self-supervision, …)
2. Planned maintenance (examples : Checks for IED’s in general, correct current flows through
the IED in all phases, correct voltage appears in the IED as required, accuracy of current/voltage
and time measurements and algorithms, local and remote control and indications, other control
functions as tap change control etc., additional checks for protection, routine trip testing, etc…)
IED essentially need to be tested for availability, but correct acquisition of analogue values has to
be verified. Drifting of component characteristics due to ageing is eliminated because the
processing is digital.
Level 2: actions requiring simple procedures and / or auxiliary equipment which are simple to use
(example: replacement of a I/O circuit board).
Level 3: requiring complex procedures and / or auxiliary equipment which are complex to use
(example: replacement of an IED).
8
Level 4: actions or procedures implying the use of a particular technology and / or the use of
dedicated equipment (example: software or firmware upgrade).
Level 5: Actions based on procedures implying a special expertise, using particular technologies
and or factory based auxiliary equipment (example: re-engineering the device).
The utility maintenance strategy can identify different departments and / or third parties depending on the
level of maintenance for a given task. Example: level 1 and 2 – operation department, level 3 and 4 –
specialised teams within the utility, level 5 – vendor.
2.6 Self-Supervision
Self-Supervision is a very important feature of DSAS and especially of numerical protection relays.
The integrity of hardware and software is monitored by plausibility routines and watch dogs. An IED
should not become dysfunctional without recognising failures and reporting them, to facilitate remedial
actions. In addition, associated external circuits are also covered by the monitoring of IED’s and
associated communications infrastructure for the DSAS :
• The integrity of instrument transformers, i.e. current and voltage transformers and their
associated circuits are checked for plausibility and correct phase sequence.
• The A/D converters are checked for integrity and accuracy.
• On-load directional verification and calculation of load impedance for availability.
• The DC supply is checked for availability.
• The relay-to relay communication channel of teleprotection schemes for integrity and channel
latencies.
• All internal and external communication of the DSAS (communication switches, LAN,
routers…)
• Diagnosis of the IED
• Diagnosis of station computer, gateways, etc.
• The trip circuits are checked for integrity.
9
The ability to measure outside the IED allows the protection and control equipment to determine the state
of the network and guide event driven targeted maintenance.
As the amount of monitoring information is considerable, it is becoming more difficult to analyse all the
information available. Especially the enhanced communication features with IEC61850 require adequate
tools and expertise. Utilities will have to ensure the availability of the expertise to analyse the information
available and decide the action required.
As mentioned before, the principle of self-supervision can expand beyond the DSAS itself. Permanent
monitoring may instantly detect when maintenance is needed, not only on the DSAS itself, but also on the
primary system. This contributes to optimising the balance between preventive and corrective
maintenance. Therefore cost savings and improvement of reliability with respect to conventional time-
based management is possible using corrective maintenance based on self-supervision.
2.8 Reliability
Reliability [12] on device- or system level depends on the components used and the DC-supply. MTTF
(Mean Time To Failures) may be considerably increased if a redundant power supply or redundant
components (IED, switches …) are used. Beside that reliability also depends on the particular system
architecture.
MTTR (Mean Time To Repair) is the time required to perform the repair plus the time required to initiate
the repair action. The time required to perform the repair basically depends on the system architecture
and thus on the design chosen by the vendor. The time to initiate the repair action depends on the
organisation put into place by the utility and may vary greatly, also in function of the nature and the
consequences of the failure and the geographic location of the substation.
MTTR for failures not covered by self-supervision is significantly longer.
10
3 Life cycle strategies of DSAS
11
DSAS life cycle
en d
m an
n
tio
ts
re y
se uir lua
ui teg
an ion
cq va
ds
eq a
l r str
e
er
en nct
m or e
ce
ce
e
e
en
nc
nc
ov
an
d
pp to ct
u
na
na of
en of tena
ng
e
d
g
Su on du
en
te
ct
ke
r io
tio n
in
ni
nt
nt
in
e y
n
nc itio
le
o
a
io
on
tiv ba
ai
ai
ta
ai
pe
r
iss
m
cis f p
m
m
fu f in
SA issi
er
m
ive
Ad tive
e
y
De o
ev n
De
m
tiv
lie
om
nt
Pr itio
ct
m
t
co
i
c
ar
en
ra
r re
r re
m
d
Cu T
De
T
st
St
ev
ar
Co
FA
Co
Co
Pr
W
Re
pa R SW Re
Bi Pr ir ha epl -u me pla
a p c c
dd oj rd c e gra han e b
i ng ec wa p de ica att
t re ar
pe pe ts l c erie
r of le s
rio io an a
d d in nd
g filt
er
s ,
In-service period
Several events or actions regarding maintenance can take place during the in-service period of the DSAS:
• Planned actions or events
End of the warranty period,
Preventive maintenance actions (replace batteries, filters or other hardware components,
mechanical cleaning),
Addition or modification of functionalities and SW & HW Upgrades,
Addition or modification of bays,
Assessment of DSAS behaviour and experience feed back for the collection and analysis of
failures and elaboration of failure statistics.
• Unplanned actions
Corrective maintenance
Repair of equipment
Software update further to identified problems
Today, a globally shared expectation is that the in-service period of DSAS is about 20 years (cf. §11). In
some cases, utilities may be faced with a raise of the failure rate, obsolescence or lack of support before
the end of lifetime of the equipment. Utilities may then be forced to refurbish the DSAS before the end of
its "nominal" lifetime.
12
process at a time is activated. Such problems, if not critical to the process are often left for the next
maintenance period. This will increase the maintenance period beyond what was originally anticipated and
may impact the operation of the process, i.e. down time and substation availability. To alleviate such
problems it is important to use a proven, tested design, especially on critical installations, such as feeders
supplying sensitive industrial plants, hospitals, military sites etc and reliable, tried and tested equipment.
The system must be tested extensively during generic qualification tests, FAT, SAT and final
commissioning. Since once the system is in service, it may not be possible to carry out any remedial work
easily until the next planned outage – hence, impacting on the maintenance schedule. The design must
be traceable accounting for all modifications and all parties must be kept up to date.
For this reason, the design and test phase is important for the subsequent maintenance period. It is not
possible or very difficult not to take into account maintenance after the system design. The system should
thus be designed with maintenance in mind, not as an after thought. Where possible it is advantageous to
involve the maintenance team as early as possible in all the phases of the project development.
Depending on the organisation of the utility, this may allow them to be fully trained when the system
comes on line and not afterwards following a lengthy training process.
The component parts and tools of the DSAS should be easy to operate, not require a complete strip down
for simple maintenance procedures, be operator friendly and take full advantage of maintenance design
features i.e. CB control, monitoring, watchdogs, communications statistics, trending etc.
One aspect that can make a big difference is the test facilities that are available for the maintenance
Engineers. Having a test platform to simulate faults can save time and prevents testing on a live system.
13
4.1 General considerations about Maintenance Strategies
System maintenance is concerned with the conservation of the system hardware and software through
maintenance programs. These identify hardware and software resolutions which address operational
problems. Replacement parts have to be replenished or stocks managed depending on requirements.
Utilities can adopt different strategies to perform the maintenance of DSAS:
• Outsourcing of maintenance activity by subcontracting to a vendor or another service provider.
• Performing the maintenance of DSAS completely by its own staff.
• A mixture between the two above mentioned possibilities. In this case, only particular activities
(management of spare-part stock, complexity of tasks, etc…) are subcontracted depending on the
needs of the utility. Utility personnel can take simple substitutions of devices or cards in charge.
A combination of maintenance carried out by the utility's staff and maintenance carried out by equipment
manufacturer is particularly useful in substations which are in easy reach of operation- or maintenance
staff. The utility staff is often trained to deal with failures not requiring complicated repairs. If this is
unsuccessful, specialised staff either from the utility or the vendor is engaged. This allows the utility to at
least have some understanding of the equipment, without the responsibility of fixing system critical issues.
System reliability management through maintenance contracts aims at obtaining a contractual guarantee
that the system will remain fully functional throughout the contract period. In addition, it can manage
through an auditable process, version control of software, setting changes, drawing amendments etc.
In order to maximise the lifecycle of a DSAS it is important that maintenance is undertaken, particularly in
the case of aging equipment. Maintenance is often dictated by the environment, criticality of load,
resources, technology, availability of equipment and cost. As for other systems, lack of maintenance may
result in premature failure of the DSAS.
During its lifetime the DSAS is also likely to evolve, for example, increased or decreased functionality,
change in system/process conditions or implementation of new technology or new functions. These will
all impact on the DSAS design, hardware, software and configuration. If this action is ignored it can lead
to many unnecessary system problems. More often, updates are left until the next maintenance period;
this can leave the DSAS in a vulnerable position until the problem is rectified.
The following areas need to be considered when developing a maintenance strategy :
• Type and scope of maintenance (corrective or preventive (cf §2.5.1, §4.5)
Corrective maintenance is often more labour intensive and requires a different type of
organisation. The maintenance strategy has to define a balance or ratio between preventive
maintenance and expected corrective maintenance.
• Health and Safety
It is often a legal, or regulation obligation and / or a utility strategy to ensure that staff are
adequately trained for the job they have to perform and all necessary paper work is in place i.e.
risk assessments, method statements etc.
• Contribution of utility staff and vendor (or third party)
For corrective maintenance, the utilities have to define the strategy for dealing with failures of the
DSAS. Strategies depend on the action required to remove the problem. This may imply its own
staff and / or maintenance contracts. The ideal coverage is 24/7, 365 days per year.
• Training and knowledge of the utility staff depending on its contribution for maintenance (cf. §5)
• Availability of maintenance- and test procedures
Invasive actions (modification / replacements) should be performed based on written and
validated procedures which also take into account contingencies. The line of action in this case
(emergency procedure / back up) has to be planned. Also, corrective maintenance has to be
based on written guides covering procedures and tests for standard maintenance actions on the
DSAS.
14
• Hard-, firm- configuration- and software version management (cf. §7.2),
• Other documentation required for the maintenance of the system over its lifetime.
• Use of remote access (cf. §4.7) including authorization.
• Characteristics of the portfolio of substations owned by the utility
The number of substations of different types (electromechanical, electronic, digital) and the
repartition of equipment of different vendors requires specific know-how, tools, spare-parts etc..
• Scheduling including combination of DSAS maintenance with the maintenance of HV equipment.
• Operating restrictions due to the network
In some cases, the DSAS or parts of it can only be accessed during `out of hours` i.e. weekends
and nights. In other cases, the feeders have to be de-energised in order to perform maintenance.
That implies that planned maintenance should take into account these constraints.
• Operating restrictions due to the maintenance
Maintenance action can lead to operational restrictions of the primary system (in particular
switching restrictions).
• Physical Access to equipment and Authorization
The person granting the physical access and the authorization and the person carrying out the
isolation may not be the same person carrying out the maintenance.
• Organisation to elaborate Fault Analysis and Statistics
The analysis and compilation of statistics of primary and secondary faults are important for
maintenance management.
• Spare-part and obsolescence management (cf § 9.6).
• Maintenance for equipment installed in small numbers
It may be preferably outsourced to the vendor in order to avoid the maintenance of the associated
knowledge within the utility.
• Number of vendors (and/or systems) by issuing frame contracts over a defined period. This
influences on the number of different skills, tools, devices to be maintained.
15
If maintenance is outsourced, some of this work can be transferred to maintenance service provider. The
analysis and statistics of faults is usually done by the utility or asset manager.
For primary faults, it is important to analyse if the DSAS worked properly, eg. the protection and control
system worked correctly.
For failures in the DSAS, it should be verified if there are similar faults elsewhere or if there is a common
failure type of this component. This includes inspection of failed equipment (e.g. forensic) and analysis of
the behaviour after failure. This allows detection of hidden faults and can lead to modification of the
installed or new equipments of the same type [14]. The exchange of information between utility and DSAS
manufacturer is important for the success of this process.
16
These contracts cover service characteristics and parameters such as:
- Delays for response and repairs ,
- Time period during which maintenance service will be provided (maintenance of the knowledge
and of the ability to maintain it by the vendor during the lifetime of the DSAS),
- Site access and security procedures,
- Use and limitations of remote access,
- Reference values for MTTF and MTTR (cf. §2.7),
- Periodic review of the maintenance Contract and of DSAS failure statistic,
- Information and Procedures of implementation of corrections and patches including those based
on generic problems identified for the DSAS.
In any case, the subcontracted maintenance has to be evaluated and verified by the utility. This
verification can be realized by
- Audits and verification of documents,
- Verification of reference values and improvements,
- Detailed verification on a sample (bays).
17
latter ones. This warranty period may have an impact on the application of the maintenance
contract.
• Regrouping of several substations in one maintenance contract. This is usually done if several
substations are purchased together. Maintenance of existing systems can also be included in
contracts for new DSAS.
• In some contracts, a system is put into place at the vendor for qualification, trouble-shooting and
training purposes. This system may later be used as spare-part or deployed in a substation at the
end of the contract. The decision to do this should be taken between the two parties. It is also
possible to install such system at the utility where it may be used to prepare maintenance
operations or for training purposes.
• The extensions of the warranty with respect to the basic delivery contract and commitments for
availability of spare-parts (cf [6]).
• It has to be determined to what extent the vendors guarantees the support for the installed DSAS
and the necessary spare-parts, including the ability of the vendor to change software or hardware
of the DSAS over its complete lifetime. This goes beyond the ability to repair the system over its
lifetime.
18
• Planned actions by utility staff can be disturbed if the same personnel has to cover also corrective
maintenance.
If maintenance is performed by utility staff, it is still important to have some kind of internal service
agreements and maintenance cost evaluation. Also different kinds of internal maintenance guidelines are
required (cf. §6.2). Detailed test guidelines can be in internal cases so accurate that they contain terminal
strip numbers for the different tests.
Modern DSAS systems are so complex that corrective maintenance with own staff can usually handle only
hardware problems and software problems by substituting IED or defect card of device. More complex
software problems or bugs especially in the control side of DSAS usually require assistance of the vendor
of the system. Some kind of contract or agreement with vendors is needed for this kind of usually rare but
difficult software problems (cf §4.3.1).
Depending on the utility organisation, there may be several roles with associated access rights to DSAS.
These roles include administrator, maintenance engineer, control engineer, protection engineer, DSAS
system engineer, operator and viewer. Due to their internal organisation and their maintenance strategy,
some utilities may try to separate the domains of these different roles. However, it may be useful to
simplify and to define only a small number of access profiles.
19
routine test interval (in years)
Figure 4-1: Example of total maintenance costs per year as a function of routine test interval (in years)
for a given system when the coverage of the self-check function (TA) varies between 0 to
99%. [10]
The depth of preventive maintenance can be adjusted to the device to be tested. However, a reduction of
50% of the tests to be performed does not automatically lead to a reduction of 50% of the associated
costs. The time to go to the substations and the time to prepare the bay or devices for the test is invariable
whatever the test actions performed.
Where possible it is better to limit the amount of invasive testing, since sometimes additional problems
can be created as a result of having to operate tests on equipment. It is important to clearly define what is
covered to make sure that maintenance staff does not perform unnecessary maintenance.
Historically, periodic maintenance tests have been performed for protection relays and less for substation
control systems. With the self-supervision and, furthermore, with functional integration of protection and
control functions in the same IED, this distinction is less justified in DSAS. Many utilities have decided to
continue with periodic tests for protection functions for safety reasons, although with an increase of the
test interval [cf. §11]. There is a trend towards self-supervision based corrective maintenance both for
protection and for control functions.
20
tripping circuits and telecom circuits (cf. §2.6). Over 90 % coverage can be achieved at least for internal
IED self check with numerical IED’s.
Self-supervision functions of DSAS give opportunity to decrease the amount of preventive maintenance,
since most of the faults occurring in the system are detected by self-supervision itself. This means that the
Mean Time To Repair is significantly shorter than in conventional SAS where faults may remain
undetected for a long time. This allows to increase maintenance test intervals and also to decrease the
amount of testing on modern relays. The self-supervision results thus in a shift of balance away from
preventive maintenance towards corrective maintenance.
Nowadays, most corrective maintenance is initiated by the self-supervision of the DSAS after the
detection of a failure or of a malfunction.
In order to be able to perform corrective maintenance in a satisfactory way, the following conditions have
to be verified:
• Availability and accessibility of spare-parts (cf §9),
• Setting-, parameter- and configuration version management with access for maintenance staff (cf.
§7.2),
• Skill and training of maintenance staff (cf §5),
• Support from vendors if necessary (cf §4.3),
• Availability of updated documentation and procedures (cf §6.2),
• Availability of tools and software (cf. §6.6)
Depending on the nature of the fault, corrective maintenance is often time-critical. This has to be taken
into account in the above mentioned points.
Based on the use of dialup (fixed phone line, GSM or GPRS modems)
This technology transmits the necessary information over the public or utility-owned
telephone network.
21
o Connection to the Internet
The wide availability and low costs make the option of using the public network
infrastructures to start growing as the selected solution for the connection to the
substations.
Today, remote access based maintenance is basically used to access data and for remote diagnostics of
faults or failures (cf. §11). Remote testing or remote modification of settings or configuration is not widely
used today. It is however probable that these features will become more important in the future, due to the
technological developments and cost pressure.
As a result of the modern communication standards as the IEC 61850, a new generation of Substation
Automation Systems has been introduced, where all the equipment is connected to the same LAN, and
where this LAN can be easily connected to the corporate networks or the Internet at the substation level.
The expansion of the network and internet communications used in the DSAS make it necessary to
analyse in detail the strategies to be implemented to protect the substations in two main aspects:
Accessibility of the information.
Data exchanged.
The characteristics of the different solutions will require analysis of the security constraints and
implementation of fire-walls and other preventive measures. The details and actual trends in network
security are not in the scope of this document but will have to be taken into account when implementing
the remote connection to the DSAS.
Specific working groups and committees are working on the security problem and the cyber security and
can be referred for further details. (e.g. WG15 of IEC TC57, B5.38 [11]). This issue is very sensitive for
many utilities and its solution is a condition for a wider use of remote access.
22
Each utility has its own policy for the scope of remote access, including the authorisation given to vendors.
The acceptable use of remote access is different for each utility (change protection parameter settings,
global read-only access,…). There is however a tendency to increase the scope of utilisation of remote
access. Most applications are supported by the available technology.
23
have to take into account these generic time constraints in order to make sure that the architecture of the
DSAS and the characteristics of its different components allow to meet them.
The specification also should contain requirements covering the following subjects:
- Ability to add or to accommodate an extension of the substation,
- Possibility to add new or to modify existing functions,
- Spare part availability (including verification of version compatibility, cf. §9.2),
- Services for system and hardware maintenance (cf. §4.3),
- Acceptable constraints for software version update during operation of the substation,
- Maintenance tools (cf. §6.6),
- Documentation for maintenance operation including test procedures (cf. §6.2).
- Ability to test the functions of the DSAS by maintenance staff and the conditions of this testing (cf.
§6.3).
Level of specifications and maintenance principles can vary in large scale between large and small
utilities, between different voltage levels and depending on type and importance of substations.
In certain cases utilities decide to specify / approve one firmware version for the product which they
purchase. With this approach they reduce problems that could be derived from having different versions
installed. This approach may be recommended for fixed applications without need for extra functionalities.
However, a application may be difficult to maintain fixed over the lifetime of the DSAS.
As for the other technical requirements in the DSAS specification, the approval- and test process of the
utility has to include the verification of the respect of the requirements concerning maintenance.
24
updates are required in order to make sure that maintenance and testing can be performed correctly on
the system.
There are different types of documents related to the maintenance of a control system :
• Documentation of the DSAS (architecture, circuit diagrams, …),
• Maintenance procedures and guidelines,
• Documentation tracing the modifications of the system and its components,
• Maintenance actions, both corrective and periodic.
These different documents have to be identified in the specification. Any modification of the DSAS may
require an update of all or of a part of the documentation.
The implementation of a document for the control system is paramount to the success and efficiency of
any maintenance contract. It is important to clearly define the necessary paper work and who controls it.
Concerning the maintenance procedures and guidelines, the following types can be identified :
• Detailed testing guidelines for test teams, In detailed test guidelines, maintenance personnel can
find which devices and IED’s have to be tested. It gives also instructions and limits how to test
different kind of systems, devices and functions (distributed, centralized, individual functions).
They also define documentation which has to be produced after each test. Acceptance levels
should also be defined in advance. It can include requirements concerning testing tools and
devices. Most detailed testing guidelines are used for qualification and commissioning tests, the
same one or slightly limited testing guidelines can be used for periodic maintenance tests.
• Test guidelines for particular situations. Particular situations include IED replacement, IED version
update, DSAS version update, Network Control Center system change or replacement situation,
NCC telecommunication protocol change, DSAS extension. These particular situations are often
more problematic from a testing point of view and sensitive to the DSAS. They can be related to
DSAS maintenance or upgrading.
25
Table 6-1: Requirements with respect to maintenance related constraints
# Requirement Maintenance related constraint
1. Explicit prohibition of common Be able to perform maintenance operation on one
modes between certain equipment of the DSAS without de-energizing a feeder
functionalities or another part of the substation.
This type of requirement is especially important for a
high degree of Functional Integration.
Example : maintenance of Main 1 Protection requires
Main 2 Protection to be operational if the feeder cannot
be de-energised.
2. Include documentation for Availability of documents for operators and staff to
operation and maintenance in the support failure research and maintenance operation.
deliverables of the DSAS
3. Include documentation for test 1. Support test after corrective maintenance or during
procedure preventive maintenance.
2. Make sure that testing of all functions (especially
distributed functions) is possible without de-energising
a part or the whole substation.
4. Connectors as bay- equipment Simple, safe and secure connection and reconnection
interface for maintenance and repair. Avoid reconnection errors.
Guarantee electric continuity after reconnection.
5. Implementation of special test Minimal impact of tests on DSAS operation
modes in order to facilitate the test
of a given equipment or function
6. Switches for disconnection of the Visible and securable separation required by safety
bay from voltage sources (eg. regulations and operational procedures.
Connection to secondary side of
vt's, inter-bay signal exchange by
wires, etc).
7. Use of disconnection switches Facilitate maintenance and repair operation
inside the bay cabinet for testing
8. Identification of connectors and pins Avoid reconnection errors. Guarantee coherence
in the schemes and as labels between (standard) schemes and bay circuits.
associated to connectors.
9. Interface with primary equipment Special disposition for command, voltage and current
circuits in order to facilitate separations for test
purposes.
10. Use of fuses to open voltage Visible and securable separation required by safety
circuits and switches to shunt regulations and operational procedures.
current circuits
11. Dimensions and mechanical Facilitate access to the bay and its equipment
characteristics
12. Ergonomic considerations Facilitate maintenance
26
The testing procedure has thus to consider the implementation of the functions and cannot, in some
cases, be performed as an almost independent procedure for each functionality.
In the case of Functional Integration, in some countries there is "one IED per bay" for control and
protection, especially on MV level. Functional integration makes testing procedures often more difficult to
perform. Best way for these tests is to take the bay out of service, if possible. Two redundant identical IED
may help to mitigate this problem.
Taking into account the severe constraints for availability of a significant part of a substation at any given
time, the only - and last - possibility for a complete unit test of some distributed functions is perhaps at the
commissioning of the DSAS of the substation. This is due to the fact that afterwards it will not be possible
to put simultaneously all concerned equipments in maintenance mode. In case of refurbishment with
migration, the last possible moment might even be the Factory Acceptance Test. However, FAT does
normally not cover the connection of HV equipment and telecontrol. General qualification on prototypes in
case of the purchase of several substation can also help to test distributed functions before
commissioning.
In DSAS with Functional Integration, a test of a given function may lead to the unavailability of the other
functions implemented in the same equipment [2]. If this is unacceptable or too onerous, alternative test
procedures have to be considered.
One possibility is to perform a set of test verifying each elementary function of the equipment concerning
the function to be tested : binary and analog inputs, information coming from the local communication
network, proper configuration of the equipment, proper operation of the CPU, binary and analog outputs,
information sent to the local communication network (Figure 6-1).
A consistent subset of these tests thus can cover all aspects of a complete unitary test of the function
without having the same impact on the availability of the other functions implemented in the same
equipment. Distributed Functions are in most cases also functions concerned by Functional Integration, if
they are implemented in several equipments together with other functions. The sequence of the tests
shown in figure 6-1 is random.
Equipment #1 Equipment #2
S/S Bus Interface S/S Bus Interface
Analog In Analog In
Test 1 : S/S NW
Filtering - Filtering -
Signal treatment Signal treatment
Test 2 S/S Interface Eq1
CPU Test 3 S/S Interface Eq2 CPU
Memory Test 4 Analog In Eq1 Memory
Configuration Test 5 Analog In Eq2 Configuration
Binary I/O Binary I/O
Function Function Test 6 Sig. Treatment Eq1 Function Function
#1 #2 Test 7 Sig. Treatment Eq2 #1 #2
Test 8 CPU –Config. F1Eq1
Test 9 CPU –Config. F1Eq2
27
6.5 Test coverage
[4] gives detailed recommendation of the test which should be performed in the different phases of the life
cycle of a DSAS, including tests in the context of preventive or corrective maintenance.
This paragraph does not aim to give extensive recommendations about the tests related to maintenance
operation, but rather some suggestions that may be taken into account by utilities. A general principle is
that, after a corrective maintenance, the tests performed should aim at verifying that all the functions by
the replaced or repaired equipment are again completely available. If during the operation connectors or
switches have been operated, the tests also should verify that the contacts are again closed, including
especially the contacts and cables connecting the HV equipment.
The utility and the vendor have to agree about the necessary tests after performing a Software upgrade,
depending on the included changes.
1. Commissioning and qualification test versus maintenance test
Commissioning and qualification are the most important tests concerning the life cycle of DSAS or
individual IED. A lot of hidden failures can be avoided with appropriate and complete commissioning or
qualification tests. Maintenance tests can not have the same coverage as commissioning test. This limits
the extent of the maintenance test. The maintenance tests do not have the same objective as
commissioning or qualification tests. There are also often time schedule limits for periodic maintenance.
2. Tests with energized and de-energized feeders
It is possible to perform DSAS maintenance tests with the primary either energized or de-energized. Many
companies think that the maintenance of secondary systems should be done with de-energized primary
system because this allows a complete coverage of the test (proper trip circuit testing, measuring circuit
testing, circuit breaker and disconnector operational testing, alarms). However, de-energizing is often not
possible in heavily loaded and/or radial networks, limiting the maintenance test scope. In meshed
networks and less loaded networks maintenance de-energizing the primary equipment of the bay to be
tested is in general possible.
The preparation of the maintenance may be more difficult in case of energized feeder (be sure to avoid
unwanted tripping and more paperwork with dispatching). In some cases, it may be possible to combine
planned maintenance of DSAS with planned maintenance of primary equipment. Some distributed
functions, like busbar protection can only be tested during maintenance with at least some feeders
energized
3. Improving of maintenance test methods for numerical IED’s
There is a significant difference in maintenance testing between numerical and traditional relays and IEDs.
With numerical IED’s the testing of measuring circuits is facilitated because of clear measuring on-line
screens and features. Use of setting based files helps especially when testing IEDs. Using prepared
COMTRADE files or disturbance recording files from recorded faults provide new possibilities for testing.
28
• Stand alone tools or tools implemented in the HMI of the DSAS providing support for changing the
parameters and modifying the configuration of the DSAS.
• Secondary injection devices. For some maintenance procedures, "soft" injection of sample values
via process bus (IEC 61850-9-2 or equivalent) may also be used.
• Primary injection devices comparable to those used for protection devices in order to verify the
operation of protections and control algorithm implemented in the DSAS. These devices may
inject real currents and voltages at the terminals of the HV equipments.
• Tools providing the possibility for a "soft" test of the different functions implemented in the DSAS.
The utilities should state in their specifications the requirements and the constraints for the different types
of maintenance which may occur in the lifetime of the Digital SAS and ask the vendor to provide tools
enabling these types of maintenance action. The ergonomics of the tools, the safety of the operators and
requirements in order to avoid unwanted events (eg. untimely circuit-breaker or disconnector operation)
should also be covered by the specifications. They may also include requirements aiming at the possibility
of using existing tools (in particular injection) for the maintenance of the Digital SAS. The specifications
should also require that the maintenance tools associated to the DSAS enable maintenance operation
according to the conditions discussed in §6.1 to §6.5 above.
Thought should also be given to license issues for maintenance tools, which should be included in the
delivery scope of the DSAS and requirements concerning the use of portable maintenance computers for
which some utilities may have restrictions. The maintenance tools have to be available over the
complete lifetime of the DSAS and thus require also support of their own (update, obsolescence
strategies, repair…), which may be included in the DSAS specifications and / or in the maintenance
contract.
CIGRE WG B5.31 has described a generic process that can be applied for managing settings over the life
of a protection device or system. This generic process can be used regardless of company structure or
whether parts are outsourced, providing hand off of information is clearly defined in an organization with
defined roles and responsibilities [13].
29
7.2 Impact of settings, firmware and hardware versions on maintenance
It is a challenge for the utilities to ensure that the settings and the parameters in the DSAS systems are
identical to those stored in the office. This uncertainty about the validity of the settings stored in the office
can affect the security of the electrical system and generate mistakes on routine activities.
Strict management and quality control are required to assure that the combination of the DSAS hardware
version, firmware version, scheme logic and settings files are compatible and correct for the scheme and
can be modified in a controlled way to cope with changes. This generally means that a utility standardises
on one version which is extensively tested and all necessary support documents are generated.
During the lifetime of the DSAS, a manufacturer will inevitably bring out new versions of the components
and there may be a need to update devices installed and commissioned at site. This change may involve
changing the hardware version, firmware version, programmable logic version/configuration tool or
settings file/settings tool. In general the new IED version will consist of a combination of these depending
on the nature of the change or improvements. This change is usually strongly resisted by the utility as it is
preferable to maintain one version of the numerical IED installed across the utility population, for ease of
management (this is also visible from the responses to the questionnaire §11). There is also the impact of
the disruption caused in visiting sites to carry out the upgrade, the resources and impacts on system
access and availability and spare-part management. In some case the upgrade has to be applied because
the reason for the change is to mitigate a detrimental effect on the performance of a function of the DSAS.
There is a risk of regression of existing DSAS functions when modifying or upgrading system versions. A
new hardware version/firmware version is required to undergo an acceptance procedure which may
include testing to confirm the functionality and performance have not been affected. This generally
involves a level of factory testing of all DSAS functions as used by the utility, which may include specific
dynamic tests. The outputs will be new set of test results, test and support documentation to manage the
DSAS. The aim of this testing is to prove that the existing settings and functionality is unchanged by this
new release. If new settings files are required to match the new firmware, these must also be tested. It is
expected that a full set of functional tests are carried out, including secondary injection.
Similar considerations also concern the maintenance tools of DSAS, including those which are not directly
implemented in the DSAS (cf. §6.6).
Once the utility is satisfied with the DSAS performance and all necessary documentation has been
produced the firmware can be deployed at site either under circuit outage or protection depletion
conditions. Method statements, Risk Assessments and Test Schedules have to be produced to manage
the risks of installing new firmware on already commissioned circuit protection schemes. Depending on
the nature of the modification, before installing the firmware the protection IED outputs (including any
communication trip outputs) may have to be isolated from the system to prevent any unwanted operation
during the upgrade.
After extraction of the current settings files, the IED can be powered down and its firmware upgraded. The
settings should then be re-applied or new settings file installed as appropriate. If a new settings file has
been installed this must be verified. Finally the IED can be put back into service, with any final on load
commissioning tests deemed necessary and the setting files and test records returned to the engineering
centre.
Furthermore, with multifunction numerical relays the settings include the configuration and interconnection
of those functions that have traditionally been handled by wiring connections on a panel. To fully benefit
from achieving advantages with reduced wiring, it is necessary to properly handle changes in
configuration and avoid errors. This requires that the accompanied setting-, parameter- or configuration
changes are fully documented and controlled.
30
8 Considerations regarding enhancement and modifications of DSAS
after commissioning
31
These modifications can be identified as "upgrading maintenance" (cf. 2.5.1). It is very important that the
design of the DSAS allows these modifications and that they are carried out with future maintenance in
mind. The main problem is of course to make sure that these modifications can be implemented over the
whole lifetime of the DSAS, including the period when the system can no longer be sourced by the vendor
i.e. it is obsolete. It is therefore recommended that the utilities and the vendors agree on the possibility
and the procedure covering this kind of modification in existing DSAS. This agreement may be included in
the maintenance contract. It is also possible to agree that some modifications may imply the use of new
generation equipment. This in turn may require a definition of the qualification or acceptance tests and
clauses aiming at guaranteeing a compatibility between actual DSAS and possible substitution devices.
Standards such as IEC 61850 facilitate upgrading maintenance in any stage of the DSAS lifetime.
As a general rule, every modification of a DSAS after its commissioning should be preceded by a proper
preparation limiting the risk of the operation. This preparation should comprise:
• Validated written procedure including risk evaluation and fall-back scenarios.
• Depending on the modification : validation on a test platform.
• In case of modification performed by vendor : proper authorization by utility.
These general rules apply for modifications done on-site and for modifications done by remote access.
32
• Complete product(s): in the case of configurable products, spare product(s) are preferably having
hardware and or software configuration that covers several products, i.e. as general as possible
for the actual installation.
• Complete IED´s with factory default configuration
Utilities and vendors have to define together the appropriate scope and consistency of spare parts
(complete devices, spare boards, ..) based on a risk- and cost analysis and on the characteristics of the
equipments and its location in the DSAS and the network (are single boards easy to replace ?). This
evaluation can be performed by the vendor as a deliverable of the purchase order of the system.
During the lifetime of the DSAS, the general knowledge gathered by the manufacturer from repair
statistics, combined with the knowledge gathered from a utility (cf. §4.2) may contribute to determine and
to improve the content and volume of spare-parts needed.
Information exchange between manufacturer and utility about repair statistics is an approach that may be
efficient in order to determine the type and number of spare-parts for a certain installation, however this
will require that both manufacturer and utility agree to exchange information about repair- and failure
statistics. There are nowadays obstacles for sharing this information. Statistics about outages,
disturbances and repairs, both of the power grid itself and the secondary devices, are sensitive
information for every company. Utilities and vendors may include agreements concerning the exchange of
these in formations in the tender-, purchase- or maintenance contracts.
1. Engineering tools for setting, configuration etc. may require old operating systems and
connectors no longer supported and no more running on the PCs used by the utilities
IT-organisation.
2. If tools require passwords, handling and storage must be secured on a utility level, not
on a personnel level.
4. It has to be made sure that the software and/or hardware version of spare-part module
are the same or compatible compared to the faulty module. The installed IEDs may
have been upgraded with new software, but the spare-parts may have been forgotten.
It is recommended to start upgrade with the spare-parts and to include this aspect in
the maintenance contracts.
1. The design of the devices should enable a simple and fast standard exchange for
spare-parts where specific configurations is not required. This replacement should be
performed preferably without having to use specific configuration- and setting tools.
This may imply a copy of the configuration and settings stored apart from the device.
33
The concept of a "super"-device as a spare-part may be antagonistic to a simple
"standard exchange" of a device.
2. It has to be made sure that available spare-parts are compatible with the different
versions of the devices in the DSAS they may have to be inserted. This implies a
backward compatibility of these parts, since several hard- and software versions may
coexist in one same substation. This backward-compatibility has to be guaranteed by
the vendor. If a backward-compatibility cannot be maintained, the vendor has to inform
the customers concerned by this limitation.
It is recommended to include this clause in the maintenance contract.
1. Pre-configuration: the spares are already pre-configured when stored. This gives the
advantage that time for repair will be short. Disadvantage may be that a lot of spares
need to be stored due to different configurations.
2. Event driven: spares will be configured when needed. This gives the advantage that the
number of spares will be reduced. Disadvantage may be that it will take time to
configure the spare when it is needed and consequently the time to repair can be long.
Remote diagnosis if available can be helpful to determine which board is affected. This
helps the service crew to take the right spares and how to configure. This can reduce
the MTTR and the size of spare-part stock.
3. Auto-configuration when spare parts are inserted. This is not something that is
available today, but should be a consideration for future designs.
In any case hardware and software configuration and settings of spare parts have to be
checked before using them.
Utilities use their own service organisation with established routines for handling of spare-parts
and related issues like storage, personalisation/configurations etc. Even if the spare-part stock is
managed by the utility, the utility usually still relies on the vendor for the repair of the components.
If third parties are using these parts for maintenance of the DSAS, the utility also has to provide
access to them.
2. Done by vendor/manufacturer
Vendor/manufacturer has the possibility to deliver spare-parts on demand from a general stock or
after production of the actual spare-part. If no spare-parts are available on stock or if production of
new parts is not possible, repair of the broken part is a third possibility. Another possibility is a
customer dedicated stock provided as a service by the vendor/manufacturer i.e. a consignment
stock. Service like this will ensure that the stock of spare-parts will be up to date with respect to
both number of spares and version revision handling due to changes of the spares. In the above
cases vendor/manufacturer does not interact in the actual replacement of the broken part.
Vendors know well the equipment and the technology and may thus have an advantage in the
management of the spare-part stock.
3. Done by third party
A third party taking a general responsibility with established routines for handling of spare-parts
and related issues like storage personalisation/configurations etc. This has advantages and
34
inconvenients comparable to the outsourcing of maintenance activity. It basically comes down to a
trade-off between cost and control. This has to be defined in the asset management policy of the
utility.
In general, there will be several of these strategies applied for the spare part management of different
devices and parts of the DSAS.
In any case, it should be made sure that the spare-part stock is correctly maintained and periodically
verified. The entity which stores the equipment, has to provide space, check the stock, ensure that it
is tamper proof, in good working order and make sure that the correct stock and numbers is available
and kept it up to date.
On the other hand, modern electronics, once being exposed to increased temperatures during its
normal operation can be damaged and it may be critical to use it on some other place, or even to
move it. The correct state of the components and devices has to be verified before using them as
spare parts.
Handling and storage of electronic parts must be done with respect to ESD (Electro Static
Discharge), temperature, environment etc in accordance with the manufacturers recommended
procedure.
For numerical equipment, there is some experience with ageing parts. Except some components like
power supply or I/O boards, an exchange of specific digital components is in general more difficult.
Electronic components have today relatively short commercial life-time compared to the required life-
time of the DSAS. This forces the manufacturers to act and take decision on each change in the
electronic component life-cycle. The result of such decisions can be either to buy enough
components LTB (Last Time Buy) for the remaining life-time of the DSAS including need for spare-
part or find a component that replaces the obsolete one. Replacing a component can result in a
design change with corresponding test and verification.
35
10 Case of systems based on different communication standards
Although modern DSAS can support a great variety of protocols by means of integrated gateway
functions, there are some impacts of these protocols on the maintenance strategy.
The key to these impacts is tightly associated with the data models:
• Many existing protocols have their roots in RTU applications and are using signal-oriented data
models. These models are containing a list of anonymous signals, arranged in only few function
groups like messages, commands, values, etc. Typical examples for such protocols are IEC
60870-5-10x or DNP 3.0 or Modbus, and most proprietary protocols.
• In contrast there are the domain specific data models, which are object-oriented and reflect the
structure of their domains (e.g. DSAS as a domain). These data models define hierarchical
relations between the single data objects and therefore cater for a strong data consistency even
over many years of operation. A prominent example is the IEC 61850.
Basically both kinds of data model have their place in substation automation: whereas the signal-oriented
models are easy to use in refurbishment of RTUs, the object-oriented data models are very efficient with
new, distributed station automation systems.
Independent from its design the data model of a substation has to be a subset of an overall data model of
a whole utility. As many DSAS – users have equipment from different vendors, they sometimes even
define their own data model, which is independent of the vendor specific tools and proprietary data
models. Such solutions can be mainly found in big utilities with large grids to maintain.
One of the main benefits of the IEC 61850 is that it defines a data model, which is independent from the
system and even from the used communication. This saves the investment of engineering for a very long
period, even beyond the end of lifetime of an installed station automation system. This concept makes the
data model future proof. However, many implementations today cannot make use of a complete
independent data model yet.
The IEC 61850 has a great impact on maintenance. This is due to the fact that this standard goes beyond
the scope of a communication protocol and defines the data model of the DSAS. The use of GOOSE and
Sampled Values („process bus“) reduces panel wiring and as a consequence changes testing strategies.
Instead of conventional measuring instruments new software tools are to be used. They provide
comfortable features for simulation, test logs and reproduction of tests, but also ask for trained experts
using them.
The need to disconnect single devices from all sides during test is now simplified and reduced, e.g. by
setting testbits according to the standard. Therefore it has to be ensured, that all IEDs and functions in a
system are treating the testbits correctly.
In case of proprietary protocols, there is also the impact of maintaining knowledge how to
change/enhance data to be transmitted. Whereas global communication standards are stable for many
years, which makes maintenance much easier on the long run, proprietary protocols are typically
implemented in fewer applications and often are issued in several versions and even project specific
implementations. Increased maintenance efforts may be the consequence.
The tools required to maintain systems using these protocols may be vendor specific and controlled by the
manufacturer. On the other hand, tools are in general available for open or standardised protocols.
A special situation appears, when a device or remote station is to be renewed, and the user has to
coordinate different vendors using a proprietary protocol. This can happen, when one or more IEDs, an
RTU or a control center have to be exchanged due to their end of lifetime. See also "Exchange".
36
Table 10-1 Overview of the impact of communication standards on DSAS maintenance
Task Impact of proprietary Impact of standard Impact of
protocols protocols IEC 61850
Remote maintenance Sometimes defined, Not defined in Vendor dependent, but
but often limited standard, but easily easy to implement and
service supported by transmit
transparent
transmission
Spare part stocks In some cases special Standard components Standard components,
components required e.g. star couplers, e.g. switches, etc.
switches , etc.
Training Available from original Available from many Available from many
vendor only, depending service providers, service providers, long-
on prod. lifecycle long-term supported term supported
Testing, Tools from original Tools available from Tools available from
Assessment vendor and vendors and vendors and
independent suppliers independent suppliers, independent suppliers,
in some cases, compatibility is clearly conformance is part of
dependent on product defined standard
lifecycle, comp. with
other vendors complex
Modification / Vendor-spec. tools and With vendor-spec. Vendor-spec. tools and
Enhancement -knowledge depend. on tools, long-term –knowledge, long-term
lifecycle. Data knowledge mostly knowledge will be
exchange between available, simple data available, further
tools only project exchange (csv) often learning necessary,
specific, use of available, use of data exchange
gateways to other interoperability lists between tools (XML)
devices. increasingly supported,
differences in
interpretation of
standard
Update or upgrade Vendor specific, Vendor specific, Vendor specific,
depending on product depending on product depending on product
life cycle life cycle life cycle
Migration to new Supported by some Supported by some No experience yet
concept/architecture vendors vendors
Renewal (depending on Gateway function Easy due to standards, Easy exchange by
external interfaces) necessary, exchange simple exchange of retaining the data
of data model mostly data model model defined in
needs adoption standard
Support from vendor Different levels of Different levels of Different levels of
support available. support available. support available. Long
Expertise will fade out Long term expertise term expertise to be
after defined period of existing. expected.
product discontinuation
Table 10-1 gives an overview of the impact of communication standards on DSAS maintenance. The
following comments can be given:
37
Remote maintenance is getting more important due to unmanned substations and economic reasons (cf.
§4.7). It is available in modern automation systems, mostly associated with the use of proprietary tools
and therefore with proprietary communication. The access to the system can be located on station level
(single point of entry to the system), or at the location of a control center, or from a different place like e.g.
a maintenance center. Even mobile access with a notebook computer from a service vehicle can occur for
better flexibility of the service staff.
Remote maintenance can include remote diagnosis, remote change of parameters or settings, remote
testing, and even remote update or remote firmware loading.
Different performance of the basic communication structure results in either integrated or separated
solutions of remote maintenance. Whereas it is easy to use transparent transmission methods over fast
communication like e.g. IEC -101/104 or 61850, other standards like e.g. IEC 103 are very limited in
performance and ask for reduced or separated remote access.
Spare part stocks are also depend on the communication structure. Whereas with proprietary protocols
sometimes special devices, like specified modems or interface converters can be difficult to obtain after
some time, for standard communication there are compatible devices on the market for long term access.
This applies for star couplers, switches, routers and similar devices.
Training on systems with proprietary communication is typically available from original vendors only,
which means a dependency on the related product lifecycle. In contrast there are many offers from service
providers for training on standard protocols and they are supported on long-term. Only in cases, where a
newer standard is substituting another standard on the markets, the availability of training offers for the
substituted standard will decrease.
Testing and Assessment are also dependent on the type of communication: whereas for proprietary
protocols tools are available from original vendors only, and therefore dependent on the respective
product lifecycle, for standard protocols there are also tools from independent suppliers on the market.
Furthermore there are clear defined criteria for interoperability for standard protocols, so it is easier to
service systems over a long lifetime. In case of IEC 61850 conformance testing is even part of the
standard.
Modification or enhancements always depend on vendor specific tools. Using proprietary protocols
means either to stick to the original vendor for enhancements, or use of gateways to other devices. The
availability of knowledge and support for a system is strongly dependent on the product lifecycle. Data
exchange between tools of different vendors is usually project specific.
Using standard protocols usually means long-term availability of knowledge about the communication
part, which represents the most important function of station automation systems. Enhancement is easier
due to interoperability lists and data exchange between tools is often supported by import / export of csv-
files. In case of IEC 61850 this is even easier, because data exchange between tools by use of XML-files
is defined in the standard and engineered data is independent from products and communication. Anyway
the complexity and flexibility of IEC 61850 results in the need for special trained experts and differences in
interpretation of the standard are still possible, but will decrease with further progress of future editions of
the standard.
Update or upgrade of a system is depending on the specific product life cycle independent of the chosen
communication.
Migration to new concept/architecture is supported by some vendors and is specifically dependent on the
involved products. Migration strategies are often related with the involved communication and new
standards, as IEC 61850, are often drivers for new migration strategies.
Some devices on the market can be upgraded to the new standard, and there are strategies to exchange
old devices with new ones, which can support both old and new communication, and furthermore can
adapt and import the old engineering data so that a stepwise renewal of systems with a minimized effort,
cost and interruption of operation can be achieved.
Renewal of a substation automation system is often related with the compatibility of external
communication interfaces, e.g. to existing control centers or neighbouring substations. Whereas
proprietary protocols need a gateway function and additional effort to harmonize the data models, plus
38
more effort for testing and commissioning, standard protocols make it easier. Also the exchange of the
data models is much more simple using communication standards.
Support from vendor is available on different levels, depending on the kind of service contract, which is
concluded. Anyway the expertise on a system will fade out after a defined period from product
discontinuation. This period is subject to contract negotiation, but there are recommendations included in
IEC 61850.
For standardized communication long-term expertise is existing and is also to be expected for new
standards like IEC 61850.
39
utilities seem to try to synchronize the replacement of HV equipment (every 40 years) and the
replacement of DSAS. This leads to an expected lifetime of DSAS of about 20 years.
• Half of the utilities do not expect an impact of IEC 61850 on DSAS maintenance in short term.
Most utilities think that there is a potential to improve maintenance, but there are also problems
identified concerning updating maintenance:
- modification of interlocking and goose messages when adding new IEDs (upgrading
maintenance).
- proper handling of alarms related to GOOSE-message faults.
- impact of IED faults on functions using information from the concerned IED (eg. busbar
interlocking).
• There seems to be a general consensus among the utilities that the use of IED allows to reduce
the maintenance time spent locally in substations. This reduction is estimated at an average of
50%.
• About half of the utilities state that they use a comprehensive asset management system for
DSAS, which is utilized by several of their departments.
40
11.4 Training of DSAS Maintenance
• About half of the utilities consider that the training courses offered by the vendors cover their
needs, sometimes a specification is required. This in turn means that about half of the utilities do
think that the training courses do not correspond to their needs. One possible interpretation is that
the training courses are too focused on single IEDs without talking about the system.
12 Conclusion
A considerable number of Digital Substation Automation Systems are already in operation, and utilities
worry about the most efficient, cost effective and safe way of maintaining them during their lifetime. There
is a general consensus within the industry that maintenance strategies cannot be the same as those
employed for conventional SAS and would need to be adapted to meet the their needs: this is related to
the particular characteristics and implemented technologies of DSAS.
The organisation of the maintenance activity of utilities has also to be adapted to this new context. This
includes long-term maintenance contracts and the review of the possibility of outsourcing a part of this
task to vendors or third parties.
System standardisation and optimisation of maintenance is a continuous task for all utilities.
It also appears that maintenance of DSAS has to be taken into account at the system design stage. The
system has to be designed with maintenance in mind. DSAS also require specific training, test methods
and procedures, documentation and skilled staff. The expression of a "paradigm change" was recently
employed to describe the amplitude of this evolution.
One key element is to acquire and to maintain a sufficient knowledge of the staff in charge of DSAS
maintenance. This requires to adapt the content of the training courses and to plan periodic updates. In
this context, the training courses for maintenance staff should more concentrate on how to solve the
problems than on how to build the systems.
These requirements also apply to DSAS based on IEC 61850. In addition, this standard may help to make
the maintenance for DSAS of different vendors more homogenous. Its application supports a self-
documentation of the configuration and of some other information of the DSAS (but not all). At the same
time, the use of these models and the use of goose messages and test-bits has a significant impact on
maintenance of DSAS. But the reality of thousands of "legacy" DSAS deployed by utilities must not be
forgotten. The main stakes of DSAS maintenance in the near future concern these systems.
It has been demonstrated that the version management of DSAS configuration, both for hard- and
software is of major importance for the long-term maintenance of these systems. In practice, only the
utilities are able to put into place version management systems that properly reflect reality in the field.
Remote access is believed to play a more important part in the future, in spite of security issues which
have to be addressed. It may be possible to perform on-line updates of centralised databases of the
utilities containing setting and configuration information using standardised data models and remote
access.
Much work is clearly to be done in the field of maintenance testing in order to obtain adequate tools and
validated and standardised testing procedures. Maintenance procedures and test procedures after
maintenance operations should be included in the deliverables of the DSAS documentation. Attention has
also to be paid to the maintenance tools, which may experience similar constraints concerning the
maintenance of their hard- and software than the DSAS itself.
The issues concerning life-time maintenance of DSAS for utilities can be summarised as follows: Utilities
should not try to apply the organisation and procedures put into place for maintenance of conventional
SAS. Maintenance of Digital SAS has to be adjusted and adapted.
41
13 References
[1] V. Leitloff, N. Petit: "Determination of functional specifications based on analysis of
constructive constraints of substation control equipment."
SC B5 colloquium Calagry, September 2005
[2] SC B5.13 TB "Acceptable Functional Integration"
[3] V. Leitloff , P. Bongrain : Accepted Functional Integration in RTE's Specifications of digital
DSAS
SC B5 colloquium Madrid, October 2007
[4] SC B5 WG B5.32 TB "Functional testing of IEC based systems"
[5] SC B5 WG B5.18 TB "Guidelines for specification and evaluation of SAS systems"
[6] IEC 61850-4 Part 4 Communication networks and systems in substations – Part 4: System
and Project management
[7] SC B5 WG B5.08 TB "Refurbishment Strategies based on (Total) Cost of Ownership,
Operational Risk and Technical Constrains"
[8] SC B5 WG B5.06 (PETRINI M, LOPEZ DE VIÑASPRE A, LOUKKALAHTI M., LEITLOFF V.:
"The impact of digital technology on the maintenance of Substation Automation Systems"
Paper #104 Study Committee B5 Colloquium October 19-24, 2009 Jeju Island, Korea
[9] Volker LEITLOFF (Special Reporter): Special Report on Preferential Subject 1 – "Strategies
for the Life-Time Maintenance of Substation Automation Systems"
Study Committee B5 Colloquium October 19-24, 2009 Jeju Island, Korea
[10] Harri Rytkönen : Master of thesis "Defining the Optimum Routine Test Interval for Control
and Protective Systems in Substations", August 1997
[11] SC B5 WG B5.38 TB "The impact of implementing cyber security requirements using IEC
61850"
[14] Paul GILL: Electrical Power Equipment Maintenance and Testing, 2nd edition, CRC Press,
2009
42
14 Annexe 1: Restitution of questionnaire
WG B5.06 - Questionnaire for utilities on Maintenance of Digital
Substation Automation Systems
Digital SAS are systems using digital communication on substation level where the main functions are
implemented in Intelligent Electronic Devices such as bay controllers or digital protections. Some
elements, e.g. particular protections, of different technologies, may be part of these systems.
General information
Maintenance principles
3. Which maintenance principles are utilized by your company for Digital SAS:
• corrective maintenance (event-triggered) Y : 92%
• preventive maintenance
o periodic maintenance Y : 100%
o condition based maintenance Y : 50%
o reliability based maintenance Y : 33%
• updating maintenance, (e.g. improvement of reliability or performance)
Y : 75%
• security maintenance, (e.g. improvements due to regulation/policy for physical or cyber
security) Y : 42%
• other strategy (please explain) Y : 8%
4. In case of periodic maintenance of the DSAS : What are the periods for maintenance?
• visual inspection : average 10 months
• mechanical maintenance (e.g. dust cleaning) : average 2.2 years
• testing of protections relays : average 3.5 years
5. Do you use remote maintenance (remote monitoring and/or remote configuration and setting)
for DSAS? Y : 67%
If yes:
• Did you change the periods for local maintenance in the substations as a consequence of
implementation of remote maintenance? Y : 100%
• Do you have a dedicated communication network for remote maintenance?
Y : 42%
• Is it physically separated from other networks and the internet? Y : 42%
• Do you use public Internet for maintenance purposes? Y : 8%
• Do you have an enhanced data security concept? Y : 50%
43
6. Does your company directly (not the vendor) use diagnosis tools for the maintenance of Digital
SAS (not only for protections)
• locally Y : 75%
• remote read only Y : 58%
• remote read and write Y : 25%
8. Do you authorize the vendor to use diagnosis tools for the maintenance of Digital SAS
o locally Y : 83%
o remote read only Y : 42%
o remote read and write Y : 25%
11. Do you expect the introduction of IEC 61850 will change your maintenance principles?
Y : 50%
12. Modern digital substation automation reduces wiring, interposing relays and other mechanical
parts. These DSAS also benefit from of self-supervision. According to your experience, is local
maintenance in the substation reduced due to Intelligent Electronic Devices? Y : 83%
What percentage ? average 57%
13. Do you collect data for any statistic survey and analysis (national / international / vendor
driven) on reliability and lifecycle of DSAS? Y : 50%
Is one of the products of this analysis a kind of DSAS "health index" ? Y : 25%
14. Do you make use of a comprehensive Asset management system for DSAS, which are
utilized by other departments within your company, like grid-planning, grid-operation,
maintenance, workforce management, purchasing, etc.? Y : 42%
44
Maintenance organisation
15. For upgrading maintenance (addition, evolution), do you use a central database for all
substations with digital SAS,
• comprising of the complete data model Y : 50%
• comprising the settings parameters Y : 58%
• comprising signals and other configuration data Y : 50%
• providing also configuration management Y : 25%
• providing also management of settings Y : 42%
• comprising design documentation Y : 50%
• comprising user manuals Y : 50%
17. In case of maintenance contract: is the vendor of an DSAS also the contractor for
maintenance? Y : 58%
18 Do you have a spare – part strategy for the devices of Digital Substation Automation Systems?
Y : 83%
If yes:
• do you have your own spares on store? Y : 67%
• do you have a special contract on spares with the vendors? e.g. for different cases, like
repair/exchange/express delivery, all-in or case-by-case. Y : 50%
• do you utilize a local store for spares, which belong to the vendor?
Y : 17%]
19. If the DSAS system involves interfacing to protection, is this a different skill set and level of
authorization in your company. Y : 25%
45
Maintenance methods
20. Do you load every available software-update into your Digital SAS?
• never Y : 0%
• only, when a severe bug has been detected in your application, Y : 67%
• on recommendation of the vendor Y : 83%
• always Y : 0%
21. Do you use or do you think it is useful to have software support for:
Training
22. Do you consider that the training courses on maintenance of DSAS offered by the vendors do
cover your needs? Y : 42%
46
15 Annexe 2: B5.06 Colloquium paper
47
Study Committee B5 Colloquium
October 19-24, 2009
104 Jeju Island, Korea
2
Advantages of insourcing are: high knowledge level and skill of the maintenance staff in the
utility, local switching operation permissions of the maintenance staff with no need of
additional supervision of the subcontracted personnel, shorter response times, faster
information flow inside the utility. Drawbacks of insourcing are: no strong commitment to
reduce maintenance costs and the need to train people (with all the related cost) in order to
guarantee the necessary competence.
The main advantage of outsourcing is the reduction of responsibility and training cost for the
utility. The main drawback of outsourcing is the dependence on the vendor (or third party)
and the consequent risk, specially in case of bad performance of DSAS and/or not fulfillment
of contract requirements by the maintenance company.
There are also intermediate positions, consisting in a combination of the previous two
strategies, e.g. insourcing all the maintenance activities but the management of spare parts
stock and of complex tasks, that can be subcontracted to the vendor or a third party.
Utilities, according to their requirements, can negotiate and sign maintenance contracts with
the vendors, normally for a defined period, but with a time extension option: usually, a
minimum of three years for unitary products and five years for systems is desirable, mainly
due to set up costs, as internal training, simulator investments, etc., but there are also cases
of contracts that cover the whole system post sale phase for ten-fifteen years. The contract
and the related level of support selected will be also driven by the size and complexity of the
DSAS.
3
• The specification has to be oriented in order to enable a system design whose need for
maintenance is reduced to a minimum and whose maintenance can be performed in the
simplest possible way (e.g. requirements may concern self-monitoring functions,
configurability, interoperability, remote access, redundancy, function availability and
device reliability);
• The specification must be written taking in consideration the strategy for maintenance
(insourcing/outsourcing), that has to be defined in advance; the requirements concerning
training shall also be defined in the specification according to this strategy.
A positive approach seems to be involving people from the maintenance department in the
specification phase and people from design team in the maintenance activity; this helps the
latter to “feel” the impact of specification on the whole life of the system, to understand which
are the bottlenecks, to avoid some mistakes, and the former to know the system in advance
and to be already trained at the time of the installation.
4
level (the last simulated): the more correctly is the switchgear simulated, the best is the
result of the FAT. This approach is usually adopted by many transmission utilities, but
some distribution utilities can’t operate in this way because in MV projects all the IED’s
are directly sent to the switchgear factory, to be installed into secondary cabinets of bays.
• SAT: Taking into account the severe constraints for availability of a significant part of a
substation at any given time, the only possibility for a complete unit test of some
distributed functions can be at the SAT of the DSAS of the substation; this is due to the
fact that afterwards it will not be possible to put simultaneously all concerned equipments
in maintenance mode (in case of refurbishment of the substation on itself it may also not
be possible to put in maintenance mode all the equipments at the same time during
SAT). During SAT all the system functions and connections should be tested.
• Planned/periodical tests: Usually only performed for protection IED´s. At this level the
impact of self-check function has the largest influence on modern maintenance for
secondary devices. The self-check function of numerical IED´s gives the opportunity to
decrease the amount of planned maintenance.
5
outage/fault is a quite hard task, much harder is the task of comparing the related costs (of
maintenance and of the outages): this is a problem common to all utilities and vendors, so a
way to find a solution to it should pass through the sharing of information at international
level.
11. Conclusions
The introduction of DAS brings changes for maintenance in terms of subject, content, time
periods. The dependence of utilities from vendors surely increases at the beginning, but its
level can be mitigated, according to the utility strategy, by means of an adequate training
program. The whole life cycle of a DSAS, from the philosophy to the decommissioning, has
to be rethought in a more integrated way and human resources always must be considered
at the centre of the process: for instance, maintenance people, if involved, can help in the
specification phase, and people from the design team can learn form the maintenance
activity and develop systems that are easy to maintain. Everything has to be thought,
decided and planned in advance: for example, everything that can be tested during the
DSAS Qualification Process, or during FAT, contributes to simplify the post sale life of a
system and, therefore, its maintenance. More generally, involving all the people working on
this process in all the phases of the process itself, sharing their experience and information,
is a fundamental approach. Last but not least it’s the need to think of the new technology not
only as the renewal of the old one, but as a real new approach, that, on one side, can solve
some problems related to the conventional SAS and, on the other side, induces new
problems: one of them is the reduction of the duration of the system life cycle, that is a critical
issue, both for utilities and for vendors, and strongly impacts on maintenance as well.