Available online at www.sciencedirect.

com
Energy
Procedia
Energy Procedia 00 (2011)
Energy 000–000
Procedia 12 (2011) 27 – 36
www.elsevier.com/locate/procedia

ICSGCE 2011: 27–30 September 2011, Chengdu, China

Terrestrial-Satellite Hybrid Backbone Communication

Network for Smart Power Grid
Chuang Denga*, Xingquan Xiaoa, Zhong Fua, Ge Liua, Hongchang Yanga,
Junyong Liub
a
Sichuan Electric Power Corporation , SGCC, Chengdu, China
b
Department of Electrical Engineering, Sichuan University, Chengdu, China

Abstract

A smart power system demands a strong and self-healing communication system with greater capacity. In China, as
the ultra high voltage transmission lines and large scale interconnected grids are expanding throughout the nation,
disaster-tolerant communication network is essential for reliable system operation and control. In this paper, we
propose a hybrid communication network utilizing both terrestrial and satellite links to improve the system reliability.
The adaptive link selection protocol at network layer is developed to facilitate the network integrity. A modification
of transport protocol over satellite communication is also proposed to better exploit the limited bandwidth resources.
The performance analysis and field trial of the proposed scheme based on the practical network configuration in
Sichuan power grid is thoroughly studied. The results showed a great bandwidth saving of satellite resources without
loss of network integrity.
© 2011 Published by Elsevier Ltd. Open access under CC BY-NC-ND license.
Selection and/or peer-review under responsibility of University of Electronic
Science and Technology of China (UESTC).
Keywords: Hybrid, Satellite Communication, Backbone Network, Smart Grid.

1. Introduction

Power system communication networks are essential for system operations and control. The physical
transmission modalities for power system communications include wired terrestrial wired
communications such as optical fibers and power line communication (PLC), as well as wireless
communications such as satellite communications (SC), 3G or wireless sensor networks (WSN). For wide
area network (WAN) which is the backbone network which connects power generators, substations and
control center together, a wide-band digital data network is required to reliably transmit and control the

* Corresponding author. Tel.: +86-28-68133322.

E-mail address: eedeng@gmail.com.

1876-6102 © 2011 Published by Elsevier Ltd. Selection and/or peer-review under responsibility of University of Electronic Science and Technology of
China (UESTC). Open access under CC BY-NC-ND license.
doi:10.1016/j.egypro.2011.10.006
228 ChuangChuang
Deng etDeng et al. / Energy
al. / Energy Procedia
Procedia 12 (2011)
00 (2011) 000–00027 – 36

system [1]. In China, a dedicated communication network for power systems utilizing optical fiber
composite to optical fiber (OPGW) for transmission medium is largely deployed throughout the nation.
Compared to PLC, optical fiber could provide greater communication capacity and network robustness.
However, these wired terrestrial communications are very vulnerable to the natural disaster. The
communication cables are distributed along the long-distance transmission line through many
geographically tough areas. In 2008 Sichuan earthquake, one third of generators and substations lost
contact with control center due to the destruction of the optical fiber communication networks, an
important cause of blackout in the catastrophe[2]. After the earthquake, the satellite communication was
proposed to use as the backup communication for power system to resist potential natural disaster.
The modern satellite communication system could provide reliable broadband digital communications
[3]. The installation of satellite communication is of great mobility and flexibility. Due to its wireless
nature without terrestrial relay, satellite communication has a strong capacity against natural disasters.
However, the radio frequency resources for satellite communication are limited, making satellite
communication infeasible as a prime backbone network for power systems. A better solution is to use
satellite communication as a backup only when optical fiber fails. Based on this scenario, several
problems are needed to be addressed for a feasible and economical operation. First, the automatic link
switch between terrestrial and satellite must be established for network self-healing functionality. We
developed an adaptive link selection protocol to realize automatic link switch based on the IP-based data
network. We also developed revised TCP protocol based on the space communication protocol standards
– transmission protocol (SCPS-TP)[4-6]. This revision greatly improved the performance over lossy
wireless channels with large latency. Our results showed that bandwidth efficiency is improved by 60%
under typical channel state in practice. Through the methods above, we have achieved a stronger hybrid
communication network for power systems with greater capacity against disaster and economical
investments.

2. Satellite Communication for Power Systems

Fig 1 shows the illustration of the network setup. One of the modern satellite communication systems
utilizes a very small aperture terminal (VSAT) ground station for physical layer communications. By
using small-size RF units and highly integrated satellite modem, a VSAT ground station could provide
IP-based broadband communications which can be readily infused into existing networks. There are some
distinct properties needed to be addressed for satellite communication compared to terrestrial networks.

Fig.1. Illustration of hybrid terrestrial-satellite network for power system backbone communications
 ChuangDeng
Chuang Dengetetal.al./ Energy
/ EnergyProcedia
Procedia0012(2011)
(2011)000–000
27 – 36 329

Unlike optical fiber communication which could provide extremely large bandwidth, the satellite
communication suffers from many factors shared by all wireless communication technologies. The
transmit power of both ground station and satellites are constrained due to practical considerations such as
size and cost. The scarce radiofrequency resources for satellite communication also impose serious
limitations for bandwidth. As governmental and commercial demands on satellite communication grow,
the limited bandwidth occupancy will face a lot more contentions from end users. The cost is also an
important consideration. The rate of bandwidth usage is very high and proportion to usage duration,
making the cost of operation hard to cut down. Based on the limits above, satellite communication cannot
be used as the prime backbone communication and is often used as a backup for cables. A balance
between cost and integrity must be found to economically and reliably operate the systems. For satellite
SCADA system in practical power grids in Sichuan, the average bandwidth needed for each substation is
around 20 to 200kbps. A 2Mbps satellite channel could sustain 50 substations to transmit simultaneously,
around 30% of all important generators and substations for Sichuan grids.

2.1. Latency

The greatest distinct between satellite communication and other wired/wireless communications are the
transmit latency. The most commercial satellite communication systems use satellite at geosynchronous
orbits which is 36,000 km high above the equator since the overall cost for such systems is low compared
to near-earth orbits satellite systems. This distance naturally introduces the round-trip latency at about
250ms. In power systems, EMS/SCADA systems update the grid information every several seconds. So
they are sufficiently tolerable to latency induced by satellite communication. Although some delay-
sensitive services cannot rely on satellite communication (e.g. protective information), the EMS/SCADA
could provide enough grid operation information necessary for control center when emergency happens.

2.2. Bit-Error Rate

The internet mainly relies on terrestrial communications such as optical fibers which have an extremely
low bit error rate due to low noise of its transmission medium. The satellite channel, however, often
reaches a BER at 10e-5 to 10e-7 due to unstable wireless environments. The low quality of satellite link
will cause the data loss due to packet corruptions which is much more often than in terrestrial
environments. The error correction mechanism at transport and network layer must be introduced to
address the problem.

2.3. Network Topology

The star topology is often used in satellite communications. In star topology, a point to multiple points’
mode is used. This is especially suitable for many industrial applications including power systems where
multiple substations send data directly to a single control center. Mesh topology is also used in some
scenarios. In mesh network, each point may connect to multiple points. In our system, we mainly study
the communications between control center and remote substations. Thus a star topology will be used.

2.4. Multiple-Access Control

The main MAC for satellite are time-division multiple access (TDMA) and frequency-division multiple
access). TDMA allows multiple end users access the channel with a single carrier. In TDMA, slotted
access and contention based carrier sensing multiple access can be used for MAC. For this mode,
430 ChuangChuang
Deng etDeng et al. / Energy
al. / Energy Procedia
Procedia 12 (2011)
00 (2011) 000–00027 – 36

bandwidth efficiency is better exploited especially for sparse data. Since the contention based access
mechanism is in use, additional latency due to data collisions will be introduced. FDMA allocate a
determined carrier for each user, ensuring a dedicated transmission channel which is contention free. For
non-intermittent high throughput data transmission, FDMA will secure better performance at the cost of
high bandwidth resources. In practical power systems, each substation may require a large and continuous
data flow for system information. However, as a backup for cable communication, satellite networks are
idle when terrestrial cable communication works. An always-on FDMA mode will bring unnecessary cost
in bandwidth occupancy. Based on the circumstances above, we propose a hybrid MAC scheme described
below.
During standby period a narrowband TDMA carrier is established to manage the data network. In such
mode, only low throughput network management data are transmitted. Each remote substation maintains
a standby mode. When cable communication fails, the substation is triggered to establish a dedicated
broadband carrier, entering FDMA mode for high throughput data transmission. When data transmission
is end, the substation could change back to idle mode. This cost effective hybrid mode eliminates the
unnecessary bandwidth occupancy without compromise on network performance.
Bandwidth Pool

Carrier 1 Carrier 2 Carrier N

Substation N

Substation 2

Substation 1 Substation N Substation 1 Working Mode

Substation 2

Standby Mode

Fig. 2. TDM/FDM hybrid mode for satellite communications systems

2.5. Communication Interface and Network Protocol

In power systems, Ethernet based IEC60875-5-104 is mostly used in satellite SCADA/EMS systems.
This can be readily connected to satellite modems which also use standard Ethernet and IP protocol. The
satellite modems have limited routing capability since they are basically the physical layer devices. In
practical systems, a complicated data network with a variety of network layer protocol is used. For
instance, multiple Label Switch and Virtual Private Network (MPLS/VPN) are used to securely
differentiate different type of services and QoS control. This MPLS label cannot pass through the satellite
modem due to its limited process capability at network layer. We developed a new protocol to handle this
problem to be elaborated in subsection III.
 ChuangDeng
Chuang Dengetetal.al./ Energy
/ EnergyProcedia
Procedia0012(2011)
(2011)000–000
27 – 36 531

Fig. 3. Illustration of system setup at satellite links

3. Communication Protocol for Hybrid Network

3.1. Network Setup

The setup of terrestrial-hybrid data network of power system is illustrated in fig X. Aside from the
existing terrestrial configurations, the satellite link is added. The network has N virtual private network
(VPN)[7] to support different types of services. Different MPLS labels are added to the data from these
VPNs to enable MPLS routing through label switch router (LSR) in the data network. Since the satellite
modem has minimum network functionality, it does not support MPLS labeled switch. To overcome this
problem, MPLS gateway pair is used to address the MPLS label in order for the packet to pass through
the satellite modem. The detail will be elaborated in next section. TCP accelerator is to modify the TCP
protocol to optimize the transport performance and will be elaborated later.

3.2. Satellite Routing Protocol

The terrestrial networks are used for primary communications. As designed, satellite link is used as
backup channel. The dynamic routing algorithm (OSPF) will allocate greater link selection weight on
terrestrial routes based on bandwidth, latency and other network properties. When the terrestrial link is
disabled, OSPF will automatically switch the next hop to satellite links. When packets arrive at satellite
modem, the modem will switch to working mode, forward the packet to destination mode and establish a
dedicated channel (SCPC) for p2p communication by looking up preconfigured static routing table. After
recovery of terrestrial links, the link selection will automatically switch back to original routes. The
automatic link switch between terrestrial and satellite network function can be realized by the
abovementioned method.

3.3. MPLS Label Processing

The satellite modems are mainly designed for convert baseband Ethernet signals to radiofrequency
electromagnetism. Although these modems are generally using Ethernet and IP as the data interface, they
only have limited L2/L3 process capacity compared to dedicated switch and router. The practical
632 ChuangChuang
Deng etDeng et al. / Energy
al. / Energy Procedia
Procedia 12 (2011)
00 (2011) 000–00027 – 36

considerations such as manufactory technology and economical design are the probable causes. Some
advanced and modern network technologies such as MPLS protocols are not implemented in such
modems. To overcome this obstacle, we developed a dedicated device to process the MPLS label.
Fig 4 shows the basic mechanism for MPLS gateway. To establish label switch route, routers will
exchange MPLS labels using BGP labels. When these BGP labels pass through the MPLS gateway, they
will be copied and the corresponding MPLS label information will be stored in MPLS gateway. The
source/destination IP and the corresponding MPLS label will form a lookup table. The packet from
terminal A pass through the MPLS enabled router. MPLS label is then added to the packet and the packet
is forward to satellite link. The gateway will drop the MPLS label and the packet is now in standard IP
format. This standard IP packet passes through the satellite modem and arrives at the other end. By
passing the MPLS gateway B, the standard IP packet will be recovered to MPLS packet based on the look
up table stored in the gateway. This method is a very straightforward way to address the problem. The
gateways themselves do not exchange MPLS labels. Instead, they only learn MPLS labels from neighbor
PE router. Thus this method is very fast and bandwidth efficient.

Fig. 4. The working procedures of MPLS gateway adding/dropping MPLS labels from packets.

3.4. Validation

We build up a test environment completely based on the real setup of Sichuan Power Automation Data
Network. We use five routers to simulate the setup of one control center and four substations. Two VPN
are setup for the network. VPN1 is for realtime services (e.g. EMS, WAMS, VoIP, etc) with data rate
around 500kbps. VPN2 is for non-realtime services, with burst data rate 100kbps every 10 seconds. The
satellite link bandwidth is set to 128kbps for standby mode and 2048kbps for working mode. A
geosynchronous satellite covering Sichuan is used. At first the terrestrial links are used to support the
dataflow. After the dataflow is stabilized, we cut down the terrestrial links and test the link establishment
time of the satellite network. After the satellite link is established, we recover the terrestrial link and test
the link recovery time. Within 30 tests, the average link establishment time is 13 seconds and average link
recovery time is 9 seconds. The results showed that the network setup and MPLS gateway can
successfully achieve the designed goal. The link selection between terrestrial and satellite is fully
automatic. Although the switch process may cause several seconds communication failure, they are
worthwhile compared to the improvement of the network robustness due to backup satellite link.
 ChuangDeng
Chuang Dengetetal.al./ Energy
/ EnergyProcedia
Procedia0012(2011)
(2011)000–000
27 – 36 733

4. Performance Enhancement

In satellite SCADA/EMS systems, the system information are reliably transported using TCP at
transport layer. Traditional TCP is mainly used in internet which is running in terrestrial environments.
However, satellite communication has several significant differences at physical and network layers,
making standard TCP insufficient to sustain efficient transport performance. The major differences
between terrestrial and satellite links and proposed solution are listed in table 1.

Table 1. Performance index comparison of terrestrial and satellite communication

Performance Index Terrestrial Satellite

Bit Error Rate 10e-9 10e-5
Round-trip Delay ~30ms(typical) >550ms
Up/Down Rate 1:1 to 10:1 >100:1
Data Loss Cause Congestion Link outage

The abovementioned different properties of terrestrial and satellite communication drives to several
modification of traditional TCP to better adapt the satellite environment. Based on the satellite SCPS-TP
protocol, several TCP protocol modifications are proposed below.

4.1. Selective Negative Acknowledgement (SNACK)

In standard TCP, the sender will only retransmit the packet after it receives the ACK of previous
packets. The traditional TCP uses cumulative acknowledgements which only acknowledge the last
received ordered packets. The high BER and large round trip delay may cause many misordered packets
at the receiving window, leaving many packet “holes” to be filled in between. However, retransmission of
traditional ACK mechanism will retransmit some duplicate packets which have already been received.
We replace the TCP acknowledgement (ACK) into Selective Negative Acknowledgement (SNACK).
SNACK uses the normal ACK header and an additional variable-length field to identify the packet
“holes”. When an out-of-sequence queue at the receiver exists, the SNACK will identify the packet
“holes” by defining two parameters: offset and size. The offset identifies the hole position from the ACK
number and size identifies the “hole length” by MSS unit. When a SNACK is received, the sender
selectively resends the lost packet. If multiple “holes” exist at receiver, the variable-length option field of
SNACK can include them all in a single ACK packet, thus enabling the multiple “holes” to be filled at a
single retransmission, thus greatly improve the transport efficiency. Since no duplicate packets at
receiver are received, the transmission rate can be greatly accelerated.

4.2. Window Scaling

The standard TCP often runs on terrestrial network which has small RTT and transmission latency. In
such scenario, the sender does not need a large window to buffer the packets. With the low latency, ACK
from receiver will return to sender even before the window being filled. While in satellite environment,
the large latency may cause the sender to wait ACK from receiver since sender’s window is full of
buffered packets. In such case, expand the sender’s window size is essential for efficient transportation. In
standard TCP, the window size is normally set to 16kbytes. The maximum throughput for such setup is
computed as
Window Size/RTT = 16kbytes/550ms ≈ 230kbps (1)
834 ChuangChuang
Deng etDeng et al. / Energy
al. / Energy Procedia
Procedia 12 (2011)
00 (2011) 000–00027 – 36

This rate indicates the highest throughput in satellite link using standard TCP. It means that satellite
bandwidth over 230kbps cannot be fully exploited under standard TCP. We must modify the window size
to break up this limit. In practice, to achieve the designed maximum throughput of the system, we modify
the window size to 128kbytes.

4.3. Avoiding Slow Start

The internet mainly relies on terrestrial communications such as optical fibers which have an extremely
low bit error rate due to low noise of its transmission medium. The satellite channel, however, often
reaches a BER at 10e-5 to 10e-7 due to unstable wireless environments. Thus link outage due to high
BER often happens. In standard TCP, sender will mistakenly take the cause of the packet loss as
congestion and will reduce the transmission rate (i.e. reduce the congestion sliding window) and double
the transmission timer to avoid further congestions. In fact, as FDMA satellite system provides the
dedicated link for each sender at the physical layer, there is hardly any data low due to congestion in such
satellite system. On the other hand, most data loss can attribute to BER. For a packet loss due to BER, the
sender should retransmit the packet instantly without changing the transmit rate and timer. There is no
need for a slower pace of transmission since congestion is not the cause of the packet loss.

Fig. 5. Standard setup for end-to-end TCP connections. Fig. 6. Modified setup for TCP fraud connections.

4.4. ACK Frequency Reduction

The goal of EMS/SCADA data is to collect information from the substations. This telemetry system
typically utilizes a high-rate uplink for information and a low-rate downlink for query. This will introduce
an asymmetrical channel in satellite link (e.g. 500kbps for uplink and 5kbps for downlink). If the
information is transmitted under TCP, the ACK traffic will be greatly generated and overrun the query
traffic in the downlink. We use the configurable RTT-related frequency to ACK the received packets
rather than instantly acknowledgement the every segments. This mechanism does not depend on ACK
clocking therefore allows the acknowledgement rate to be reduced. Combined to SNACK to selectively
choose the missing packets in out-of-order-sequence, we can greatly reduce the acknowledgement
frequency to avoid overload in downlink.

4.5. Practical Configuration

The normal TCP connection is shown in figXX. In this setup, the TCP establishes an end-to-end
connection and use flow/congestion control mechanism to reliably transmit the data. Notice that the
connection is passing through the satellite link. The standard TCP cannot cope with the high BER and
large latency problems in satellite link, resulting in poor performance and inefficient bandwidth
 ChuangDeng
Chuang Dengetetal.al./ Energy
/ EnergyProcedia
Procedia0012(2011)
(2011)000–000
27 – 36 935

utilization.
In some practical scenario, the direct modification of TCP protocol at end users is prohibited due to
security considerations. In order to leave the end users intact, we use a TCP fraud method to transport the
data from original end user to accelerator. By adding TCP accelerator to the satellite link, the original
end-to-end TCP link is cut into three segments: Between the sender/receiver and TCP accelerator, a TCP
connection remains. Between two TCP accelerators, the modified protocol replaces the original one. In
such configuration, the accelerator may send fraud ACK to “fool” sender before the real ACK arrives
from the other end. Consequently the sender will continuously send the datagram to the accelerator. The
TCP fraud cleverly transports the datagram from sender to accelerator with maximum transmission rate,
leaving the original end user unmodified. The modified TCP protocol is then applied on the connection
between the two accelerators.

4.6. Performance Evaluation

We use the same network setup as the last section to test the transmission rate and evaluate the bandwidth
efficiency before and after the TCP modifications. We use the satellite channel simulator to simulate
different network conditions by setting different values of channel parameters including round-trip delay,
bit-error rate, up/down link asymmetric rate. The parameter setup is listed in Table 2.

Table 2. Channel parameter setup for performance evaluation

Channel Parameter Value

Bit Error Rate 10e-5, 10e-6, 10e-7, 10e-8
Round-trip Delay 550ms
100kbps, 200kbps, 500kbps, 1Mbps,
Channel Bandwidth
1.5Mbps
Up/Down Link Rate 1:1, 2:2, 5:1, 10:1,50:1

Figure 7 shows the data throughput using standard and modified TCP under a typical satellite channel
condition. We can see the great improvement on data throughput using modified transport protocol,
demonstrating the validity of the proposed scheme.

Fig. 7. Performance evaluation of TCP and modified satellite Fig. 8. Performance evaluation of TCP and modified
SCPS-TP under RTT 550ms, BER 1e-7 and satellite SCPS-TP under different BERs
up/down link ratio 10:1.

Figure 8 shows the performance of TCP acceleration under different bit-error rate. We can see the
greater improvement on through is reached under high bit-error rate.
Figure 9 shows the improvement of data throughput under asymmetric channels. It can be inferred
36
10 ChuangChuang
Deng etDeng et al. / Energy
al. / Energy Procedia
Procedia 12 (2011)
00 (2011) 000–00027 – 36

from the results that TCP acceleration greatly improves the bandwidth efficiency and enhances the
performance under poor channel conditions.

Fig. 9. Performance evaluation of TCP and modified satellite SCPS-TP under different up/down link ratio (up/down 1,2,5,10,50
respectively)

5. Conclusion

In this paper, we proposed a hybrid terrestrial-satellite communication networks for power system
automations. This configuration is essential in enhancing the network integrity and robustness under
tough natural environment in China. The automatic link switch mechanism is developed to enable the
self-healing capacity of the network. At the transport layer, TCP is modified to better adapt to
characteristics of the satellite channel. The modified protocol can greatly improve the transport
performance over satellite channels thus making a practical and economical hybrid network possible. The
network configuration has already been implemented in Sichuan power grids and showed a great
improvement on power system communication networks.

Acknowledgment

This work was supported in part by Sate Grid Corporation of China (SGCC) under project WG1-2010-
6.

References

[1] C.H.Hauser, D.E.Bakken, and A.Bose, "A Failure to Communicate: Next-Generation Communication Requirements,
Technologies, and Architectures for the Electric Power Grid" IEEE Power and Energy Magazine, pp. 47-55, Mar/Apr. 2005.
[2] C.Li, “Application of Satellite Communication System to Power Dispatching in Mountainous Area”, Automation of
Electric Power Systems, pp.22-25, Vol 6(22), Dec.2002
[3] J.J.Spilker, “Digital Communication by Satellite”, Englwood Cliffs, N.J., Prentice-Hall, 1977, p650.
[4] R.C.Durst, P.D.Feighery, and E.J.Travis, “User’s Manual for the satellite SCPS Transport Protocol”, Washington C3
Center, McLean, VA, Project No.03970638, Mar.1997.
[5] R.Wang, S.Horan, S.Kota, and B.Sun, “An Experimental Evaluation of satellite SCPS-TP over Lossy GEO-Space Links”,
in Proc.2006 IEEE Global Communication conf., pp.316-320.
[6] R.C.Durst, G.J.Miller, and E.J.Travis, “TCP extentions for space communications”, Wireless Networks, Vol.3 pp.389-403,
1997.
[7] H.Lee, J. Hwang, B. Kang, and K.Jun, "End-To-End QoS Architecture for VPNs: MPLS VPN Deployment in a Backbone
Network" Intl’ Conf on Parallel Process Workshops, pp. 479, 2000.