Virtual I/O Server Shared Ethernet Adapter Failover

Why do we need a Shared Ethernet Adapter (SEA) failover device?

Virtual I/O Server (VIOS) partition provides an Ethernet bridge between partitions using
virtual Ethernet and a physical adapter that has a connection to the local network. The
physical adapter is allocated to the Virtual I/O server partition.

If the VIOS partition fails, this bridge between the outside network and virtual network
will be broken.

The SEA failover provides a backup for this bridge connection. It would be located on a
different VIOS.


What has changed?

Enhancements to the virtual Ethernet to support multiple trunk adapters on the same
VLAN. Prior to this new feature, only one trunk adapter per VLAN could be configured.

To support multiple trunk adapters on the same VLAN, the adapters must have a
different priority.

HMC enhanced to configure a priority value for the virtual Ethernet adapter that have
trunk capability.

PHYP changed to ensure only one trunk adapter is active per VLAN. PHYP does not
check priority values so, the virtual Ethernet driver was enhanced to have the ability to
query the adapter priority and determine if an adapter is active.










Operation:

Only two SEA devices can be created.

A control channel is created between the two SEAs which creates a failover domain.

A control channel enables failover protocol communications. The failover protocol
ensures the SEA with the highest priority is the active adapter. This ensures that multiple
SEA devices do not bridge at the same time.

The highest priority device is the primary SEA and the other device in the failover
domain is the backup SEA. When the VIO server with the primary SEA fails or the
primary SEA detects a physical adapter failure, the failover protocol will ensure that the
backup SEA becomes active until the primary recovers.


The virtual adapter properties is where the settings for trunk adapter are.














This picture (above) shows a server with three partitions and a Virtual I/O Server
partition. All the partitions use a virtual Ethernet connection between them, and also
connect to the VIOS partition. The VIOS partition has a physical Ethernet adapter that
has a connection to the network (internet) and also a route to the virtual LAN. This
physical adapter is defined as the Shared Ethernet Adapter and provides connection to the
internet from the partitions LPAR 1,2 and 3 using VLAN 10. While this is a good way to
share an adapter for network connectivity, it does have some disadvantages. If the VIOS
partition fails, the other partitions will lose the external network connection.
VLAN 10
IO Server 1
Power 5 System
Ethernet Switch
LPAR 1 LPAR 2 LPAR 3
Internet
Internet
Standalone Server
Shared
Ethernet
Adapter
VLAN 10
IO Server 1
Power 5 System
Ethernet Switch
LPAR 1 LPAR 2 LPAR 3
Internet
Internet
Standalone Server
Shared
Ethernet
Adapter
Control Channel
Shared
Ethernet
Adapter
IO Server 2
Power 5 System
Ethernet Switch
Standalone
Server
LPAR 1 LPAR 2 LPAR 3
Internet
Internet
VLAN 10
Shared
Ethernet
Adapter
IO Server 1
Control Channel
Shared
Ethernet
Adapter
IO Server 2
Shared
Ethernet
Adapter
IO Server 2
Power 5 System
Ethernet Switch
Standalone
Server
LPAR 1 LPAR 2 LPAR 3
Internet
Internet
Internet
Internet
VLAN 10
Shared
Ethernet
Adapter
IO Server 1
Shared
Ethernet
Adapter
IO Server 1
Adding a second VIOS partition with another physical SEA will add a level of
redundancy to the server for the external network connection.

Each SEA will be comprised of a physical adapter and one or more virtual Ethernet
adapters (one is enough, but it is possible to include more virtual Ethernets that serve
different VLAN IDs). These virtual Ethernets must be created as trunk adapters: trunk
adapters will receive all the traffic on their VLAN that is not addressed to any known
MAC address. Furthermore, all SEA devices in a failover domain must define a control
channel virtual Ethernet: this virtual Ethernet has a VLAN ID different from all other
VLAN IDs on the Squadrons box, and for security reasons (to prevent protocol
interference) must only be defined as part of the SEA in the same failover domain.

Previously, there could only be one trunk adapter per VLAN ID. It is now possible to
define more than one trunk adapter for the same VLAN ID, but only one of them will be
marked as active at any one time. Each trunk adapter will have an associated priority
value that is assigned by the user and is used by the SEA failover protocol to determine
which trunk adapter should be activated preferentially over the other. This is configured
through the HMC.
The next step in the setup procedure is creating and configuring the SEA device. If a SEA
has several virtual Ethernets, it will make sure that all of them are trunk adapters;
furthermore, they all must have the same priority value. The priority will help the
protocol determine which SEA should behave as the primary at any particular time: the
numerically lowest priority value will mark the SEA as primary.
During the configuration, the user also specifies a high availability mode for the SEA
device. There are three modes, auto, disabled, and standby. The SEA failover protocol is
only applicable for the auto and standby modes. When configured in these modes the
SEA device will open the control channel and start the failover protocol.



















In a failure scenario:

The failover protocol takes a SEA device through six possible states and uses four
message types. All message type packets are sent via the control channel of the virtual
Ethernet.

The bridge function is the ability to bridge unicast or broadcast packets through the
physical interface. The SEA device has three bridging modes: BRIDGE_ALL, when set
will allow the SEA to bridge all packets; BRIDGE_UNICAST means to bridge only
unicast packets; and BRIDGE_NONE, where all bridging ceases. BRIDGE_ALL and
BRIDGE_UNICAST will only occur in the PRIMARY state. BRIDGE_NONE will
occur in all other states. When the SEA device changes its state into PRIMARY from the
INIT state, it has a one second window where it will perform the BRIDGE_UNICAST
mode. Once that time is up, it can begin to switch to BRIDGE_ALL mode.

The initial state for a SEA device is INIT. The three steady states are PRIMARY,
BACKUP, and LIMBO where the physical adapter failure causes the SEA device to
remove itself until such time that the physical adapter comes back up. The two transitory
states are RECOVERY and NOTIFY. The RECOVERY state is the intermediary state
used by a backup device (BACKUP) that is going to become primary (PRIMARY) when
the backup receives a keep alive with lower priority. The NOTIFY state is the
intermediary state used by a primary (PRIMARY) device that is going to become a
backup device (BACKUP).

Control Channel
Shared
Ethernet
Adapter
IO Server 2
Power 5 System
Ethernet Switch
Standalone
Server
LPAR 1 LPAR 2 LPAR 3
Internet
Internet
VLAN 10
Shared
Ethernet
Adapter
IO Server 1
Control Channel
Shared
Ethernet
Adapter
IO Server 2
Shared
Ethernet
Adapter
IO Server 2
Power 5 System
Ethernet Switch
Standalone
Server
LPAR 1 LPAR 2 LPAR 3
Internet
Internet
Internet
Internet
VLAN 10
Shared
Ethernet
Adapter
IO Server 1
Shared
Ethernet
Adapter
IO Server 1
A keep-alive packet is sent out to the broadcast address by the current primary SEA every
half second: the purpose of this packet is to let the backups know that the primary is still
up and running, and that no failover is necessary.

A recovery packet is sent out by a recovering SEA in the INIT state or a backup SEA in
the BACKUP state. The purpose of this packet is to indicate to the other SEA that a
higher priority SEA is recovering and will soon take over.

A notify packet is sent out in response to a recovery packet by an existing primary in the
PRIMARY state, to indicate that it agrees that a higher priority SEA can become the
primary.

A limbo packet is sent out in response to an adapter failure. Its use is to inform the other
server to immediately take over and become the primary no matter what state it is in.

When a SEA comes up in a mode where the failover protocol is activated, it will listen on
the control channel for two seconds for a keep-alive packet in order to determine if a
primary SEA is already active on the network. It will do the following:

(1) If no keep-alive packets are received (i.e. this is the first SEA to come up, this sea
came up after the primary went down, or both SEA devices are coming up
simultaneously), after two seconds, the SEA assumes the role of the primary and begins
sending keep-alive packets. At this point in time, the SEA device(s) can
BRIDGE_UNICAST. The primary SEA device(s) waits one second before it does
BRIDGE_ALL, just in case the backup SEA (which might think it is the primary) came
up at the same time and also sends out keep-alive packets.

(2) If a keep-alive packet is received and its priority is higher, then there is a working
primary and no further steps need to be taken. The SEA with the lower priority will
become the backup and does not do bridge any form of traffic.

(3) If a keep-alive packet is received that is announcing a lower priority (which would
happen if the original primary went down and then came up again), it means that a
recovery must occur and the primary SEA must become the active SEA once again. In
this case, the SEA will continuously send a recovery packet on the control channel. The
new primary will not bridge any multicast or broadcast packets; instead, it will wait until
it receives a notify packet from the current primary saying it is safe to bridge without
creating routing loops. Once it receives confirmation, the new primary (in RECOVERY
state) can make itself the primary and begin sending out keep-alive packets. The current
primary (in NOTIFY state) will stop sending notify packets once it receives the keep-
alive from the new primary and take the role of the backup. When it becomes the backup,
it will issue a call to take down the adapter and bring it right back up. This is to ensure
the switch will send the packets to the new primary.

(4) If the SEA does not receive a notify after sending the max number of recovery
packets allowed, it assume the SEA that is responsible for sending notify packets went
down before it was able to send notify. The SEA in the RECOVERY state will then
switch to PRIMARY.

(5) It is also possible for an abnormal case that the SEA in a PRIMARY state does not
hear or ignores the recovery packet but continues to send keep-alive. If this is the case,
the SEA that is in the RECOVERY state should not stay in RECOVERY state but should
go to the BACKUP state once it reaches its recovery iteration count. Since the SEA in the
RECOVERY state is able to hear the keep-alive, it indicates there is a PRIMARY SEA
and that is working but is unwilling to give up its activeness.

Each message type is only sent when the SEA device is in the corresponding state. There
is only one instance where both SEA devices can be primary and that is when they both
come up from the INIT state. Therefore, in any other circumstances when a SEA is in the
primary state, it will ignore keep-alives.

During the initialization process, if for some reason more than one SEA has the same
priority value, the MAC address of the control channel virtual adapter will be used to
break the tie (the one with the numerically higher MAC address will declare itself the
primary and begin sending keep-alive packets).

The protocol also takes into account the possibility of a physical adapter failure. When
the physical adapter goes down, a registered callback function is invoked by the physical
adapter. The SEA must stop sending keep-alives if it is the primary and begin sending out
limbo packets. The SEA will continuously send limbo packets until it receives a keep-
alive. In any other states, the SEA switches directly to the LIMBO state without sending
out any packets. When the adapter notifies the SEA it has recovered and is now up, the
protocol will be restarted. The SEA device that receives a limbo packet can switch right
to the PRIMARY state and immediately send out keep-alive packets.

Another provision that is available is the ability for the administrator to force the SEA
into a disabled mode. This is the backward compatible scenario where a SEA is not part
of a failover domain. In this mode the SEA does not follow the failover protocol and goes
directly to becoming a primary. A user can misconfigure a SEA device by setting up a
SEA in disabled mode incorrectly, and detection of this is not possible.

The failover process takes some minimal time interval to complete; therefore, the
administrator should attempt to reduce the occurrences of failover. However, the priority
is a value chosen to indicate a preference of one SEA device as being active over another
(maybe due to the resources allocated to the SEA); this is why a SEA should be
reactivated once it becomes inactive even though this will cause a minor interruption.






Problem Resolution:

I f SEA c r eat i on f ai l s, ver i f y:
None of the underlying adapters are being used (i.e. they do not have an IP address configured over them,
or are part of another SEA or EtherChannel)

All the virtual adapters have the trunk setting enabled

All virtual adapters and the physical adapter have the same checksum offload setting (all enabled or all
disabled) use lsattr E -l <adapter> -a chksum_offload

To see device-specific statistics, the command entstat d <SEA_adapter> will show:
-------------------------------------------------------------
ETHERNET STATISTICS (ent9) :
Device Type: Shared Ethernet Adapter

[ Generic network statistics ]

--------------------------------------------------------------
Statistics for adapters in the Shared Ethernet Adapter ent9
--------------------------------------------------------------
Number of adapters: 2

SEA Flags: 00000001
<THREAD >

VLAN Ids :
ent1: 1

Real Side Statistics:
Packets received: 0 Packets bridged: 0 Packets consumed: 0
Packets fragmented: 0 Packets transmitted: 0 Packets dropped: 0

Virtual Side Statistics:
Packets received: 0 Packets bridged: 0 Packets consumed: 0
Packets fragmented: 0 Packets transmitted: 0 Packets dropped: 0

Other Statistics:
Output packets generated: 0 Output packets dropped: 0 Device output
failures: 0
Memory allocation failures: 0 ICMP error packets sent: 0 Non IP packets
larger than MTU: 0
Thread queue overflow packets: 0

High Availability Statistics:
Control Packets in: 0 Control Packets out: 1247

Type of Packets Received:
Keep-Alive Packets: 0 Recovery Packets: 0 Notify Packets : 0
Limbo Packets: 0 State: PRIMARY Bridge Mode: All
Number of Times Server became Backup: 0 Number of Times Server became
Primary: 0
High Availability Mode: Auto Priority: 1