Palo Alto Networks

TechConnect

High Availability
Von Nguyen
Webinar Agenda
Active/Passive HA
 Overview
 Configuration
 Active/Active HA
 Overview
 Configuration
 HA Monitoring
 Troubleshooting

•Page 2
Active/Passive HA

© 2012 Palo Alto Networks. Proprietary and Confidential

Active/Passive HA Overview

• Supported Modes:
- Layer 2, Layer 3, Virtual Wire
• Links:
- HA1, HA2
• Device States:
- Initial, Active, Passive, Non-functional, Suspend
• Synchronization of:
- State-full sessions, Certificates, Response Pages, Configuration
- Not synchronized: Admin accounts, HA configuration
• 2 Unit cluster, same model

•Page 4
Active/Passive HA Operation

Primary Path

HA2
Secondary Path HA1

Control Plane Data Plane

Sync Configuration Sync Active Sessions

•Page 5
 HA Configuration
Group ID for Device with lower
HA pair Priority will be
elected Active

Different from Ping across Device can

Mgt IP HA1 link resume Active
after recovery

Only encrypts
HA1 link info

Enables stateful
synchronization
across HA2 link

“Auto” (for L3
interfaces) or
“Shutdown”

•Page 6 © 2012 Palo Alto Networks. Proprietary and Confidential.

Control/Data Link

Control & Data link backup

•HA1 •ethernet1/1

Gateway specification

Configurable link support

•HA2 •ethernet1/2

Page 7 |
Heartbeat Backup – Split Brain Protection

•<Heartbeat/Hello> •<Heartbeat/Hello>

•Redundant path
•DP status confirmation
•Supported on full product line
Page 8 |
Active/Active HA
A/A Agenda
• Overview
• Packet Handling
• Deployments
• Configuration
• Monitoring
• Troubleshooting
• Special Case, Wrap-Up

Page 10 |
Active/Active HA Overview
What is High Availability Active/Active?
• With A/A deployment, both HA peers are active and
processing traffic.
• A/A HA is supported only in the virtual-wire and Layer 3
modes beginning with PAN-OS 4.0.
• Such deployments are most suited for scenarios involving
asymmetric routing.
• Deployment also can be to allow dynamic routing protocols
(OSPF, BGP) to maintain active status across both peers.
• In addition to the HA1 and HA2 links used in A/P, A/A
deployments require a dedicated HA3 link. HA3 link is
used as packet forwarding link for session setup and
asymmetric traffic handling.
Page 11 |
Which to use - A/P or A/A?
What Active/Active is NOT designed for:
• A/A does NOT load balance. Load sharing can be done via
sending of traffic across each peer, but there is no load-
balancing mechanism.
• A/A will not increase performance or allow greater
capacity. At no point should traffic loads go beyond
capacity of a single stand-alone system as failover could
cause single system to become overloaded causing
possible outage.

Note: Unless Active/Active asymmetric flow or dynamic

routing capability is a requirement, for most deployments
Active/Passive is better option as it is more simple to deploy.

Page 12 |
HA Peer Connection
• Same HA1 and HA2 links as A/P.
• Add HA3, any free dataplane port with interface mode
‘HA’.
- All packet forwarding between the two devices uses HA3 link.

•HA3 •HA2

•HA1
Page 13 |
Agenda
• Overview
• Packet Handling
• Deployments
• Configuration
• Monitoring
• Troubleshooting
• Special Case, Wrap-Up

Page 14 |
Active/Active Packet Handling
In Active/Active cluster, the packet handling can be
distributed between the two peers. There are two important
functions that are handled by devices in a cluster
• Session ownership
• Session setup

Page 15 |
Session Ownership
• Session owner device can be either the firewall that
receives the first packet of a new session or the device in
an ACTIVE-PRIMARY state.
• This device is responsible for all layer 7 processing, i.e.
app-id, content-id, and threat scanning for this session.
• This device is also responsible for generating all traffic
logs for the session.

Page 16 |
Session Setup
• Session setup device is responsible for layer2 through
layer4 processing required for setting up a new session.
• Address translation is performed by session setup device.
• Session setup device is determined by configuring
“session setup load sharing” options.
• Separation of session owner and session setup devices is
necessary to avoid race conditions that can occur in
asymmetrically routed environments

Page 17 |
Packet Flow
In order to understand packet flow within a cluster, we will
discuss three different scenarios
1. New session
2. Established session
3. Asymmetric packet flow

Page 18 |
Session Setup
1. Packet arrives at one of the
devices
2. Receiving device has no
session for the packet, and
assumes ownership of the Session owner
Session setup device
Will be L7 owner
session
3. Computed hash/modulo
determines device is not
responsible for session-setup,
and forwards packet to peer
device over HA3 link
4. Session is setup and session
info and packet are returned to
session owner
5. Original device forwards 0010100010
101001001

packet out appropriate

interface
Page 19 |
Packet Flow: New Session
The sequence of steps involved in setting up
a session is listed below
1. End host sends packet to device-A.
2. Firewall examines the contents of the
packet to match it to an existing session.
3. If there is no session match, Dev-A
determines that it has received the first
packet for a new session. Therefore Dev-
A becomes the session owner.
4. Dev-A uses the configured session setup
load sharing options to identify the
session setup device. In this example we
assume the setup function is performed
by Dev-B
5. Using the HA-3 link, Dev-A sends the first
packet it received to Dev-B.
6. Dev-B sets up the session and returns the
packet to Dev-A for layer 7 processing if
any.
7. Dev-A then forwards the packet out via
the egress interface to the destination
Page 20 |
Established session
1.Packet arrives at one of
the devices
2.Receiving device has
session for the packet Session owner
Layer 7 processing
and owns the session
3. Packet is processed and
sent out via the
appropriate egress
interface

0010100010
101001001

Page 21 |
Packet Flow: Existing Session
The sequence of steps for an existing session
is listed below

1. End host sends packet to Dev-A.

2. Firewall examines the contents of the
packet to match the packet to an existing
session.
3. If there is a session match, Dev-A
processes the packet and sends the
packet out via the egress interface to the
destination

Page 22 |
Established Session – Packet Arriving at non
session owner device
1.Packet arrives at one of
the devices
0010100010
101001001

2.Receiving device has a

session for the packet but Session owner
Layer 7 processing
it is owned by peer device
3.Receiving device
forwards packet over the
HA3 link to the owner for
processing
4.Owner processes packet
1. In vwire packet is sent back to
receiving device
2. In L3 if owner has route to
destination, packet is forwarded
out

Page 23 |
Packet Flow: Asymmetric Flow - L3
The sequence of steps for an assymetric
packet flow

1. Dev-B receives a packet.

2. Receiving device has a session for the
packet but it is owned by peer device,
Dev-A.
3. Dev-B forwards packet over the HA3 link
to the Dev-A for processing.
4. In layer3 deployment , Dev-A processes
packet and forwards it to destination if it
has the route.

Page 24 |
Packet Flow: Asymmetric Flow – V-Wire
The sequence of steps for an assymetric
packet flow

1. Dev-B receives a packet.

2. Receiving device has a session for the
packet but it is owned by peer device,
Dev-A.
3. Dev-B forwards packet over the HA3 link
to the Dev-A for processing.
4. In Vwire deployment in order to preserve
the forwarding path, Dev-A processes the
packet and returns to Dev-B, to be
transmitted out the egress interface to the
destination.

Page 25 |
Agenda
• Overview
• Packet Handling
• Deployments
• Configuration
• Monitoring
• Troubleshooting
• Special Case, Wrap-Up

Page 26 |
Deployment: V-Wire
• Simplest solution to implement high
availability
• Firewalls are installed between L3
devices. These are often used in
conjunction with dynamic routing
protocols which will fail traffic over to the
other cluster member if needed.
Note: Implementing A/A HA in v-wire
mode in a layer2 sandwich will result in
switching loops if Spanning Tree Protocol
is not enabled on the switches. It is
recommended to deploy A/A in v-wire in a
layer3 topology.

Page 27 |
Deployment: Layer 3
Layer3 deployment supports virtual IP addressing, NAT,
and use of dynamic routing protocols for redundancy.
Active/Active cluster can be deployed in several different
scenarios in layer3 mode as described below
• Floating IP
• ARP load sharing
• Mixed mode (combine both floating IP and ARP load
share)

Page 28 |
Deployment: L3 Floating IP
• Floating IP can move between HA devices when a link
failure or device failure occurs.
• Interface on device in cluster that owns floating IP
responds to ARP requests with a virtual MAC.
• Floating IPs are recommended when VRRP-like
functionality is required.
• Floating IPs can be used for VPNs and source NAT
allowing for persistent connections when a failure
occurs.
• Each interface on firewall has its own IP and a floating
IP. Interface IP remains local to the device but floating IP
address can move between the devices.
• End hosts are configured to use floating IP as default
gateway allowing traffic to be load balanced within the
cluster.
• External load balancers can also be used to load
balance traffic between firewalls within the cluster.
• If failover occurs, gratuitous ARP is sent out by the
functional device. Once device recovers, floating IP and
VMAC will move back to the original device.

Page 29 |
Deployment: L3 ARP Load Sharing
• HA pair to share an IP address and provide gateway
services.
• All hosts are configured with single gateway IP. ARP
requests for gateway IP are responded to with a virtual
MAC address from a single device in the pair.
• Each device will have unique virtual MAC address
generated for the shared IP.
• The device that responds to ARP request is determined
by computing hash or modulo of source IP of the ARP
request.
• Once end host receives ARP response from device, it
caches the MAC address and all traffic from host is
routed via the firewall that responded with VMAC. Life
time of ARP cache is dependent on end host OS.
• ARP load-sharing should be used only when a Layer 2
separation exists between firewalls and end hosts.
• If link or device failure, floating IP and VMAC moves over
to the functional device. Gratuitous ARP is sent out by
the functional device.

Page 30 |
Deployment: L3 Mixed Mode
• It is possible to have some of interfaces configured with
floating IPs and some with shared IPs for ARP loading
sharing.
• Cluster can be configured with ARP load sharing IPs,
configured for hosts on the LAN segment, and floating IP
configured on upstream WAN edge routers.

Page 31 |
Agenda
• Overview
• Packet Handling
• Deployments
• HA States
• Configuration
• Monitoring
• Troubleshooting
• Special Case, Wrap-Up

Page 32 |
Active/Active Configuration
• First step, set the HA mode to active-active.
Device > High Availability; Setup

• ID: HA group ID. Both devices must have the same group ID. HA group-ID is used to calculate virtual MAC.
• Mode: Choose active-active from the drop down list.
• Device-id: Select unique device from drop down list (0 or 1). Device-ID remains local to the device and does not
transition between devices during failover. This field is also used to calculate VMAC.
• Peer HA IP Address: IP address of HA1 control link on peer device.
• Backup Peer HA IP Address: IP address of backup control link on peer device. This field is optional.
• Enable Config Sync: Enabled by default, required to synchronize configuration between devices in cluster.

Page 33 |
HA Control and Data Links
• Same as Active/Passive

•PA-1 •PA-2

•Control
Link

•Data
Link

Page 34 |
HA3 Link
Used for packet forwarding between session owner and
session setup device.

• HA3 link is L2 link and uses MAC-in-MAC encapsulation.

• Aggregate interfaces can be configured as HA3 link (4000
and 5000 series only) for redundancy of HA3 link.
• Interface mode must be HA to use as HA3 link.
Note: Because of overhead associated with encapsulation on HA3 link,
switch ports connecting HA3 link must be configured to support jumbo
frames.
Page 35 |
Configuring ARP Load Sharing
Device > High Availability > Virtual Address

• Click on “Add” to add a new virtual address.

• From interface drop down list choose appropriate interface, and click
“Add”.
• Choose Type to “arp-load-sharing”. In this example we choose “ip-
modulo” as ARP Load Sharing Type.
Page 36 |
Configuring Floating IP
Device > High Availability > Virtual Address

• Click “Add” to add a new virtual address.

• From interface drop down list choose appropriate interface, and click
“Add”.
• Choose Type to be “floating”. Device priority determines which device
will own the floating IP address.
• Configure two floating IP address, one for each device, with different
priorities as shown above. Address with lower numeric value will have
highest priority.
Page 37 |
Monitoring
Settings are same for Active/Passive and Active/Active:
• Heartbeat polling
• Link monitoring
• Path monitoring

Page 38 |
Configuring Link Monitoring
• Device > High Availability; Link Monitoring

“Any” or “All” failure

conditions will
cause failover

Page 39 |
Configuring Path Monitoring
• Device > High Availability; Path Monitoring

“Any” or “All” failure

conditions will
cause failover

“Vwire”, “VLAN”, “VR”

Page 40 |
Agenda
• Overview
• Packet Handling
• Deployments
• Configuration
• Troubleshooting
• Special Case, Wrap-Up

Page 41 |
Troubleshooting
• CLI show commands:
admin@PA-2(active-primary)> show high-availability ?
> all Show high-availability information
> control-link Show control-link statistic information
> dataplane-status Show dataplane runtime status
> flap-statistics Show high-availability preemptive/non-functional
flap statistics
> interface Show high-availability interface information
> link-monitoring Show link-monitoring state
> path-monitoring Show path-monitoring statistics
> state Show high-availability state information
> state-synchronization Show state synchronization statistics
> transitions Show high-availability transition statistic
information
> virtual-address Show Active-Active virtual address status

• Logs:
- less mp-log ha_agent.log
- show log system

Note: For HA issues, be sure to always get data from BOTH peers as
issues may be on either device.
Page 42 |
HA CLI Commands
• Force configuration and session synchronization to peer
admin@student1> request high-availability sync-to-remote

• Fail HA master to peer and make system ineligible to be

master
admin@student1> request high-availability state suspend

• Re-enable HA on suspended system

admin@student1> request high-availability state functional

• Show HA status
admin@student1> show high-availability state
admin@student1> show high-availability link / path -monitoring
Troubleshooting Sessions
Session flow from host 172.35.2.4 to host 10.1.1.250.
admin@PA-2(active-primary)> show session all filter destination-port 23
--------------------------------------------------------------------------------
ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port])
Vsys Dst[Dport]/Zone (translated IP[Port])
--------------------------------------------------------------------------------
19485 telnet ACTIVE FLOW NS 172.35.2.4[56484]/trust-l3/6 (10.1.1.101[57558])
vsys1 10.1.1.250[23]/untrust-l3 (10.1.1.250[23])

From session table, we see that host 172.35.2.4 is translated to IP

10.1.1.101, floating IP on PA-2 which is device-id 1

admin@PA-2(active-primary)> show session id 19485 | match HA

session synced from HA peer : False
session owned by local HA A/A : True

PA-2 is session owner.

Page 44 |
Global Counter
Show counter global for Active/Active related packets.
admin@PA-2(active-primary)> show counter global filter aspect aa delta yes
Global counters:
Elapsed time since last sampling: 24.406 seconds

name value rate severity category aspect description

--------------------------------------------------------------------------------
ha_aa_session_setup_peer 1 0 info ha aa Active/Active: setup session on
peer device
ha_aa_pktfwd_rcv 1 0 info ha aa Active/Active: packets received
from peer device
ha_aa_pktfwd_xmt 1 0 info ha aa Active/Active: packets forwarded
to peer device
--------------------------------------------------------------------------------
Total counters shown: 3
--------------------------------------------------------------------------------

Page 45 |
Viewing Floating IPs
• “show high-availability virtual-address” can be used to
view all configured floating IP addresses.

admin@PA-1(active-primary)> show high-availability virtual-address

Total interfaces with virtual address configured: 2
Total virtual addresses configured: 4
-----------------------------------------------------------------------------
Interface: ethernet1/2 Virtual MAC: 00:1b:17:00:01:11
10.1.1.100 Active:yes Type:floating
10.1.1.101 Active:no Type:floating
-----------------------------------------------------------------------------
Interface: ethernet1/1 Virtual MAC: 00:1b:17:00:01:10
172.35.2.100 Active:yes Type:arp-load-sharing
-----------------------------------------------------------------------------

Page 46 |
Manual failover
Same as A/P except will determine Primary/Secondary.
• GUI:

• CLI (on active peer):

request high-availability state suspend
request high-availability state functional

Page 47 |
Logs and Packet Captures
• All traffic logs are logged by session owner.
• When session owner fails, peer device will become
session owner and will handle logging.
• If preempt is enabled and should failed device recover
before session ends, it will take back ownership of the
session and handle logging.

Page 48 |
Agenda
• Overview
• Packet Handling
• Deployments
• Configuration
• Monitoring
• Troubleshooting
• Special Case, Wrap-Up

Page 49 |
 PA-200 – A/P HA-Lite

 Supports limited A/P functionality “HA-Lite”

 Uses MGMT port as HA1 link for heartbeats and config sync
 No HA2 or HA3 link supported, no session sync

Page 50 |
 For More Information

• Active/Passive HA Tech Note:

https://live.paloaltonetworks.com/docs/DOC-1160
• Active/Active HA Tech Note:
https://live.paloaltonetworks.com/docs/DOC-1756
• Designing Networks with Palo Alto Networks firewalls:
https://live.paloaltonetworks.com/docs/DOC-2561

Page 51 |
 THANK YOU !!

•Upcoming TechConnect Webinars:

•Go to www.paloaltonetworks.com/partner site to register.

Page 52 |