QV1203 StudentGuide

V3.1.0.
cover
Front cover
PowerHA for AIX (HACMP) I:

Installation and Initial
Configuration
(Course Code QV120)
Student Notebook
ERC 3.1
UNIX Software Service Enablement

Student Notebook
August 2009 Edition
The information contained in this document has not been submitted to any formal IBM test and is distributed on an “as is” basis without
any warranty either express or implied. The use of this information or the implementation of any of these techniques is a customer
responsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. While
each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will
result elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.
© Copyright International Business Machines Corporation 2007, 2009. All rights reserved.
This document may not be reproduced in whole or in part without the prior written permission of IBM.
Note to U.S. Government Users — Documentation related to restricted rights — Use, duplication or disclosure is subject to restrictions
set forth in GSA ADP Schedule Contract with IBM Corp.
V3.1.0.1
Student Notebook
TOC Contents
Course Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Unit 1. Introduction to PowerHA for AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1

Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3
IBM's HA Solution for AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4
A Highly Available Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
PowerHA's Topology Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-6
Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7
Supported Networking Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-8
Un-Supported/Limited Support Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-9
PowerHA’s Resource Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-10
Resource Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-11
Solution Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-12
AIX's Contribution to High Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-13
Supported Shared Storage Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-14
Some Clusters Do Not Have Shared Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-15
So What is PowerHA Really? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-16
Additional Features of PowerHA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-17
Some Assembly Required . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-18
Just What Does PowerHA Do? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-19
What Happens When Something Fails? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-20
What Happens When a Problem is Fixed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-21
Standby: With Fallback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-22
Standby: Without Fallback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-23
Mutual Takeover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-24
Concurrent: Multiple Active Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-25
Fundamental PowerHA Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-26
Cluster Requirements and Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-27
High Availability Vs. Disaster Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-28
Disaster Recovery: Limited Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-29
Disaster Recovery: Extended Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-30
Things PowerHA Does Not Do . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-31
When is PowerHA Not the Correct Solution? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-32
What Do We Plan to Achieve in this Course? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-33
Overview of the Implementation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-34
Sources of PowerHA Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-35
Review (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-36
Review (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-37
Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-38
Unit Summary 1 of 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-39
Unit Summary 2 of 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-40
Unit 2. Configuring Networks for PowerHA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
© Copyright IBM Corp. 2007, 2009 Contents iii

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
Student Notebook
How PowerHA Uses Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-3

PowerHA Configures Highly Available Service Addresses . . . . . . . . . . . . . . . . . . . .2-4
PowerHA Monitors Networks and Detects Failures . . . . . . . . . . . . . . . . . . . . . . . . .2-5
Heartbeat Packet Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-6
Heartbeat Rings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-7
Failure Detection Versus Failure Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-8
Failure Diagnosis Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-9
What If All Heartbeat Packets Stop? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-10
Clusters Should Have Multiple Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-11
Failure Recovery and Reintegration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-12
PowerHA Networking Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-13
Network Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-14
PowerHA Topology Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-15
PowerHA Communication Hardware Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-16
PowerHA Address Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-17
Node Name Considerations (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-18
Node Name Considerations (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-19
Configuration Rules for Heartbeating (Boot Addresses) . . . . . . . . . . . . . . . . . . . . .2-20
Boot Address Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-21
Boot Address Rule Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-22
Persistent IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-23
IP Address Takeover (IPAT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-24
Two Ways to Implement IPAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-25
Configuration Rules: IPAT via IP Aliasing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-26
IPAT via IP Aliasing in Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-27
IPAT via IP Aliasing After an Interface Fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-28
IPAT via IP Aliasing After a Node Fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-29
IPAT via IP Aliasing: Distribution Preferences . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-30
IPAT via IP Aliasing Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-31
IPAT via IP Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-32
Service Address Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-33
IP Naming Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-34
Name Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-35
Other Configurations: Etherchannel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-36
Other Configurations: Virtual Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-37
PowerHA View of Virtual Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-38
Other Configurations: Single Adapter Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-39
Netmon and netmon.cf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-40
Talk to Your Network Administrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-41
Changes to AIX Boot, PowerHA v5.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-42
Changes AIX Boot: 5.3, 5.4 and 5.4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-43
Changes to /etc/inittab: 5.3, 5.4 and 5.4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-44
Changes to AIX Boot: 5.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-45
Changes to /etc/inittab: 5.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-46
Common TCP/IP Configuration Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-47
How Are Users Affected by IPAT? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-48
What About the User's Computer? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-49
Gratuitous ARP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-50
iv PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

V3.1.0.1
Student Notebook
TOC What if Gratuitous ARP Is Not Supported? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-51

Hardware Address Takeover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-52
Configuration Rules: Non-IP Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-53
IP Version 6 (IPv6) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-54
IPv6 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-55
IPv6 Brief Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-56
Configuring IPv6 Service and Persistent Addresses . . . . . . . . . . . . . . . . . . . . . . . 2-57
IPv6 Service Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-58
IPv6 Persistent Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-59
IPv6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-60
Review 1 of 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-61
Review 2 of 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-62
Review 3 of 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-63
Review 4 of 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-64
What We Are Going to Do in This Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-65
Why Do We Use Three Interfaces per Node? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-66
What We Are Going to Do in This Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-67
Unit Summary (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-68
Unit summary (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-69
Unit 3. Configuring Applications for PowerHA . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1

Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2
How to Define an Application to PowerHA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-3
Application Considerations 1 of 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
Application Considerations 2 of 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5
Shared Storage Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-6
Writing Start and Stop Scripts 1 of 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-7
Writing Start and Stop Scripts 2of 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8
Resource Group Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-9
Startup Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-10
Online on All Available Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-11
Fallover Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-12
Fallback Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-13
Valid Combinations of Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-14
Dependent Applications/Resource Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-15
Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-16
Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-17
Unit 4. PowerHA installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1

Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Steps for Successful Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3
Where Are We in the Implementation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-4
First Step in Installation: Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-5
PowerHA Filesets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-6
Remember the Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7
Remember to Check for Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8
Things to Check Before Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9
Install PowerHA Client Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10
© Copyright IBM Corp. 2007, 2009 Contents v

Student Notebook
How Does PowerHA Fit into AIX? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-11

PowerHA Components and Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-12
Cluster Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-13
Cluster Secure Communication Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-14
Cluster Communication Daemon (clcomd) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-15
clcomd: Standard Connection Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-16
RSCT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-17
PowerHA From an RSCT Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-18
Heartbeat Rings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-19
PowerHA’s SNMP Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-20
Cluster Information Daemon (clinfo) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-21
Highly Available NFS Server Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-22
Shared External Disk Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-23
Review 1 of 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-24
Review 2 of 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-25
Unit Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-26
Unit 5. Initial PowerHA Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1

Unit Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-2
What We Are Going to Achieve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-3
Where Are We in the Implementation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-4
The Topology Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-5
Configuration Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-6
Planning and Base Configuration Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-7
Starting at the Very Beginning... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-8
Almost There... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-9
The Top-Level PowerHA SMIT Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-10
The Standard Configuration Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-11
Configure Nodes to a Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-12
What Did We Get? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-13
Now Define Highly Available Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-14
Configure Service Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-15
Adding Service IP Address(es) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-16
Add appA-svc Service Address (1 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-17
Configure Application Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-20
Add appA Application Server (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-21
Add appA Application Server (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-22
Configure Volume Groups (Optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-23
Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-24
Adding the appAgroup Resource Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-25
Setting Name, Nodes, and Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-26
Adding Resources to appAgroup (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-27
Adding Resources to appAgroup (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-28
Synchronize Your Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-29
Test Your Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-30
What Do We Have at This Point? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5-31
vi PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

V3.1.0.1
Student Notebook
TOC Extending the Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-32

Extended Topology Configuration Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-33
Menus to Configure a Non-IP Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-34
Defining a Non-IP Network (1 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-35
Remove A Network From PowerHA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-38
Defining Persistent Node IP Labels (1 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-39
Synchronize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-42
Save Configuration: Snapshot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-43
Save Configuration: XML File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-44
Two-Node Cluster Configuration Assistant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-45
What the Two-Node Assistant Gives You . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-46
Where Are We in the Implementation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-47
Starting Cluster Services (1 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-48
Removing a Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-52
We're There! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-53
Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-54
Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-55
Unit summary 1 of 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-56
Unit Summary 2 of 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-57
Appendix A. Checkpoint solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1
Appendix B. HACMP 5.5 release_notes and README . . . . . . . . . . . . . . . . . . . . . B-1
Appendix C. IPAT via IP Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-1
Appendix D. Configuring target mode SSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-1
© Copyright IBM Corp. 2007, 2009 Contents vii

Student Notebook
viii PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

V3.1.0.1
Student Notebook
pref Course Description

PowerHA for AIX (HACMP) I: Installation and Initial Configuration
Brief Description
This course is part of a PowerHA curriculum designed to prepare students to support
customers using PowerHA. This course is an introductory course designed to prepare
students to install and configure a highly available cluster using PowerHA Version 5.5 on
an IBM pSeries server running AIX V6.1 or V5.3.
Audience Description
This course is intended for AIX technical support personnel and AIX system
administrators.
Course Objectives
On completion of this course, students should be able to:
- Describe the capabilities of PowerHA for AIX
- Configure AIX networks for PowerHA
- Configure PowerHA for AIX with a two-node standby cluster with one resource
group (one active node and one standby node)
- Start and stop cluster services
- Move a resource group from one node to the other.
Course Prerequisites
Students attending this course are expected to have AIX system administration, TCP/ IP,
LVM storage and disk hardware implementation skills. These skills are addressed in the
following courses (or can be obtained through equivalent education and experience):
- AHQV011: AIX Problem Determination I: Boot Issues
- AHQV012: AIX Problem Determination II: LVM Issues
- AHQV511: TCP/IP on AIX Problem Determination I: Configuration
Course Topics:
- Introduction to PowerHA for AIX
- Configuring Networks for PowerHA
- Configuring Applications for PowerHA
- Installing PowerHA
- Initial PowerHA Configuration
© Copyright IBM Corp. 2007, 2009 Course Description ix

Student Notebook
Course Agenda
(Agenda times are just estimates.)
(1:00) Welcome
(2:00) Unit 1 - Introduction to PowerHA for AIX
(1:00) Exercise 1 - Exploring PowerHA
(2:00) Unit 2 - Configuring Networks for PowerHA
(1:00) Unit 2 - Continued
(2:00) Exercise 2- Configure Networks for PowerHA
(1:00) Unit 3 - Configuring Applications for PowerHA
(1:00) Unit 4 - Installing PowerHA
(2:00) Unit 5 - Configuring PowerHA
(3:00) Exercise 3- Configure PowerHA
x PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

V3.1.0.1
Student Notebook
pref Text highlighting

The following text highlighting conventions are used throughout this book:
Bold Identifies file names, file paths, directories, user names and
principals.
Italics Identifies links to Web sites, publication titles, is used where the
word or phrase is meant to stand out from the surrounding text,
and identifies parameters whose actual names or values are to
be supplied by the user.
Monospace Identifies attributes, variables, file listings, SMIT menus, code
examples of text similar to what you might see displayed,
examples of portions of program code similar to what you might
write as a programmer, and messages from the system.
Monospace bold Identifies commands, daemons, menu paths, and what the user
would enter in examples of commands and SMIT menus.
© Copyright IBM Corp. 2007, 2009 Course Description xi

Student Notebook
xii PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

V3.1.0.1
Student Notebook
Uempty Unit 1. Introduction to PowerHA for AIX
What This Unit Is About

This unit introduces the concepts of high availability and PowerHA for
AIX.
What You Should Be Able to Do

After completing this unit, you should be able to:
• Describe basic concepts and terminology of PowerHA
(formerly HACMP)
• Describe the components of a PowerHA cluster
• Explain how PowerHA operates in some typical cluster
configurations
• Discuss considerations and limitations of PowerHA
How You Will Check Your Progress

• Review quiz
• Exercise
References
http://www-03.ibm.com/systems/p/library/hacmp_docs.html
PowerHA manuals
http://www.software.ibm.com/webapp/set2/sas/f/hacmp/home.html
Service packs for PowerHA
http://w3.tap.ibm.com/w3ki03/display/hacmp/PowerHA+for+AIX
PowerHA Wiki
Contains links to many PowerHA resources.
http://www.ibm.com/common/ssi:
IBM Sales Manual
Select country and language;
Choose the HW & SW desc (Sales Manual, RPQ); Click
Advanced Search;
Enter PowerHA in the Title: field;
Click Search at the bottom of the page
© Copyright IBM Corp. 2007, 2009 Unit 1. Introduction to PowerHA for AIX 1-1
Student Notebook
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/Web/Flashes
Techdoc Flash web page
Search for PowerHA or HACMP to find latest on supported
hardware and other technical developments which have not yet
been added to the manuals.
http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1611&start=0
PowerHA (Formerly known as HACMP) Technical Forum
A highly technical forum focussing on concepts, and concerns
with using PowerHA for High Availability.
This is a good place to post questions as this forum is
monitored by the PowerHA support group.
1-2 PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

V3.1.0.1
Student Notebook
Uempty
Unit Objectives
• Describe basic concepts and
terminology of PowerHA (formerly HACMP)
• Describe the components of a PowerHA cluster
• Explain how PowerHA operates in some typical cluster
configurations
• Discuss considerations and limitations of PowerHA
Figure 1-1. Unit Objectives QV1203.1
Objectives
In this unit we introduce PowerHA , describe the concepts, components, abilities and
limitations of PowerHA, and provide an overview of basic PowerHA cluster
configurations.
PowerHA terminology
In this course we will use the following terminology:
- PowerHA will mean any version and release of the PowerHA product.
- PowerHA x will mean version x and any release of that version
- PowerHA x.y will mean a specific version and release
Student Notebook
IBM's HA Solution for AIX

• PowerHA Characteristics
– High Availability Cluster Multiprocessing
– Based on cluster technology
– Provides two environments:
•Serial (High Availability): the process of ensuring an application is
available for use through the use of serially accessible shared data
and duplicated resources
•Parallel (Cluster Multiprocessing): concurrent access to shared data
Figure 1-2. IBM's HA Solution for AIX QV1203.1
PowerHA characteristics
IBM’s PowerHA product is a mature and robust technology for building a high
availability solution. A high availability solution based upon PowerHA provides
automated failure detection, diagnosis, recovery and reintegration. With an appropriate
application, PowerHA can also work in a concurrent access or parallel processing
environment, thus offering excellent horizontal scalability.

V3.1.0.1
Student Notebook
Uempty
A Highly Available Cluster
Fundamental Concepts
clstrmgr clstrmgr
urce
Reso up
gro Shared Storage
Node Node
Fallover
A Cluster is comprised of
•Topology (nodes, networks, network adapters)
•Resources (items being made available: applications, volume groups, etc.)
Figure 1-3. A Highly Available Cluster QV1203.1
Fundamental concepts
PowerHA is based on the fundamental concepts of cluster, resource group, and cluster
manager (clstrmgrES).
Cluster
A cluster is comprised of nodes, networks and network adapters. These objects are
referred to as topology objects.
Resource group
A resource group is typically comprised of an application, network address, and a
volume group using shared disks. These objects are referred to as resource objects.
clstrmgrES
The cluster manager daemons together are the software components that
communicate with each other to control on which node a resource group is activated or
where the resource group is moved on a fallover based on parameters set up by the
administrator. The cluster manager runs on all the nodes of the cluster.
Student Notebook
PowerHA's Topology Components

Ne
IP ork
tw
N tw
Communication
on or
N
e
-IP k
Interface
n
tio
ica
un e
m evic
m
Co D
No
de
r
st e
Cl u
The topology components consist of a cluster, nodes and the

technology which connects them together.
Figure 1-4. PowerHA's Topology Components QV1203.1
• Topology components
- Cluster:
• Nodes (Power Systems servers)
• Networks (connections between the nodes)
• Communication interfaces (e.g.: Ethernet or token-ring network adapters)
• Communication devices (e.g.: /dev/hdisk for heartbeat on disk non-IP network).
• Nodes
In this context, any IBM Power Systems server which is a member of a PowerHA
cluster.
• Networks
Networks consist of IP and non-IP networks. The non-IP networks ensure cluster
monitoring can be done if there is a total loss of IP communication. Non-IP networks are
strongly recommended to be configured in PowerHA, and are required to provide true
high availability. In PowerHA 5.4 the importance of non-IP networks is emphasized by
the issuance of warnings when a cluster verification is performed and non-IP networks
are missing.

V3.1.0.1
Student Notebook
Uempty
Nodes
All Power Systems servers work with PowerHA in IBM
any combination of nodes within a cluster.

Performance requirements will usually constrain
the choice of systems to systems with similar
performance capabilities. p5-595
IBM Power 595
IBM
IBM
pSeri
IBM Power 570

es
p5-570 IBM Power 575

server
IBM
HC
R
U6 IBM
pSeri
es
p5-550/550
server
HC
R
server
U6
pSer
ies
I
B
M
IBM
server
A minimum of four free

p5-520/520 adapter slots is
IBM Power 550 recommended.
IBM Power 520

IBM Power Blade
Figure 1-5. Nodes QV1203.1
Supported nodes
As you can see, the range of systems that supports PowerHA is, well - everything. The
only requirement is that the system should have at least four adapter slots spare (two
for network adapters and two for disk adapters). The internal Ethernet adapter fitted to
most entry level Power Systems servers cannot be included in the calculations. It
should be noted that even with four adapter slots free, there is still be a single point of
failure as the cluster is only able to accommodate a single TCP/IP local area network
between the nodes.
PowerHA 5 works with Power Systems servers in a “no-single-point-of-failure” server
configuration. For a current list of systems that are supported with the version of
PowerHA you wish to use, please see the Sales manual at:
http://www.ibm.com/common/ssi.
Student Notebook
Supported Networking Environments
Ethernet / Etherchannel
Token Ring PC
PC Server
Server Server Traffic Flow Server

Server
FDDI
Server Server
Server
Non -IP
Traffic Flow
Heartbeat on Disk
Server
RS232/422
Target mode SSA
Target mode SCSI
Figure 1-6. Supported Networking Environments QV1203.1
Supported TCP/IP-based networks (called IP networks)

- Ethernet
- EtherChannel
- Token ring
- FDDI
- SP Switches
- ATM
- ATM LAN Emulation
Supported non-IP networks

- Heartbeat on Disk (using Enhanced Concurrent Mode Volume Groups)
- RS232/422
- Target Mode SCSI (TMSCSI)
- Target Mode SSA (TMSSA)

V3.1.0.1
Student Notebook
Uempty
Un-Supported/Limited Support Networks
• Un-supported networks:
í Serial Optical Channel Converter (SOCC)
í Serial Line Internet Protocol (SLIP)
í Fibre Channel Switch (FCS)
í 802.3 Ethernet (et interface type)
í Virtual IP Address (VIPA) interface
í Aggregate IP interface (ml0) with SP Switch2
• Limited support:
í Internet Protocol Version 6 (IPv6)
• Un-supported prior to HACMP v5.5
• Supported for service addresses only in HACMP v5.5
Figure 1-7. Un-Supported/Limited Support Networks QV1203.1
Un-supported/limited support networking environments

- Serial Optical Channel Converter (SOCC)
- SLIP
- Fibre Channel Switch (FCS)
- 802.3 ethernet
- Virtual IP Address (VIPA) facility of AIX
The pseudo IP address provided by VIPA cannot be reliably monitored by RSCT
or PowerHA. VIPA can be configured and used outside of PowerHA, but when
using VIPA on a PowerHA cluster node, ensure that the VIPA is configured on
subnets that are completely different from the subnets used by PowerHA.
- Aggregate IP Interface with the SP Switch2
With the SP Switch2 you have css0 and css1, PSSP allows you to configure an
Aggregate IP switch. This is an ml0 interface with its own IP address. This ml0
interface is not supported by PowerHA.
- IPv6
Prior to PowerHA v5.5, there is no support of IPv6. In PowerHA v5.5, IPv6 is
supported for service addresses.
Student Notebook
PowerHA's Resource Components

• Must include everything needed to run the application on
a node, for example:
íApplication server:
• Identifies scripts used by PowerHA to start and stop the
application
íService IP address(es):
• IP address used by clients to access the application
íVolume group(s):
• Shared storage for the application (if needed)
íFile system:
• File systems to be mounted (if needed)
íAnd so forth
• Resources for an application are grouped together into a
resource group and controlled as a unit.
Figure 1-8. PowerHA’s Resource Components QV1203.1
Resources
- Application Server - the application itself must be identified. The use of the term
Application Server in PowerHA describes the start/stop methods (scripts) for the
application. It is an object that points to the start/stop methods.
- Service IP Address/Label - IP address or hostname users use to connect to the
application. This IP address/hostname becomes a resource in the resource group
as it must be associated with the same node that is running the application. More
than one Service IP Address may be configured for a resource group.
- Volume Group - if the application requires shared disk storage, this storage is
contained within volume groups.
- Filesystem - an application often requires that certain filesystems be mounted.
In addition to the resources listed in the figure, you can also associate the following with
a resource group:
- NFS mounts - NFS filesystems to be mounted by the node running the application
- NFS exports - NFS filesystems to be exported by the node running the application
- Attributes - that can be assigned to control various aspects of the group’s behavior

V3.1.0.1
Student Notebook
Uempty
Resource Groups
Nodes where the •Application server
resource group can run •Service IP address
•Volume group
NODE LIST
•File system
How PowerHA will •and so forth
choose which node to RESOURCES
run the resource group
RUN POLICIES
ce G roup
Resour
Figure 1-9. Resource Groups QV1203.1
Resource group
A resource group is a collection of resources treated as a unit by PowerHA, including:
- The resources needed by an application
- The nodes they can potentially be activated on
- The policies the cluster manager should use to decide which node to choose during
startup, fallover, and fallback.
A cluster may have more than one resource group (usually one for each application),
thus allowing for very flexible configurations.
Student Notebook
Solution Components
Not Just PowerHA
Ethernet / Etherchannel
PC EMC
DS4000
RS/6000
RS/6000
Server Server
IBM
SAN
DS8000 RS/6000
Fibre
Figure 1-10. Solution Components QV1203.1
Not just PowerHA

The final high availability solution is more than just PowerHA. A high availability solution
must include:
- A reliable operating system (AIX)
- Applications that are tested to work in a high availability cluster
- Storage devices
- Appropriate selection of hardware
- Trained administrators
- Thorough design and planning

V3.1.0.1
Student Notebook
Uempty
AIX's Contribution to High Availability
9Object Data Manager (ODM)
9System Resource Controller (SRC)
9Logical Volume Manager (LVM)
9Journaled File System (JFS/JFS2)
9Online JFS Backup (splitlvcopy)
9Work Load Manager (WLM)
9Quality of Service (Qos)
9External Boot
9Software Installation Management (installp, smit, websmit)
9Reliable Scalable Cluster Technology (RSCT, RMC)
Figure 1-11. AIX's Contribution to High Availability QV1203.1
AIX contributions
AIX offers many availability benefits, for example, logical volume manager mirroring of
all data types, the journaled filesystem, and AIX’s capability to execute an online JFS
backup. AIX has a dynamic, multi-threaded kernel which allows configuration changes
to the operating system while the computer is running.
How often have you had to reboot an AIX system in order to configure a new hardware
device or extend a filesystem?
Student Notebook
Supported Shared Storage Environments

Most PowerHA clusters require shared storage.
SSA Adapter
RS/6000
RS/6000
DS4000
IBM
SSA
Maximum 25m
Host
System
SCSI DS8000
Controller RS/6000
SCSI SCSI SCSI SCSI

Module Module Module Module
Disk Disk Disk Disk
Drive Drive Drive Drive Storage Area Network (SAN)
Fibre Channel or
SCSI iSCSI
Virtual SCSI
(using Virtual I/O Server (VIOS))
Figure 1-12. Supported Shared Storage Environments QV1203.1
Shared storage environments

PowerHA is largely unconcerned about the disk storage that you select. Supported
technologies include SSA, SCSI, Fibre Channel and iSCSI. Most IBM storage is
supported with PowerHA and although there is no formal agreement, third-party storage
such as EMC and Hitachi can be used, although some custom modifications may be
required.
Note that data availability is not ensured by PowerHA. This must be done either through
the redundancy features of a storage device or through AIX LVM mirroring.
• For a complete list of supported IBM devices see the Sales Manual pages:
http://www.ibm.com/common/ssi .
• For the latest on supported hardware and other technical developments which
have not yet been added to the Sales Manual, see the Techdoc Flash pages
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/Web/Flashes .
• For information on a third party storage device, see the vendor’s web page.

V3.1.0.1
Student Notebook
Uempty
Some Clusters Do Not Have Shared Disks
Clusters providing firewall services do not usually have shared disks.

Can you think of any other examples?
Figure 1-13. Some Clusters Do Not Have Shared Disks QV1203.1
Shared disk not required

Think about the solutions that do not require shared disk. It’s not always the case that a
highly-available cluster requires disk.
Clusters providing firewall services do not usually have shared disks as this ensures
that if an attacker is able to crash one of the firewall boxes then they are presented with
a fresh box on the backup node. While it is true that if they can crash the first node then
they can probably crash the second node, there is little point in making it easier by
allowing them to start with a node that possibly retains some of the cracks that they’ve
managed to exploit in the first node.
Student Notebook
So What is PowerHA Really?

•An application which:
–Controls where resource groups run
–Monitors and reacts to events
•Does fallover
•Does reintegration
–Provides tools for cluster wide configuration and synchronization
clcomdES CSPOC and configuration tools
Cluster Manager Subsystem

SNMP
Topology Resource Event
SMUX
manager manager manager
peer
RSCT (Topology Svcs / RMC) snmpd clinfoES

clstat
Figure 1-14. So What is PowerHA Really? QV1203.1
PowerHA core components

PowerHA comprises a number of software components:
- The cluster manager (clstrmgrES) is the core process. The cluster manager
includes a topology manager to manage the topology components, a resource
manager to manage resource groups, an event manager with event scripts that
works through the RMC facility and RSCT to react to failures.
- In PowerHA 5.3 and later, the cluster manager contains the SNMP SMUX Peer
function (previously provided by the clsmuxpd) which provides cluster status to
clstat or to any SNMP manager, such as Tivoli, BMC or OpenView.
- The optional clinfoES daemon provides an API for communicating between the
cluster manager and your application. Clinfo provides remote monitoring capabilities
and can respond to a status change in the cluster. Clinfo can run on both servers
and clients (the source code is provided). The clstat command uses clinfoES to
display status via ASCII, Xwindows, or Web browser interfaces.
- In PowerHA 5.X, clcomdES allows the configuration tools to communicate to the
other nodes in a secure manner without using rsh and the /.rhost files.

V3.1.0.1
Student Notebook
Uempty
Additional Features of PowerHA
OLPW Configuration
webSMIT Assistant
Verification
Auto tests
clstrmgrES
CSPOC
DARE SNMP
RG in
WPAR
Tivoli Application
NetView Monitoring
PowerHA is shipped with utilities to simplify configuration, monitoring,

customization and cluster administration.
Figure 1-15. Additional Features of PowerHA QV1203.1
Additional features
- C-SPOC simplifies administration by providing SMIT menus to perform
administration tasks across all nodes in the cluster: including LVM, start/stop cluster
services, move a RG to another node or bring a RG offline or online.
- Dynamic Automatic Reconfiguration Event (DARE): Configuration changes can
be made to the cluster while the cluster is running.
- PowerHA plug-ins are available to monitor your cluster from Tivoli NetView.
- Application monitoring should be used to monitor the cluster’s applications and
restart them should they fail.
- Resource Group in WPAR: Beginning in PowerHA 5.5, PowerHA integrates with
the WPAR feature in AIX 6. A resource can be run within a WPAR.
- Verification is provided at PowerHA startup time, as part of synchronization, as a
manual process and a daily Automatic Cluster Configuration Monitoring function.
- The two node configuration assistant facility allows you to configure an PowerHA
cluster with very little input.
- Administration is made easier by the use of Online Planning Worksheets (OLPW)
and a WebSMIT interface.
Student Notebook
Some Assembly Required
Customized
Pre-event scripts
Application
PowerHA core events
•Application start script
•Application stop script
•Application monitor (optional) Customized
post-event scripts
• PowerHA is not an out of the box solution

• PowerHA's flexibility allows for complex customization in order
to meet availability goals
Figure 1-16. Some Assembly Required QV1203.1
Customization required
Application start and stop scripts: At a minimum, you must provide application start
and stop scripts to allow PowerHA to control your application.
Application monitors: It is highly recommended that you also configure application
monitors. The monitors can be custom scripts, or you can do process monitoring using
RMC.
PowerHA Smart Assists: These are a priced feature which provide start, stop and
monitor scripts for three popular applications (DB2, Oracle and Websphere). In
PowerHA 5.4 and later there is an API provided that allows 3rd party application
vendors to write Smart Assists.
Pre- and post-event scripts: PowerHA is shipped with event scripts (PowerHA core
events) which handle failure processing. In most cases, these core events will do all
that is needed. However, if you need to provide some special fallover behavior for your
environment, you can further customize PowerHA using pre- and post-event scripts.

V3.1.0.1
Student Notebook
Uempty
Just What Does PowerHA Do?
PowerHA basic functions:

– Monitors communication adapters/devices for 3 failures
– Moves resource groups
– Additional monitoring can be configured:
• Application monitoring: monitors health of your application(s)
• AIX error notification: LVM quorum loss, disk failures, any errlog error
• User Defined Events: almost any event in the AIX environment
Figure 1-17. Just What Does PowerHA Do? QV1203.1
PowerHA detects three kinds of network related failures

- NIC: A communications adapter or device failure
- Node: A node failure (all communication adapters/devices on a given node)
- Network: A network failure (all communication adapters/devices on a given network)
Additional monitoring
ERROR NOTIFICATION (via the AIX errdemon):
- Quorum loss: Moves affected RG to another node if quorum is lost
- Optional: You can add "automatic error notification" to monitor for disk failures
- Optional: You can monitor for anything that creates an entry in the errlog
APPLICATION MONITORS (via RMC or via a script which the user creates):
- Optional: Provides a mechanism to monitor that your application is running.
If not running, PowerHA tries to restart it. If it still fails, PowerHA moves the RG to
another node.
USER DEFINED EVENTS (via Resource Monitoring and Control (RMC)):
- Optional: UDEs can be created to use RMC to monitor for almost any possible
event in AIX. (For example, a file system full or a process not running, etc.)
Student Notebook
What Happens When Something Fails?
• How the cluster responds to a failure depends on what has failed, what
the resource group's fallover policy is, and if there are any resource
group dependencies
• The cluster's configuration is determined by the application's
requirements
• Typically another component of the same type takes over duties of
failed component (for example, another node takes over from a failed
node)
Figure 1-18. What Happens When Something Fails? QV1203.1
How PowerHA responds to a failure

PowerHA generally responds to a failure by using a still available component to take
over the duties of the failed component. For example, if a node fails, then PowerHA
initiates a fallover, an action which consists of moving the resource groups which were
previously on the failed node to a surviving node. If a Network Interface Card (NIC) fails,
PowerHA usually moves any IP addresses being used by clients to another available
NIC. If there are no remaining available NICs, PowerHA initiates a fallover. If only one
resource group is affected then only the one resource group is moved to another node.

V3.1.0.1
Student Notebook
Uempty
What Happens When a Problem is Fixed?
?
• How the cluster responds to the recovery of a failed component depends
on what has recovered, what the resource group's fallback policy is, and
what resource group dependencies there are
• The cluster's configuration is determined by the application's
requirements
• Cluster administrator may need to indicate/confirm that the fixed
component is approved for use
Figure 1-19. What Happens When a Problem is Fixed? QV1203.1
How PowerHA responds to a recovery

When a previously failed component recovers, it must be reintegrated back into the
cluster (reintegration is the process of PowerHA recognizing that the component is
available for use again). Some components, like NICs, are automatically reintegrated
when they recover. Other components, like nodes, may not be reintegrated until the
cluster administrator explicitly requests the reintegration (by starting the PowerHA
daemons on the recovered node, starting cluster services and possibly moving the
resource group or bringing it online).
Student Notebook
Standby: With Fallback
node1 fails A node2 fails
node1 node2
A A
One node is primary
node1 node2 node1 node2
(no change)
node1 returns node2 returns
A A

Figure 1-20. Standby: With Fallback QV1203.1
Standby
In standby configurations one (or more) nodes have no workload. The application
(resource group (RG)) normally runs on a primary or home node and the node with no
workload is the secondary or standby node.
- The startup policy indicates the home node
- The fallover policy indicates how to select the fallover node
- The fallback policy indicates automatic or manual fallback to the primary node
when the primary node recovers
There are several drawbacks:
- One node is not used (this is ideal for availability but not for utilization)
- A second outage on fallback
This can be extended to more nodes in two ways
- All nodes except one have applications and one node is the standby node.
- The resource group could be configured to have multiple backup nodes. If the
primary node fails, the RG is moved to the first standby node. If that fails, the second
standby node would be used. And so forth.

V3.1.0.1
Student Notebook
Uempty
Standby: Without Fallback
A node2 returns
node1 fails
node1 node2
A
Minimize downtime A
node1 returns node2 fails
A
node1 node2
Figure 1-21. Standby: Without Fallback QV1203.1
Minimize downtime
A resource group can be configured to not fall back to the primary node (or any other
higher priority node) when it recovers. This avoids the second outage which results
when the fallback occurs.
The cluster administrator can request that PowerHA move the resource group back to
the higher priority node at an appropriate time or it can simply be left on its current node
indefinitely (an approach which calls into question the terms primary and secondary, but
which is actually quite a reasonable approach in many situations).
Extending to more nodes

Extending a standby configuration without fallback can result in multiple applications
ending up on the node that stays up the longest.
Student Notebook
Mutual Takeover
A B
node1 fails node1 node2 node2 fails
Very common A B
B A
node1 node2
node1 node2
node1 returns node2 returns

(with Fallback) (with Fallback)
A B
node1 node2
Figure 1-22. Mutual Takeover QV1203.1
Mutual takeover
An extension of the primary node with a secondary node configuration is to have two
resource groups, one failing from right to left and the other failing from left to right. This
is referred to as mutual takeover.
Mutual takeover configurations are very popular configurations for PowerHA since they
support two highly available applications at a cost which is not that much more than
would be required to run the two applications in separate stand-alone configurations.
Issues
Each cluster node probably needs to be somewhat larger than the stand-alone nodes
as they must each be capable of running both applications, possibly in a slightly
degraded mode, should one of the nodes fail. If you are involved in the design and
planning of a cluster, it is important that these issues get discussed. If not properly
sized, the application performance may be unacceptable in a fallover situation when
both applications are running on one node.

V3.1.0.1
Student Notebook
Uempty
Concurrent: Multiple Active Nodes
node1, node2 and

node3 are all running
Application A, each using
a separate IP Address A A A
node1 node2 node3
If nodes fail, the application remains

continuously available as long as there
A A A are surviving nodes to run on.
node1 node2 node3
Fixed nodes resume running their copy of
the application.
Application must be designed to run simultaneously on multiple nodes.

This has the potential for essentially zero downtime.
Figure 1-23. Concurrent: Multiple Active Nodes QV1203.1
Concurrent mode
PowerHA also supports resource groups in which the application is active on multiple
nodes simultaneously. In such a resource group, all nodes run a copy of the application
and share simultaneous access to the disk. This style of cluster is often referred to as a
concurrent access cluster or concurrent access environment.
The application must be specifically designed to support concurrent access.
Service addresses
Since the application is active on multiple nodes, each node must have its own address
for clients to connect. There is no PowerHA service address resource for concurrent
resource groups. The client systems must be configured to select which server address
to communicate with, and be prepared to switch to another address should the one that
they’re dealing with stop functioning (presumably, because the node has failed). It is
also possible to configure an IP multiplexer between the clients and the cluster which
redistributes the client sessions to the cluster nodes, although care must be taken to
ensure that the IP multiplexer does not itself become a single point of failure.
Student Notebook
Fundamental PowerHA Concepts

• Topology: networking components
• Resources: the entities which are being made highly available
• Resource group: a collection of resources which PowerHA
controls as a single unit
– A given resource may only appear in at most one resource group
• Resource group policies:
– Startup policy: which node the resource group is activated on
– Fallover policy: determines target when there is a failure
– Fallback policy: determines target when fallback occurs
• Customization is the process of augmenting PowerHA, defining
the topology and resources in the PowerHA ODM (typically via
SMIT) and implementing scripts which PowerHA invokes at
appropriate times
Figure 1-24. Fundamental PowerHA Concepts QV1203.1
Terminology
A clear understanding of the above concepts and terms is important as they appear
over and over again both in the remainder of the course and throughout the PowerHA
documentation, log files and smit screens.

V3.1.0.1
Student Notebook
Uempty
Cluster Requirements and Capabilities
• Applications:
– Must be manageable via scripts (start/restart and stop)
– Can be restarted via monitoring
• Resource groups:
– Must be serviced by at least two nodes
– Can have different policies
– Can be migrated (manually or automatically) to rebalance loads
• Clusters:
– Must have at least one IP network and should have one non-IP
network
– Need not have any shared storage
– Can have any combination of supported nodes *
– For disaster recovery, the cluster may be split across two sites
•May or may not require replicating data (PowerHA/XD).
* Application performance requirements and other operational issues

almost certainly impose practical constraints on the size and
complexity of a given cluster.
Figure 1-25. Cluster Requirements and Capabilities QV1203.1
Importance of planning
Planning, designing, configuring, testing and operating a successful PowerHA cluster
requires considerable attention to detail. In fact, a careful methodical approach to all the
phases of the cluster’s life cycle is probably the most important factor in determining the
ultimate success of the cluster.
Methodical approach
A careful methodical approach takes into account the relevant points above, and many
other issues which are discussed in this course and in the PowerHA documentation.
Student Notebook
High Availability Vs. Disaster Recovery

•Implementing cluster sites can extend PowerHA
to a disaster recovery solution
•High availability (HA)

High availability is the capability to provide access to
applications regardless of local failures, whether these
failures are in the business processes, in the physical
facilities, or in the IT hardware or software.
•Disaster recovery (DR)

Disaster recovery is the capability to recover a data center at
a different site if a disaster destroys the primary site or
otherwise renders it inoperable. With a disaster recovery
solution processing resumes at a different site and on
different hardware.
Figure 1-26. High Availability Vs. Disaster Recovery QV1203.1
High availability vs. disaster recovery

As generally used, the term high availability (HA) refers to keeping an application
available at one site.
Disaster recovery (DR) refers to keeping applications available even if a site is
destroyed or rendered inoperable. Processing will be resumed in another location.
PowerHA and disaster recovery

As we have been discussing in this unit, PowerHA can be used to provide high
availability solutions.
With the included cross-site LVM mirroring feature, PowerHA can provide disaster
recovery solutions for limited distances.
With the optional priced PowerHA Extended Distance (PowerHA/XD) feature, PowerHA
can be used to provide extended distance disaster recovery solutions.
In the next two visuals, we introduce PowerHA’s DR capabilities. However, cross-site
LVM mirroring and PowerHA/XD will not be covered in any detail in this class.

V3.1.0.1
Student Notebook
Uempty
Disaster Recovery: Limited Distance
•Localized separation of data centers (sites)
– Limited distance: PowerHA for AIX
• Each site sees same LUNs (half local, half remote)
• LVM mirroring is used to copy data to remote site
• PowerHA Cross-site LVM mirroring capability can and should be used
• Max distance limited by bandwidth and latency of the SAN being used
WAN
LVM
1 1’
Mirroring
SAN
North South
Campus Campus
Figure 1-27. Disaster Recovery: Limited Distance QV1203.1
DR for limited distances

PowerHA’s cross-site LVM mirroring feature is very viable for cross-town, cross-campus
availability/disaster recovery requirements. This implementation blurs the line between
high availability and disaster recovery.
The definition of a disaster, in this case, must be quite local. In this case, the disaster is
limited to a data center, building, power grid, section/neighborhood of the city, and so
forth.
This approach can be seen as an in-between solution, because it solves the problems
of failure at the levels noted, and also allows for a more localized fallover for scheduled
maintenance, meaning substantially reduced impact on users.
A practical definition of limited distance is about 15km (approx. 9 miles).
Student Notebook
Disaster Recovery: Extended Distance

•Geographic separation of data centers (sites)
– Extended distance (Geographic Clustering Solution):
PowerHA/XD
•Distance can be unlimited
•Local access to local storage, replicated remotely
•Application, network, and disk (in some cases) independent
•Automated site failover and reintegration
•A single cluster across two sites
GLVM
Metro Mirror
1 1’
HAGEO*
Data Replication
Toronto Brussels
* HAGEO has been replaced by PowerHA/XD GLVM V5.5
Figure 1-28. Disaster Recovery: Extended Distance QV1203.1
• DR for extended distances

The PowerHA/XD priced feature provides three software solutions for disaster recovery:
- PowerHA/XD for Metro-Mirror uses IBM TotalStorage ESS/DS/SVC volumes that
use Peer-to-Peer Remote Copy (PPRC) to copy data to a remote site for disaster
recovery purposes. The physical distance between sites is limited to the capabilities
of the ESS/DS/SVC hardware, practical maximum: ~300km.
- PowerHA/XD for Geographic Logical Volume Manager (GLVM) uses a TCP/IP
network to copy data to a remote site. With PowerHA/XD 5.5 and AIX 6.1.D, GLVM
provides synchronous mirroring (limit: 100-200km) and asynchronous mirroring
(unlimited distance). Asynchronous mirroring means that the disks at the standby
site can be out of sync with the primary site. Note: practical distance limits exist
based on network throughput and how far out of sync the customer can tolerate.
- HACMP/XD HAGEO has been replaced by PowerHA/XD GLVM V5.5. GLVM offers
all of the features of HAGEO technology while being easier to configure, more
reliable, and better integrated with PowerHA and AIX LVM. Customers currently
using HAGEO are recommended to develop a strategy for moving from HAGEO to
GLVM.

V3.1.0.1
Student Notebook
Uempty
Things PowerHA Does Not Do
TSM
• Backup and restoration

• Time synchronization
• Application specific configuration
• System Administration tasks unique to each node
Figure 1-29. Things PowerHA Does Not Do QV1203.1
Things PowerHA does not do

PowerHA does not automate your backups, neither does it keep time in sync between
the cluster nodes. These tasks do require further configuration and software. For
example, Tivoli Storage Manager for backup and a time protocol such as xntp for time
synchronization.
Student Notebook
When is PowerHA
Not the Correct Solution?
• Zero downtime required
– Maybe a fault tolerant system is the correct choice
– 7x24x365, PowerHA occasionally needs to be shut down for
maintenance
– Life-critical environments
• Security issues
– Too little security
•Lots of people with the ability to change the environment.
– Too much security
•C2 and B1 environments may not allow PowerHA to function as
designed.
• Unstable environments
– PowerHA cannot make an unstable and poorly managed
environment stable
– PowerHA tends to reduce the availability of poorly managed
systems
Figure 1-30. When is PowerHA Not the Correct Solution? QV1203.1
Zero downtime
If you truly need zero down time, PowerHA is not the solution. It takes a few minutes for
fallover and PowerHA may need to be occasionally shut down for maintenance, An
example of zero down time may be the intensive care room. Also PowerHA is not
designed to handle many failures at once.
Security issues
PowerHA may not be able to operate in some highly secure environments.
Unstable environments
The prime cause of problems with PowerHA is poor design, planning, implementation,
or administration. If you have an unstable environment, with poorly trained
administrators, easy access to the root password, and a lack of change control and
documented operational procedures, PowerHA is not the solution for you.
With PowerHA, the only thing more expensive than employing a professional to plan,
design, install, configure, customize, and administer the cluster is employing an
amateur.

V3.1.0.1
Student Notebook
Uempty
What Do We Plan to
Achieve in this Course?
Your mission in this course is to build a two-node highly available
cluster using two previously separate System p systems, making a
single application highly available in a standby configuration.
This course focuses on configuring the networks for PowerHA.
Configuring shared storage will be covered in the next course.
A A
Figure 1-31. What Do We Plan to Achieve in this Course? QV1203.1
Goals
During this course you will design, plan, configure, customize, and administer a
two-node high availability cluster running PowerHA 5.5 on an IBM Power Systems
server. You will learn how to build a standby environment for one application.
Some classroom environments will involve creating the cluster on a single Power
Systems server between two LPARs. Although this is not a recommended configuration
for production, it provides the necessary components for a fruitful PowerHA
configuration experience.
Student Notebook
Overview of the Implementation Process

• Plan and design
– Focus on eliminating single points of failure
• Configure AIX
– Networks (IP interfaces, /etc/hosts, non-IP networks and
devices)
– Storage (adapters, LVM volume group, filesystem)
– Applications start and stop scripts
• Install the PowerHA filesets and reboot
• Configure the PowerHA environment on one node
– Topology
•Cluster, node names, PowerHA IP and non-IP networks
– Resources and Resource groups:
•Identify name, nodes, policies
•Resources: Application Server, service label, VG, filesystem
• Synchronize (copies the configuration to the other node(s))
• Start cluster services
Figure 1-32. Overview of the Implementation Process QV1203.1
Implementation process
• Work as a team. It can’t be stressed enough that it will be necessary to work with others
when you build your PowerHA cluster in your own environment. Practice here will be
useful.
• Look at AIX environment.
- For storage, configure adapters and LVM components required for application
- For networks, configure communication interfaces, devices, name resolution via
/etc/hosts and service address(es) for the application(s)
- For application build start and stop script and test outside of the control of PowerHA
• Install the PowerHA software and reboot.
• Configure the topology and resource groups (and resources).
• Synchronize, start and test.

V3.1.0.1
Student Notebook
Uempty
Sources of PowerHA Information
• PowerHA Web Site:
- http://www-03.ibm.com/systems/p/ha/
• PowerHA manuals are available online – READ THEM!
- http://www-03.ibm.com/systems/p/library/hacmp_docs.html
• Sales Manual (specifications, supported hardware, etc.):
- http://www.ibm.com/common/ssi
• Techdocs Flash web site (HW support and technical updates):
- http://www-03.ibm.com/support/techdocs/atsmastr.nsf/Web/Flashes
• /usr/es/sbin/cluster/release_notes
• USSE PowerHA live virtual classes:
- http://bvrgsa.ibm.com/projects/t/tse/Roadmap/HACMP.html
• HA and HACMP Wiki:
- http://w3.tap.ibm.com/w3ki03/display/hacmp/Home+of+The+HA+and+HACMP+Wiki
• Techlink PSdb database: - https://techlink.austin.ibm.com/psdb/

Figure 1-33. Sources of PowerHA Information QV1203.1
IBM Training PowerHA classes

IBM Training offers a number of classes in a traditional classroom environment. Two of
these classes cover topics that we do not cover in depth in the USSE virtual classes:
- HACMP Internals (AU60)
- HACMP System Administration III: Virtualization and Disaster Recovery (AU62)
For details, go to the Learning@IBM web site and search for HACMP:
https://w3.ibm.com/learning/lati/sites/explorer/home.html
Additional web sites for storage

http://www.storage.ibm.com
http://www-1.ibm.com/servers/storage/support/software/sdd.html
ftp://ftp.software.ibm.com/storage/fastt/fastt500/HACMP_config_info.pdf
Student Notebook
Review (1 of 2)
1. Which of the following are examples of topology

components in PowerHA (select all that apply)?
a. Node
b. Network
c. Service IP label
d. Hard disk drive
2. True or False?
All clusters require shared disk for storage of PowerHA log
files.
3. True or False?
All nodes in an PowerHA cluster must have equivalent
performance characteristics
Figure 1-34. Review (1 of 2) QV1203.1

V3.1.0.1
Student Notebook
Uempty
Review (2 of 2)
4. True or False?
Resource Groups may be moved from node to node.
5. True or False?
PowerHA/XD is a solution for building geographically
distributed clusters.
6. Which of the following capabilities does PowerHA not
provide (select all that apply)?:
a. Time synchronization.
b. Automatic recovery from node and network adapter
failure.
c. System Administration tasks unique to each node.
Backup and restoration.
d. Fallover of just a single resource group.
Figure 1-35. Review (2 of 2) QV1203.1
Student Notebook
Exercise
Exploring PowerHA
Figure 1-36. Exercise QV1203.1
Notes:

V3.1.0.1
Student Notebook
Uempty
Unit Summary 1 of 2
• PowerHA concepts:
– Topology: network components
• IP networks (network adapter)
• Non-IP networks (network device)
– Resources: The entities being made available
• Application server - Object that points to application start and stop scripts
• Service IP address - Address/hostname used to access application
• Storage - Volume groups, file systems, etc.
• NFS mounts / NFS exports
– Resource group (RG): A collection of resources that PowerHA
controls as a group
– Fallover: Move resource group to another node when there is a failure
– Fallback: Move resource group to "normal" node when recovery
occurs
– Resource group policies: How PowerHA will handle a resource group
in event of a failure
• Startup policy: determines target node for the resource group when PowerHA is started
• Fallover policy: determines target node when there is a failure
• Fallback policy: determines target node when recovery occurs
– Event: A change of status within a cluster
• The PowerHA system is event-driven
• When the Cluster Manager detects a change in cluster status, it initiates pre-defined
event handling and any user-defined customized processing
Figure 1-37. Unit Summary 1 of 2 QV1203.1
Student Notebook
Unit Summary 2 of 2
• PowerHA provides these basic functions:
– Monitor networks (communication adapters/devices) for failure (RSCT)
– Monitor applications for failure (application monitors)
– Monitor many other failures and events (AIX errdemon and RMC)
– Move resource groups (fallover / fallback) based on resource group policies
• Some customization is required:
– Application start and stop scripts (required)
– Application monitors (optional, but highly recommended)
– Pre- and post-event scripts (optional, if customized fallover is needed)
• Basic PowerHA configurations:
– Serial access to shared data and resources:
application runs on one node at a time
• Standby (with or without automatic fallback) - standby node takes over RG in case of failure
• Mutual takeover - multiple active nodes; RG moved to another active node in case of failure
– Concurrent access to shared data and resources:
application runs in parallel across several nodes
• PowerHA is not the right solution if:
– Zero downtime is required
– Highly secure environment is required
– Systems are unstable or poorly managed

V3.1.0.1
Student Notebook
Uempty Unit 2. Configuring Networks for PowerHA

This unit describes the PowerHA functions related to networks. You
learn which networks are supported in a PowerHA cluster and what
you have to take into consideration for planning and configuring
networks.

• Discuss how PowerHA uses networks
• Describe PowerHA network topology components and terminology
• Explain why a non-IP network is essential for any PowerHA cluster
• Describe the basic PowerHA network configuration rules
• Describe persistent IP addresses and how they can be used
• Explain and configure IP Address Takeover (IPAT):
- IPAT via IP aliasing
- IPAT via IP replacement
- Select the appropriate style of IPAT for a given context
• Describe how the AIX boot sequence changes when IPAT is
configured in a cluster
• Discuss the importance of consistent IP addressing and labeling
conventions
• Explain how user systems are affected by IPAT related operations
• Describe the ARP cache issue and several ways it can be
addressed
• Describe the limited IPv6 support available in PowerHA v5.5

• Review quiz
• Exercise
References
PowerHA manuals
http://www-947.ibm.com/systems/support/supportsite.wss/brandmain?brandind=
5000025
System p Support Site (Fix Central)
© Copyright IBM Corp. 2007, 2009 Unit 2. Configuring Networks for PowerHA 2-1
Student Notebook
Unit Objectives
• Discuss how PowerHA uses networks
• Describe PowerHA network topology components and terminology
• Explain why a non-IP network is essential for any PowerHA cluster
• Describe the basic PowerHA network configuration rules
• Describe persistent IP addresses and how they can be used
• Explain and configure IP Address Takeover (IPAT):
– IPAT via IP aliasing
– IPAT via IP replacement
– Select the appropriate style of IPAT for a given context
• Describe how the AIX boot sequence changes when IPAT is
configured in a cluster
• Discuss the importance of consistent IP addressing and labeling
conventions
• Explain how user systems are affected by IPAT related operations
• Describe the ARP cache issue and several ways it can be addressed
• Describe the limited IPv6 support available in PowerHA v5.5
Figure 2-1. Unit objectives QV1203.1
Unit objectives
This unit discusses networking in the context of PowerHA.

V3.1.0.1
Student Notebook
Uempty
How PowerHA Uses Networks
PowerHA uses networks to:
1. Configure and make highly available an application access address
(service address)
2. Monitor network interfaces to detect failures and co-ordinate event
processing (RSCT)
3. Internode configuration, verification and synchronization (clcomd)
4. Provide cluster status (SNMP)
clstat, cldump, cldisp

1 SNMP managers
en0 en1 en0 en1

2
RSCT RSCT 4
3
clcomd clcomd
clstrmgrES clstrmgrES
snmpd snmpd
Figure 2-2. How PowerHA Uses Networks QV1203.1
How PowerHA Uses Networks

• Client access to applications:
Satisfying this requirement for client access to the cluster involves a bit more than just
plugging in a network cable.
• Monitor network interfaces:
Detecting node, network and network interface card (NIC) failures imposes a number of
requirements on how the networks are designed. Being able to distinguish between
certain failures imposes yet more requirements on the network design. PowerHA uses
the Reliable Scalable Cluster Technology (RSCT) for these services.
• Internode configuration and synchronization communication:
Assuming that the requirements imposed by the first two uses are properly satisfied,
this use does not impose any additional requirements on the network design. All
communication between nodes relating to configuration is sent through the Cluster
Communications daemon, clcomd, which runs on each node.
• Cluster status:
PowerHA uses the Simple Network Monitor Protocol (SNMP) to provide status to a
number of different utilities.
Student Notebook
PowerHA Configures
Highly Available Service Addresses
• PowerHA configures application access addresses
– The application access address is called the Service Address
– Configured on top of an IP interface
– Can use either an alias or replacement method
– Address can be moved to a new interface when the underlying
interface fails
• High availability requirements:

– Multiple network interface cards (NICs) per network per node
– (Possibly) multiple networks per node
– Careful network design and implementation all the way out to the
client systems
Figure 2-3. PowerHA Configures Highly Available Service Addresses QV1203.1
High Availability requirements

• Network interface card (NIC): In order to avoid the NIC being a single point of failure,
(SPOF) each cluster node requires at least two NICs per network. The alternative is that
the loss of a single NIC would cause an outage while the application (that is, the
resource group) is moved to another node.
• Network as SPOF: The network itself is a single point of failure since the failure of the
network will disrupt the users’ ability to communicate with the cluster. The probability of
this SPOF being an issue can be reduced by careful network design.
If the network must be eliminated as a SPOF, then the cluster requires at least two
networks. This eliminates the network directly connected to the cluster as a SPOF. It is
not unusual for the users to be located some number of hops away from the cluster.
Each of these hops involves routers, switches and cabling - potential SPOFs. Truly
eliminating the network as a SPOF can become a massive undertaking. Many
organizations compromise by designing the network to ensure that no single failure
deprives all key users of their access to the cluster. In the end, there is simply no
replacement for careful network design and implementation all the way out to the users.

V3.1.0.1
Student Notebook
Uempty
PowerHA Monitors
Networks and Detects Failures
• NICs/devices are monitored by Topology Services
– Via heartbeats
• The following failures are detected:
– Network Interface Card (NIC) failure
– Network failure
– Node failure
IP network
en0 en1 en0 en1
non-IP network
node1 node2
Figure 2-4. PowerHA Monitors Networks and Detects Failures QV1203.1
What PowerHA monitors

PowerHA uses RSCT to monitor and detect network failures. RSCT sends heartbeats
over IP and non-IP networks. Using the information from RSCT, PowerHA handles
three different types of failures:
- network interface card (NIC) failures
- network failures
- node failures
Other failures
PowerHA uses AIX features such as the error notification facility to respond to other
failures (for example, the loss of a volume group can trigger a fallover) but PowerHA
does not detect them directly.
In addition, Cluster administrators can configure their own error notifications and even
User Defined Events to cause PowerHA to take action on almost any incident in AIX.
Student Notebook
Heartbeat Packet Flow

• Topology Services sends heartbeat packets to all PowerHA
defined IP interfaces (enX), non-IP devices (RS-232, hdiskX)
• Heartbeat packets are sent one way with no acknowledgement
• This is sufficient to detect all NIC, network and node failures
en0 en1 en0 en1
node1 node2
Figure 2-5. Heartbeat Packet Flow QV1203.1
Heartbeat packets
PowerHA’s primary monitoring mechanism is for Topology Services to send heartbeat
packets. They are sent from every NIC and to every NIC and to and from non-IP
devices.
Heartbeating pattern
IP and non-IP heartbeat packets are sent in a ring or loop. In a two node cluster this
amounts to heartbeat packets going in both directions.
Heartbeat failures determined by the receiver - not the sender

Heartbeat packets are not acknowledged. Instead, each node knows what the
heartbeat pattern is and simply expects to receive appropriate heartbeat packets on
appropriate network interfaces. Noticing that the expected heartbeat packets have
stopped arriving is sufficient to detect failures.

V3.1.0.1
Student Notebook
Uempty
Heartbeat Rings
• For redundancy, there should be at least two NICs on each physical network
• Each set of NICs will be organized into a heartbeat ring
(for example: en0 on all nodes in ring one; en1 on all nodes in ring two)
• Heartbeats are sent in one direction around each ring
• Addresses on different subnets are used
(for example: en0 addresses on 10.0.1/24 and en1 on 10.0.2/24)
One physical network
en0 en1 en0 en1 en0 en1

node1 node2 node3
subnet mask = 255.255.255.0
Heartbeat ring one Heartbeat ring two
10.0.1.1 10.0.1.2 10.0.2.2

en0 en0 en1 10.0.2.1 en1
subnet one subnet two
en0 en1
node3 10.0.1.3 node3 10.0.2.3
Figure 2-6. Heartbeat Rings QV1203.1
Heartbeat rings
To prevent the NIC from being a SPOF, at least two NICs are needed on each network.
In order to detect and diagnose failures, you must configure the sets of NICs into
separate subnets. For example, the first set of NICs (en0 in the drawing) are in one
subnet (10.0.1 subnet in the drawing) and the second set of NICs (en1 in the drawing)
are in a second subnet (10.0.2 subnet in the drawing).
This allows RSCT to send heartbeats to a specific NIC on a node and thus to be able to
differentiate between failures on one or the other NIC.
We will cover the rules for configuring these addresses used for heartbeating in a few
minutes.
Student Notebook
Failure Detection Versus Failure Diagnosis

• Failure detection is realizing that something is wrong
– For example, node2's en1 stopped receiving heartbeat packets
• Failure diagnosis is figuring out what is wrong
– For example, figuring out that node1's en1 NIC has failed
• PowerHA uses RSCT to do both detection and diagnosis
en0 en1 en0 en1
node1 node2
Figure 2-7. Failure Detection Versus Failure Diagnosis QV1203.1
Detection
The heartbeat patterns just discussed are sufficient to detect a failure in the sense of
realizing that something is wrong. They are not sufficient to diagnose a failure in the
sense of figuring out exactly what is broken.
For example, if the en1 interface on node1 fails as in the visual above, node1 stops
receiving heartbeat packets via its en1 interface, and node2 stops receiving heartbeat
packets via its en1 interface. node1 and node2 both realize that something has failed,
but neither of them have enough information to determine what has failed.

V3.1.0.1
Student Notebook
Uempty
Failure Diagnosis Example
• When a failure is detected, PowerHA (RSCT Topology
Services) uses specially crafted packet transmission patterns
to diagnose the actual failure by ruling out other alternatives
• Example:
1. RSCT on node1 notices that heartbeat packets are no longer
arriving via en1 and notifies node2 (which has also noticed that
heartbeat packets are no longer arriving via its en1)
2. RSCT on both nodes send diagnostic packets between various
combinations of NICs (including out via one NIC and back in via
another NIC on the same node)
3. The nodes soon realize that all packets involving node1's en1 are
vanishing but packets involving node2's en1 are being received
4. DIAGNOSIS: node1's en1 has failed
Figure 2-8. Failure Diagnosis Example QV1203.1
Diagnostic heartbeat patterns

Once one or more cluster nodes detect a failure, they share information and plan a
diagnostic packet pattern or series of patterns which will diagnose the failure.
These diagnostic packet patterns can be considerably more network-intensive than the
normal heartbeat traffic, although they usually only take a few seconds to complete the
diagnosis of the problem.
Student Notebook
What If All Heartbeat Packets Stop?

• If a node does not receive any heartbeat packets, the node cannot distinguish
between failure of the network and failure of the other node
– The node concludes that the other node is down
– The node attempts to take over resources, based on resource group policies
• If it is a network failure:
– Both nodes conclude the other node is down
– Both nodes try to take over resources
• Both nodes try to acquire shared storage: real potential for data corruption!
• This is called a partitioned cluster or split-brain cluster
en0 en1 en0 en1
node1 node2
Figure 2-9. What If All Heartbeat Packets Stop? QV1203.1
Total loss of heartbeat traffic

If a node in a two-node cluster realizes that it is no longer receiving any heartbeat
packets from the other node, then it starts to suspect that the other node has gone
down. Once it determines that it is totally unable to communicate with the other node, it
concludes that the other node has failed.
If the network fails

If the network fails, then each node concludes that the other node has failed. Each node
proceeds to take over any resource groups currently resident on the other node.
Partitioned cluster
Since each node is, in fact, still very alive, the result is that the applications are now
running simultaneously on both nodes. If the shared disks are also online to both nodes,
then the result could be a quite massive data corruption problem. This situation is called
a partitioned cluster. It is, clearly, a situation which must be avoided.

V3.1.0.1
Student Notebook
Uempty
Clusters Should Have Multiple Networks
• Multiple networks protect against a partitioned cluster
– There must be more than one network to distinguish
between: Failure of a network VS failure of a node
• Second network should be a non-IP network
– There must be a non-IP network to distinguish between:
Failure of a node's IP subsystem VS failure of a node
• Therefore, ALWAYS INCLUDE A NON-IP NETWORK!
en0 en1 en0 en1
non-IP network
node1 node2
Figure 2-10. Clusters Should Have Multiple Networks QV1203.1
Why we need more than one network

If a partitioned cluster is to be avoided then every cluster must be configured with at
least two ways for nodes to communicate with each other.
Why we need a non-IP network

It is also possible for the entire Internet Protocol (IP) subsystem to fail on a node without
the node crashing. In order to distinguish between the failure of the IP subsystem on a
node and the failure of the node itself, every cluster most be configured with a way
to communicate between nodes which does not require IP to be operational.
Both IP and non-IP networks are needed

These pathways which do not require IP are called non-IP networks. Every cluster must
be configured with enough non-IP networks to ensure that any node can communicate
with every other node (possibly by asking an intermediate node to pass along
messages) without requiring any nodes’ IP subsystem to be operational.
Student Notebook
Failure Recovery and Reintegration

• Failed components are monitored in order to detect their
recovery
• Recovered components are reintegrated back into the cluster
• Reintegration might trigger significant actions
– For example, recovery of primary node may trigger fallback of
resource group to primary node, based on resource group policies
en0 en1 en0 en1 en0 en1 en0 en1
Figure 2-11. Failure Recovery and Reintegration QV1203.1
NIC and network recovery

NIC cards and networks are automatically reintegrated into the cluster when they
recover.
Node recovery
In contrast, a node is not considered to have recovered until the PowerHA daemons
have been started up on the node. This behavior allows the user to reboot the node and
implement any necessary repair action without the PowerHA declaring failures and/or
perform automatic reintegration.
The reintegration of a component might trigger quite significant actions. For example, if
a node is reintegrated which has a high priority within a resource group then, depending
on how the resource group is configured, the resource group might fallback.

V3.1.0.1
Student Notebook
Uempty
PowerHA Networking Support
• Supported IP networking technologies:
– Ethernet
•All speeds
•Not the IEEE 802.3 frame type which uses et0, et1 ...
– Etherchannel
– FDDI
– Token-Ring
– ATM and ATM LAN Emulation
– SP Switch 1 and SP Switch 2
– Infiniband
• Supported non-IP network technologies:
– Heartbeat over Disks (diskhb)
•Requires Enhanced Concurrent Volume Group and
PowerHA 5.x
– RS232/RS422 (rs232)
– Target Mode SSA (tmssa)
– Target Mode SCSI (tmscsi)
Figure 2-12. PowerHA Networking Support QV1203.1
Supported IP networks
PowerHA supports all of the popular IP networking technologies (and a few which are
possibly not quite as popular). Note that the IEEE 802.3 Ethernet frame type is not
supported.
Supported non-IP networks

PowerHA supports four non-IP networking technologies. Heartbeat over disk and
RS-232 are the most prevalent. The advantage of heartbeat over disk is that there is no
need for additional hardware, assuming you have shared storage and are willing to
create an enhanced concurrent mode volume group. No data area is used for the
heartbeat on disk processing.
Student Notebook
Network Types
PowerHA categorizes all networks:
• IP networks:
– Network type: ether,token,fddi,atm,

hps (SP Switch or High Performance Switch),
XD_data,XD_ip (IP networks used with PowerHA/XD)
– Network attribute: public or private (Oracle only)
• Non-IP networks:
– Network type: rs232, tmssa, tmscsi, diskhb,

XD_rs232 (non-IP network used with PowerHA/XD)
Figure 2-13. Network Types QV1203.1
IP network attribute
The default for this attribute is public. Oracle uses the private network attribute
setting to select networks for Oracle inter-node communications. This attribute is not
used by PowerHA itself. See the HACMP for AIX, Planning Guide for more information.
Non-IP networks
PowerHA uses non-IP networks for:
- Alternative non-IP path for PowerHA heartbeat and messaging
• Differentiates between node/network failure
• Eliminates IP as a single point of failure
PowerHA and virtual Ethernet adapters

PowerHA 5.3 and later supports virtual Ethernet in POWER5- and POWER6-based
systems as type ether. There are some issues, which we discuss later.

V3.1.0.1
Student Notebook
Uempty
PowerHA Topology Components
• PowerHA uses some unique terminology to describe the type and
function of topology components under its control
IP lab
IP el s
Ne IP addres
two
rk
vancouver-service 192.168.5.2
TCP/IP network -Internalnet

Comm
u nicatio
n Interf
ace
Interface
Network
Interface
Interface
Network
Network
Interface
Network
Card
Card
Card
Card
ne non
tw -IP
or
ks
Communication Device
Serial Serial
Port
non IP - rs232 Port
non IP - tmssa
node1 node2
non IP - diskhb
nod
en
am
e
Figure 2-14. PowerHA Topology Components QV1203.1
Terminology
- node: An IBM system p server operating within an PowerHA cluster.
- node name: The name of a node from PowerHA’s perspective.
- IP label: For IP networks, the name specified in the /etc/hosts file or by the Domain
Name Service for a specific IP address.
- IP network: A network which uses the TCP/IP family of protocols.
- non-IP network or serial network: A point-to-point network which does not rely on
the TCP/IP family of protocols.
- communication interface: A network connection onto an IP network (slightly better
definition coming shortly).
- communication device: A port or device connecting a node to a non-IP network
(slightly better definition coming shortly).
Student Notebook
PowerHA Communication Hardware Terms

• A communication interface refers to IP-based networks and
NICs
A PowerHA communication interface is a combination of:
– A network interface (en0)
– An address (195.16.20.10)
– An IP label (node1-if1)
(address/label name resolution must be in /etc/hosts file)
• A communication device refers to one end of a point-to-point

non-IP network connection (/dev/tty1, /dev/hdisk1 or /dev/tmssa1).
• A communication adapter is an X.25 adapter used to support
a Highly Available Communication Link.
Figure 2-15. PowerHA Communication Hardware Terms QV1203.1
PowerHA network terminology

When using PowerHA SMIT, it is important to understand the difference between
communication interfaces, devices and adapters. Understanding these distinctions will
help you navigate through the SMIT menus.
- Communication interfaces:
Interfaces for IP-based networks
Note: The term communication interface in PowerHA refers to more than just the
physical NIC. From PowerHA’s point of view, a communication interface is an object
defined to PowerHA, which includes:
• The logical interface (the name for the physical NIC), such as en0
• The IP label / address
- Communication devices:
Devices for non-IP networks
- Communication adapters:
X.25 adapters

V3.1.0.1
Student Notebook
Uempty
PowerHA Address Terms
•A service address is:
–Used by clients to access an application
–Configured by PowerHA and kept highly available
–Configured either by replacement or by alias
•A service label is the IP label associated with a service address
•A service interface is an interface configured with a service address
•A non-service address or boot address is:

–An address that is stored in the ODM
–Configured by AIX at boot time
–Not used to access an application
–Used by RSCT for heartbeating (in most cases)
•A non-service label is the IP label associated with a non-service address
•A non-service interface is an interface that does not have a service IP address,
but could be used as a backup
•A persistent address is
–Configured by PowerHA by alias
–Kept highly available on the interfaces of a single node
•A persistent label is the IP label associated with a persistent address
Figure 2-16. PowerHA Address Terms QV1203.1
More terminology
• Service IP label / address: An IP label or address used by client systems to access
services running within the cluster. Used with IP Address Takeover (IPAT).
• Applications should use the service IP label / address: Non-service IP addresses
should not be used by client systems to contact the cluster’s applications. A client
system which gets into the habit of connecting to its application using a non-service IP
address will not be able to find its application after a fallover to a different node.
• Service interface: A communications interface used by clients that is configured with a
service IP label / address (either by alias or by replacement).
• Non-service (boot) IP label / address: An IP address which is stored in the AIX ODM
and configured onto a NIC at boot time.
• Non-service interface: A communications interface that is configured with a
non-service (boot) IP address. It can be used as a backup for a service IP address.
• Persistent IP label / address: An IP address monitored by PowerHA but it stays on the
node on which it is configured. It is implemented as an alias and PowerHA will attempt
to keep this IP label / address highly available on the same node.
Student Notebook
Node Name Considerations (1 of 2)

• There are several names a node can be known by: the AIX hostname,
the LPAR name, the PowerHA node name or an IP label name
• These concepts should not be confused
AIX hostname LPAR name

# hostname # /usr/bin/lparstat –i | grep Name
gastown Node Name vancouver
Partition Name lpar1
PowerHA node name

# /usr/es/sbin/cluster/utlities/get_local_nodename
vancouver
IP labels
# netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
lo0 16896 link#1 5338 0 5345 0 0
lo0 16896 127 localhost 5338 0 5345 0 0
lo0 16896 ::1 5338 0 5345 0 0
tr0 1500 link#2 0.4.ac.49.35.58 76884 0 61951 0 0
tr0 1500 192.168.1 vancouverboot1 76884 0 61951 0 0
tr1 1492 link#3 0.4.ac.48.22.f4 476 0 451 13 0
tr1 1492 192.168.2 vancouverboot2 476 0 451 13 0
tr2 1492 link#4 0.4.ac.4d.37.4e 5667 0 4500 0 0
tr2 1492 195.16.20 db-app-svc 5667 0 4500 0 0
Figure 2-17. Node Name Considerations (1 of 2) QV1203.1
Node name considerations

• Hostname: Each node within an PowerHA cluster has a hostname associated with it
that was assigned when the machine was first configured. For example, a hypothetical
machine might have been given the name gastown for hostname. The hostname value
can be displayed by the hostname command.
Note: The hostname does not have to resolve to an IP label but it is commonly
assigned to an interface via the mktcpip command. Some applications require that the
hostname be resolvable.
• LPAR name: If the node is a logical partition, then it also has an LPAR name. This is the
name known by the HMC, and can be viewed with uname -L (or lparstat -i).
• PowerHA node name: Each node within an PowerHA cluster also has a node name.
By default, the PowerHA node name will be the same as the hostname, but this is not
required.
• IP labels: An IP label is the name that is associated with an IP address using name
resolution (/etc/hosts, DNS, etc.).

V3.1.0.1
Student Notebook
Uempty
Node Name Considerations (2 of 2)
• AIX hostname, LPAR name and PowerHA node name are separate names
– They are often assigned the same value, but they do not need to be
(although it's less confusing if they are the same)
– Except if using DLPAR with PowerHA, then these three names must be
the same
• The AIX hostname is stored in the inet0 object in the ODM
(hostname and uname -n)
• The LPAR name is the name of the logical partition
(lparstat –i | grep Name or uname -L)
• The HACMP node name is the name of the node to PowerHA and is stored
in the HACMPnode object in the ODM
(get_local_nodename)
• The IP label is a name associated with an IP address

– Stored in name resolution (/etc/hosts, DNS, etc.)
– Traditionally, systems had one interface and the IP label associated with
the IP address assigned to that interface usually matched the hostname
– PowerHA nodes typically have many addresses: at least two boot
addresses, at least one service address and may have a persistent
address for each node
Figure 2-18. Node Name Considerations (2 of 2) QV1203.1
Hostname, LPAR name and PowerHA node name

These three names are not required to be the same.
If you use the Standard Configuration path in SMIT to create your cluster, the PowerHA
node name will default to be the same name as the hostname.
However, if DLPAR will be implemented with PowerHA, then the hostname, the LPAR
name and the PowerHA node name must be the same.
IP labels
Each IP address used by an PowerHA cluster almost certainly has an IP label
associated with it. In non-PowerHA systems, it is common for the system’s only IP label
to be the same as the system’s hostname. This is rarely a good naming convention
within a PowerHA cluster as there are just so many IP labels to deal with, and having to
pick which one gets a name that is the same as a node’s hostname is a pointless
exercise.
Student Notebook
Configuration Rules for

Heartbeating (Boot Addresses)
• General
– A PowerHA network consists of one or more subnets
• One subnet mask for all addresses on the same network
• One physical network (same “wire”)
– At least one direct (non-routed) connection to all nodes
– Do not place packet filter equipment between nodes
• Boot IP address rules
– Heartbeating over IP Interfaces
• Within a network, each NIC on a node must be on a different subnet
• There must be at least one subnet in common with all nodes
– Heartbeating over IP Aliases (an alternative)

• Removes subnet restrictions on all addresses
• You specify a base address for the heartbeat subnets
• PowerHA configures a set of heartbeat IP addresses via aliasing
• These subnets do not need to be routable
Figure 2-19. Configuration Rules for Heartbeating (Boot Addresses) QV1203.1
• General:
- Each node must have at least one non-routed connection with all other nodes.
- Do not place intelligent switches, routers, or other network equipment that can filter
or delay packets between cluster nodes.
- All addresses on this network must use the same subnet mask.
• Heartbeating over IP interfaces:
- Each interface on a node must be on a different subnet
If there are multiple interfaces on the same subnet, PowerHA can not select which
interface will be used for heartbeating, so it cannot reliably monitor all interfaces.
- There must be at least one subnet in common with all nodes
• Heartbeating over IP aliases:
- You specify an IP address offset to be used for heartbeating. PowerHA then
configures a set of IP addresses and subnets for heartbeating.
- You must reserve a unique address and subnet range to be used for
heartbeating. These addresses need not be routed in your network.
- Other addresses can be on any subnet. However, there may be some routing
issues if the service address is in the same subnet as the boot addresses.

V3.1.0.1
Student Notebook
Uempty
Boot Address Example
• All addresses have the same subnet mask: 255.255.255.0
• en0 addresses are on the 192.168.10/24 subnet
• en1 addresses are on the 192.168.11/24 subnet
Before starting the application resource group
192.168.11.2 (ODM)
192.168.10.1 (ODM) 192.168.11.1 (ODM) 192.168.10.2 (ODM)
node1 node2
Figure 2-20. Boot Address Example QV1203.1
Examples
The visual shows some boot IP address examples. We’ll see the service IP address
examples later, when we discuss IPAT.
Student Notebook
Boot Address Rule Examples
IP Address IP Address Valid boot addresses?

node1 node2 Assume a subnet mask of 255.255.255.0
192.168.5.1 192.168.5.2 YES
192.168.6.1 192.168.6.2
192.168.5.1 192.168.5.2 Will function, but NOT RECOMMENDED:
NICs are a single point of failure
Requires netmon.cf file or Heartbeat over IP Aliases
192.168.5.1 192.168.6.2 NO, there isn’t a direct non-routed network connection
between the two nodes.
192.168.6.1 192.168.5.3 both node2 interfaces are on same subnet; they
cannot be monitored
192.168.6.1 192.168.6.2 3rd and 4th interfaces on node1 do not have a
192.168.7.1 common subnet with another node; they cannot be
192.168.8.1 monitored
(or exclude 3rd and 4th interfaces from PowerHA
configuration)
Figure 2-21. Boot Address Rule Examples QV1203.1

V3.1.0.1
Student Notebook
Uempty
Persistent IP Addresses
• An IP address associated with a particular node
• Useful for administrative purposes:
– Provides highly available IP address associated with a particular node
– Allows external monitoring tools (for example, Tivoli) and administrative
scripts to reach a particular node
• Configured, via IP aliasing, after node synchronization
• Configured at boot time, even if cluster services are not running
• PowerHA will strive to keep a node's persistent address available on
that node -- never moved to another node
• Maximum of one persistent address per network per node
• Configuration rules:
– IPAT via Aliases: persistent address can not be in any boot subnet
– IPAT via Replacement: persistent address must be in the service
subnet
• Example:
– Boot addresses are in 192.168.10/24 and 192.168.11/24 subnets
– node1 persistent address: 9.47.87.101
– node2 persistent address: 9.47.87.102
Figure 2-22. Persistent IP Addresses QV1203.1
Persistent IP addresses (persistent labels)

Users may optionally configure persistent IP addresses. These are IP aliases that are
configured on a node and kept available as long as at least one communication
interface remains active on the associated network.
Persistent addresses are not moved as part of IPAT from node to node, but are moved
to another NIC on the same node in the event an adapter failure occurs.
The persistent address is configured at boot time, even if cluster services are not
running. However, the failure of the underlying NIC will cause the persistent IP address
to become unavailable, if cluster services are not running.
Persistent addresses are supported on the following network types:
- Ethernet
- Token Ring
- FDDI
- ATM LANE
Student Notebook
Ser
IP Address Takeover (IPAT)

• Each highly available application is likely to require its own IP
address (service IP address)
• This service address is placed in the application's resource
group
– PowerHA is responsible for ensuring that the service IP address is
available on the node currently responsible for the resource group
• IP Address Takeover (IPAT)
Moving the service address to another NIC or to another node
eel ss
Pal d ce
IIP errvviice
abdr
NF
Sy
Fil tem
S
e
s
SSe
ex
ro e
NF
G lu m
S po
up
mo rts
Vo
er
un
ts ion Serv
licat
App
Reso
u
Grou rce
p
Figure 2-23. IP Address Takeover (IPAT) QV1203.1
IP Address Takeover (IPAT)

• Service IP address:
Most highly available applications work best if the application’s IP address never
changes. This capability is provided using a feature called IP Address Takeover. A
service IP address is selected, which is associated with the application. It is placed in
the application’s resource group. PowerHA then ensures that the service IP address is
kept available on whichever node the resource group is currently active.
• Applications that don’t need IPAT:
Although very common, IPAT is optional. An example of an application which might not
require IPAT is a database server for which the client software can be configured to
check multiple IP addresses when it is looking for the server.
• Concurrent resource groups:
IPAT is not supported for resource groups configured for concurrent access as the
application in such a resource group is active on all the nodes which are currently up.
Consequently, clients of a concurrent access resource group must be capable of finding
their server by checking multiple IP addresses.

V3.1.0.1
Student Notebook
Uempty
Two Ways to Implement IPAT
• IPAT via IP Aliases:
– PowerHA adds the service address to the NIC using AIX's IP aliasing
feature. The NIC now has two addresses assigned: boot address and
service address.
ifconfig en0 alias 9.47.87.157
– This is the default
– Requires more subnets than IPAT via IP replacement
– Allows multiple service addresses to use the same NIC, so may
require less hardware that IP replacement
• IPAT via IP Replacement:

– PowerHA replaces the boot address on the NIC with the service
address
ifconfig en0 9.47.87.157
– Requires fewer subnets
– Only one service address per NIC, thus it may require more hardware
if multiple service addresses are needed
– Must be used if Hardware Address Takeover (HWAT) is required
– Must be used for ATM network type
Figure 2-24. Two Ways to Implement IPAT QV1203.1
Two ways to implement IPAT

PowerHA provides two ways to implement IPAT. We will examine the advantages and
disadvantages of each method in the next few pages.
• IPAT via IP aliasing:
AIX has the ability to have multiple IP addresses associated with a single NIC. This
ability, called IP aliasing, allows PowerHA to move service IP addresses between NICs
(or between nodes) without having to either change existing IP addresses on NICs or
worry about whether or not there is already a service IP label on the NIC.
• IPAT via IP replacement:
IPAT via IP replacement involves replacing the IP address currently on a NIC with a
service IP address. It has the limitation of supporting only one service IP label per
adapter which restricts the number of resource groups which can use IPAT and, in
practical terms, the number of service IP labels in a resource group. IPAT via
replacement supports hardware address takeover, which we discuss shortly.
Student Notebook
Configuration Rules: IPAT via IP Aliasing

• Define boot address for each NIC in the AIX ODM
– Each boot address must be in a different logical IP subnet and should be
configured in AIX (mktcpip, smit mktcpip, smit chinet)
– Define these address in the /etc/host file and configure them in PowerHA
topology as communication interfaces
• Define service addresses in /etc/hosts and in PowerHA resources
• Configuration rule:
Service address must not be in the same subnet as any boot
address
• PowerHA will configure the service address when the resource group is
started
192.168.10.1 (ODM) 192.168.11.2 (ODM)

192.168.11.1 (ODM) 192.168.10.2 (ODM)
Figure 2-25. Configuration Rules: IPAT via IP Aliasing QV1203.1
Requirements
• The following networks support IPAT via IP aliasing:
- Ethernet
- Token-ring
- FDDI
- SP switch 1 / SP switch 2
• No service IP labels on the network require hardware address takeover (HWAT)

V3.1.0.1
Student Notebook
Uempty
IPAT via IP Aliasing in Operation
• When the resource group comes up on a node, PowerHA
aliases the service IP label onto one of the node's available
(that is, currently functional) NICs (ODM)
After starting the application resource group
9.47.87.22 (alias)
192.168.11.2 (ODM)
192.168.10.1 (ODM) 192.168.11.1 (ODM) 192.168.10.2 (ODM)
Figure 2-26. IPAT via IP Aliasing in Operation QV1203.1
Operation
PowerHA uses AIX’s IP aliasing capability to alias service IP labels included in resource
groups onto network interfaces on the node which runs the resource group. With
aliasing, the boot IP address (stored in the ODM) is still present.
Note that one advantage of sorts of IPAT via IP aliasing is that the boot IP addresses do
not need to be routable from the client/user systems.
Applications should use the service address

Users should be strongly discouraged from using anything other than approved service
IP addresses when contacting the cluster as the NICs associated with these boot IP
addresses might fail or the application might move to a different node while the boot IP
addresses remain behind on the original node.
Student Notebook
IPAT via IP Aliasing After an Interface Fails

• If the NIC being used for the service address fails, PowerHA
aliases the service address onto one of the node's remaining
available (currently functional) NICs
• The eventual recovery of the failed NIC makes it available again
for future use but the address is not moved back
9.47.87.22 (alias)
192.168.10.1 (ODM) 192.168.11.1 (ODM) 192.168.10.2 (ODM) 192.168.11.2 (ODM)
Figure 2-27. IPAT via IP Aliasing After an Interface Fails QV1203.1
Interface failure
If a NIC fails, PowerHA moves the service IP addresses to another NIC, which is still
available, on the same network. If there are no remaining available NICs on the node
for the network, then PowerHA initiates a fallover for that resource group. Note:
PowerHA will choose among NICs that have been defined to PowerHA Topology.
User perspective
Since existing TCP/IP sessions generally recover cleanly from this sort of
failure/move-IP-address operation, users might not even notice the outage if they don’t
happen to be interacting with the application at the time of the failure.
Failure of all NICs on a node

If the last remaining NIC on a node fails, then PowerHA triggers a fallover for any
resource groups with service IP addresses on the failed NIC’s network.

V3.1.0.1
Student Notebook
Uempty
IPAT via IP Aliasing After a Node Fails
• If the node fails, PowerHA moves the resource group to a new
node and aliases the service address onto one of the new
node's currently functional NICs
Clients should use the service address to connect to the application. If they use a boot address, or
even a persistent address, they may not be able to connect after a failure!
(If possible, don't give them the boot addresses!)
9.47.87.22 (alias)
192.168.10.2 (ODM) 192.168.11.2 (ODM)
Figure 2-28. IPAT via IP Aliasing After a Node Fails QV1203.1
Node failure with IPAT

When a node that is running an IPAT-enabled resource group fails, PowerHA moves the
resource group to an alternative node. Since the service IP address is in the resource
group, it moves with the rest of the resources to the new node. The service IP address
is aliased onto an available (currently functional) communication interface on the
takeover node.
Node failure from the users’ perspective

The users experience a short outage and then, from their perspective, the same server
is back up and running.
You probably shouldn’t correct the user when they mention that the server was down for
a few minutes earlier when you happen to know that it is still down and undergoing
repair! Strictly speaking, the service was down for a few minutes and is now up again.
Student Notebook
IPAT via IP Aliasing:

Distribution Preferences
• Network level attribute which controls the placement of
service and persistent addresses onto physical NICs
– Load balancing
– VPN requirements
– Best effort:
If there are insufficient interfaces available to satisfy the
preference, PowerHA allocates service addresses and
persistent addresses to an existing active NIC
• Four choices
– Anti-Collocation (default)
– Collocation
– Collocation with Persistent Label
– Anti-Collocation with Persistent Label
Figure 2-29. IPAT via IP Aliasing: Distribution Preferences QV1203.1
Four possible values for this attribute

smitty hacmp -> Extended Configuration -> Extended Resource Configuration ->
HACMP Extended Resources Configuration -> Configure Resource Distribution
Preferences -> Configure Service IP Labels/Address Distribution Preference
• Anti-Collocation: This is the default. PowerHA distributes all service IP addresses
across all interfaces using a “least loaded” selection process.
• Collocation: All service IP addresses on the same network interface.
• Collocation with Persistent Label: All service IP addresses on the same interface
that is hosting the persistent IP address. This option may be useful in VPN firewall
configurations where only one interface is granted external connectivity and all IP
addresses (persistent and service) must be allocated on the same interface card.
• Anti-Collocation with Persistent Label:
PowerHA distributes all service IP addresses across all active interfaces that are NOT
hosting the persistent IP address. PowerHA will place the service IP address on the
interface that is hosting the persistent label only if no other network interface is
available.

V3.1.0.1
Student Notebook
Uempty
IPAT via IP Aliasing Summary
• Configure each node's NICs with boot addresses (each on a
different subnet)
• Assign service addresses to resource groups as appropriate
– Must be on separate subnet from boot addresses
– There is a total limit of 256 IP addresses known to PowerHA and
64 resource groups. Within those overall limits:
• There is no limit on the number of service addresses in a resource
group
• There is no limit on the number of resource groups with service
addresses
• PowerHA aliases service addresses to NICs based on
resource group rules and available hardware
• Hardware address takeover (HWAT) is not supported when
using IPAT via IP aliasing
• IPAT via IP aliasing requires gratuitous ARP support
Figure 2-30. IPAT via IP Aliasing Summary QV1203.1
Advantages
• It can require fewer physical interfaces than IPAT via Replacement
Probably the most significant advantage to IPAT via IP aliasing is that it supports
multiple service IP addresses on the same NIC and allows a node to easily support
quite a few resource groups. In other words, IPAT allows you to share several service
addresses on one interface.
Disadvantages
IPAT via alias is usually the best choice, but there are cases when it cannot be used:
• IPAT via IP aliasing does not support hardware address takeover (HWAT)
You will rely on gratuitous ARP as the means of resetting the ARP entries on IPAT. If
HWAT is required, you must use IPAT via replacement.
• IPAT via IP aliasing can require a lot of subnets
You must have a subnet for each interface and a subnet for each service IP label.
• IPAT via aliasing not supported for ATM networks
Student Notebook
IPAT via IP Replacement

• AIX boots with a boot address (ODM) on each NIC
• When PowerHA is started, it replaces one boot address with the
service address on the node where the resource group is online
• Only one service address on a NIC at a time
• If the NIC hosting a service address fails, PowerHA will attempt to
replace the boot address of another NIC, in order to maintain the
service address
• Persistent addresses are supported. They are still aliased onto the
NIC.
• Configuration rules:
– Each service address must be in the same subnet as one of the boot
subnets
– All service IP labels must be in the same subnet
– There must be at least as many NICs on each node as there are service
addresses
(If you have 2 NICs per node, you can have at most 2 service addresses)
• Advantages
– Supports hardware address takeover (HWAT)
– Requires fewer subnets
• Disadvantages
– Requires more NICs to support multiple service IP labels
– Less flexible
Figure 2-31. IPAT via IP Replacement QV1203.1
History
In the beginning, IPAT via IP replacement was the only form of IPAT available. IPAT via
IP aliasing became available when AIX could support multiple IP addresses associated
with a single NIC via IP aliasing. Because IPAT via IP aliasing is more flexible and
usually requires less network interface cards, IPAT via IP replacement is no longer the
recommended method.
Many existing cluster implementations still have IPAT via Replacement. When
upgrading to versions of PowerHA that support IPAT via Aliasing, you should consider
converting IPAT via Replacement configurations to Aliasing only if there is another
reason compelling you to do so. Otherwise, leave the IPAT via Replacement
configuration as it is. Any new implementations should strongly consider using IPAT via
Aliasing.
This visual gives a brief overview of IPAT via IP replacement. A detailed discussion can
be found in the Appendix.

V3.1.0.1
Student Notebook
Uempty
Service Address Examples
subnet mask = 255.255.255.0
Boot addresses on Boot addresses on Valid service Valid service addresses for
first node second node addresses for IPAT via IPAT via IP Replacement
IP Aliasing
192.168.5.1 192.168.5.25 192.168.7.1 192.168.5.3 and 192.168.5.97
192.168.6.1 192.168.6.25 9.37.202.213 OR
192.161.22.1 192.168.6.3 and 192.168.6.97
...
192.168.5.1 192.168.5.25 192.165.8.110 192.168.5.3

192.168.6.1 192.168.7.1 9.10.65.213 192.168.5.97
192.161.22.1
...
192.168.5.1 192.168.5.2 192.168.13.110 192.168.5.3, 192.168.5.97 and
192.168.6.1 192.168.6.2 9.37.65.213 192.168.5.13
192.168.7.1 192.168.7.2 192.107.22.1 OR
192.168.8.1 ... 192.168.6.3, 192.168.6.97 and
192.168.9.1 192.168.6.13
OR
192.168.7.3, 192.168.7.97 and
192.168.7.13
Figure 2-32. Service Address Examples QV1203.1
Service IP address rules and examples

The rules for service IP addresses are straight-forward. It comes down to what subnet
the service IP address can be in.
For IPAT via Replacement, all service IP addresses must be in a subnet that is the
same as one of the non-service IP address subnets.
For IPAT via Aliasing, the service IP addresses must be in a subnet that is different
than the non-service IP address subnets.
The table above provides some examples. Notice that for a given set of IP addresses
on NIC cards (AIX ODM), service IP labels which are acceptable for IPAT via IP aliasing
are not acceptable for IPAT via Replacement and vice-versa. Also notice that the IPAT
via Replacement column only contains subnets that are the same as the subnets in the
first two columns, while the IPAT via Aliasing column contains only subnets that are
different than the subnets in the first two columns.
Student Notebook
IP Naming Conventions
• PowerHA clusters tend to have quite a few IP labels and other
names associated with them
• Adopt appropriate labeling and naming conventions
Suggestions:
• Node-resident labels should include the node's name:
For example: node1-if1, node1-if2, node2-boot1, node2-boot2,
node1-if1, node1-if2 …
• Service IP labels that move between nodes should describe the
application rather than the node:
For example: web1-svc, infodb-svc, app1, prod1, …
• Persistent IP labels should include the node name (since they won’t
be moved to another node) and should identify that they are
persistent:
For example: node1-per, node2-per, node1adm, node1admin, …
• Why?
– Conventions avoid mistakes
– Mistakes reduce availability!
Figure 2-33. IP Naming Conventions QV1203.1
Using IP labeling/naming conventions

Again, the purpose of PowerHA is to create a highly available environment for your
applications. A naming convention can make it easier for humans to understand the
configuration. This can reduce mistakes, leading to better availability.
Never underestimate the value of a consistent labeling/naming convention. It can avoid
mistakes which can, in turn, avoid outages.

V3.1.0.1
Student Notebook
Uempty
Name Resolution
• All of the cluster's IP labels must be defined in every cluster
node's /etc/hosts file:
127.0.0.1 loopback localhost 127.0.0.1 loopback localhost

# cluster explorers # cluster explorers
# netmask 255.255.255.0 # netmask 255.255.255.0
# node1 boot addresses # node1 boot addresses

192.168.15.29 node1boot1 192.168.15.29 node1boot1
# node2 boot addresses # node2 boot addresses

# persistent IP labels # persistent IP labels

192.168.5.29 node1-per 192.168.5.29 node1-per
192.168.5.31 node2-per 192.168.5.31 node2-per
# Service IP labels # Service IP labels

192.168.15.92 xweb-svc 192.168.5.92 xweb-svc
192.168.15.70 yweb-svc 192.168.5.70 yweb-svc
# test client node # test client node

192.168.5.11 test 192.168.5.11 test
Figure 2-34. Name Resolution QV1203.1
Name resolution issues

Make sure that the /etc/hosts file on each cluster node contains all of the IP labels used
by the cluster.
If NIS or DNS is in operation, IP label lookup uses a nameserver system. However, if
the nameserver was accessed through an interface that has failed, the request does not
complete, and eventually times out. This may significantly slow down PowerHA event
processing.
To ensure that cluster events complete successfully and quickly, PowerHA disables NIS
or DNS hostname resolution by setting the following AIX environment variable during
IPAT operations: NSORDER = local
As a result, the /etc/hosts file of each cluster node must contain all PowerHA-defined
IP labels for all cluster nodes.
The easiest way to ensure that all of the /etc/hosts files contain all of the required
addresses is to get one /etc/hosts file setup correctly and then copy it to all of the other
nodes (or use the File Collections facility of PowerHA 5.x).
Student Notebook
Other Configurations: Etherchannel
n1boot1 192.168.1.1 n1boot2 192.168.2.1 n2boot1 192.168.1.2 n2boot2 192.168.2.2

en4 en5 en4 en5
sw1
sw2
en0 en1 en2 en3 en0 en1 en2 en3
Shared Storage
Heartbeat on disk
appB 192.168.3.20
appA 192.168.3.10
• IPAT via Aliasing is required for Etherchannel
Figure 2-35. Other Configurations: Etherchannel QV1203.1
Etherchannel history
Etherchannel is a “trunking” technology that allows grouping several Ethernet links.
Traffic is distributed across the links, providing higher performance and redundant
parallel paths. When a link fails, traffic is redirected to the remaining links within the
channel.
Using Etherchannel with PowerHA

- Only supported with IPAT via Alias networks
- If configured correctly, NIC failures go unnoticed by PowerHA, that is, they are
handled by the Etherchannel technology
- Minimum downtime is experienced; NIC failure should not result in Clients needing
to “reconnect”
- Gives us the performance improvement of link aggregation while allowing the
hardware to deal with NIC failures

V3.1.0.1
Student Notebook
Uempty
Other Configurations: Virtual Ethernet
Virtual I/O Server (VIOS1) AIX Client LPAR 1 Virtual I/O Server (VIOS2)
ent3 ent4 ent4 ent3

en0 (SEA) (LA)
(LA) (SEA) Control Control
Channel Channel
Frame1 ent1 ent0 ent2 ent5 ent0 ent5 ent2 ent1 ent0
(phy) (phy) (virt) (virt) (virt) (virt) (virt) (phy) (phy)
Hypervisor
Ethernet Switch Ethernet Switch
Hypervisor
ent1 ent0 ent2 ent5 ent0 ent5 ent2 ent1 ent0

Frame2 Control
Control
Channel
Channel
ent3 ent4 en0 ent4 ent3
(LA) (SEA) (SEA) (LA)
Virtual I/O Server (VIOS1) AIX Client LPAR 2 Virtual I/O Server (VIOS2)
• IPAT via Aliasing is required for virtual Ethernet adapters

• netmon.cf should be configured
Figure 2-36. Other Configurations: Virtual Ethernet QV1203.1
Where to get more information

To get more information on this configuration, please consult the following:
Redbooks:
Implementing HACMP Cookbook
http://publib-b.boulder.ibm.com/abstracts/sg246769.html?Open
Advanced POWER Virtualization on IBM System p5
http://www.redbooks.ibm.com/abstracts/sg247940.html?Open
Other education:
Q1378 (AU78), Advanced POWER Virtualization Implementation and Best Practices
Student Notebook
PowerHA View of Virtual Ethernet

net_ether_0
9.19.51.20 (service IP) (service IP) 9.19.51.21

9.19.51.10 (persistent IP) (persistent IP) 9.19.51.11
192.168.100.1 192.168.100.2
( base address) Topsvcs heartbeating ( base address)
en0 serial_net1 en0
PowerHA Node 1 PowerHA Node 2

FRAME 1 FRAME 2
Hypervisor
ent1 ent0 ent2 ent5 ent0 ent5 ent2 ent1 ent0

FRAME X
Control
Control
Channel
Channel
ent3 ent4 en0 ent4 ent3
(LA) (SEA) (SEA) (LA)
Virtual I/O Server (VIOS1) AIX Client LPAR Virtual I/O Server (VIOS2)
Figure 2-37. PowerHA View of Virtual Ethernet QV1203.1
Additional information
Single adapter Ethernet networks in PowerHA require the use of a netmon.cf file. This
will be discussed in a few minutes.
Note that there does not have to be link aggregation at the VIO Server level. You could
use a single NIC in each VIO server and rely on the other VIO server for redundancy (in
a Shared Ethernet Adapter Failover configuration, for example).

V3.1.0.1
Student Notebook
Uempty
Other Configurations: Single Adapter Nodes
• Single IP Adapter nodes may appear to reduce cluster cost
• The cost reduction is an illusion:
1. A node with only a single adapter on a network is a node with a single
point of failure - the single adapter
2. Clusters with unnecessary single points of failure tend to suffer more
outages
3. Unnecessary outages cost (potentially quite serious) money
• PowerHA needs at least two NICs per network for failure diagnosis
• Clusters with fewer than two NICs per IP network, though supported,
are not recommended
– If you have a single adapter network, PowerHA will issue warnings when
cluster verification is run
• Single adapter networks are considered OK if redundancy is provided
via another mechanism, such as:
– Virtual Ethernet, backed with two VIOS LPARs in failover configuration
– Etherchannel
• If you have a single adapter network with redundancy:
– Configure /usr/es/sbin/cluster/netmon.cf file
Figure 2-38. Other Configurations: Single Adapter Nodes QV1203.1
Single IP adapter nodes

It is not unusual for a customer to try to implement an PowerHA cluster in which one or
more of the cluster nodes have only a single network adapter (the motivation is usually
the cost of the adapter but the additional cost of a backup system with enough PCI slots
for the second adapter can also be the issue).
The situation is actually, quite simple: with the exception of implementations where
redundancy is provided via other means, any cluster with only one NIC on a node for a
given network has a single point of failure, the solitary NIC, and is not supported.
Nodes with only a single NIC on an IP network are, at best, a false economy. At worst,
they are a fiasco waiting to happen as the lack of a second NIC on one or more of the
nodes could lead to extended cluster outages and just generally strange behavior
(including PowerHA failing to detect failures which would have been detected had all
nodes had at least two NICs per IP network).
Student Notebook
Netmon and netmon.cf

• When heartbeat fails, RSCT tries to determine adapter failure
versus network failure
– RSCT calls netmon (/usr/sbin/rsct/bin/netmonAdapterHealth)
– Netmon checks the adapter status and configuration in AIX and tries to
stimulate traffic over the adapter
– One method for stimulating traffic is to send pings to the addresses
defined in /usr/es/sbin/cluster/netmon.cf
• netmon.cf helps avoid false adapter death, especially in single
adapter networks
• If you have a single adapter network, verification will issue
warnings unless you have addresses defined in netmon.cf
– Single adapter networks are still not recommended unless redundancy is
provided by some other means
• APAR IZ01331 introduces changes to netmon.cf that allow you to
associate specific external addresses with specific interfaces. This
addresses problems that can occur in a virtual Ethernet adapter
environment.
Figure 2-39. Netmon and netmon.cf QV1203.1
• APAR IZ01331: New options for netmon.cf for VIO environment

HACMP customers using VIO within their clusters have been experiencing problems
with specific scenarios where an entire CEC is unplugged from the network, but the
HACMP node within does not detect a local adapter down event, because traffic being
passed between the VIO clients looks like normal external traffic from the perspective of
the LPAR's OS.
The netmon.cf file can be configured to help avoid false adapter death events,
especially in single adapter networks. RSCT will attempt to ping the addresses in the
file in an attempt to differentiate between a local adapter failure and a network failure. In
prior releases, all addresses were used, regardless of the adapter which was
considered to have failed or which address responded to the ping request.
This change allows customers to declare that a given adapter should only be
considered up if it can ping a set of specified targets.
If you are using LPARs with virtual Ethernet adapters, see the descriptions for APAR
IZ01331 and APAR IZ44269 in Fix Central.

V3.1.0.1
Student Notebook
Uempty
Talk to Your Network Administrator
•Explain how PowerHA uses networks
•Ask for what you need:
–IPAT via IP aliasing:
•Service IP address(es) (for client connections)
•One boot IP address for each NIC (for heartbeating)
–One subnet per NIC on each node
(for example: three NICs per node requires three subnets)
–Must not be in any service subnet
–These addresses do not need to be routable
–IPAT via IP replacement:
•Service IP address(es), must be in one subnet (for client connections)
•One boot address for each NIC (for heartbeating)
–One subnet per NIC on each node
(for example: three NICs per node requires three subnets)
–One boot subnet must be the same as the service address subnet
•Service address subnet must be routable, others need not be routable
–Persistent IP address for each node (for admin access, optional)
•Cannot be on any boot subnet
•For IPAT via Replacement: must be in the service subnet
•Ask early (getting subnets assigned may take some time)
Figure 2-40. Talk to Your Network Administrator QV1203.1
• Getting IP addresses and subnets:

Unless you happen to be the network administrator (in which case, you may feel free to
spend time talking to yourself), you need to get the network administrator to provide you
with IP addresses for your cluster. The requirements imposed by PowerHA on IP
addresses are rather unusual and might surprise your network administrator, so be
prepared to explain both what you want and why you want it. Also, ask for what you
want well in advance of the date that you need it as it may take some time for the
network administrator to find addresses and/or subnets for you that meet your needs.
• What if you can’t get all the subnets you need?
It may be hard to get all the subnets you need. One thing that can make it easier (and
you should definitely explain this to your network administrator) is that only the
subnet(s) used by the service address(es) and persistent address(es) need to be
routable. Private addresses can be used for the boot addresses. They are only used for
heartbeating between the nodes and do not need to be routed at all.
Do not accept IP addresses which do not meet the PowerHA configuration rules. Even if
you can get them to appear to work, they almost certainly will not work at a point in time
when you can least afford a problem.
Student Notebook
Changes to AIX Boot, PowerHA v5.x

• The startup sequence of AIX networking is changed when IPAT
is enabled
– HACMP v5.1 introduced changes to the /etc/inittab and
/etc/rc.net files
•Changes when NICs are configured and when network daemons
are started during the boot sequence
– In HACMP v5.3, additional changes were made to inittab to
start the cluster daemons even when cluster services are not
running
– In HACMP v5.5, some changes to inittab and rc.net were
removed
• In the next few slides we will look at the changes for

– PowerHA v5.3, v5.4 and v5.4.1
– PowerHA v5.5
Figure 2-41. Changes to AIX Boot, PowerHA v5.x QV1203.1
AIX boot sequence changes

A node with a network configured for IPAT via IP replacement must not start inetd until
PowerHA has had a chance to assign the appropriate IP addresses to the node’s NICs.
Consequently, the AIX start sequence is modified slightly if a node has a resource
group which uses either form of IPAT.

V3.1.0.1
Student Notebook
Uempty
Changes to AIX Boot: 5.3, 5.4 and 5.4.1
• The startup sequence of AIX networking is changed when IPAT
/etc/inittab
is enabled
/etc/inittab /sbin/rc.boot
cfgmgr
/sbin/rc.boot /etc/rc.net
cfgmgr modified for IPAT: exit 0
/etc/rc.net /etc/rc
config NICs, mount FSs
hostname, static /usr/es/sbin/cluster/etc/harc.net
routes config NICs, config persistent
addresses, rc.net -boot,
/etc/rc start selected daemons
mount FSs /etc/rc.tcpip
changed to run level a, not run
/etc/rc.tcpip /etc/rc.nfs
start network
changed to run level a, not run
daemons
/etc/rc.nfs Start cluster services

start NFS When service addresses are configured
daemons (IPAT)
telinit -a
run level a scripts are run:
/etc/rc.tcpip
/etc/rc.nfs
Figure 2-42. Changes AIX Boot: 5.3, 5.4 and 5.4.1 QV1203.1
Changes for PowerHA 5.3, 5.4 and 5.4.1

The visual summarizes the changes. We will take a detailed look at the inittab file in the
next visual.
Student Notebook
Changes to /etc/inittab: 5.3, 5.4 and 5.4.1

init:2:initdefault:
brc::sysinit:/sbin/rc.boot 3 >/dev/console 2>&1 # Phase 3 of system boot
[ . . . ] rc.boot calls rc.net to config network devices
Added for IPAT: rc.net modified for IPAT:
harc.net configs network devices (calls rc.net -boot), rc.net exits immediately unless called with -boot
configs persistent addresses and starts selected daemons
srcmstr:23456789:respawn:/usr/sbin/srcmstr # System Resource Controller
harc:2:wait:/usr/es/sbin/cluster/etc/harc.net # HACMP for AIX network
startup Modified for IPAT:
rctcpip:a:wait:/etc/rc.tcpip > /dev/console 2>&1 Scripts
# Startthat startTCP/IP daemons
network daemons
rcnfs:a:wait:/etc/rc.nfs > /dev/console 2>&1 # Start now in runNFSlevelDaemons
a, not run at boot time
[ . . . ]
qdaemon:a:wait:/usr/bin/startsrc -sqdaemon Added when HACMP installed:
writesrv:a:wait:/usr/bin/startsrc -swritesrv rc.init starts clcomd & clstrmgrES at boot,
even if cluster services are not started
[ . . . ]
hacmp:2:once:/usr/es/sbin/cluster/etc/rc.init >/dev/console 2>&1
ctrmc:2:once:/usr/bin/startsrc -s ctrmc > /dev/console 2>&1
clinit:a:wait:/bin/touch /usr/es/sbin/cluster/.telinit # HACMP for AIX
These must be the last entries of run level a in inittab!
pst_clinit:a:wait:/bin/echo Created /usr/es/sbin/cluster/.telinit >
/dev/console # HACMP for AIX These must be the last entries of run level
a in inittab!
Modified for IPAT:
Added if Start cluster services on restart is configured: clinit entries flag HACMP that telinit -a is done
rc.cluster starts cluster services
hacmp6000:2:wait:/usr/es/sbin/cluster/etc/rc.cluster -boot -i # Bring up
Cluster
Figure 2-43. Changes to /etc/inittab: 5.3, 5.4 and 5.4.1 QV1203.1
/etc/inittab for PowerHA 5.3, 5.4 and 5.4.1

PowerHA 5.1 added the harc entry to the /etc/inittab file, which runs harc.net to
configure the network interfaces. Also, starting in PowerHA 5.1, some of the other
inittab entries have been changed to run in run-level a. These are invoked by PowerHA
when it is time for the TCP/IP daemons to run. The final two lines use the touch
command to create a marker file when all of the run-level a items have been run. The
marker file indicates to PowerHA that all of the run-level a items have completed.
PowerHA 5.3 made some additional changes to the inittab file. In PowerHA 5.3 and
later, the PowerHA daemons are running all the time, even before you start the cluster.
These daemons are started by the hacmp entry in the inittab file.

V3.1.0.1
Student Notebook
Uempty
Changes to AIX Boot: 5.5
•telinit -a no longer needed for IPAT
/etc/inittab /etc/inittab
/sbin/rc.boot /sbin/rc.boot
cfgmgr cfgmgr
/etc/rc.net /etc/rc.net
config NICs, not modified
hostname, static /etc/rc
routes mount FSs
/usr/es/sbin/cluster/etc/harc.net
/etc/rc config persistent addresses,
mount FSs cleanup NFS, start selected
daemons
/etc/rc.tcpip /etc/rc.tcpip
start network not modified
daemons /etc/rc.nfs
not modified
/etc/rc.nfs
start NFS
daemons
Figure 2-44. Changes to AIX Boot: 5.5 QV1203.1
Changes made for PowerHA 5.5

In PowerHA 5.5, the requirement for using telinit -a during IPAT operations was
removed. (This was accomplished by changes to AIX.)
Student Notebook
Changes to /etc/inittab: 5.5

init:2:initdefault:
brc::sysinit:/sbin/rc.boot 3 >/dev/console 2>&1 # Phase 3 of system boot
[ . . . ] rc.boot calls rc.net to config network devices
Added for IPAT:
harc.net configs persistent addresses, cleans up NFS and
starts selected daemons
srcmstr:23456789:respawn:/usr/sbin/srcmstr # System Resource Controller

harc:2:wait:/usr/es/sbin/cluster/etc/harc.net # HACMP for AIX network
startup Not modified
rctcpip:2:wait:/etc/rc.tcpip > /dev/console 2>&1 Scripts
# Startthat start network daemons are run
at boot time
TCP/IP daemons
rcnfs:2:wait:/etc/rc.nfs > /dev/console 2>&1 # Start NFS Daemons
[ . . . ]
qdaemon:2:wait:/usr/bin/startsrc -sqdaemon Added when HACMP installed:
writesrv:2:wait:/usr/bin/startsrc -swritesrv rc.init starts clcomd & clstrmgrES at boot,
even if cluster services are not started
[ . . . ]
hacmp:2:once:/usr/es/sbin/cluster/etc/rc.init >/dev/console 2>&1
ctrmc:2:once:/usr/bin/startsrc -s ctrmc > /dev/console 2>&1
[ . . . ]
Added if Start cluster services on restart is configured:
rc.cluster starts cluster services
hacmp6000:2:wait:/usr/es/sbin/cluster/etc/rc.cluster -boot -i # Bring up
Cluster
Figure 2-45. Changes to /etc/inittab: 5.5 QV1203.1

V3.1.0.1
Student Notebook
Uempty
Common TCP/IP Configuration Problems
• Subnet masks are not consistent for all HA network adapters
• Boot IP addresses on one node are placed on the same subnet
• Service and boot IP addresses are placed in the same subnet in
IPAT via IP aliasing networks
• Service and boot IP addresses are placed in different subnets in
IPAT via IP replacement networks
• Ethernet frame type is set to 802.3. This includes Etherchannel.
• Ethernet speed is not set uniformly or is set to autodetect
• The contents of /etc/hosts are different on the cluster nodes
• A different version of perl than is used by the PowerHA
verification tools (resulting in what appears to be a network
communications problem)
Figure 2-46. Common TCP/IP Configuration Problems QV1203.1
Configuration problems
The visual shows some common IP configuration errors to watch out for.
Student Notebook
How Are Users Affected by IPAT?

• IP address moves/swaps within a node result in a short outage
– Connection oriented sessions typically recover seamlessly
(TCP layer deals with packet retransmission)
– Connectionless sessions: user (or application) must resend
• Resource group fallovers to a new node result in a longer outage
– Connection oriented services must be reestablished
– Connectionless sessions: user (or application) must resend
• In either case:
– TCP-based services like http and SQL queries experience short
server down outage
– UDP-based services must deal with lost packets
Figure 2-47. How Are Users Affected by IPAT? QV1203.1
• How long is the outage

There are two components which contribute to the duration of the outage:
- How long it takes PowerHA to detect a failure and to diagnose the failure
Usually between 5 to 30 seconds.
- How long it takes PowerHA to recover from the failure
10-15 seconds for IPAT within a node; a few minutes or more for a fallover
• If the problem can be resolved without a fallover
The users may notice a short outage and then are able to continue with what they were
doing. They might not even notice the outage.
• If the problem requires a fallover
Users will likely notice the outage. Existing TCP/IP sessions eventually fail. Also, it
typically takes a few minutes to complete a fallover (much of this time is spent dealing
with taking over volume groups, checking file systems and recovering applications).
It may be that they see a short period of total silence followed by a clean recovery, or
they might have to reconnect to the application. What they experience generally
depends far more on how the client side of the application is designed and implemented
than on anything within the control of the cluster’s administrator.

V3.1.0.1
Student Notebook
Uempty
What About the User's Computer?
• An IPAT operation renders ARP cache entries on client systems
obsolete
• Client systems must (somehow) update their ARP caches
xweb (192.168.5.1) 00:04:ac:62:72:49

xweb (192.168.5.1) 00:04:ac:62:72:49
xweb (192.168.5.1) 00:04:ac:48:22:f4
192.168.5.1 (alias) xweb

xweb 192.168.5.1 (alias) 192.168.10.1 (ODM) 192.168.11.1 (ODM)
192.168.10.1 (ODM) 192.168.11.1 (ODM)
00:04:ac:62:72:49 00:04:ac:48:22:f4
00:04:ac:62:72:49 00:04:ac:48:22:f4
Figure 2-48. What About the User's Computer? QV1203.1
ARP cache issues

Client systems which are located on the same physical network as the cluster may find
that their ARP cache entries are obsolete after an IP address moves to another NIC (on
the same node or on a different node).
The ARP cache is a table of IP addresses and the associated network hardware
address (MAC address) of the physical network adapter. When an IP address moves to
a different physical network card, the client’s ARP cache might still have the old MAC
address. It could take the client system up to 20 minutes to realize that its ARP cache is
out-of-date and ask for an updated MAC address for the server’s IP address.
Student Notebook
Gratuitous ARP
• AIX supports a feature called gratuitous ARP
– AIX sends out a gratuitous (that is, unrequested) ARP update
whenever an IP address is set or changed on a NIC
• Other systems on the local physical network are expected to
update their ARP caches when they receive the gratuitous ARP
packet
• Remember: only systems on the cluster's local physical network
must respect the gratuitous ARP packet
• So ARP update problems have been minimized
• Required if using IPAT via Aliasing
Figure 2-49. Gratuitous ARP QV1203.1
Gratuitous ARP
Whenever an IP address associated with a NIC changes, AIX broadcasts out a
gratuitous (in other words, unsolicited) ARP update. This gratuitous ARP packet is
generally received and used by all systems on the cluster’s local physical network to
update their ARP cache entries.
Gratuitous ARP issues

Not all network technologies support gratuitous ARP. For example ATM does not
support it. In addition, operating systems which implement TCP/IP are not required to
respect gratuitous ARP packets (although practically all modern operating systems do).
Finally, support issues aside, an extremely overloaded network or a network which is
suffering intermittent failures might result in gratuitous ARP packets being lost. (If the
network is very overloaded, the cluster administrator probably has more significant
problems than the ARP cache issue.)

V3.1.0.1
Student Notebook
Uempty
What If Gratuitous ARP Is Not Supported?
• Some types of physical networks do not support gratuitous ARP
(for example: ATM)
• Some systems may ignore gratuitous APR broadcasts
(although this is rare)
• If gratuitous ARP is not supported then there are two options:

– clinfo
• can be used on the client to receive updates of changes and trigger ARP
cache updates
• can be used on the nodes to ping a list of clients or routers, forcing ARP
cache updates
– Hardware Address Takeover (HWAT)
• Suggestion:
Do not get involved with using either clinfo or HWAT to deal
with ARP cache issues until you've verified that there actually
are ARP issues
Figure 2-50. What if Gratuitous ARP Is Not Supported? QV1203.1
PowerHA supports three alternatives to gratuitous ARP

• clinfo on the client: In this case, clinfo automatically flushes ARP cache. The supplied
clinfo binary module is AIX only but it is possible to get the source code to compile on
other systems. clinfo uses snmpd to determine that a change was made.
• clinfo on all the nodes: In this case clinfo will ping clients which causes an ARP
cache update. For this purpose there is a “ping” file -- either the
/usr/es/sbin/cluster/etc/clinfo.rc script or the /etc/cluster/ping_client file.
• Hardware Address Takeover: Discussed in the next visual.
Don’t add unnecessary complexity

It should be quickly apparent if you have ARP problems (either because you are using
ATM or during cluster testing) . Don’t introduce unnecessary complexity if you don’t
have a problem.
Student Notebook
Hardware Address Takeover

• PowerHA can be configured to swap a service IP label's hardware
address between network adapters
• HWAT is incompatible with IPAT via IP aliasing because each service IP
address must have its own hardware address and a NIC can support only
one hardware address at any given time
• Cluster implementer designates a Locally Administered Address (LAA)
which PowerHA assigns to the NIC which has the service IP label
9.47.87.22 (service IP address) 192.168.11.1 (ODM)

02:05:dc:65:77:47 (LAA)
192.168.10.2 (ODM) 192.168.11.2 (ODM)
Clients use service address (9.47.87.22)

Clients' ARP cache uses LAA
(02:05:dc:65:77:47)
9.47.87.22 (service IP address)

192.168.10.2 (ODM) 02:05:dc:65:77:47 (LAA)
Figure 2-51. Hardware Address Takeover QV1203.1
Hardware address takeover (HWAT)

Hardware Address Takeover (HWAT) is the most robust method of dealing with the ARP
cache issue as it ensures that the hardware address associated with the service IP
address does not change (which avoids the whole issue of whether the client system’s
ARP cache is out-of-date).
The essence of HWAT is that the cluster administrator designates a hardware address
which is to be associated with a particular service IP address. PowerHA then ensures
that whichever NIC the service IP address is on also has the designated hardware
address.
HWAT is discussed in detail in Appendix C.

V3.1.0.1
Student Notebook
Uempty
Configuration Rules: Non-IP Network
• Non-IP networks are strongly recommended in order to provide an
alternate communication path between cluster nodes in the event of an
IP network failure or IP subsystem failure
• You must provide a non-IP communication path, possibly via
intermediate nodes, between every pair of nodes in the cluster
• With more than three nodes you can configure the non-IP network
topology using one of the following layouts:
– Mesh: Each node is connected to all other nodes. This is the most robust, but,
requires the most hardware.
– Ring or loop (RECOMMENDED): Each node is connected to its two adjacent
neighbors. Each node, has two non-IP connections for heartbeating.
– Star (NOT RECOMMENDED): One node is connected to all other nodes. This is the
least robust; the center node becomes a single point of failure for all the associated
networks.
net_rs232_04
net_rs232_01 net_rs232_02 net_rs232_03

Example: ring non-IP network topology
Figure 2-52. Configuration Rules: Non-IP Network QV1203.1
Non-IP networks
Non-IP networks are point-to-point: each connection between two nodes is considered
a network and a separate non-IP network label for it is created in PowerHA.
For example, the visual shows four RS232 networks, in a ring configuration, connecting
four nodes to provide full cluster non-IP connectivity.
Planning non-IP networks

You can configure heartbeat paths over the following types of networks:
- Serial (RS-232)
- Disk heartbeat (over an enhanced concurrent mode disk)
- Target Mode SSA
- Target Mode SCSI
See the HACMP Installation Guide for requirements for each type on non-IP network.
Student Notebook
IP Version 6 (IPv6)
• IPv6 is not supported prior to PowerHA 5.5
• Limited support for IPv6 in PowerHA 5.5:
– Only for service addresses and persistent addresses
• NICs must have IPv4 boot address used for heartbeating
• IPv6 addresses allowed in netmon.cf
– ether network type only
– Only IPAT via aliases networks
– AIX 6.1D or later
– IPv6 address can not be used for HMC (DLPAR/CuoD)
– IPv6 service address can not be included in any WPAR enabled
resource groups
– No NFS v4 exports using IPv6 service address (NFS limitation)
Figure 2-53. IP Version 6 (IPv6) QV1203.1

V3.1.0.1
Student Notebook
Uempty
IP Version 6 History
• Why IPv6?
– Main reason for redesigning IP was the predicted IPv4 address
exhaustion
– Network Address Translation (NAT) provides relief but has its own
drawbacks
• How fast is IPv6 being adopted?
– A 2008 study by Google indicated that implementation was still less
than one percent of Internet-enabled hosts in any country
• The leaders were Russia (0.76%), France (0.65%), Ukraine (0.64%),
Norway (0.49%), and the United States (0.45%).
• Why IPv6 now for PowerHA?
– In US, the June 2003 “Stenbit memo” signaled the intent for
Department of Defense to transition to IPv6 by 2008.
– Other countries have similar initiatives.
– All IBM products must support or plan to support IPv6
• Realistic outlook is that IPv4 and IPv6 will coexist for decades
Figure 2-54. IPv6 History QV1203.1
Student Notebook
IP Version 6 Brief Overview

• Same basic technology as V4
– More similarities than differences
• Larger (128 bit) addresses and extended address space
– IPv4 address space is theoretically 232 (~4 X 109)
– IPv6 address space is theoretically 2128 (~3.4 X 1038)
• Address notation:
– 16 octets compared to 4 octets for IPv4 – requires changes to internal data
structures
– Written as 8 groups of 4 hex characters
Example: fe80:0:0:0:a00:5aff:fef8:b966
– Colon separator instead of dot notation
• Particular challenge for scripts / smit that have used colon separator
– Notation allows compression by removing 0’s
• fe80:0:0:0:a00:5aff:fef8:b966 can be written as
fe80::a00:5aff:fef8:b966
• Subnet notation
– IPv6 uses prefix notation to indicate the network portion of the address
Example: fe80:0:0:0:a00:5aff:fef8:b966/64
indicates that the left most 64 bits are the network portion of the address
• Link local address
– The link-local address prefix (fe80::/10) specifies that the address is only
valid in the scope of a given local link
– Analogous to the Autoconfiguration IP addresses (169.254.0.0/16) in IPv4
Figure 2-55. IPv6 Brief Overview QV1203.1

V3.1.0.1
Student Notebook
Uempty
Configuring IPv6
Service and Persistent Addresses
• Configured through the same set of SMIT panels
• New field for “prefix length” – mandatory for IPv6
Add a Service IP Label/Address configurable on Multiple Nodes (extended)
Type or select values in entry fields.

Press Enter AFTER making all desired changes.
[Entry Fields]
* IP Label/Address [] +
Prefix Length [] #
* Network Name [] +
Alternate HW Address to accompany IP Label/Address []
Add a Persistent Node IP Label/Address

[Entry Fields]
* Node Name ppstest1
* Network Name [] +
* Node IP Label/Address [] +
Prefix Length [] #
Figure 2-56. Configuring IPv6 Service and Persistent Addresses QV1203.1
Student Notebook
IPv6 Service Address

• AIX configures boot address (CuAt) - RSCT uses this for
heartbeat
• autoconf6 adds link local address
– Automatically configures IPv6 network interfaces at boot time
– Run by /etc/rc.tcpip
• When RG started, PowerHA adds the IPv6 service address
as an alias on top of the base NIC addresses
• PowerHA moves service address after failure
en1
192.9.201.10 AIX boot address
fe80::c862:65ff:fee7:e2f7 Link local address
2001::1 Service address
Figure 2-57. IPv6 Service Address QV1203.1

V3.1.0.1
Student Notebook
Uempty
IPv6 Persistent Address
• AIX configures boot address (CuAt) - RSCT uses this for
heartbeat
• autoconf6 adds link local address
• PowerHA adds the IPv6 persistent address as an alias
• PowerHA adds the IPv6 service address as an alias
• PowerHA moves service and persistent address after failure
en1 AIX boot time address

192.9.201.10
Link local address
fe80::c862:65ff:fee7:e2f7
ef80::8 Persistent IP
2001::1 Service IP
Figure 2-58. IPv6 Persistent Address QV1203.1
Student Notebook
IPv6 References
• Documentation, reference, standards, or other publications, URLs,
etc. related to IPv6:
– Ipv6 @ Wikipedia:
http://en.wikipedia.org/wiki/Ipv6
– IPv6 Information Page:
http://www.ipv6.org/
– IPv6 Forum:
http://www.ipv6forum.com/
– “Format for Literal IPv6 Addresses in URL’s”:
http://www.ietf.org/rfc/rfc2732.txt
– IPv6 on AIX White Paper:
http://www.ibm.com/servers/aix/whitepapers/aix_ipv6.pdf
Figure 2-59. IPv6 References QV1203.1

V3.1.0.1
Student Notebook
Uempty
Review 1 of 4
1. How does PowerHA use networks (select all which apply)?
a. Provide client systems with highly available access to the cluster's
applications
b. Detect and diagnose failures
c. Provide cluster status
d. Communicate between cluster nodes
e. Monitor network performance
2. Using information from RSCT, PowerHA only directly handles
three types of failures: ______________, ______________,
______________.
3. True or False?
• Heartbeat packets must be acknowledged or a failure is assumed to have
occurred.
4. True or False?
• Clusters should include a non-IP network.
Figure 2-60. Review 1 of 4 QV1203.1
Student Notebook
Review 2 of 4
5. True or False?
• Clusters must always be configured with a private IP network for PowerHA
communication.
6. Which of the following are true statements about communication
interfaces? (select all that apply)
a. Has an IP address assigned to it using the AIX TCP/IP SMIT screens
b. Might have more than one IP address associated with it
c. Sometimes but not always used to communicate with clients
d. Always used to communicate with clients
7. True or False?
• Persistent node IP labels are not supported for IPAT via IP replacement.
8. True or False?
• Each NIC on each physical IP network on each node is required to have an
IP address on a different logical subnet (unless using heartbeat over IP
alias).
9. True or False?
• A single cluster can use both IPAT via IP aliasing and IPAT via IP
replacement.

V3.1.0.1
Student Notebook
Uempty
Review 3 of 4
10.True or False?
• All networking technologies supported by PowerHA support IPAT via IP
aliasing.
11.True or False?
replacement.
12.If node1 has NICs with the IP addresses 192.168.20.1 and
192.168.21.1 and node2 has NICs with the IP addresses
192.168.20.2 and 192.168.21.2, then which of the following are
valid service IP addresses if IPAT via IP aliasing is being used?
(select all that apply)
a.(192.168.20.3 and 192.168.20.4) OR (192.168.21.3 and 192.168.21.4)
b.192.168.20.3 and 192.168.20.4 and 192.168.21.3 and 192.168.21.4
c. 192.168.22.3 and 192.168.22.4
d.192.168.23.3 and 192.168.24.3
Student Notebook
Review 4 of 4
13. If node1 has NICs with the IP addresses 192.168.20.1 and
192.168.20.2 and 192.168.21.2 then which of the following are
valid service IP addresses if IPAT via IP replacement is being
used? (select all that apply)
a. (192.168.20.3 and 192.168.20.4) OR (192.168.21.3 and 192.168.21.4)
b. 192.168.20.3, 192.168.20.4, 192.168.21.3 and 192.168.21.4
c. 192.168.22.3 and 192.168.22.4
d. 192.168.23.3 and 192.168.24.3
14. True or False?
• Clients are required to exit and restart their application after a fallover.
15. True or False?
• All client systems are potentially directly affected by the ARP cache issue.
16. True or False?
• clinfo must not be run both on the cluster nodes and on the client
systems.

V3.1.0.1
Student Notebook
Uempty
What We Are Going to Do in This Class
student hdisk0
rootvg client
en1 en2 en0
laptops Virtual disk - client only
External Network: 9.47.X.X (netmask = 255.255.255.0)

Internal Network: 192.168.X.X (netmask = 255.255.255.0)
en0 en1 en2 Resource Group = appAgroup en1 en2 en0
Node Name = node1 Home node name = node1 Node Name = node2
en1 (boot) = node1-if1
Startup Policy = home node en1 (boot) = node2-if1
192.168.1.1 Fallover Policy = next prio node 192.168.1.2
en2 (boot) = node1-if2 Fallback Policy = never fallback en2 (boot) = node2-if2
192.168.2.1 Service IP = appA-svc 192.168.2.2
Persistent = node1-per 192.168.3.10 Persistent = node1-per
192.168.3.1 192.168.3.2
Application server = appA
en0 (login) = appA start script = /tmp/qv20/startA en0 (login) =
appA stop script = /tmp/qv120/stopA

Non-IP Network: Non-IP Network:
Device = /dev/hdisk1 Device = /dev/hdisk1
non-IP network
hdisk1 hdisk2
hdisk0 hdisk0
rootvg rootvg
Virtual disk - node1 only Virtual disk - node2 only

Virtual disks - shared: node1 & node2
(hdisk1 is for heartbeating; hdisk2 is unused)
Figure 2-64. What We Are Going to Do in This Class QV1203.1
• Goal for this class

- In this class, you will create a simple two-node cluster with one resource group in a
standby configuration
- It differs from the cluster you worked with in the first exercise:
• Only one resource group.
• No shared storage for that resource group, instead the startA script just logs to a
local directory on each node.
Student Notebook
Why Do We Use Three Interfaces per Node?

student hdisk0
rootvg
client 9.47.8x.x3
en1 en2 en0
laptops Virtual disk - client only
External Network: 9.47.8X.X (netmask = 255.255.255.0)

Internal Network: 192.168.X.X (netmask = 255.255.255.0)
en0 en1 en2 Resource Group = appAgroup en1 en2 en0
= node1
Node Name = node1
Home node name = node1 Node Name = node1
= node2
en0 (boot)
en1 = node1-if1
= node1-if1
Startup Policy = home node en1 (boot) = node2-if1
= node2-if1
192.168.1.1
192.168.1.1 Fallover Policy = next prio node 192.168.1.2
192.168.1.2
en1 (boot)
en2 = node1-if2
= node1-if2 Fallback Policy = never fallback en2 (boot) = node2-if2
= node2-if2
192.168.2.1
192.168.2.1 ServiceIP
Service IP = appA-svc 192.168.2.2
192.168.2.2
Persistent = node1-per
= node1-per 192.168.3.10
192.168.3.10 Persistent = node1-per
= node2-per
192.168.3.1
192.168.3.1 192.168.3.2
192.168.3.2
Application server = appA
en0 (login) = 9.47.8x.x1 appA start script = /tmp/qv20/startA en0 (login) = 9.47.8x.x2
appA stop script = /tmp/qv120/stopA

Non-IP Network: Non-IP Network:
Device = /dev/hdisk1 Device = /dev/hdisk1
non-IP network
hdisk1 hdisk2
hdisk0 hdisk0
rootvg rootvg
Virtual disk - node1 only Virtual disk - node2 only

Virtual disks - shared: node1 & node2
(hdisk1 is for heartbeating; hdisk2 is unused)
Figure 2-65. Why Do We Use Three Interfaces per Node? QV1203.1
Most commonly, PowerHA clusters have two NICs per node. Boot addresses often use
private addresses (not routable). Service and persistent addresses are usually routable.
In our class environment, all the addresses used by PowerHA are private addresses.
And we have three interfaces on each node. This is NOT required by PowerHA. Why
did we do it this way?
- We need multiple clusters (one for each team)
- If all the clusters were on the same physical network, we would need different
cluster addresses for each team. This made the lab instructions confusing.
- Instead we isolated each cluster: Each cluster, and one client, are within a p520
system.
- Cluster addresses (boot, service and persistent) are all private addresses and on a
private network. This would be pretty unusual in a production cluster, but is very
useful for our classroom.
- The third NIC (en0) is connected to the external network and is not used by
PowerHA at all. It is only used by us to connect to the LPARs. This “external login”
address is the only routed address in our configuration. This would not be normal for
a production cluster.

V3.1.0.1
Student Notebook
Uempty
What We Are Going to Do in This Exercise
Configuring Networks for PowerHA

Part 1: Remove the existing cluster configuration
•Remove the cluster from both nodes
•Remove the shared volume groups on both nodes (vgA and vgB)
•Unconfigure en1 and en2 on both nodes
Part 2: Configure and test IP networking for PowerHA
(IPAT via Aliases)
•Configure boot addresses on en1 and en2
•Verify /etc/hosts has all the cluster addresses (boot, service and
persistent) and is the same on both nodes
•Verify name resolution for all IP labels in /etc/hosts
Part 3: Configure a disk heartbeat non-IP network
•Verify the shared storage
•Create an Enhanced Concurrent volume group on node1
•Import the volume group on node2
•Verify that heartbeating will work using this volume group
Figure 2-66. What We Are Going to Do in This Exercise QV1203.1
Student Notebook
Unit summary (1 of 2)
• PowerHA uses networks to:
– Provide highly available client access to applications in the cluster
– Detect and diagnose NIC, node, and network failures using RSCT
– Communicate with PowerHA daemons on other nodes
– Provide cluster status using SNMP
• All PowerHA clusters should have a non-IP network
– Differentiate between node, IP subsystem and network failures
– Prevent cluster partitioning
• PowerHA networking terminology
– Service IP label/address: HA address used by client to access application
– Boot (non-service) IP label/address: Applied to NIC at boot time; stored in
AIX ODM
– Persistent IP label/address: Node bound HA address for admin access to a
node
– Communication interface: NIC used in an IP network
– Communication device: Device used in non-IP network
– Communication adapter: X.25 adapter used in a HA communication link
– IP Address Takeover (IPAT): Moves service IP address to working NIC
after a failure
• IPAT via aliasing: Adds the service address to a NIC using IP aliasing
• IPAT via replacement: Replaces the non-service address with the service address
Figure 2-67. Unit Summary (1 of 2) QV1203.1

V3.1.0.1
Student Notebook
Uempty
Unit summary (2 of 2)
• PowerHA has very specific requirements for subnets
– All addresses on one network must use the same subnet mask
– IPAT via aliasing
• NICs on a node must be on different subnets (boot addresses)
• There must be at least one subnet in common with all nodes
• Service addresses must be on different subnet than any boot address
• A service address can be on same subnet with another service address
– IPAT via replacement
• NICs on a node must be on different subnets (boot addresses)
• Each service address must be in same subnet as one of the boot addresses
• Multiple service addresses must be in the same subnet
– Heartbeating over IP alias (any form of IPAT)
• Service and non-service addresses can coexist on the same subnet, or be on
separate subnets
• One subnet per NIC required for heartbeating; these do not need to be routed
• PowerHA can update local clients’ ARP cache after IPAT
– Gratuitous ARP (default)
– clinfo on clients
– clinfo on server nodes
– Hardware address takeover (HWAT),
requires IPAT via IP Replacement
Figure 2-68. Unit summary (2 of 2) QV1203.1
Student Notebook

V3.1.0.1
Student Notebook
Uempty Unit 3. Configuring Applications for PowerHA

This unit describes the considerations for making an application highly
available in a PowerHA environment.

• List and explain the requirements for an application to be
supported in a PowerHA environment
• Describe the PowerHA start and stop scripts
• Describe the resource group behavior policies supported by
PowerHA

• Review quiz
References
PowerHA manuals
© Copyright IBM Corp. 2007, 2009 Unit 3. Configuring Applications for PowerHA 3-1
Student Notebook
Unit Objectives
• List and explain the requirements for an application to be

supported in a PowerHA environment
• Describe the PowerHA start and stop scripts
• Describe the resource group behavior policies supported by
PowerHA

V3.1.0.1
Student Notebook
Uempty
How to Define an Application to PowerHA
• Create start/stop scripts; make executable; copy to all nodes; test on
all nodes
• Create resources in PowerHA
– Application server: start and stop scripts (and optionally monitor(s))
– Service address
• Create Resource Group
– Participating Nodes
– Run time policies: where to run
• Add resources to the group
– Application Server
– Service Address
– Volume Group
• Synchronize
Resource Group
Node 1 Node 2 Node 3
Shared
Disk
Figure 3-2. How to Define an Application to PowerHA QV1203.1
Steps to define an application to PowerHA

• Create and test the start and stop scripts
• Create a PowerHA resource called an application server:
The application server defines a start and a stop script for the application. Optionally,
you can create application monitors as part of the application server.
• Create a PowerHA resource group:
- Define the resource group:
• Define a list of nodes on which the application can run (The default priority is
determined by the order of the list. The first node listed is called the home node.)
• Define run time policies that control where the application runs
- Add resource(s) to the resource group (These are the resources that PowerHA will
move during a fallover.):
• Application server name
• Service address
• Volume group
Student Notebook
Application Considerations 1 of 2
• Automation
– No manual intervention required
• Dependencies
– Using names unique to one node
– Node based licensing
– Dependencies with other applications
• Interference
– Conflicts with PowerHA
Figure 3-3. Application Considerations 1 of 2 QV1203.1
• Automation
One key requirement for an application to function successfully under PowerHA is that
the application must start and stop without any manual intervention. Since the cluster
daemons call the start and stop scripts, there is no option for interaction.
• Dependencies
Consider what dependencies your application has that might prevent it from running in
a cluster environment. For example:
i. Referring to a locally attached device
ii. Hard coding such as /dev/tty0 which may not be the same on another node
iii. Using a hostname which is not the same on other nodes
iv. One application must be up before another one
v. Applications that must both run on the same node
• Interference
An application may execute properly on both the primary and standby nodes. However,
when PowerHA is started, a conflict with the application or environment could arise that
prevents PowerHA from functioning successfully. Two areas to look out for are: Using
IPX/SPX Protocol and manipulating network routes.

V3.1.0.1
Student Notebook
Uempty
Application Considerations 2 of 2
• Robustness
– Application can restart after hardware failure
– Is there cleanup that needs to be done by your scripts?
• Implementation
– Data storage
– Effective scripts
– Using crontab or inittab
• Monitoring using PowerHA
– Startup monitor: Checks that app has started
– Long running monitor: Checks that app is still running
– If not running: attempt to restart on this node; fallover to other
node
– Used to be overlooked
– Nearly mandatory for
•Unmanaged resource groups
•Non-disruptive Startup/Upgrade
• See Applications and HACMP in the Planning Guide
Figure 3-4. Application Considerations 2 of 2 QV1203.1
• Robustness
An application under PowerHA should meet robustness characteristics, such as
successful start after hardware failure, survival of real memory loss, and so forth.
• Implementation
- Where should data be stored, private storage or shared storage?
- Writing effective scripts (We will look at script considerations in a few minutes)
- Using inittab and crontab (inittab is processed before PowerHA is started and
crontab is local to a each node. Time must be synchronized if using crontab.)
• Monitoring using PowerHA
- Startup monitor:
• Checks if application is already running. Prevents multiple instances when
returning from unmanaged state or during non-disruptive startup or upgrade.
• Checks if application has started. If not, restart or fallover to another node.
- Long running monitor:
• Checks periodically that application is running. If not, restart or fallover to another
node.
Student Notebook
Shared Storage Considerations

• Where should data go? • How is access controlled?
– Private Storage Only the node running the application
• Operating system components should be able to access the data
– Shared Storage – Standard volume group:
• Dynamic data used by the
hardware reserve/release
application
– It Depends – Enhanced concurrent volume group:
• Configuration files RSCT
• Application binaries
• License files
Node Node
1 2
LVM LVM
odm odm
pvid
Device
access shared access
pvid
Device
hdisk hdisk
Adapter disks Adapter
VGDA
rootvg
rootvg pvid rootvg
rootvg
private private
Figure 3-5. Shared Storage Considerations QV1203.1
• Where should data go?:

For some data, the answer is clear. For other cases, it depends on your specific needs.
- Private storage must be used for the operating system components
- Shared storage must be used for dynamic data used by the application
- It depends: Configuration files, license files and application binaries
License files deserve a special mention. If using node locked licenses, then you
should use private storage. In any case, you must learn the license requirements of
the application to make a proper determination.
• Access control:
In a serial (non-concurrent) environment, the application runs on only one node at a
time and modification or even access from another node could be catastrophic.
PowerHA and AIX provide two means for access control:
- Standard VGs, access is controlled using hardware based mechanisms
- Enhanced concurrent VGs, access is controlled using RSCT
We will discuss these issues in the QV125 - PowerHA for AIX (HACMP) II:
Administration class.

V3.1.0.1
Student Notebook
Uempty
Writing Start and Stop Scripts 1 of 2
•Start script issues
– Script should perform any preparation needed to start the
application.
– Script must handle any errors or cleanup from previous
termination. Is data recovery needed? You should assume data is
in an unknown state.
– Start script is run in background. If it fails to start the application,
PowerHA is unaware, even if the script returns an error.
Consider an application startup monitor.
– Script should check if application is already running and not start
application if it is already running (unless multiple instances are
desired).
– When starting an application with multiple instances, only start the
instances applicable for each node. Certain database startup
commands read a configuration file and start all known databases
at the same time; this may not be what is desired.
Figure 3-6. Writing Start and Stop Scripts 1 of 2 QV1203.1
Writing start and stop scripts

Start and stop scripts which perform correctly on a single node, may need some
changes when run in a cluster environment. Thorough testing is always recommended.
The following notes expand on some of the considerations mentioned in the visual:
• Application start scripts should perform any preparation needed to start the application.
Manual intervention will not be available during fallover.
• Start scripts should not assume the state of the application or the application’s data.
The start script is used for the initial start of the application and also for restart after an
application crash. Your script should verify and handle any irregular conditions that may
occur.
• The cluster manager does not check the exit status of start scripts. PowerHA does not
know if the application fails to start unless you create an application startup monitor.
• Your start script may need to check if the application is already running. Another way to
handle this may be an application startup monitor.
Student Notebook
Writing Start and Stop Scripts 2 of 2

•Stop script issues
–Script should make sure the application is really stopped.
–Script should exit with RC=0, if application is stopped.
For example, if application was already stopped, stop script should not
return an error.
–Script should not kill an HACMP process.
For example via misuse of the grep command.
•Location of scripts
–Scripts must be in same directory and executable on all nodes in the RG.
•Correct coding
–Scripts must start by declaring a shell (for example, #!/bin/ksh).
–Variables must be explicitly declared (not set via /etc/environment)
–Consider carefully before exiting with RC0.
•Using Assists
–Smart Assists (a priced feature): Provide SMIT based configuration for:
WebSphere, DB2, and Oracle Real Application Server (RAC)
–Plugin filesets (included in PowerHA): Provide sample start/stop scripts for:
DHCP, DNS and print services
•See Applications and HACMP in the Planning Guide
Figure 3-7. Writing Start and Stop Scripts 2of 2 QV1203.1
Writing start and stop scripts - continued

• Application stop scripts must make sure the application is really stopped. It must not exit
until the application is totally stopped as PowerHA will start to unmount filesystems and
release other resources as soon as the stop script terminates. The attempt to release
these resources might fail if there are remnants of the application still running.
• All scripts must start by declaring a shell. If not, the script will not run and no error is
generated in PowerHA. This includes any script that will be run by PowerHA, including:
start/stop scripts, customized event scripts and application monitors.
• You must explicitly declare any variables. Scripts run by PowerHA will not pick up
variables from /etc/environment or root’s .profile file.
• Consider carefully before having any script (to be run by PowerHA) exit with a non-zero
return code. This will cause an event_error: further cluster event processing will be
blocked until this is addressed and manual intervention is required. If it truly is a serious
error, then this may be appropriate, but it is usually better to try to handle the error in the
script or to provide some mechanism for notification.
Note: The return code for start scripts is ignored.

V3.1.0.1
Student Notebook
Uempty
Resource Group Policies
• Three initial policies
– Startup Policy
– Fallover Policy
– Fallback Policy
• Additional run-time options

– Settling time (Startup)
– Delayed Fallback (Fallback)
Figure 3-8. Resource Group Policies QV1203.1
• There are three initial policies

- Startup
When cluster services are started on a node, the cluster manager examines the
Startup policy for each resource group that can run on this node, to determine if it
should activate the resource group and start the application.
- Fallover
If there is a node failure, the Fallover policy determines which node should activate
the resource group and start the application.
- Fallback
If a higher priority node for the resource group is started after a fallover, the Fallback
policy determines if the resource group should be stopped and restarted on the
higher priority node.
• There are two run-time options which can modify the policies
- Settling time affects one of the Startup policies
- Delayed Fallback affects how the Fallback policy works
Student Notebook
Startup Policy
•Online On Home Node Only
– Resource group will only be started on the first node in the nodelist. If
unavailable, the resource group will not be started automatically.
•Online On First Available Node

– Resource group will be started on the first node to join the cluster
– Can be modified using a Settling Time
• Start on home node. Wait for Settling Time if home node is not up. Then start
on highest priority node that is up.
• Online Using Node Distribution Policy

– Start on first available node, but only one resource group can run on a node
• Online on All Available Nodes

– Start resource group on all nodes in the resource group's nodelist
Figure 3-9. Startup Policy QV1203.1
• Online on home node only

When starting cluster services, only start this resource group (RG) on the home node. If
the home node is not available, the RG is not automatically started.
• Online on first available node
When starting cluster services, start the RG on the first node to join the cluster.
- Settling time
Start the RG on the home node, if available. If this is not the home node, wait up to
the duration of the settling time interval to see if the home node joins the cluster. At
the end of the interval, start the RG on the highest priority node available.
• Online using node distribution policy
Start on first available node, but only one RG can be active on a given node. If two or
more resource groups with this startup policy are offline at the time when a node joins,
the node acquires the RG that has the least number of nodes in its nodelist. After
considering the nodelist, PowerHA sorts the list of RGs alphabetically.
• Online on all available nodes
Start the RG on every node that belongs to this RG’s nodelist.

V3.1.0.1
Student Notebook
Uempty
Online on All Available Nodes
• Application runs on all available nodes concurrently
– No fallover/fallback – just less/more nodes running the
application
• Resource group restrictions:

– No JFS or JFS2 filesystems (only raw logical volumes)
– No service IP Labels / Addresses (which means no IPAT)
– Application must provide own lock manager
• Potential to provide essentially zero downtime
Figure 3-10. Online on All Available Nodes QV1203.1
• Online on All Available Nodes

When starting cluster services, start the resource group on this node if it is in the
resource group’s nodelist. In this case, it does not matter if the resource group is
already active on another node and the application ends up being started on all nodes
where cluster services are started. This policy is also referred to as concurrent access.
• Resource group restrictions
There are restrictions when defining a resource group that will use this policy:
- The data can not be part of a jfs or jfs2 type logical volume, it must be defined as
a raw logical volume.
- You cannot include a service address in the resource group definition.
- The application must provide a lock manager to ensure data isn’t being updated
simultaneously from multiple nodes. Oracle Real Application Server (RAC) is an
application that can use this type of startup policy.
• Potential to provide essentially zero downtime
Because the application is running on multiple nodes, the loss of a node does not result
in the loss of the application.
Student Notebook
Fallover Policy
•Fallover To Next Priority Node In The List
– Example: If the nodelist is node2,node1,node3 and node2 fails, PowerHA
will attempt to start the resource group on node1. If node1 is not up, PowerHA
will try node3.
•Fallover Using Dynamic Node Priority
– Fallover node is determined dynamically. You can choose:
•cl_highest_mem_free (choose node with most available memory)
•cl_highest_idle_cpu (choose node with most available CPU cycles)
•cl_lowest_disk_busy (choose node with least disk activity)
– Example:
• Dynamic Node Priority Policy is set to cl_highest_mem_free
• The resource is running on node1 and node1 fails
• PowerHA will chose node2 or node3 based on which node has the most free
memory
– Only useful for clusters with three or more nodes. For two node clusters, Next
Priority Node and Dynamic Node Priority are the same.
•Bring Offline (On Error Node Only)
– Only for Online on All Available Nodes resource groups (concurrent
access)
– The resource group goes offline only on the error node but remains online on
the other nodes
Figure 3-11. Fallover Policy QV1203.1
• Fallover to next priority node in the list

Only for serial access resource groups: Fallover to the next node in the resource
group’s nodelist.
• Fallover using dynamic node priority
Only for serial access resource groups: Fallover to the least busy node. Dynamic node
priority is useful in a cluster that has more that two nodes. You can choose one of the
following three methods to have PowerHA choose the fallover node dynamically:
i. cl_highest_mem_free (choose node with most available memory)
ii. cl_highest_idle_cpu (choose node with most available CPU cycles)
iii. cl_lowest_disk_busy (choose node with least disk activity)
• Bring offline (on error node only)
Only for concurrent access resource groups: Bring the resource group offline on the
node that has the problem, but keep it online on the other nodes. This choice is only
available for concurrent access resource groups.

V3.1.0.1
Student Notebook
Uempty
Fallback Policy
•Fallback To Higher Priority Node In The List

– Resource group will fallback when a higher priority node joins
the cluster
– Can use a Delayed Fallback Timer to fallback at a
specified time
•Never Fallback
– Resource group will NOT fallback when a higher priority node
joins the cluster
Figure 3-12. Fallback Policy QV1203.1
• Fallback to higher priority node

When cluster services start on a node, PowerHA looks to see if there is a resource
group with this node in the nodelist and which is currently active on another node. If this
policy is configured and the joining node is higher in the nodelist, then the resource
group is moved to the higher priority joining node.
- Delayed fallback timer
A runtime fallback timer policy can be set to a time in the future when the fallback
should happen. You can specify a daily, weekly, monthly, or yearly fallback time. Or
you can specify a specific one-time fallback date and time.
Example: If a node in a cluster failed, and was later repaired, you may want to
integrate the node into a cluster during off-peak hours. After starting the node,
PowerHA automatically moves the resource group at the specified time.
• Never fallback
The resource group is NOT moved when a higher priority node joins the cluster. The
cluster administrator can manually move the resource group when desired, or leave it
running on the current node.
Student Notebook
Valid Combinations of Policies
Startup Behavior Fallover Behavior Fallback Behavior

Online on home node – Fallover to next priority node – Never fall back
only (OHNO) – Fallover using Dynamic Node – Fall back to higher
Priority priority node
Online using node – Fallover to next priority node – Never fall back
distribution policy – Fallover using Dynamic Node
Priority
Online on first available – Fallover to next priority node – Never fall back
node (OFAN) – Fallover using Dynamic Node – Fall back to higher
Priority priority node
Online on all available – Bring offline (on error node – Never fall back
nodes only)
Figure 3-13. Valid Combinations of Policies QV1203.1
Valid combinations
PowerHA allows you to configure only valid combinations of startup, fallover, and
fallback behaviors for resource groups.
Preferences are not the only factor in determining node

In addition to the node preferences described by the policy combinations in the visual,
other issues may determine the resource groups that a node acquires.

V3.1.0.1
Student Notebook
Uempty
Dependent Applications/Resource Groups
RG 1
Parent RG
Child RG Child RG
RG 2 RG 3
Parent RG Parent RG
Child RG
RG 4
•Parent/Child Dependency
– Online: Parent will be started before the child. If the parent cannot be started, the child will
not be started
– Offline: Child will be stopped before the parent
– A resource group can have more than one child or more than one parent
– One resource group can be the parent of another resource group
– A child resource group can be a parent of a third resource group
•Location Dependency
– A resource group may only be on the same node/site or on a different node/site than
another resource group
•Implemented as Run-Time Policy
Figure 3-14. Dependent Applications/Resource Groups QV1203.1
• Parent / Child dependencies

In PowerHA 5.2 and higher, it is possible to have cluster wide resource group
online/offline dependencies.
i. Parent will be brought online before child; child cannot be started if parent does
not start
ii. An RG can have multiple parents or multiple children
iii. Parent will be brought offline after child
iv. Parent/child can be on different nodes
v. A child RG can be a parent to a third RG
• Location dependencies
In PowerHA 5.3 and higher it is also possible to specify resource location dependencies
i. Online on same node
ii. Online on different nodes
iii. Online on same site
Student Notebook
Review
1. True or False
Applications are defined to PowerHA in a configuration file that lists
what binary to use.
2. What policies would be the best to use for a 2-node “active-active” cluster
using IPAT to minimize both applications running on the same node?
a. home, next, never
b. first, next, higher
c. distribution, next, never
d. all, error, never
e. home, next, higher
3. Which type of data should not be placed in private data storage?
a. Application log data
b. License file
c. Configuration files
d. Application binaries
4. True or False
Start and stop scripts which are tested on one node can be counted
on to perform correctly under PowerHA. No further testing is needed.
Figure 3-15. Review QV1203.1

V3.1.0.1
Student Notebook
Uempty
Unit Summary
• To define an application to PowerHA, you must:
– Create an application server resource (start and stop scripts)
– Create a resource group (node list, policies, resources)
• Considerations for putting an application under PowerHA control
– Automation
– Dependencies
– Interference
– Robustness
– Implementation details
– Monitoring
– Shared storage requirements
• Considerations for start and stop scripts
– Environment
– Multiple instances
– Script location
– Error handling
– Coding issues
• Resource group policies control how PowerHA manages the
application
– Startup policy (with optional Settling timer)
– Fallover policy
– Fallback policy (with optional Delayed fallback)
• Resource group dependencies
– Parent/child dependencies control order of startup and shutdown
– Location dependencies control whether resource groups are started on same node or
same site
Figure 3-16. Unit Summary QV1203.1
Student Notebook

V3.1.0.1
Student Notebook
Uempty Unit 4. PowerHA installation

This unit describes the installation process for PowerHA 5.5 for AIX.

• State where installation fits in the implementation process.
• Describe how to install PowerHA 5.5
• List the prerequisites for PowerHA 5.5
• List and explain the purpose of the major PowerHA 5.5
components.

Accountability:
• Review quiz
• Machine exercise
References
PowerHA manuals
© Copyright IBM Corp. 2007, 2009 Unit 4. PowerHA installation 4-1

Student Notebook
Unit Objectives
• State where installation fits in the implementation process
• Describe how to install PowerHA 5.5
• List the prerequisites for PowerHA 5.5
• List and explain the purpose of the major PowerHA
components

V3.1.0.1
Student Notebook
Uempty
Steps for Successful Implementation
• Proper planning is critical to a successful implementation
– Special care should be taken when installing PowerHA on a system
which is in production
Step Step Description Comments

1 Plan Use planning worksheets and documentation.
2 Assemble hardware Install adapters, connect shared disk and network.
3 Install AIX Ensure you update to the latest maintenance level.
4 Configure networks Requires detailed planning.
5 Configure shared storage Set up shared volume groups and filesystems.
6 Install HACMP Install on all nodes in the cluster (don't forget to install
latest fixes).
7 Define/discover the cluster Review what you end up with to make sure that it is
topology what you expected.
8 Configure application You will need to write your start and stop scripts.
servers
9 Configure cluster resources Refer to your planning worksheets.
10 Synchronize the cluster Ensure you "actually" do this.
11 Start HACMP Watch the console for messages.
12 Test the cluster Document your tests and results.
Figure 4-2. Steps for Successful Implementation QV1203.1
• Steps to building a cluster

- You should plan and follow a methodical process which includes eventual testing
and documentation of the cluster.
- There is general agreement among the experts that the first step in configuring a
successful cluster is to plan the cluster carefully.
• Considerations
- It is often best to configure the cluster’s resources iteratively. Get basic resource
groups working first and then add the remaining resources gradually, testing as you
go, until the cluster does everything that it is supposed to do.
- If you leave the configuration of the shared storage (step 5 above) until after you’ve
created the topology (step 7) you can use C-SPOC to configure the shared storage.
- If the application is installed, configured and tested prior to installing and configuring
PowerHA then most issues which arise during later cluster testing are probably
PowerHA issues rather than application issues. On the other hand, if PowerHA is
installed and configured first, the applications can be installed into the exact context
that they will ultimately run in. When to install and configure the applications is just
one more point that will have to be resolved during the cluster planning process.

Student Notebook
Where Are We in the Implementation?

9Plan for network, storage, and application resource groups
– Eliminate single points of failure
9Define and configure the AIX environment
– Networks (IP interfaces, /etc/hosts, non-IP networks and
devices)
– Application start and stop scripts
• Install the PowerHA filesets
• Configure the PowerHA environment
– Topology
– Resources:
•Application Server
•Service labels
– Resource group:
•Identify name, nodes, policies
•Resources: Application Server, service label, VG, filesystem
– Synchronize then start cluster services
Figure 4-3. Where Are We in the Implementation? QV1203.1
What we have done so far

In the previous units, we planned and built the network and application environments for
our cluster.
The planning and configuration of PowerHA shared storage are covered in the
PowerHA for AIX (HACMP) II: Administration class.
So we are now ready to install the PowerHA filesets.

V3.1.0.1
Student Notebook
Uempty
First Step in Installation: Documentation
• PowerHA manuals can be obtained from the CD or from:
• HACMP for AIX 5L V5.5 Planning Guide
SC23-4861-11 11/2008
– Contains Planning Worksheets: print and fill in by hand
– Online Planning Worksheets application
• Can be used in planning, then applied to create the cluster
• Can be used to document an existing cluster
• Runs on AIX or Windows (Java 1.3.0 or later)
• Install from CD or from /usr/es/sbin/cluster/worksheets directory
• Instructions in the Planning Guide
• HACMP for AIX 5L V5.5 Installation Guide
SC23-5209-02 11/2008
• Release notes:
– On the CD as README_hacmp5.5
– Installed as /usr/es/sbin/cluster/release_notes
– There may be additional notes, such as:
README5.5.0.UPDATE, release_notes_xd, etc.
Figure 4-4. First Step in Installation: Documentation QV1203.1
• PowerHA manuals:
HACMP for AIX: Concepts and Facilities Guide SC23-4864
HACMP for AIX: Planning Guide SC23-4861
HACMP for AIX: Installation Guide SC23-5209
HACMP for AIX: Administration Guide SC23-4862
HACMP for AIX: Troubleshooting Guide SC23-5177
HACMP for AIX: Programming Client Applications SC23-4865
HACMP for AIX: Master Glossary SC23-4867
• PowerHA/XD manuals:
HACMP/XD GLVM Planning and Administration Guide SA23-1338
HACMP/XD: Metro Mirror Planning and Administration Guide SC23-4863
• PowerHA Smart Assist manuals:
HACMP for AIX: Smart Assist for WebSphere User's Guide SC23-4877
HACMP for AIX: Smart Assist for Oracle SC23-5178
HACMP for AIX: Smart Assist for DB2 SC23-5179
HACMP for AIX: Smart Assist Developer's Guide SC23-5210

Student Notebook
PowerHA Filesets
Here are some of the PowerHA 5.5 filesets:
cluster.adt.es cluster.es.cfs
+ 5.5.0.0 ES Client CLINFO Samples + 5.5.0.0 ES Cluster File System Support
+ 5.5.0.0 ES Client Clstat Samples cluster.es.plugins
+ 5.5.0.0 ES Client Include Files + 5.5.0.0 ES Plugins - Name Server
+ 5.5.0.0 ES Client LIBCL Samples + 5.5.0.0 ES Plugins - Print Server
+ 5.5.0.0 ES Web Based Monitor Demo
cluster.doc.en_US.es + 5.5.0.0 ES Plugins - dhcp
+ 5.5.0.0 HAES PDF Documentation-U.S. English cluster.es.worksheets
+ 5.5.0.0 HAES Web-based HTML Documentation - + 5.5.0.0 Online Planning Worksheets
U.S. English cluster.hativoli
cluster.es.client
+ 5.5.0.0 ES Client Libraries
+ 5.5.0.0 HACMP Tivoli Client
+ 5.5.0.0 ES Client Runtime + 5.5.0.0 HACMP Tivoli Server
+ 5.5.0.0 ES Client Utilities cluster.license
+ 5.5.0.0 Web based Smit + 5.5.0.0 HACMP Electronic License
cluster.es.server cluster.man.en_US.es
+ 5.5.0.0 ES Cluster Test Tool
+ 5.5.0.0 ES Server Diags + 5.5.0.0 ES Man Pages - U.S. English
+ 5.5.0.0 ES Server Events cluster.msg.en_US
+ 5.5.0.0 ES Server Utilities + 5.5.0.0 HACMP CSPOC Messages - U.S.
+ 5.5.0.0 ES Two-Node Configuration Assistant English
+ 5.5.0.0 ES Base Server Runtime [...]
cluster.es.cspoc
+ 5.5.0.0 ES CSPOC Commands
+ 5.5.0.0 ES CSPOC Runtime Commands
•Your requirements will determine what you
+ 5.5.0.0 ES CSPOC dsh install
•Reboot is required after initial installation
Figure 4-5. PowerHA Filesets QV1203.1
• Fileset considerations
- smit install_latest will not show the msg filesets: use install_all and select
the filesets.
- cluster.es contains both client and server components. You can install either or
both. Normally cluster nodes get both and clients get the client filesets.
- The same filesets should be installed on all nodes or Verify will give warnings.
- You should install the documentation filesets on at least one non-cluster node
(ensuring that the PowerHA PDF-based documentation is available even if none of
the cluster nodes will boot could prove REALLY handy someday).
- Some of the filesets require other products such as Tivoli or NetView. You should not
install these filesets unless you have these products. HAView is never installed on
the cluster node, it is installed on the NetView server.
- The cluster.es.cfs fileset can only be used if GPFS is installed.
- You may not need the plug-ins.
• Reboot after install
The installation or upgrade of PowerHA filesets normally requires rebooting the nodes.
Starting in PowerHA 5.4 PTF 1, you can install subsequent updates without rebooting
the system or disrupting applications.

V3.1.0.1
Student Notebook
Uempty
Remember the Prerequisites
• Always check:
– HACMP for AIX: Installation Guide Complete list of prerequisites
– Release notes / README files Any updated prerequisites
• When using AIX Version 5.3:
– AIX V5.3 with Technology Level 9
– RSCT 2.4.10.0
• When using AIX Version 6.1:
– AIX V6.1 with Technology Level 2 Service Pack 1 and APAR IZ31208
– RSCT 2.5.2.0
• AIX filesets
– The Installation Guide lists a number of AIX filesets that are needed for
PowerHA in any case. Some of these are not part of a standard base install
and must be installed before you install PowerHA.
– Other prerequisites are only required for specific needs
• Enhanced concurrent mode Volume Groups:
– bos.rte.lvm (at required TL version)
– bos.clvm.enh (at required TL version)
• Online Planning Worksheets
– AIX 5L Java Runtime Environment
• And so forth
– See the Installation Guide and release notes
Figure 4-6. Remember the Prerequisites QV1203.1
Prerequisites
Listed above are the minimum prerequisites. As time goes by, these will almost
certainly be superseded by later levels.
Always check the Installation Guide for a complete list of prerequisites. Then check the
release notes for any additional prerequisites or requirements.

Student Notebook
Remember to Check for Updates

• http://www.ibm.com/systems/support/
Select System p; select Cluster updates; select PowerHA for AIX X.X
Figure 4-7. Remember to Check for Updates QV1203.1
Check for fix packs

Since you are unlikely to want to upgrade a new cluster anytime soon, it is generally
wisest to start with the latest available AIX and PowerHA patches.
Finally, it’s always a good idea to call IBM support and ask if there are any known issues
with the versions of AIX and PowerHA that you plan to install/upgrade. Indicate that you
intend to install the latest PowerHA PTF or fix pack and ask if it’s known to be stable.
Depending on the timing of your installation, it might be advisable to either stay one
maintenance level behind on AIX and/or PowerHA, or it may be wise to wait for an
imminent maintenance level for AIX and/or PowerHA.

V3.1.0.1
Student Notebook
Bevore
Uempty
Things to Check Before Configuration
• Code installation
– Correct filesets and prerequisites have been installed
– Documentation is installed and accessible
• Network setup
– /etc/hosts file is configured on all nodes correctly
– Name resolution works
– IP and non-IP networks are configured
– Subnets configured correctly
• The subnet mask is identical
• All interfaces are on different subnets
– Routing configured correctly
– Test connectivity
• Shared storage configured correctly
• You have a written plan describing configuration and testing
procedures!
Figure 4-8. Things to Check Before Configuration QV1203.1
Create a checklist of items to check before starting configuration

The visual shows a basic checklist of items which you should verify before starting to
configure an PowerHA cluster. It is not a complete list as each situation is different. You
should develop your own checklist during the cluster planning process and then verify it
just before embarking on the actual configuration of the cluster.

Student Notebook
Install PowerHA Client Machine

• PowerHA client filesets can be installed on non-cluster node systems (AIX)
– Monitor the cluster nodes (using clstat)
– Use clinfoES to update ARP cache
• Set up network
– Route to service address
– Name resolution
• Install prerequisites:
– bos.adt.libm
– bos.adt.syscalls
– bos.data
• Install PowerHA client filesets:
– cluster.adt.es
– cluster.es.client
– cluster.msg.en_US.es
– cluster.man.en_US.es
• Configure /usr/es/sbin/cluster/etc/clhosts
– clhosts on cluster nodes: 127.0.0.1
– clhosts on client: addresses of the cluster nodes
– Can copy /usr/es/sbin/cluster/etc/clhosts.client from cluster node and
rename
• Test connectivity
Figure 4-9. Install PowerHA Client Machine QV1203.1
Client machine properties

A client machine is a node running AIX and only the client filesets from PowerHA. It can
be used to monitor the cluster nodes as well as to test connectivity to an application
during fallover or to be a machine that is used to access a highly available application
Installing and setting up the client machine

Make sure the network is setup so that the client machine can access the cluster nodes.
If the client machine is on the same LAN then choose an address that is in the same
subnet as the service address that you want to access. If it’s on a different LAN, make
sure you configure routing so that it can access the service address.
Also make sure clinfo is setup (clhosts file) to be able to find the cluster node(s). A
clhosts file is generated for clients when you synchronize the cluster. The name of the
file is /usr/es/sbin/cluster/etc/clhosts.client

V3.1.0.1
Student Notebook
Uempty
How Does PowerHA Fit into AIX?
Here are the layers of software on an PowerHA 5.5 cluster node:
Application Layer
Contains the highly available applications that use
PowerHA services
PowerHA Layer
Provides highly available services to applications
RSCT, RMC Layer

Provides monitoring, event management and
coordination of subsystems for PowerHA clusters
AIX Layer
Provides operating system services (SRC, snmpd)
LVM Layer TCP/IP Layer

Manages disk space at Manages communication
the logical level at the logical level
Figure 4-10. How Does PowerHA Fit into AIX? QV1203.1
• The application layer

Any application or service which is being made highly available.
• The PowerHA layer
Provides a number of services to the application layer, including:
- Tracking the state of the cluster in cooperation with the other cluster nodes
- Initiating fallovers and other recovery actions as required
- (Optionally) monitoring the applications and initiating recovery if they fail
- Doing “whatever else it takes” to make the applications highly available
Note that the applications running within the application layer, as a rule, are “blissfully
unaware” of the existence of PowerHA or RSCT.
• The RSCT layer
Monitors the state of the cluster’s topology; coordinates the response to cluster events
and keeps PowerHA informed of cluster status. RSCT is distributed with AIX.
• The AIX and below layers
All of the operating system services provided by AIX to programs running on the
operating system. Of course this includes: the programs at the application layer, the
programs at the PowerHA layer and the programs at the RSCT layer. PowerHA uses
many AIX facilities. The two that are highlighted are LVM and networking.

Student Notebook
PowerHA Components and Features

• The PowerHA software has the following components:
– Cluster Manager
– Cluster Secure Communication Subsystem
– Reliable Scalable Cluster Technology, and Resource
Monitoring and Control (RSCT and RMC)
– SNMP monitoring programs
– Cluster Information Program
– Highly Available NFS Server
– Shared External Disk Access
Figure 4-11. PowerHA Components and Features QV1203.1
PowerHA components and features

The visual shows PowerHA components and features. We will discuss each in the
following visuals.

V3.1.0.1
Student Notebook
Uempty
Cluster Manager
• Runs on each cluster node
– PowerHA 5.3 and later: always running, even if cluster services are not
started
• Implemented by the clstrmgrES subsystem
• Primarily responsible for responding to cluster events:
– Recover from software and hardware failures
– Respond to user-initiated events:
•Request to online/offline a node
•Request to move/online/offline a resource group
•And so forth
• A client to RSCT
• Provides snmp retrievable status information
• Started in /etc/inittab and always running
Figure 4-12. Cluster Manager QV1203.1
The cluster manager’s role

The cluster manager is, in essence, the heart of the PowerHA product. Its primary
responsibility is to respond to events. From this responsibility flows most of the features
and facilities of PowerHA. For example, in order to respond to unexpected events, it is
necessary to know when they occur. It is the job of the RSCT component to monitor for
certain failures.
In PowerHA 5.3 and later the clstrmgrES subsystem is always running.

Student Notebook
Cluster Secure
Communication Subsystem
• Runs on each cluster node
– PowerHA 5.3 and later: always running, even if cluster services not started
• Implemented by the clcomdES subsystem
• Provides communication infrastructure for configuration tools:
– C-SPOC
– Verification and synchronization
– Discovery
– WebSMIT
• PowerHA provides several security options:
– Connection Authentication
• Standard
– Authentication
– Uses /usr/es/sbin/cluster/rhosts file and PowerHA ODM files
• Kerberos (SP only)
– Authentication and privacy (encryption)
• Virtual Private Networks (VPN) using persistent labels
– Authentication and privacy (encryption)
– VPNs are configured within AIX
– PowerHA is then configured to use VPNs
– Message authentication or Message authentication and Encryption
• Provided via RSCT (ctcas subsystem)
• PowerHA provides methods for key distribution
Figure 4-13. Cluster Secure Communication Subsystem QV1203.1
• Cluster communication subsystem

Introduced in PowerHA 5.1, the cluster secure communication subsystem provides
connection level security for the PowerHA configuration tools, eliminating the need for
either /.rhosts files or a Kerberos configuration on each cluster node. The need for
these /.rhosts files had been a source of concern for many customers.
This facility goes beyond eliminating the need for the /.rhosts files. You have options to
make clcomd quite secure.
• Leaving clcomd running
There may be a temptation to stop clcomd (based on concerns about security).
However, clcomd can be made quite secure and bear in mind that clcomd makes more
than just verification and synchronization possible. It also supports the File Collections
and Automatic Verification functions in PowerHA. It is recommended that you leave it
running. The benefit of these services probably outweighs the potential security risk.
• Only supported for PowerHA generated requests
Finally, this subsystem is not supported for use by user commands outside of the
cluster manager and CSPOC.

V3.1.0.1
Student Notebook
Uempty
Cluster Communication Daemon (clcomd)
• Provides secure node-to-node communications without use
of /.rhosts
• Caches coherent copies of other nodes' ODMs
• Long term socket connections on tcp port 6191
• Implements the principle of least privilege:

– Nodes no longer require root access to each other
• Started via /etc/inittab entry
• Managed by the SRC

– startsrc, stopsrc, refresh
Figure 4-14. Cluster Communication Daemon (clcomd) QV1203.1
clcomd basics
This daemon replaces a number of ad hoc communication mechanisms with a single
facility thus funneling all cluster communication through one point. This makes it
feasible to use a VPN to send traffic between nodes.
Efficient node-to-node communications and data gathering

clcomd makes the verification and synchronization processes more efficient by:
- Caching coherent copies of each nodes’ ODM, which reduces the amount of
information which must be transmitted during a verification operation.
- Maintaining long term socket connections between nodes, which avoids the
necessity to create and destroy many short term sessions as was needed when
using rsh and other similar mechanisms.
Verification and synchronization may still take a few minutes to complete as comparison
processing and resource manipulation may be occurring.

Student Notebook
clcomd:
Standard Connection Authentication
•clcomd's authentication algorithm uses:
– Special rhosts file: /usr/es/sbin/cluster/etc/rhosts
– PowerHA ODM
• Upon receiving a connection request:
– If the rhosts file is missing: connection is rejected
– If the rhosts file exists, then check the PowerHA ODM
• If HACMPadapter and HACMPnode are populated:
– Does requestor match an entry in the ODM files?
»No: Request is rejected
»Yes: Connect back and ask for the node name
If node name matches the ODM entry: request is accepted
• If HACMPadapter and HACMPnode are empty: (new cluster)
– If the cluster rhosts file is empty: Allow any connection request
– If the rhosts file is populated: Allow requests only from addresses in the file
• Potential security hole during installation / initial configuration

– Installed rhosts file is empty - allowing any connection
– When cluster is configured, the rhosts file is populated and HACMPadapter
and HACMPnode are populated
– More secure installations should populate the rhosts file with only the cluster
node IP address(es)
Figure 4-15. clcomd: Standard Connection Authentication QV1203.1
Each connection request is authenticated as follows:

• If the cluster rhosts file is missing:
The connection is refused.
• If the cluster rhosts file exists and HACMPadapter and HACMPnode are populated:
The target clcomd daemon authenticates the in-bound session by checking the
request’s source IP address against a list of addresses in the HACMPadapter and
HACMPnode ODM files. If there is a match, the target daemon connects back and
requests the node name. The connection is only allowed if the node name matches the
ODM entry. In this case, the contents of the cluster rhosts file do not matter (but it must
exist).
• If the cluster rhosts file exists and HACMPadapter and HACMPnode are empty:
Cluster is not yet configured.
- If the cluster rhosts file is empty, allow initial cluster configuration request
- If the cluster rhosts file is populated, allow connection requests only from the
addresses in the file. This closes a potential security hole that exists between
installation and initial configuration of the cluster.

V3.1.0.1
Student Notebook
Uempty
RSCT
• Reliable Scalable Cluster Technology
• Included with AIX
• Provides:
– Scalability to large clusters
– Cluster Failure notification
– Coordination of changes
• Key components used by PowerHA:
– Topology Services
•Heartbeat services and reliable messaging service
– Group Services
•Coordinates/monitors cluster state changes
– RMC: Resource Monitoring and Control
•Provides process monitoring, dynamic node priority variables and
user-defined events
• Cluster Manager is an RSCT (group services) client
Figure 4-16. RSCT QV1203.1
• What is RSCT?
Reliable Scalable Cluster Technology (RSCT) is a set of software components that
together provide a comprehensive clustering environment for AIX and Linux. RSCT is
the infrastructure used by a variety of IBM products to provide clusters with improved
system availability, scalability, and ease of use.
• What RSCT provides to PowerHA:
- Failure detection and diagnosis for topology components (nodes, networks and
network adapters)
- Notification to the cluster manager of events which it has expressed an interest in -
primarily events related to the failure and recover of topology components
- Coordination of the recovery actions involved in dealing with the failure and recovery
of topology components (in other words, fallovers, fallbacks and dealing with
individual NIC failures by moving or swapping IP addresses)
- Monitoring of many different AIX components to provide:
• Application monitoring
• CPU, memory and disk I/O values for fallover with dynamic node priority
• User Defined Events

Student Notebook
PowerHA From an RSCT Perspective

RMC Monitors • Configuration tools CLUSTER
• C-SPOC COMMUNICATIONS
• Verification DAEMON To/from other nodes
Host • Synchronization clcomdES
Monitor • Discovery
(memory • WebSMIT
& CPU)
Disk I/O
Monitor CLUSTER MANAGER
RSCT ~
File RMC clstrmgrES Recovery
System Programs
Monitor
Network ctrmc
Device
Monitor
CPU
Monitor
CPU
Monitor
Recovery
Commands
RSCT RSCT
Group ~
Topology
Services Services PowerHA Event
topsvcs grpsvcs Scripts
Group Membership
heartbeats messages Event Subscription
Voting Protocols between nodes
To/from other nodes

Figure 4-17. PowerHA From an RSCT Perspective QV1203.1
• Topology Services
Builds heartbeat rings to detect, diagnose and report state changes to Group Services.
Topology Services also transmits RSCT-related messages between cluster nodes.
• Group Services
Group Services reports failures to the cluster manager as it becomes aware of them
from Topology Services. The cluster manager then drives cluster-wide coordinated
responses to the failure through the use of the Group Services voting protocols.
• RMC Monitors
These monitors report state changes related to monitored entities to the RMC Manager.
• RMC Manager
Receives notification from the monitors. Notifies RSCT clients of those events in which
they have expressed an interest. The PowerHA cluster manager is an RSCT client and
registers with both the RMC manager and Group Services.
• Cluster manager
The cluster manager responds to events through the use of PowerHA’s recovery
programs and event scripts. Running the scripts across the nodes in the cluster
coordinated via Group Services.

V3.1.0.1
Student Notebook
Uempty
Heartbeat Rings
en0 on each node en1 on each node
25.8.60.6 25.8.60.5 25.8.65.6 25.8.65.5
25.8.60.2 25.8.60.4 25.8.65.2 25.8.65.4
25.8.60.3 25.8.65.3
• Heartbeat is one way in order of high to low IP address
Figure 4-18. Heartbeat Rings QV1203.1
• RSCT Topology Services functions

Topology Services is responsible for the detection and diagnosis of topology component
failures. Heartbeat packets are sent between interfaces. Rather than send heartbeat
packets between all combinations of interfaces, Topology Services sorts the IP
addresses of the interfaces on a given logical IP subnet and then arranges to send
heartbeats in a ring from high to low IP addresses in the sorted list. For non-IP networks
(like RS-232 or heartbeat on disk), addresses are assigned to the devices that form the
endpoints of the network and are used by Topology Services like IP addresses for
routing/monitoring the heartbeat packets.
• Example
In the example, each node has two interfaces:
- The en0 interfaces on each node are in the 25.8.60 subnet and Topology Services
sorts the addresses from high to low and sends heartbeats in a ring:
25.8.60.6 --> 25.8.60.5-->25.8.60.4-->25.8.60.3-->25.8.60.2-->25.8.60.6
- The en1 interfaces on each node are in the 25.8.65 subnet and Topology Services
sorts the addresses and sends heartbeats in a ring:
25.8.65.6 --> 25.8.65.5-->25.8.65.4-->25.8.65.3-->25.8.65.2-->25.8.65.6

Student Notebook
PowerHA's SNMP Support

• PowerHA uses SNMP to provide:
– Notification of cluster events
– Cluster configuration/state information
• Support in PowerHA 5.3 and later is provided by the cluster
manager
– A client (smux peer) of AIX's SNMP daemon (snmpd)
– Prior to 5.3: clsmuxpd
• Support consists of:
– Maintaining a management information base (MIB)
– Responding to SNMP queries for PowerHA information
– Generating SNMP traps
• Used by clinfoES and HAView
• Available to any SNMP manager
(such as the snmpinfo command)
Figure 4-19. PowerHA’s SNMP Support QV1203.1
PowerHA support of SNMP

In PowerHA 5.3 and later, the cluster manager component provides cluster status
information via the Simple Network Management Protocol (SNMP). This allows the
cluster to be monitored via SNMP queries and SNMP traps.
In addition, PowerHA includes an extension to the Tivoli NetView product called
HAView. This extension can be used to make Tivoli NetView PowerHA-aware.
The clinfo daemon as well as any SNMP manager (such as the snmpinfo command)
can access PowerHA information using SNMP.
Configuring SNMP in order to provide cluster status is discussed in detail in the
PowerHA for AIX (HACMP) III: Administration and Problem Determination Concepts
class.

V3.1.0.1
Student Notebook
Uempty
Cluster Information Daemon (clinfo)
• An SNMP-aware client to the cluster manager
• Provides
– A cluster information API to the PowerHA SNMP
manager
•Focused on providing PowerHA cluster information
•Easier to work with than the SNMP APIs
– Support for ARP cache issues
• Is used by:
– The clstat command
– Customer written utility/monitoring tools
• Implemented as the clinfoES subsystem
Figure 4-20. Cluster Information Daemon (clinfo) QV1203.1
• What the clinfo daemon provides

The clinfo daemon provides an Application Program Interface (API) which can be
used to:
- Monitor the cluster
- Manage the ARP cache issue for systems that do not support gratuitous ARP
The clinfo daemon can run on PowerHA cluster server nodes or on any machine
which has the clinfo code installed.
clinfo must be running on a node or client machine in order to use any of the clstat
related commands (clstat, xclstat, clstat.cgi)
• Starting clinfo on a server node:
- Start it along with the rest of the PowerHA daemons from SMIT
- Use startsrc
• Starting clinfo on a client system:
- Use the /usr/es/sbin/cluster/etc/rc.cluster script
- Use startsrc

Student Notebook
Highly Available NFS Server Support

• The cluster administrator can:
– Define NFS exports at directory level to all clients
– Define NFS mounts and network to PowerHA nodes
– Specify export options for PowerHA to set
• PowerHA preserves file locks and dupcache across
fallovers
• Limitations
– Lock support is limited to
•Two node clusters (NFSv2/v3)
•32 nodes (NFSv4)
– Resource group is only active on one node at a time
• HACMP 5.4.1 extends support for NFSv4
Figure 4-21. Highly Available NFS Server Support QV1203.1
• PowerHA NFS support

The PowerHA software provides the following availability enhancements to NFS
operations:
- Reliable NFS server capability that allows a backup processor to recover current
NFS activity should the primary NFS server fail, preserving the locks on NFS
filesystems and the duplicate request cache
- Ability to specify a network for NFS mounting
- Ability to define NFS exports and mounts at the directory level
- Ability to specify export options for NFS-exported directories and filesystems
• Limitations
- The locking function is available only for two-node clusters
(32 nodes with NFSv4)
- The resource group must be non-concurrent - active on one node at a time

V3.1.0.1
Student Notebook
Uempty
Shared External Disk Access
• Provides two types of shared disk support:
– Serial access resource group:
•Varied on by one node at a time under the control of PowerHA
•LVM or RSCT ensures no access by 2 nodes at once
•Two types of volume groups:
– Non-concurrent VG (standard VG)
– Enhanced Concurrent VG running in non-concurrent mode
– Concurrent access resource group:
•Used by concurrent applications writing to raw logical volumes
•Two types of volume groups:
– SSA Concurrent Mode VG
(no longer supported in HACMP 5.4 and later)
– Enhanced Concurrent VG running in concurrent mode
• bos.clvm.enh fileset required for Concurrent/Enhanced
Concurrent
Figure 4-22. Shared External Disk Access QV1203.1
Shared disk support

As you know by now, PowerHA supports shared disks.
The planning and configuration of PowerHA shared storage are covered in the
PowerHA for AIX (HACMP) II: Administration class.

Student Notebook
Review 1 of 2
1. What is the first step in implementing a cluster?
a. Order the hardware
b. Plan the cluster
c. Install AIX and PowerHA
d. Install the applications
e. Take a long nap
2. True or False?
PowerHA 5.5 is compatible with any version of AIX v5.x or v6.x.
3. True or False?
Each cluster node must be rebooted after the PowerHA software is
installed.

V3.1.0.1
Student Notebook
Uempty
Review 2 of 2
4. Which component detects an adapter failure?
a. Cluster manager
b. RSCT
c. clcomd
d. clinfo
5. Which component provides SNMP information?
a. Cluster manager
b. RSCT
c. clsmuxpd
d. clinfo
6. Which component is required for clstat to work?
a. Cluster manager
b. RSCT
c. clcomd
d. clinfo
7. Which component removes requirement for the /.rhosts file?
a. Cluster manager
b. RSCT
c. clcomd
d. clinfo

Student Notebook
Unit Summary
• Before you install PowerHA, you should
– Plan your cluster
– Configure AIX: shared storage, networks and application start/stop scripts
• Install PowerHA software
– Determine which PowerHA filesets you will need and determine prerequisites
– Install software on cluster nodes (and optionally on clients (clinfo))
• PowerHA 5.5 includes the following major components
– Cluster Manager (clstrmgr)
• Heart of PowerHA. Responds to events (changes in cluster status) and takes action
(fallover, fallback, etc)
– Cluster Secure Communication Subsystem (clcomd)
• Provides secure node-to-node communications without /.rhosts
– RSCT and RMC
• Failure detection; notification to cluster manager; co-ordination of recovery
– SNMP monitoring programs
•clstrmgr is an smux peer, managing the PowerHA MIB for the SNMP daemon
• PowerHA status is available to clinfo and SNMP manager programs
– Cluster Information Program (clinfo)
• Provides an API to access PowerHA status via SNMP
• Supports clstat group of commands and customer written utilities
– Highly Available NFS Server
• Availability enhancements to NFS
– Shared External Disk Access
• Serial access shared disks (one node at a time)
• Concurrent access shared disks
Figure 4-25. Unit Summary QV1203.1

V3.1.0.1
Student Notebook
Uempty Unit 5. Initial PowerHA Configuration

In this unit, you will learn how to configure a cluster using the
PowerHA SMIT interface. You will learn how to perform simple and
more advanced cluster configuration. You will also learn how and
when to verify and synchronize your cluster.

• Build a standby and mutual takeover PowerHA 5.5 cluster
- Use the Initialization and Standard Configuration path (standard
path)
- Use the Two-Node Cluster Configuration Assistant (two-node
assistant)
• Configure PowerHA topology to include:
- IP address takeover via alias
- Non-IP networks (rs232, diskhb)
- Persistent address
• Verify, synchronize, and test a cluster
• Save cluster configuration

• Review questions
• Machine exercises
References
PowerHA manuals
© Copyright IBM Corp. 2007, 2009 Unit 5. Initial PowerHA Configuration 5-1
Student Notebook
Unit Objectives
• Build a standby and mutual takeover PowerHA 5.5 cluster
– Use the Initialization and Standard Configuration path
(standard path)
– Use the Two-Node Cluster Configuration Assistant
(two-node assistant)
• Configure PowerHA topology to include:
– IP address takeover via alias
– Non-IP networks (rs232, diskhb)
– Persistent address
• Verify, synchronize, and test a cluster
• Save cluster configuration
• Objectives
This unit will show how to configure a two-node hot standby cluster, with a heartbeat
over disk non-IP network, using both the standard and extended path and the two-node
configuration assistant. It will then demonstrate how to start and stop PowerHA cluster
services. It will also show the additional steps necessary to build a mutual takeover
cluster configuration. To accomplish this, you will see how to modify the configuration of
the cluster by adding a resource group, adding a persistent IP label, adding a heartbeat
on disk non-IP network and synchronize the changes. The final step is making a
snapshot backup of the cluster configuration.
You will be walked through the methods of configuring the cluster, starting with the
standard path and using the extended path where necessary followed by the simplest,
most limited method, i.e., the Two-Node Configuration Assistant. While using the
Initialization and Standard Configuration path, you will also see what additional
resources are required to build a cluster configuration to support a mutual takeover.

V3.1.0.1
Student Notebook
Uempty
What We Are Going to Achieve
A two node standby configuration (active / passive)
– Resource group appAgroup with node1 as its home (primary) node
and node2 as its backup node
– Application server appA in appAgroup
Fallover
IP network
node2
node1
appA appA node2-per
node1-per
Y Y
non-IP network
heartbeat over disk
Figure 5-2. What We Are Going to Achieve QV1203.1
• Configuring either a standby or a mutual takeover configuration

You will configure a two-node standby cluster. You will be guided through the process
using the standard path with extended path used only when needed. To adapt this to a
mutual takeover cluster, you need only add the steps that involve creating a second
resource group and its content and designate that it will have node2 as primary and
node1 as backup.
The standard path gives you the ability to use the pick lists and it automates some
steps. It requires that you have a solid understanding of your environment and the way
PowerHA works to successfully configure the cluster.
• Two-node configuration assistant
The two-node assistant can be used to create a simple hot-standby cluster, with one
resource group only. That one resource group will contain all the non-rootvg volume
groups present on the node where the configuration is being done. We will discuss both
the standard path and the assistant in the lecture. In the exercise, you will use the
standard path to configure your cluster.
The cluster will be tested for reaction to node, network, and network adapter failure.
Student Notebook

9Plan for network, storage, and application
– Networks (IP interfaces, /etc/hosts, non-IP)
9Install the PowerHA filesets and reboot
¾Configure the PowerHA environment
– Topology
– Resources, resource group, attributes:
•Resources: Application Server, service label
•Resource group: Identify name, nodes, policies
•Attributes: Application Server, service label, VG, filesystem …
– Synchronize
• Start PowerHA
• Test configuration
• Save configuration
• Ready for configuration

Now that the PowerHA filesets are installed, we can start to configure PowerHA.
• Where do we go from here?
As mentioned on the previous visual, we will configure a hot standby configuration with
one application and one resource group using the Standard Configuration method.
Finally, in this topic, we will use the extended path to deal with some initial configuration
choices that cannot be done with the standard path.

V3.1.0.1
Student Notebook
Uempty
The Topology Configuration
• Here's the key portion of the /etc/hosts file that we'll be using in this unit:
192.168.1.1 node1-if1 # node1's first interface IP label
192.168.2.1 node1-if2 # node1's second interface IP label
192.168.3.1 node1-per # persistent node IP label on node1
192.168.1.2 node2-if1 # node2's first interface IP label
192.168.2.2 node2-if2 # node2's second interface IP label
192.168.3.2 node2-per # persistent node IP label on node2
192.168.3.10 appA-svc # the IP label for the application
# normally resident on node1
• Hostnames: node1, node2
• node1's network configuration (defined via smit chinet) :
en0 - 192.168.1.1
en1 - 192.168.2.1
• node2's network configuration:
en0 - 192.168.1.2
en1 - 192.168.2.2
• These network interfaces are all connected to the same
physical network
• The subnet mask is 255.255.255.0 on this network
• An enhanced concurrent mode volume group “vgA" has been created to
support the appA application and will be used for a non-IP heartbeat over
disk network
Figure 5-4. The Topology Configuration QV1203.1
• A sample network configuration

Every discussion must occur within a particular context. The above network
configuration is the context within which the first phase of this unit will occur. Please
refer back to this page as required over the coming visuals.
Note that the addresses are set to support IPAT via Aliasing. The service address would
have replaced the address of one of the interface adapters if IPAT via Replacement was
to be used.
Also note that an understanding of the physical layout of the adapters in each system is
critical to ensure that the cable attachments are going to the correct network interface in
AIX. This is true whether you’re dealing with a standalone system or an LPAR with
adapters in drawers.
For virtual Ethernet adapters, you must make sure that the virtual adapter is correctly
bridged by the Shared Ethernet Adapter(s) in the VIO server LPAR(s) and that the
physical adapter(s) in the VIO servers are correctly connected to the switch.
For Logical Host Ethernet Adapters (LHEA), you must make sure that the physical HEA
port is correctly connected to the switch.
Student Notebook
Configuration Methods
• PowerHA provides two menu paths with three methods
– Initialization and Standard Configuration
•Two-node cluster configuration assistant
– Limited configuration
» Only 2 node hot standby cluster
– Builds cluster configuration based on AIX configuration
» All adapter addresses are used
» All volume groups assigned to one resource group
– Creates everything needed for simple cluster (Topology,
Resources, Resource Group)
» No persistent addresses
» Creates non-IP network using Heartbeat over Disk ( if enhanced concurrent
mode volume group present)
•Standard configuration
– Topology done in one step
– You then must configure resource groups and synchronize
– Desirable means to create more than 2 node hot standby cluster
– Extended Configuration
• More steps but provides access to all the options
• Needed for persistent address and non-IP network configuration
Figure 5-5. Configuration Methods QV1203.1
• Two-Node Cluster Configuration Assistant

With this method, all the steps of Standard Configuration are done at once including
adding a non-IP disk heartbeat network if you created an enhanced concurrent volume
group. Note that this is a simple two-node configuration with one resource group
containing all configured volume groups. This can be a starting point for creating a more
robust cluster but should not be viewed as a shortcut to creating a cluster without a
thorough understanding of how PowerHA works.
• Standard Configuration
With this method you must do the following:
i. Topology (simplified via “Configure an HACMP Cluster and Nodes”)
ii. Configure Resources and Resource Groups
iii. Verify and Synchronize
• Extended Configuration
With this method you follow similar steps as the Standard Configuration but Topology
has more steps and there are many more options. Some options can only be done
using this method such as adding a non-IP network.

V3.1.0.1
Student Notebook
Uempty
Planning and Base Configuration Review
• In AIX, configure boot (non-service) addresses on all systems
– Addresses on same system are in different subnets
• Plan service IP Label (appA-svc)
– IPAT via Aliases: Different subnet than the boot subnets
– IPAT via Replacement: Same subnet as one boot subnet
• Ensure /etc/hosts:
– Contains boot, service and persistent address for all systems
– Is the same file on all systems.
• Create the volume group(s) to be used by the application(s) (vgA)
– Enhanced Concurrent Mode Volume group(s) recommended
• Plan communication path to the nodes (node1-if1, node2-if1)
• Plan Application Server name (appA)
• Ensure Application Server start and stop scripts
– Exist on both nodes
– Are executable
Figure 5-6. Planning and Base Configuration Review QV1203.1
• Network configuration
Configure boot addresses in AIX and ensure that all the addresses (interface, service
and persistent) are in the /etc/hosts files. If you do this first, then SMIT can discover the
IP addresses / IP labels and populate the SMIT pick lists for you.
• Storage configuration
In the previous exercise, you created an enhanced concurrent volume group prior to
configuring the cluster. Another approach: once you have created the cluster topology,
you can create any volume groups using the PowerHA SMIT menus and C-SPOC will
automatically create the volume group on all your cluster nodes in one step.
• Communication path (boot address)
Both the 2-node assistant and the Standard Path use one of the boot addresses on
each node to identify the nodes. In the 2-node assistant, you must identify the second
node using one of the boot addresses (it assume this system is the first node). In the
Standard Path, you must identify all the nodes using the boot address.
• Application server
You will need to name the application server and identify the start/stop scripts. The
scripts must be executable and in the some location on both nodes.
Student Notebook
Starting at the Very Beginning...

System Management
Move cursor to desired item and press Enter.
Software Installation and Maintenance
Software License Management
Devices
System Storage Management (Physical & Logical Storage)
Security & Users
Communications Applications and Services
Print Spooling
Advanced Accounting
Problem Determination
Performance & Resource Scheduling
System Environments
Processes & Subsystems
Applications
Installation Assistant
Cluster System Management
Using SMIT (information only)
F1=Help F2=Refresh F3=Cancel F8=Image

F9=Shell F10=Exit Enter=Do
Figure 5-7. Starting at the Very Beginning... QV1203.1
• Starting at the main SMIT menu

Here’s the top level SMIT screen which everyone familiar with AIX would know.
PowerHA is under the “Communications Applications and Services” selection because,
presumably, it is considered to be primarily a networking or communications related
product.
More often, this menu (and the next one shown on the next visual) would be skipped by
entering the command smit hacmp or smitty hacmp but for the sake of completeness
we will start at the main SMIT menu.

V3.1.0.1
Student Notebook
Uempty
Almost There…
Communications Applications and Services

TCP/IP
NFS
HACMP for AIX

Figure 5-8. Almost There... QV1203.1
• HACMP shows up on the communications SMIT menu

Once you’ve installed the PowerHA software, a new menu selection appears in the
Communications Applications and Services menu.
Student Notebook
The Top-Level PowerHA SMIT Menu

# smit hacmp
HACMP for AIX
Initialization and Standard Configuration

Extended Configuration
System Management (C-SPOC)
Problem Determination Tools

Figure 5-9. The Top-Level PowerHA SMIT Menu QV1203.1
• The main PowerHA SMIT menu

This is the top level PowerHA SMIT menu. You’ll find it often simplest to get here using
the SMIT fastpath shown above.
As implied by the # prompt, there is little point in being here if you don’t have root
privileges!

V3.1.0.1
Student Notebook
Uempty
The Standard Configuration Method

Configuration Assistants
Configure an HACMP Cluster and Nodes
Configure Resources to Make Highly Available
Configure HACMP Resource Groups
Verify and Synchronize HACMP Configuration

Display HACMP Configuration
HACMP Cluster Test Tool

Fell F10=Exit Enter=Do
What we’ll see, step-by-step

Start with Configure an HACMP Cluster and Nodes
Figure 5-10. The Standard Configuration Method QV1203.1
• The initialization and standard configuration menu

For very simple, two-node, hot-standby, one resource group, one volume group
configurations, you can use the Two-node Configuration Assistant. However, it is good
practice to use the Initialization and Standard Configuration path for all your cluster
configurations because it requires you to be aware of the details of your configuration.
• Synchronization
Configuration changes are made on one node and do not take effect until they are
verified and synchronized. During synchronization, the files are propagated to the other
nodes and the changes become active. If changes are made on one node but not
synchronized and then more changes are made on a second node and then
synchronized, the changes made on the first node are lost.
Recommendation: Choose one node as your “administration node”.
• The method
The menu shows the tasks in the order they are to be performed: Create the cluster,
configure resources and add the resources to a resource group.
• Configuration assistants
In addition to the 2-node assistant, PowerHA provides, via priced feature, “Smart
Assists” for WebSphere, Oracle, and DB2.
Student Notebook
Configure Nodes to a Cluster

Configure Nodes to an HACMP Cluster (standard)

[Entry Fields]
* Cluster Name [qv120Cluster]
New Nodes (via selected communication paths)[node1-if1 node2-if1] +

Currently Configured Node(s)

Figure 5-11. Configure Nodes to a Cluster QV1203.1
• Input for the standard configuration method

Assuming your network planning and setup was done correctly you need only decide on
a name for the cluster and choose one IP address/hostname for each node that will be
in the cluster, including the node where you see this screen.
• New Nodes (via selected communication paths)
The communication path should be one of the boot (non-service) IP address / labels on
each node.
You can select the interfaces from a pick list (created from the local /etc/hosts file).
• Currently Configured Node(s)
At this point in time, this field is empty. If you were adding nodes to an existing cluster,
this field would show the existing nodes.
• Node name
The IP label used in the communication path is not necessarily the PowerHA node
name that will be assigned to the node. The hostname of each node is used as the
PowerHA node name. (If you need the node node to be different from the hostname,
you must add the node manually using the Extended Path menus.)

V3.1.0.1
Student Notebook
Uempty
What Did We Get?
•# /usr/es/sbin/cluster/utilities/cltopinfo
Cluster Name: qv120Cluster
Cluster Connection Authentication Mode: Standard
Cluster Message Authentication Mode: None
Cluster Message Encryption: None
Use Persistent Labels for Communication: No
There are 2 node(s) and 2 network(s) defined
NODE node1:
Network net_ether_01
node1-if1 192.168.1.1
node1-if2 192.168.2.1
NODE node2:
node2-if1 192.168.1.2
node2-if2 192.168.2.2
No resource groups defined
Figure 5-12. What Did We Get? QV1203.1
• Configure Nodes to a Cluster creates the basic cluster topology

This step creates the cluster, adds the nodes and the boot addresses (non-service
addresses). These topology objects are created based on the addresses/subnet masks
that are configured on the adapters in the nodes specified in the previous screen. The
configuration scripts connect to each node, discover the interfaces that are configured
and send patterns of network packets to determine which interfaces are connected to
the same network.
• Configuration only made to one node until we synchronize
This exists only on the node where the command was run. The cluster does not actually
get created until we synchronize. Later we will see the synchronization process.
• What is not done at this point
Notice that there is no non-IP network and there are no resources and no resource
groups yet when using the standard configuration method.
Student Notebook
Now Define Highly Available Resources

smit hacmp -> Initialization and Standard Configuration


Figure 5-13. Now Define Highly Available Resources QV1203.1
• Not done yet

Since Configure an HACMP Cluster and Nodes only does the topology, there is more
to do:
- Configure Resources to Make Highly Available
Resources must be created (application server(s), service address(es), volume
groups, etc.)
- Configure HACMP Resource Groups
• A resource group must be created with policies and attributes
• Resources must be added to the group
- Verify and Synchronize
The cluster definitions must be propagated to the other node(s).
• Add non-IP networks and persistent addresses
The Extended Configuration menus must be used to add non-IP heartbeat
network(s) and to configure persistent addresses.

V3.1.0.1
Student Notebook
Uempty
Configure Service Addresses
Configure Service IP Labels/Addresses

Configure Application Servers
Configure Volume Groups, Logical Volumes and Filesystems
Configure Concurrent Volume Groups and Logical Volumes

Figure 5-14. Configure Service Addresses QV1203.1
• Follow the order in the menus

Again, the process will be to follow the menus. The first step is to define the Service IP
address(es).
Student Notebook
Adding Service IP Address(es)

Add a Service IP Label/Address

Change/Show a Service IP Label/Address
Remove Service IP Label(s)/Address(es)

Figure 5-15. Adding Service IP Address(es) QV1203.1
• Configure Service IP Label/Address

This is the menu for managing service IP labels and addresses within the standard
configuration path.

V3.1.0.1
Student Notebook
Uempty
Add appA-svc Service Address (1 of 3)
Add a Service IP Label/Address (standard)

[Entry Fields]
* IP Label/Address [] +
Prefix Length [] #
* Network Name [] +
ňņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņŉ
Ň IP Label/Address Ň
Ň Ň
Ň Move cursor to desired item and press Enter. Ň
Ň Ň
Ň (none) ((none)) Ň
Ň client (9.47.87.113) Ň
Ň node1-per (192.168.3.1) Ň
Ň node2-per (192.168.3.2) Ň
Ň appA-svc (192.168.3.10) Ň
Ň client-app (192.168.3.15) Ň
Ň appB-svc (192.168.3.20) Ň
Ň Ň
Ň F1=Help F2=Refresh F3=Cancel Ň
F1=Help Ň F8=Image F10=Exit Enter=Do Ň
F5=Reset Ň /=Find n=Find Next Ň
F9=Shell Ŋņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņŋ
Figure 5-16. Add appA-svc Service Address (1 of 3) QV1203.1
• Selecting the service address

This is the PowerHA SMIT screen for adding a service IP address in the standard
configuration path.
If you use F4 (or ESC-4) in the IP Label/Address field you get a list of the IP labels
which were found in /etc/hosts but not yet associated with NICs in PowerHA.This could
be quite a long list depending on how many entries there are in the /etc/hosts file.
Although, in practice, the list is usually fairly short as /etc/hosts on cluster nodes tends
to only include IP labels which are important to the cluster.
The service IP address that we intend to associate with the appAgroup resource
group’s application is appA-svc.
Student Notebook

[Entry Fields]
* IP Label/Address [appA-svc] +
Prefix Length [] #
* Network Name [] +
Ň CONTEXTUAL HELP Ň
Ň Ň
Ň Press Enter or Cancel to return to the application. Ň
Ň Ň
Ň Enter the prefix length for the service IP. For IPv4 service IP Ň
Ň the prefix length must, either, concur with the prefix length of the Ň
Ň underlying HACMP network, or left blank for its auto-population. Ň
Ň However, for IPv6 service IP, the user must supply appropriate prefix Ň
Ň length (ranging from 1 to 128). Ň
Ň Ň
F1=Help Ň F1=Help F2=Refresh F3=Cancel Ň
F5=Reset Ň F8=Image Enter=Do Ň
• Prefix Length field

PowerHA 5.5 adds a new field to this SMIT screen. The Prefix Length field is
required for IPv6 service addresses. It is optional for IPv4 addresses. If you fill it in for
an IPv4 address, make sure the prefix length matches the subnet masks you had set on
the interfaces. (All addresses in one PowerHA network must use the same subnet
mask.)

V3.1.0.1
Student Notebook
Uempty

[Entry Fields]
* IP Label/Address [appA-svc] +
Prefix Length [] #
* Network Name [net_ether_02] +
Ň Network Name Ň
Ň Ň
Ň Ň
Ň net_ether_01 (9.47.87.0/24) Ň
Ň net_ether_02 (192.168.2.0/24 192.168.1.0/24) Ň
Ň Ň
F1=Help Ň F8=Image F10=Exit Enter=Do Ň
F5=Reset Ň /=Find n=Find Next Ň
Repeat the process for every service address to be

configured if mutual takeover cluster, i.e. defining appB-svc
• Choosing the network

You must specify which network this service address will be associated with. Although
you can manually type in the network name, it’s easier to use F4 to show the pick list,
which shows IP network(s) defined on this cluster.
Notice that the pick list shows the IP subnets associated with each network. This is
useful information at this point as we must specify a service IP label which is not in
either of these subnets in order to satisfy the rules for IPAT via IP aliasing.
• Create the service address
Once we’re sure that this is what we intend to do, press Enter to define the service IP
address. The service address is then available from a pick list when you add resources
to a resource group later.
• Mutual takeover configuration
If creating a mutual takeover configuration, you would repeat the step to define the
service IP label for the application that will run on the other node.
Student Notebook
Configure Application Server


Figure 5-19. Configure Application Server QV1203.1
• Application server
Continuing to follow the menus, the next step is to define the application server(s).

V3.1.0.1
Student Notebook
Uempty
Add appA Application Server (1 of 2)

Add an Application Server

Change/Show an Application Server
Remove an Application Server

Figure 5-20. Add appA Application Server (1 of 2) QV1203.1
Student Notebook
Add appA Application Server (2 of 2)

Add Application Server
[Entry Fields]
* Server Name [appA]
* Start Script [/usr/HTTPServer/bin/apachectl start]
* Stop Script [/usr/HTTPServer/bin/apachectl stop]
F1=Help F2=Refresh F3=Cancel F4=List

F5=Reset F6=Command F7=Edit F8=Image
Repeat the process for every Application Server to be

configured if mutual takeover cluster, i.e. creating appB
Figure 5-21. Add appA Application Server (2 of 2) QV1203.1
• Add Application Server

An application server has a name and consists of a start script and a stop script. Use
the full path for the script names. Once defined here, the application server will be
available from a pick list when adding resources to a resource group later.
If creating a mutual takeover configuration, you would repeat this step to define the
application server for the application that will run on the other node.
• Location of scripts
- The start and stop scripts must must be executable and must reside on a local
non-shared file system on each node defined in the resource group or you will
verification errors and will not be able to synchronize the cluster.
- If you are using the auto-correction facility of verification, the start/stop scripts from
the node where they exist will be copied to the other node(s).
- PowerHA 5.2 and later provides a file collections facility to help keep critical files in
sync across the nodes in the cluster. We will discuss using file collections in the
PowerHA for AIX (HACMP) III: Administration and Problem Determination Concepts
course.

V3.1.0.1
Student Notebook
Uempty
Configure Volume Groups (Optional)
• Shared volume groups can be created before you start PowerHA
configuration
• Once you have created the cluster topology, you can use
C-SPOC to create the volume groups and import them on all nodes
in the resource group
• If you choose to, follow the menus…



Create volume groups for every application, regardless of which

system the application will normally run if mutual takeover cluster
Figure 5-22. Configure Volume Groups (Optional) QV1203.1
• Volume group creation

Creating the shared volume groups can be done outside of the cluster configuration
process or as part of the cluster configuration process. In the exercise, you created the
volume group for appA outside of PowerHA.
Once you have created the cluster topology, you can create your volume group(s) from
within PowerHA, using the cluster single point of control (C-SPOC) feature. You can do
this from within the Initialization and Standard Configuration menu path (as
shown in the visual), or from within the System Management (C-SPOC) menu.
In general, it is recommended that you use C-SPOC to create your volume groups.
C-SPOC saves work by automatically importing the new volume group on all of your
cluster nodes. We will learn much more about C-SPOC in the PowerHA for AIX
(HACMP) II: Administration course.
If creating a mutual takeover configuration, you would repeat this step, creating the
volume groups needed for both applications.
Student Notebook
Discovery
• Information for SMIT pick lists is created by the Discovery process
• Discovery is initially run when you create the cluster topology
• Discovery should be re-run when you make changes to the network
or LVM configuration
• After creating volume groups using C-SPOC:
– Prior to PowerHA 5.5: Run discovery to make VGs available in pick lists
– PowerHA 5.5: Discovery is run automatically
• Use Extended Configuration menu to run Discovery
Discover HACMP-related Information from Configured Nodes

Extended Topology Configuration
[ . . . ]
Figure 5-23. Discovery QV1203.1
• PowerHA pick lists

When assigning resources to resource groups, selections are available from a pick list
(F4) in SMIT. Using the pick lists saves you time and avoids typing errors. In some
cases, you must use the pick list. In other cases, you can also type in the information
manually. The pick list information for PowerHA is kept in two text files:
- /usr/es/sbin/cluster/etc/config/clvg_config: volume group information
- /usr/es/sbin/cluster/etc/config/clip_config: IP information
These files are generated by the Discovery process. Discovery is run and the pick list
files are created when you initially create the cluster. If you make changes to the AIX
configuration (network configuration as well as LVM configuration), you should run
Discovery again to update the pick list files.
Prior to PowerHA 5.5, if you create volume group(s) using C-SPOC, you need to re-run
Discovery. This requires using the Extended Configuration menu shown above.
In PowerHA 5.5 and later, if you create volume group(s) using C-SPOC, the pick list
files are updated automatically.

V3.1.0.1
Student Notebook
Uempty
Adding the appAgroup Resource Group

Add a Resource Group
Change/Show a Resource Group
Remove a Resource Group
Change/Show Resources for a Resource Group (standard)

Figure 5-24. Adding the appAgroup Resource Group QV1203.1
Student Notebook
Setting Name, Nodes, and Policies

*Resource Group Name [appAgroup]

*Participating Nodes(Default Node Priority) [node1 node2]
Startup Policy Online On Home Node O> +
Fallover Policy Fallover To NextPrio> +
Fallback Policy Fallback To Higher Pr> +

Repeat the process for every resource group to be configured

if mutual takeover cluster, i.e. creating appBgroup with node2, node1
Figure 5-25. Setting Name, Nodes, and Policies QV1203.1
• Add a Resource Group

We’ll call this resource group appAgroup. It will be defined to operate on two nodes -
node1 and node2. node1 is specified first to indicate that it is the home or highest
priority node.
The policies will be chosen as listed in the visual. Depending on the type of resource
group and how it is configured, the relative priority of nodes within the resource group
might be quite important.
For a mutual takeover cluster, add the second resource group, listing node2 first so that
it is the home node for the other resource group.

V3.1.0.1
Student Notebook
Uempty
Adding Resources to appAgroup (1 of 2)
Change/Show a Resource Group
Remove a Resource Group
Change/Show Resources for a Resource Group (standard)
+--------------------------------------------------------------------+
¦ Select a Resource Group ¦
¦ ¦
¦ Move cursor to desired item and press Enter. ¦
¦ ¦
¦ appAgroup ¦
| appBgroup ¦
¦ ¦
¦ F1=Help F2=Refresh F3=Cancel ¦
¦ F8=Image F10=Exit Enter=Do ¦
¦ /=Find n=Find Next ¦
+--------------------------------------------------------------------+
Figure 5-26. Adding Resources to appAgroup (1 of 2) QV1203.1
• Adding resources to a resource group

Select the Change/Show Resources for a Resource Group (standard) to add
resources to a resource group. When the Select a Resource Group popup appears,
select which resource group you want to work with and press Enter.
Student Notebook
Adding Resources to appAgroup (2 of 2)

Change/Show All Resources and Attributes for a Resource Group
[Entry Fields]
Custom Resource Group Name appAgroup
Participating Node Names (Default Node Priority) node1 node2
Startup Behavior Online On First Avail>
Fallover Behavior Fallover To Next Prio>
Fallback Behavior Fallback To Higher Pr>
Service IP Labels/Addresses [appA-svc] +
Application Servers [appA] +
Volume Groups [vgA] +
Use forced varyon of volume groups, if necessary false +
Filesystems (empty is ALL for VGs specified) [] +

Repeat previous two steps for every configured resource group

if mutual takeover cluster, i.e. appBgroup
Figure 5-27. Adding Resources to appAgroup (2 of 2) QV1203.1
• Adding resources to a resource group

- Service IP label: Any service addresses needed (appA-svc)
- Application server: Any application servers to be run with the RG (appA)
- Volume group(s): Any VGs needed by the application(s) (vgA)
- File systems: Any file systems which should be mounted (ALL).
The default is to mount all file systems in the included volume group(s). This is the
best choice, unless you need to exclude some file systems. This way you don’t have
to continue to update the resource group as you add file systems to the volume
group.
Don’t forget to press Enter to actually add the resources to the resource group.
If creating a mutual takeover configuration, you would repeat this step, adding the
resources to the second resource group.

V3.1.0.1
Student Notebook
Uempty
Synchronize Your Changes


Figure 5-28. Synchronize Your Changes QV1203.1
• Verify and synchronize

Once you’ve defined or changed the cluster’s topology and/or resources, you must
verify and synchronize to make the configuration active in the cluster. The verification
process collects AIX configuration information from each cluster node and uses this
information, the cluster’s current configuration (if there is one) and the proposed
configuration to verify that the proposed configuration is valid.
• Using Standard Configuration to synchronize
This menu choice acts immediately in Standard Configuration. There are some
additional options available in the Extended Configuration menus which we will not
cover here.
It is possible to override verification errors but only if you are using Extended
Configuration. Deciding to do so is a decision which must be approached with the
greatest of care as it is VERY unusual for a verification error to occur which can be
safely overridden.
Student Notebook
Test Your Cluster


Figure 5-29. Test Your Cluster QV1203.1
• Testing your cluster

You must test your newly configured cluster for proper functioning. You should test your
cluster after every change to the environment, whether directly related to the cluster or
not. It is highly recommended that you create a comprehensive test plan prior to
configuring your cluster to be used during the test phase. The Cluster Test Tool can be
part of that test plan.
• PowerHA cluster test tool
This test facility is disruptive to the cluster (the application will be interrupted). So you
want to run it when not running cluster services or schedule a maintenance window.
The Standard Configuration automated test procedure performs four sets of tests in
the following order:
a. General topology tests
b. Resource group tests on non-concurrent resource groups
c. Resource group tests on concurrent resource groups
d. Catastrophic failure test
See the Administration Guide Chapter 7 for more details.

V3.1.0.1
Student Notebook
Uempty
What Do We Have at This Point?
• We DO have a cluster configured with the following:
– Two nodes defined
– One network defined with interface and service addresses
(Actually we have two networks:
net_ether_01 (en0) and net_ether_02 (en1 and en2) )
– Application server objects defined containing the start and stop
scripts for all the applications to be made highly available
– Volume groups defined to contain the data for the applications
– Resource group(s) with node priority, the service label(s),
application server(s) and volume group(s) for the applications
– All this has been synchronized
• We DON’T have: A non-IP network

• We DON’T have: Persistent IP addresses
• In addition, a snapshot of our work would be prudent
• We need to remove net_ether_01 from PowerHA
Figure 5-30. What Do We Have at This Point? QV1203.1
• What we have created so far

We have accomplished a large portion of the cluster configuration. The nodes, IP
addresses (service and non-service), networks, application servers, volume groups and
resource groups have been configured. This configuration has been synchronized
across the cluster nodes. We indicate that some level of testing could be performed at
this point. You can wait until after we do the rest of the configuration to test everything,
or break it up as we have it here.
• What’s left?
- We should create at least one non-IP network to protect against a partitioned
cluster in the event of a network failure.
- Although optional, it is usually useful to create a persistent address to provide
reliable administration access to the cluster
- Finally, it is always a good idea to create backups after producing this much good
work. It is advisable to create both a snapshot of the cluster configuration and a
mksysb of the systems.
- In our lab environment, we do not want the network associated with en0 to be under
PowerHA control. We should remove that network from the PowerHA configuration.
Student Notebook
Extending the Configuration


Extended Resource Configuration
Extended Cluster Service Settings
Extended Event Configuration
Extended Performance Tuning Parameters Configuration
Security and Users Configuration
Snapshot Configuration
Export Definition File for Online Planning Worksheets
Import Cluster Configuration from Online Planning Worksheets File
Extended Verification and Synchronization

F1=Help F2=Refresh F3=Cancel

F8=Image
Figure 5-31. Extending the Configuration QV1203.1
• PowerHA Extended Configuration path

Here’s the top level Extended Configuration menu. We need to use this path in order
to perform some steps that cannot be done using Standard Configuration, such as
defining a non-IP network, adding a persistent label and saving the configuration data.
We will explore these steps in the next few visuals.
There are a number of other tasks which can only be done from the Extended
Configuration menu.
Using the Extended Configuration menu is covered in the PowerHA for AIX
(HACMP) III: Administration and Problem Determination Concepts course.

V3.1.0.1
Student Notebook
Uempty
Extended Topology Configuration Menu
Configure an HACMP Cluster

Configure HACMP Nodes
Configure HACMP Sites
Configure HACMP Networks
Configure HACMP Communication Interfaces/Devices
Configure HACMP Persistent Node IP Label/Address
Configure HACMP Global Networks
Configure HACMP Network Modules
Configure Topology Services and Group Services
Show HACMP Topology

Figure 5-32. Extended Topology Configuration Menu QV1203.1
• Extended Topology Configuration menu

In the next few visuals we will show how to use this menu to:
- Configure a non-IP network
- Add persistent addresses
Configure HACMP Persistent Node IP Label/Address
- Remove an HACMP network from the PowerHA configuration
- Remove the cluster from a node
Student Notebook
Menus to Configure a Non-IP Network

Add Communication Interfaces/Devices

Change/Show Communication Interfaces/Devices
Remove Communication Interfaces/Devices
Update HACMP Communication Interface with Operating System
Settings

Figure 5-33. Menus to Configure a Non-IP Network QV1203.1
• Getting to the non-IP network configuration menus

Non-IP networks are elements of the cluster’s topology so we’re in the topology section
of the extended configuration path’s menu hierarchy.
A non-IP network is defined by specifying the network’s end-points. These end-points
are called communication devices so we have to head down into the communication
interfaces/devices part of the extended topology screens.

V3.1.0.1
Student Notebook
Uempty
Defining a Non-IP Network (1 of 3)
Settings
+--------------------------------------------------------------------------+
¦ Select a category ¦
¦ ¦
¦ ¦
¦ Add Discovered Communication Interface and Devices ¦
¦ Add Predefined Communication Interfaces and Devices ¦
¦ ¦
F1¦ /=Find n=Find Next ¦
F9+--------------------------------------------------------------------------+
Don't risk a potentially catastrophic partitioned cluster by using cheap rs232 cables!
Figure 5-34. Defining a Non-IP Network (1 of 3) QV1203.1
• Deciding which “Add” to choose

The first question we encounter is whether we want to add discovered or pre-defined
communication interfaces and devices. The automatic discovery that was done when
the added the cluster nodes earlier would have found the rs232 or Enhanced
Concurrent volume group devices so we pick the “Discovered” option.
Student Notebook

Settings
+--------------------------------------------------------------------------+
¦ ¦
¦ ¦
¦ # Discovery last performed: (Feb 12 18:20) ¦
¦ Communication Interfaces ¦
¦ Communication Devices ¦
¦ ¦
F9+--------------------------------------------------------------------------+
• Is it an interface or a device?
Now we need to indicate whether we are adding a communication interface or a
communication device. Non-IP networks use communication devices as end-points
(/dev/tty for example) so select Communication Devices to continue.

V3.1.0.1
Student Notebook
Uempty
Press Enter and HACMP defines a new non-IP network with these communication
devices. Choose the hdisks for the Heartbeat over Disk network too.
+--------------------------------------------------------------------------+
¦ Select Point-to-Point Pair of Discovered Communication Devices to Add ¦
¦ ¦
¦ Move cursor to desired item and press F7. Use arrow keys to scroll. ¦
¦ ONE OR MORE items can be selected. ¦
¦ Press Enter AFTER making all selections. ¦
¦ ¦
¦ # Node Device Device Path Pvid ¦
¦ node1 hdisk1 /dev/hdisk1 000b4a7cd1.. ¦
¦ node2 hdisk1 /dev/hdisk1 000b4a7cd1.. ¦
¦> node1 tty1 /dev/tty1 ¦
¦> node2 tty1 /dev/tty1 ¦
¦ node1 tmssa1 /dev/tmssa1 ¦
¦ node2 tmssa2 /dev/tmssa2 ¦
¦ ¦
¦ F7=Select F8=Image F10=Exit ¦
F1¦ Enter=Do /=Find n=Find Next ¦
F9+--------------------------------------------------------------------------+
• What are the options?

Non-IP networks are point-to-point. You must select the two end points, one on each
node. The visual shows selecting RS232 devices. In our lab cluster, we will use a
diskhb network. Target mode SSA and target mode SCSI can also be used.
Prior to using the SMIT screen, you should have tested the devices you will be using in
the non-IP network:
- RS-232: Connect cables and test using the stty command.
- diskhb: Create enhanced concurrent mode VG and test using the
/usr/sbin/rsct/bin/dhb_read command
• More than two nodes
With more than two nodes, you must create non-IP networks that form a loop. In a four
node cluster, you would need four non-IP networks. For example, if using diskhb, you
would need four networks (four disks):
- node1 to node2 using hdisk1
Student Notebook
Remove A Network From PowerHA

smit hacmp -> Extended Configuration ->
Extended Topology Configuration -> Configure HACMP Networks ->
Add a Network to the HACMP Cluster

Change/Show a Network in the HACMP Cluster
Remove a Network from the HACMP Cluster
Manage Concurrent Volume Groups for Multi-Node Disk Heartbeat
ňņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņŉ
Ň Select a Network to Remove Ň
Ň Ň
Ň Ň
Ň net_diskhb_01 Ň
Ň net_ether_01 (9.47.87.0/24) Ň
Ň net_ether_02 (192.168.3.0/24 192.168.2.0/24 192.168.1.0/24) Ň
Ň Ň
Ň F8=Image F10=Exit Enter=Do Ň
F1=Help Ň /=Find n=Find Next Ň
F9=Shell Ŋņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņņŋ
Figure 5-37. Remove A Network From PowerHA QV1203.1
• Remove a network
When you use the Standard Configuration path to create the cluster, all interfaces
found will be included in the PowerHA topology. If there are some interfaces that you
don’t want PowerHA to use, you can remove them from the configuration.
In our lab environment, we have an extra interface (en0) on each node that we use for
access to the cluster. Since en0 is not on the same physical network as en1 and en2,
PowerHA creates a separate network for it. Since it is a single adapter network,
PowerHA will give warnings about this network being a single point of failure every time
we synchronize the cluster. Further, since we are not actually using this network in
PowerHA, we can remove it from the PowerHA configuration and get rid of the
warnings.

V3.1.0.1
Student Notebook
Uempty
Defining Persistent Node IP Labels (1 of 3)
Configure HACMP Persistent Node IP Label/Addresses


Change / Show a Persistent Node IP Label/Address
Remove a Persistent Node IP Label/Address

Figure 5-38. Defining Persistent Node IP Labels (1 of 3) QV1203.1
• Add a Persistent Node IP Label/Address

Defining a persistent node IP label on each cluster node allows the cluster
administrators to contact specific cluster nodes (or write scripts which access specific
cluster nodes) without needing to worry about whether the service IP address is
currently available or which node it is associated with.
The (slight) risk associated with persistent node IP labels is that users might start using
them to access applications within the cluster. You should discourage this practice as
the application might move to another node. Instead, users should use the IP address
associated with the application (that is, the service IP label that you configure into the
application’s resource group).
Student Notebook
Configure HACMP Persistent Node IP Label/Addresses

Change / Show a Persistent Node IP Label/Address
Remove a Persistent Node IP Label/Address
+--------------------------------------------------------------------------+
¦ Select a Node ¦
¦ ¦
¦ ¦
¦ node1 ¦
¦ node2 ¦
¦ ¦
+--------------------------------------------------------------------------+
• First you select a node

V3.1.0.1
Student Notebook
Uempty
Press Enter and then repeat for the node2 persistent IP label.

[Entry Fields]
* Node Name node1
* Network Name [net_ether_01] +
* Node IP Label/Address [node1-per] +
Prefix Length [] #

• Filling out the add a persistent node IP label/address menu

Once you’re on this screen, select the appropriate IP network from the Network Name
field and Node IP Label/Address that you want to use from the pick lists. Press Enter
to finish the operation.
You can repeat to choose a persistent label for the other node(s)
Student Notebook
Synchronize
smitty hacmp -> Extended Configuration

[Entry Fields]
* Verify, Synchronize or Both [Both] +
* Automatically correct errors found during [No] +
verification?
* Force synchronization if verification fails? [No] +

* Verify changes only? [No] +
* Logging [Standard] +
Figure 5-41. Synchronize QV1203.1
• Extended Verification and Synchronization menu

The extended path version presents options not available using the standard path.
- Verify, Synchronize or Both - This option is useful to verify a change without
synchronizing it. Which allows you to verify changes before committing them to the
cluster. Synchronizing without verifying is almost certainly a foolish idea except in
the most exotic of circumstances.
- Automatically correct errors found during verification? - This feature can fix
certain errors. By default it is turned off. Only available when cluster services are not
started.
- Force synchronization if verification fails? - Almost always a very bad idea.
Make sure that you really and truly MUST set this option to Yes before doing so.
- Verify changes only? - Yes causes verification to focus on aspects of the
configuration which changed since the last synchronization. As a result, the
verification will run slightly faster. This might be useful during the mid to early stages
of cluster configuration. It seems rather risky once the cluster is in production.
- Logging - you can increase the amount of logging related to this verification and
synchronization by setting this option to “Verbose”. This can be quite useful if you
are having trouble figuring out what is going wrong with a failed verification.

V3.1.0.1
Student Notebook
Uempty
Save Configuration: Snapshot
Create a Snapshot of the Cluster Configuration

Change/Show a Snapshot of the Cluster Configuration
Remove a Snapshot of the Cluster Configuration
Restore the Cluster Configuration from a Snapshot
Configure Custom Snapshot Method
Convert Existing Snapshot For Online Planning Worksheets

[Entry Fields]
* Cluster Snapshot Name [qv120Cluster] /
Custom Defined Snapshot Methods [] +
Save Cluster Log Files in snapshot No +
* Cluster Snapshot Description [qv120 class cluster]
Figure 5-42. Save Configuration: Snapshot QV1203.1
• Saving the cluster configuration

You can save the cluster configuration to a snapshot file or to an XML file, and the
cluster can be restored from either file. The XML file can also be used with the online
planning worksheets and potentially with other applications. This visual looks at the
snapshot method and the next visual looks at the XML method.
• Creating a snapshot
smit hacmp -> Extended Configuration -> Snapshot Configuration
A snapshot captures the PowerHA ODM files which allows you to recover the cluster
configuration.
A previously saved cluster snapshot can be applied to your cluster using the Restore
the Cluster Configuration from a Snapshot menu.
Creating a cluster snapshot is highly recommended BEFORE and AFTER making
cluster changes It only takes a few seconds and consumes little disk space, but it could
save you considerable time if you need to return to a previous configuration. Make sure
you give your snapshot a meaningful name and include any notes in the description
field so that you can identify the snapshot you want to restore.
Student Notebook
Save Configuration: XML File

smitty hacmp -> Extended Configuration
Export Definition File for Online Planning Worksheets
[Entry Fields]
* File Name [/var/hacmp/log/cluster.haw] /
Cluster Notes []

Change/Show a Snapshot of the Cluster Configuration
Remove a Snapshot of the Cluster Configuration
Restore the Cluster Configuration from a Snapshot
Configure Custom Snapshot Method
Convert Existing Snapshot For Online Planning Worksheets
Figure 5-43. Save Configuration: XML File QV1203.1
• Creating an XML file

Using Extended Configuration, you have two choices to save the cluster
configuration directly to an XML file
- Export Definition File for Online Planning Worksheets
This save the current cluster configuration to an XML file.
- Convert Existing Snapshot For Online Planning Worksheets
This saves an existing snapshot to an XML file.
Once created, you can use the Online Planning worksheets to get an updated view of
the configuration or change the configuration.
• To apply an XML file to your cluster:
smit hacmp -> Extended Configuration ->
Import Cluster Configuration from Online Planning Worksheets File

V3.1.0.1
Student Notebook
Uempty
Two-Node Cluster Configuration Assistant
Two-Node Cluster Configuration Assistant
[Entry Fields]
* Communication Path to Takeover Node [node2-if1] +
* Application Server Name [appA]
* Application Server Start Script [/usr/HTTPServer/bin/apachectl start]
* Application Server Stop Script [/usr/HTTPServer/bin/apachectl stop]
* Service IP Label [appA-svc] +
Prefix Length [] #

Figure 5-44. Two-Node Cluster Configuration Assistant QV1203.1
• The two-node cluster configuration assistant

This menu creates a simple two-node, hot-standby cluster. If your network is setup
correctly and you have configured a shared enhanced concurrent mode volume group,
this menu will build a complete two-node cluster including Topology, Resources,
Resource group and a non-IP network using heartbeat over disk. Also, synchronization
is done and you are all ready to start cluster services on both nodes.
You must specify the information shown in the visual. The Prefix Length field is
optional for an IPv4 service address.
Student Notebook
What the Two-Node Assistant Gives You

# /usr/es/sbin/cluster/utilities/cltopinfo
Cluster Name: appA_cluster
Cluster Connection Authentication Mode: Standard
Cluster Message Authentication Mode: None
Cluster Message Encryption: None
Use Persistent Labels for Communication: No
There are 2 node(s) and 2 network(s) defined
NODE node1:
node1-if1 192.168.1.1
node1-if2 192.168.2.1
Network net_diskhb_01
node1_hdisk1_01 /dev/hdisk1
NODE node2:
node2-if1 192.168.1.2
node2-if2 192.168.2.2
Network net_diskhb_01
node2_hdisk1_01 /dev/hdisk1
Resource Group appA_group
Startup Policy Online On Home Node Only
Fallover Policy Fallover To Next Priority Node in List
Fallback Policy Never Fallback
Participating Nodes node1 node2
Service IP Label appA-svc
Figure 5-45. What the Two-Node Assistant Gives You QV1203.1
• What the two-node assistant does

- Only one application and 2 nodes are supported.
- You need to pre-configure the shared volume group(s) before you run the assistant.
- The cluster topology is created based on the existing interfaces. You may have to
remove interfaces that you don’t want PowerHA to have.
- If an enhanced concurrent VG exists, a diskhb non-IP network is created.
- One resource group is created. The specified application server and all shared VGs
are added to it.
- The node where you run the assistant becomes the home node (highest priority) for
the resource group that is created.
- Names will be created for the cluster, resource group and application server based
on the application server name supplied.
- The RG policies are set to
• Online on Home Node Only
• Fallover to Next Priority Node in List
• Never Fallback.
- No persistent addresses are created

V3.1.0.1
Student Notebook
Uempty
9Plan for network, storage, and application
– Networks (IP interfaces, /etc/hosts, non-IP)
9Install the HACMP filesets and reboot
9 Configure the HACMP environment
– Topology
•Cluster, node names, HACMP IP and non-IP networks
– Resources, resource group, attributes:
•Resources: Application Server, service label
•Resource group: Identify name, nodes, policies
•Attributes: Application Server, service label, VG, filesystem
– Synchronize
¾Start HACMP
• Test configuration
• Save configuration
• Cluster configuration is implemented

Wow, we are all done except for starting PowerHA. We have briefly covered testing and
saving the configuration, but these topics are covered in more detail in the HACMP
Administration Guide.
Student Notebook
Starting Cluster Services (1 of 4)
HACMP for AIX

Problem Determination Tools
F1=Help F2=Refresh F3=Cancel

F8=Image
Figure 5-47. Starting Cluster Services (1 of 4) QV1203.1
• How to start PowerHA cluster services

Starting PowerHA involves a trip to the top level PowerHA menu since we need to go
down into the System Management (C-SPOC) part of the tree.
It might be worth pointing out that if you use the Web based SMIT for PowerHA fileset
then there is a navigation menu that allows you to skip from one menu path to another
one without having to go “back to the top”.
After a few times, you will probably learn to use the command smit clstart or smitty
clstart to bypass this menu and the next 2 menus.

V3.1.0.1
Student Notebook
Uempty

Manage HACMP Services
HACMP Communication Interface Management
HACMP Resource Group and Application Management
HACMP Log Viewing and Management
HACMP File Collection Management
HACMP Security and Users Management
HACMP Logical Volume Management
HACMP Concurrent Logical Volume Management
HACMP Physical Volume Management
Open a SMIT Session on a Node

Student Notebook
Manage HACMP Services
Start Cluster Services

Stop Cluster Services
Show Cluster Services


V3.1.0.1
Student Notebook
Uempty
# smit clstart
[Entry Fields]
* Start now, on system restart or both now +
Start Cluster Services on these nodes [node1,node2] +
* Manage Resource Groups Automatically +
BROADCAST message at startup? true +
Startup Cluster Information Daemon? true +
Ignore verification errors? false +
Automatically correct errors found during Interactively +
cluster start?

• Startup choices
There are a few choices to make. For the moment we will just recommend the defaults
except selecting both nodes and turning on the Cluster Information Daemon.
• Remember the fast path
Notice the smit clstart fastpath. This is often much faster than working your way
through the menu tree.
Student Notebook
Removing a Cluster
•Stop cluster services (smitty cl_stop)
•Remove cluster using Extended Topology Configuration on first node
•Repeat on second node
•Remove contents of the cluster rhosts file - on both nodes
# > /usr/es/sbin/cluster/etc/rhosts
Add/Change/Show an HACMP Cluster

Remove an HACMP Cluster
Reset Cluster Tunables

Figure 5-51. Removing a Cluster QV1203.1
• Starting over
If you have to start over, you can:
- Stop cluster services on all nodes
- Use Extended Configuration, as shown above to remove the cluster from this
node
Note: This must be done separately on each node. (Once you’ve removed the
cluster configuration from a node, it is no longer in the cluster so you cannot
synchronize the change to the other nodes.)
- Remove the entries (but not the file) from /usr/es/sbin/cluster/etc/rhosts (on all
nodes)

V3.1.0.1
Student Notebook
Uempty
We're There!
• We’ve configured a two-node hot standby cluster containing:
– A RG(X) with node1 (primary), application, service address, VG
– A non-IP network, persistent address
• We’ve shown how to extend to a two-node mutual takeover cluster
– Add RG(Y) with node2 (primary), application, service address, VG
– Add persistent address
node1 node2
RG(X) X X
Y Y RG(Y)
Each resource group is configured to use IPAT via IP aliasing.

This particular style of cluster (mutual takeover with IPAT) is, by far,
the most common style of HACMP cluster.
Figure 5-52. We're There! QV1203.1
• We can configure hot standby

We've finished configuring a two-node PowerHA cluster with a resource group
operating in a hot standby configuration. This configuration includes a resource group
containing an application server, a service address, and an enhanced concurrent
volume group. Further, for detecting an IP failure, a non-IP network using heartbeat
over disk was added to the configuration as well as a persistent address for
administrative purposes.
• We can configure mutual takeover
In addition, we learned how to extend the hot standby configuration to the very common
mutual takeover configuration. The term “mutual takeover” derives from the fact that
each node is the home node for one resource group and provides fallover (that is,
takeover) services to the other node. This is, without a doubt, the most common style of
PowerHA cluster as it provides a reasonably economical way to protect two separate
applications.
Student Notebook
Review
1. In which of the top level HACMP menu choices is the menu for starting and
stopping cluster nodes?
a. Initialization and Standard Configuration
b. Extended Configuration
c. System Management (C-SPOC)
d. Problem Determination Tools
2. In which of the top level HACMP menu choices is the menu for defining a non-IP
heartbeat network?
3. True or False?
It is possible to configure HACMP faster by having someone help you on the
other node.
4. True or False?
You must always specify exactly which filesystems you want mounted when
you put resources into a resource group.
Figure 5-53. Review QV1203.1

V3.1.0.1
Student Notebook
Uempty
Exercise
Configure PowerHA
Part 1: Application Start and Stop Scripts

Part 2: Configure the cluster
Part 3: Start cluster services
Part 4: Failure testing
Figure 5-54. Exercise QV1203.1
Notes:
Student Notebook
Unit Summary 1 of 2
• The steps to implement an HACMP cluster are:
– Plan
– Configure AIX for HACMP (networks, storage, applications)
– Install HACMP software and reboot
– Configure PowerHA
(topology, resources, resource groups, attributes)
– Synchronize
– Start PowerHA services
– Test
• There are two menu paths used for PowerHA configuration
– Initialization and Standard Configuration
• Two-Node Cluster Configuration Assistant
– Automates process, but you don't see the details
– Simple two-node, hot-standby, one resource group configuration
– Automatic synchronization
• Standard configuration
– Preferred method for most initial cluster configuration, forces you to see the details of your
configuration
– Includes the most commonly used options
– Synchronization must be done manually
– HACMP Cluster Test Tool
– Extended Configuration
• More steps, but provides access to all options
• Required for node persistent address and non-IP network configuration
Figure 5-55. Unit summary 1 of 2 QV1203.1

V3.1.0.1
Student Notebook
Uempty
Unit Summary 2 of 2
• Pick one node as your administration node and make all changes
from that node
• Start HACMP services
– Cluster Management (C-SPOC) -> Manage HACMP Services ->
• Removing a cluster (if you have to start over)
– Stop cluster services
– Remove the cluster
Extended Configuration -> Extended Topology Configuration
-> Configure an HACMP Cluster -> Remove an HACMP Cluster
– Remove the entries (but not the file) from /usr/es/sbin/cluster/etc/rhosts on
all nodes
Student Notebook

V3.1.0.1
Student Notebook
AP Appendix A. Checkpoint solutions

Unit 1 - 1 of 2
Review (1 of 2) Solutions
1. Which of the following are examples of topology

components in PowerHA (select all that apply)?
a. Node
b. Network
c. Service IP label
d. Hard disk drive
2. True or False?
All clusters require shared disk for storage of PowerHA log
files.
3. True or False?
All nodes in an PowerHA cluster must have equivalent
performance characteristics
© Copyright IBM Corp. 2007, 2009 Appendix A. Checkpoint solutions A-1

Student Notebook
Unit 1 - 2 of 2
Review (2 of 2) Solutions
4. True or False?
Resource Groups may be moved from node to node.
5. True or False?
PowerHA/XD is a solution for building geographically
distributed clusters.
6. Which of the following capabilities does PowerHA not
provide (select all that apply)?:
a. Time synchronization.
b. Automatic recovery from node and network adapter
failure.
c. System Administration tasks unique to each node.
Backup and restoration.
d. Fallover of just a single resource group.
A-2 PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

V3.1.0.1
Student Notebook
AP Unit 2- 1 of 4
Review 1 of 4 Solutions
1. How does PowerHA use networks (select all which apply)?
a. Provide client systems with highly available access to the cluster's
applications
b. Detect and diagnose failures
c. Provide cluster status
d. Communicate between cluster nodes
e. Monitor network performance
2. Using information from RSCT, PowerHA only directly handles
three types of failures: Network interface card (NIC) failures,
Node failures, Network failures.
3. True or False?
• Heartbeat packets must be acknowledged or a failure is assumed to have
occurred.
4. True or False?
• Clusters should include a non-IP network.

Student Notebook
Unit 2 - 2 of 4
5. True or False?
• Clusters must always be configured with a private IP network for PowerHA
communication.
6. Which of the following are true statements about communication
interfaces (select all that apply)?
a. Has an IP address assigned to it using the AIX TCP/IP SMIT screens
b. Might have more than one IP address associated with it
c. Sometimes but not always used to communicate with clients
d. Always used to communicate with clients
(Communication interfaces on private IP networks are not intended to be
used by clients.)
7. True or False?
• Persistent node IP labels are not supported for IPAT via IP replacement.
8. True or False?
• Each NIC on each physical IP network on each node is required to have an
IP address on a different logical subnet (unless using heartbeat over IP
alias).
9. True or False?
• A single cluster can use both IPAT via IP aliasing and IPAT via IP
replacement.

V3.1.0.1
Student Notebook
AP Unit 2 - 3 of 4
10. True or False?
aliasing.
11. True or False?
replacement.
192.168.20.2 and 192.168.21.2, then which of the following are
valid service IP addresses if IPAT via IP aliasing is being used?
(select all that apply)
a. (192.168.20.3 and 192.168.20.4) OR (192.168.21.3 and 192.168.21.4)
b. 192.168.20.3 and 192.168.20.4 and 192.168.21.3 and 192.168.21.4
c. 192.168.22.3 and 192.168.22.4
d. 192.168.23.3 and 192.168.24.3

Student Notebook
Unit 2- 4 of 4
192.168.20.2 and 192.168.21.2 then which of the following are
valid service IP addresses if IPAT via IP replacement is being
used (select all that apply)?
a. (192.168.20.3 and 192.168.20.4) OR (192.168.21.3 and 192.168.21.4)
b. 192.168.20.3, 192.168.20.4, 192.168.21.3 and 192.168.21.4
c. 192.168.22.3 and 192.168.22.4
d. 192.168.23.3 and 192.168.24.3
14. True or False?
– Clients are required to exit and restart their application after a fallover.
15. True or False?
– All client systems are potentially directly affected by the ARP cache issue.
16. True or False?
– clinfo must not be run both on the cluster nodes and on the client
systems.

V3.1.0.1
Student Notebook
AP Unit 3 -
Review Solutions
1.True or False
Applications are defined to PowerHA in a configuration file that lists
what binary to use.
2.What policies would be the best to use for a 2-node “active-active” cluster
using IPAT to minimize both applications running on the same node?
a. home, next, never
b. first, next, higher
c. distribution, next, never
d. all, error, never
e. home, next, higher
3.Which type of data should not be placed in private data storage?
a. Application log data
b. License file
c. Configuration files
d. Application binaries
4.True or False
Start and stop scripts which are tested on one node can be counted on
to perform correctly under PowerHA. No further testing is needed.

Student Notebook
Unit 4- 1 of 2
1. What is the first step in implementing a cluster?*
a. Order the hardware
b. Plan the cluster
c. Install AIX and PowerHA
d. Install the applications
e. Take a long nap
*There is some dispute about whether the correct answer is b or e -.
A disconcerting number of clusters are implemented in the order a, b, c, d, e (how can
you possibly order the hardware if you do not yet know what you are going to build?) or
even just a, c, d (cluster implementers who skip step b rarely have time for long naps).
2. True or False?
PowerHA 5.5 is compatible with any version of AIX v5.x or v6.x.
Only AIX 5.3 and 6.1.
3. True or False?
Each cluster node must be rebooted after the PowerHA software is
installed.

V3.1.0.1
Student Notebook
AP Unit 4- 2 of 2
4. Which component detects an adapter failure?
a. Cluster manager
b. RSCT
c. clcomd
d. clinfo
5. Which component provides SNMP information?
a. Cluster manager
b. RSCT
c. clsmuxpd
d. clinfo
6. Which component is required for clstat to work?
a. Cluster manager
b. RSCT
c. clcomd
d. clinfo
7. Which component removes requirement for the /.rhosts file?
a. Cluster manager
b. RSCT
c. clcomd
d. clinfo

Student Notebook
Unit 5-
Review Solutions
1. In which of the top level HACMP menu choices is the menu for starting and
stopping cluster nodes?
2. In which of the top level HACMP menu choices is the menu for defining a non-IP
heartbeat network?
3. True or False?
It is possible to configure HACMP faster by having someone help you on the
other node.
4. True or False?
You must always specify exactly which filesystems you want mounted when
you put resources into a resource group.

V3.1.0.1
Student Notebook
AP Appendix C-
Review Solutions
1. For IPAT via replacement (select all that apply)
a. Each service IP address must be in the same subnet as one of the non-service
addresses
b. Each service IP address must be in the same subnet
c. Each service IP address cannot be in any non-service address subnet
2. True or False?
If the takeover node is not the home node for the resource group and the
resource group does not have a Startup policy of Online Using Distribution
Policy, the service IP address replaces the IP address of a NIC with an IP
address in the same subnet as the subnet of the service IP address.
3. True or False?
In order to use HWAT, you must enable and complete the ALTERNATE
ETHERNET address field in the SMIT devices menu.
4. True or False?
You must stop the cluster in order to change from IPAT via aliasing to IPAT via
replacement.

Student Notebook

V3.1.0.1
Student Notebook
AP Appendix B. HACMP 5.5 release_notes and

README
release_notes
====================================================================
Release Notes for Power HA for AIX (HACMP) Release 5.5 November, 2008
====================================================================
These Release Notes contain the latest information about the

Power HA for AIX software, formerly known as IBM High Availability
Cluster Multiprocessing or HACMP. IBM PowerHA represents a new name
but remains the same premier high availability product. This renaming
better aligns HACMP with the new IBM Power Systems Software initiative.
This document and most of the product documentation continue to refer
to HACMP.
These release notes discuss:
o Enhancements and new features included in this release
o Installation and migration considerations
o Software prerequisites
o Configuration limitations
o Notes on functionality
o Documentation
o Product man pages
o Accessing IBM on the web
o Feedback
======================================================
Enhancements and new features included in this release
======================================================
o Asynchronous mirroring option for HACMP/XD GLVM
© Copyright IBM Corp. 2007, 2009 Appendix B. HACMP 5.5 release_notes and README B-1
Student Notebook
This option lets you take advantage of asynchronous mirroring of

data to a backup site which can reduce the impact on throughput at the
primary site by allowing writes to complete on the primary before
there is acknowledgment from the backup site. This can smooth out
peaks in throughput at the primary with the disadvantage of increased
probability of data divergence after a failure.
For more information on this feature, the planned availability date
and required APARs, refer to the release_notes_xd file.
o Support for additional configurations with HACMP/XD SVC PPRC and VIO
See the release_notes_xd file for more details.
o WebSMIT Gateway
WebSMIT requires an HTTP server to enable a browser to connect
to and manage the HACMP cluster nodes. In prior releases the HTTP
server had to run on one of the cluster nodes which created
security concerns for some customers.
The WebSMIT gateway feature lets you run the HTTP server on a machine
outside of the cluster to be managed by WebSMIT. A simple registration
process is required to specify the cluster nodes to be managed
by the gateway.
o WebSMIT Enterprise View

In conjunction with the new Gateway feature you can now use WebSMIT
to monitor and manage multiple clusters using the new Enterprise view.
A simple registration process lets you define the clusters to
be monitored. WebSMIT displays the status of multiple clusters as icons
which you can select to activate all the WebSMIT status
and management functions for that cluster. The Enterprise view
is ideal for customers who need to manage multiple clusters
across an enterprise from a single management interface.
More information about the new features of WebSMIT is available in

the Administration Guide.
o Support for IPv6 service and persistent labels

IPv6 is the next generation internet protocol and the United States
government has mandated it as a required feature for all new
procurements. In this release you can configure IPv6 addresses
for service labels and persistent labels in a network which has
IPv4 addresses for boot labels. While HACMP cannot support a purely
IPv6 environment, this feature is well suited for customers migrating
their IPv4 environments to IPv6 where both v4 and v6 addresses will
be used at the same time.
B-2 PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

V3.1.0.1
Student Notebook
AP IPv6 will require AIX 6.1 on all nodes. Please see the section
on Installation prerequisites for specific information.
o Miscellaneous improvements to functionality and RAS

Several improvements are included in this release:
- COD support for POWER 5 and POWER 6 architectures. This support
is limited to the subset of capabilities that are common with
POWER 4.
- Removal of “telinit” handling. HACMP no longer manipulates the
AIX run level through telinit. This is no longer required with
later versions of AIX and its removal simplifies and speeds up IPAT.
- Miscellaneous improvements in the WebSMIT GUI.
- Specify resource groups directly from SMIT panels for managing
volume groups with HACMP. Associating the resource group used to
require a second, mandatory step.
- RAS improvements for tracing, remote logging and memory usage by
the cluster verification process.
- Name size. When you configure a cluster you specify names for
entities like the nodes, resource groups and the cluster itself.
With this release you can now use names that are 64 characters
in length, compared to 32 characters for prior releases.
=========================================
Installation and Migration Considerations
=========================================
o When migrating from a prior release you have several options to choose
from based on your overall upgrade plan and downtime requirements.
- “Static” migration. Here you stop all nodes in the cluster

and update all nodes before restarting. This requires an outage
of your application but is most conducive if you are planning
additional updates like moving to a new AIX level or installing
new hardware (which requires an outage anyway).
- “Snapshot” migration. Installing the new version of HACMP will

automatically migrate any existing configuration, however, you
may want to take a snapshot of the configuration then apply that
snapshot later. This is a handy option if you are updating AIX
with the “complete overwrite” option.
- “Rolling” migration. You can reduce the amount of downtime required

for your upgrade by using the rolling migration option. Here you stop
each node with the “Move resource groups” option, update that node,
Student Notebook
restart, and repeat for all nodes in the cluster. Although there
is an outage while the application moves between nodes, this is less
disruptive than the options above. If you are migrating from HACMP 5.3
or later you can use rolling migration.
- “Non-disruptive upgrade” migration. If you are migrating from

HACMP 5.4 or 5.4.1 and have no other updates (AIX, RSCT, application
software, etc) planned, you can use the non-disruptive upgrade option.
Here you stop each node with the “Unmanage resource groups” option,
install the new software, restart the node, and repeat for all nodes
in the cluster. Although there is no application outage, this option
is really only useful if you are updating the HACMP software and
nothing else.
HACMP supports static and snapshot migrations from release 5.3

and forward. Migration from releases prior to 5.3 may also be
possible with manual intervention. See the man page for
clconvert_snapshot for more information.
HACMP supports rolling migration from HACMP 5.3 and above.
HACMP supports non-disruptive upgrade from version 5.4 and 5.4.1.
==============================================
Software prerequisites
==============================================
When using AIX Version 5.3:

AIX V5.3 with Technology Level 9
RSCT 2.4.10.0
When using AIX Version 6.1:

AIX V6.1 with Technology Level 2 Service Pack 1 and APAR IZ31208
RSCT 2.5.2.0
The new Asynchronous Replication option for GLVM will require:

AIX V6.1 with Technology Level 2 Service Pack 1 and APAR IZ31208
PowerHA V5.5 with APAR IZ31205 and IZ31207
RSCT 2.5.2.0
==============================================
Configuration Limitations
==============================================

V3.1.0.1
Student Notebook
AP HACMP is a flexible and extensible solution for your high availability

needs. You can configure any size cluster up to:
Maximum nodes per cluster: 32

Maximum number of sites: 2
Maximum resource groups: 64
======================
Notes on Functionality
======================
-------------------------------------------------------------------------
Enable grace periods and restart nfsd after installing cluster.es.nfs.rte
-------------------------------------------------------------------------
This note applies only to 64 bit systems. You may ignore this note if all
of your cluster nodes are 32 bit.
The NFS daemon nfsd must be restarted on each cluster node with grace periods
enabled after installing cluster.es.nfs.rte before configuring NFSv4 exports.
This step is required otherwise NFSv4 exports will fail to export with the
misleading error message
exportfs: <export_path>: No such file or directory
The following commands enable grace periods and restart the NFS daemon.
chnfs -I -g on -x
stopsrc -s nfsd
startsrc -s nfsd
Please note that this will impact the availability of all exported filesystems
on the machine, therefore the best time to perform this step is when all
resource groups with NFS exports are offline or failed over to another node
in the resource group.
o Fast failure detection
- The fast failure detection function is not supported on SSA disks.

SSA disks are not supported on AIX 6.1.
o SNMP
- HACMP updates SNMP during installation. Therefore HACMP stops
Student Notebook
and starts the SNMP daemon during installation and deinstallation.

This will require that you restart any SNMP client applications
that communicate with the node on which you are installing HACMP.
o COD
- When changing a resource group and reducing the number of cpu’s or memory,
the change does not take effect immediately when the dare event runs, it
only affects the next failover (where CuOD is used).
o WPAR naming
- Due to a restriction in AIX, a WPAR-enabled resource group cannot have a

name longer than 25 chars.
o HAGEO
- HACMP/XD IP-based mirroring known as HAGEO has been replaced by

PowerHA/XD GLVM V5.5. GLVM offers all of the features of the former
HAGEO technology while being much easier to configure, more reliable,
and better integrated with PowerHA as well as the native AIX LVM.
Customers currently using HAGEO are recommended to develop a
strategy for moving from HAGEO to GLVM.
Note also that HAGEO is not supported on AIX 6.1.
For more information refer to the release_notes_xd file.
o eRCMF
- HACMP/XD no longer includes the filesets to support eRCMF.

For more information refer to the release_notes_xd file.
o Event emulation
- Event emulation is no longer supported.
o Oracle Smart Assist and AIX 6.1
- Oracle Application Server 10gR1 (9.0.4) is not supported on AIX 6.1
o Problems during snapshot apply
- When applying a snapshot to a system with an existing configuration you

may see messages related to “migcheck” and “cl_connect”. If this occurs
you will need to restart the clcomd subsystem with:

V3.1.0.1
Student Notebook
AP stopsrc -s clcomdES
startsrc -s clcomdES
o WebSMIT
- The new gateway function cannot be used with nodes running prior releases
of HACMP. This support will require a change in a service pack for the
earlier releases.
- When registering multiple clusters for use with the Enterprise view, each
cluster must have a unique cluster ID.
- The gateway can only register a configured cluster - you cannot use
WebSMIT through the gateway to configure a cluster from scratch (though
you can do so with WebSMIT without the gateway).
IBM intends to lift these restrictions in upcoming service packs.
o XD notes on functionality can be found in the release_notes_xd file
o HACMP cluster-wide C-SPOC password administration does not

support use of the feature allowing passwords longer than 8
characters which became available with the Loadable Password
Algorithm as part of AIX 53 TL 7.
===================================
Documentation
===================================
HACMP for AIX: Concepts and

Facilities Guide SC23-4864
HACMP for AIX: Planning Guide SC23-4861
HACMP for AIX: Installation Guide SC23-5209
HACMP for AIX: Administration Guide SC23-4862
HACMP for AIX: Troubleshooting Guide SC23-5177
HACMP for AIX: Programming Client

Applications SC23-4865
HACMP for AIX: Master Glossary SC23-4867
For PowerHA/XD GLVM customers:
Student Notebook
HACMP/XD GLVM Planning and Administration

Guide SA23-1338
For HACMP/XD Metro Mirror customers:

HACMP/XD: Metro Mirror Planning and
Administration Guide SC23-4863
For PowerHA Smart Assist customers:
HACMP for AIX: Smart Assist for WebSphere

User’s Guide SC23-4877
HACMP for AIX: Smart Assist for Oracle SC23-5178
HACMP for AIX: Smart Assist for DB2 SC23-5179
HACMP for AIX: Smart Assist Developer’s Guide SC23-5210
Online Documentation:
The HACMP documentation is available directly on the product media and at

http://publib.boulder.ibm.com/infocenter/systems/scope/aix/index.jsp
------------------------------------------------------
Viewing and installing the documentation files
------------------------------------------------------
You can view the PDF documentation without installing the product.
Simply copy the PDFs to your desired location or view them directly from
the media.
If you prefer you can install the documentation by following these steps:
1. At the command line, enter: smit install_selectable_all

SMIT asks for the input device/directory for software.
2. Select the CD ROM drive from the picklist and press Enter.
3. On the next SMIT screen with the cursor on “Software to Install”,

press the F4 key.
4. SMIT lists the image cluster.doc.en_US fileset with its

subdirectories:
The individual lines under the image name

V3.1.0.1
Student Notebook
AP (e.g., cluster.doc.en_US.es.pdf) are the filesets that can be

installed.
Image cluster.doc.en_US.es
--------------------------
cluster.doc.en_US.es.html HACMP Web-based HTML
Documentation - U.S. English
cluster.doc.en_US.es.pdf HACMP PDF
Image cluster.doc.en_US.glvm
----------------------------
cluster.doc.en_US.glvm.html HACMP GLVM HTML Documentation
- U.S. English
cluster.doc.en_US.glvm.pdf HACMP GLVM PDF Documentation
- U.S. English
Image cluster.doc.en_US.pprc
----------------------------
cluster.doc.en_US.pprc.html PPRC Web-based HTML
cluster.doc.en_US.pprc.pdf PPRC PDF Documentation
- U.S. English
Image cluster.doc.en_US.assist
------------------------------
cluster.doc.en_US.assist.db2.html HACMP Smart Assist for
DB2 HTML Documentation
- U.S. English
cluster.doc.en_US.assist.db2.pdf HACMP Smart Assist for
DB2 PDF Documentation
- U.S. English
cluster.doc.en_US.assist.oracle.html HACMP Smart Assist for
Oracle HTML Documentation
- U.S. English
cluster.doc.en_US.assist.oracle.pdf HACMP Smart Assist for
Oracle PDF Documentation
- U.S. English
cluster.doc.en_US.assist.websphere.html HACMP Smart Assist for
WebSphere HTML
Documentation
- U.S. English
cluster.doc.en_US.assist.websphere.pdf HACMP Smart Assist for
WebSphere PDF
Student Notebook
Documentation
- U.S. English
Once you install the documentation, store it on a server that is

accessible through the Internet. You can view the documentation in
the Mozilla Firefox browser.
NOTE: Installing all of the documentation requires about

46 MB of space in the /usr filesystem.
(PDF files = 26 MB, HTML files = 20 MB.)
5. Select all filesets that you wish to install and execute the
command.
The documentation is installed in the following directory:
/usr/share/man/info/en_US/cluster/HAES
o HTML documentation
- The installation media currently contains back level versions of

the HTML documentation. A future service pack will update these filesets
to the latest versions of the HTML documentation.
==================
Product Man Pages
==================
Man pages for HACMP commands and utilities are installed in the
following directory:
/usr/share/man/cat1
Use
man [command-name]
to access the man page.
========================
Accessing IBM on the Web
========================
o Access IBM’s home page at:

V3.1.0.1
Student Notebook
AP http://www.ibm.com
HACMP main page:

http://www.ibm.com/systems/power/software/availability/aix/index.html
Miscellaneous pages:
http://www.ibm.com/systems/p/ha/faq3.html
http://www.ibm.com/systems/p/library/hacmp_docs.html
IBM publishes Redbooks for a wide range of topics including HACMP

http://www.redbooks.ibm.com
========
Feedback
========
IBM welcomes your comments. You can send comments and questions via e-mail to:
hafeedbk@us.ibm.com
README5.5.0.UPDATE
===========================================
README for HACMP Version 5.5 Service Pack 2
===========================================
Content of this README:
o APAR IZ31205: Asynchronous mirroring option for GLVM

o APAR IZ49934: varyonvg failed to bring up the async Vg on remote site
o APAR IZ41204: Enabling Internet MIB tree for clstat or cldump to work
o APAR IZ37766: Loadable Password Algorithm
o APAR IZ01331: New options for netmon.cf for VIO environment
o APAR IZ43939: NON EXISTING SMIT PANELS FOR WPARS REFERENCED IN
ADMIN GUIDE
o APAR IZ01809: Multi-node disk heartbeat
o APAR IZ49879: Changes in NFS mount default options
o Dealing with an Aynchronous I/O cache logical volume failure
o Accessing IBM on the Web
o Feedback
====================================================================
o APAR IZ31205: Asynchronous mirroring option for GLVM
Student Notebook
====================================================================
Geographic Logical Volume Manager or GLVM provides data replication over

IP networks for use in disaster recovery solutions. GLVM supports two
modes for replication:
Synchronous replication
With this mode, application writes do not complete until the data
has been written at both sites. Network bandwidth and latency
affect the application performance.
Asynchronous replication
The asynchronous replication option decouples the application
performance from the network. Remote writes are cached locally
and completed asynchronously, which can hide peeks in load
(until the cache fills). However, data can be lost on unplanned
site fallover.
More information can be found in the

Geographic LVM: Planning and administration guide
Asynchronous replication relies on a new feature in AIX called mirror pools.

Mirror pools are available on AIX 61, therefore the asynchronous option
is only available with AIX 61.
Customers should order the following APARs to receive all the required
updates for this new feature: IZ31205 and IZ31207
====================================================================
o APAR IZ49934: varyonvg failed to bring up the async Vg on remote site
====================================================================
If there are stale partitions on the recovery node, customers may see
varyonvg command failures in hacmp.out. To address this problem,
retreive and install APAR IZ49934. This applies to Async GLVM (and
hence AIX 61) only.
====================================================================
o APAR IZ41204: Enabling Internet MIB tree for clstat or cldump to work
====================================================================
clstat or cldump will not start if the internet MIB tree is not
enabled in snmpdv3.conf file. This behavior is usually seen in
AIX 6.1 onwards where this internet MIB entry was intentionally
disabled as a security issue. This internet MIB entry is required to

V3.1.0.1
Student Notebook
AP view/resolve risc6000clsmuxpd (1.3.6.1.4.1.2.3.1.2.1.5) MIB sub tree

which is used by clstat or cldump functionality.
There are two ways to enable this MIB sub tree(risc6000clsmuxpd) they
are:
1) Enable the main internet MIB entry by adding this line in
/etc/snmpdv3.conf file
VACM_VIEW defaultView internet - included -
But doing so is not advisable as it unlocks the entire MIB tree
2) Enable only the MIB sub tree for risc6000clsmuxpd without enabling
the main MIB tree by adding this line in /etc/snmpdv3.conf file
VACM_VIEW defaultView 1.3.6.1.4.1.2.3.1.2.1.5 - included -
Note: After enabling the MIB entry above snmp daemon must be restarted
with the following commands as shown below:
1) stopsrc -s snmpd
2) startsrc -s snmpd
After snmp is restarted leave the daemon running for about two minutes
before attempting to start clstat or cldump.
====================================================================
o APAR IZ37766: Loadable Password Algorithm
====================================================================
HACMP cluster-wide C-SPOC password administration does not support use
of the feature allowing passwords longer than 8 characters which became
available with the Loadable Password Algorithm as part of AIX 53 TL 7.
====================================================================
o APAR IZ01331: New options for netmon.cf for VIO environment
====================================================================
HACMP customers using VIO within their clusters have been

experiencing problems with specific scenarios where an
entire CEC is unplugged from the network, but the HACMP
node within does not detect a local adapter down event,
because traffic being passed between the VIO clients
looks like normal external traffic from the perspective
of the LPAR’s OS.
There is already a restriction against two HACMP nodes in

the same cluster using the same VIO server, because this
would mean heartbeats can be passed between the nodes
via the server even when no real network connectivity
Student Notebook
exists. The problem addressed by this APAR is not the

same as that issue, although there are similarities.
In HACMP, heartbeating is used as a reliable means of

monitoring an adapter’s state over a long period of time.
When heartbeating is broken, a decision has to be made as
to whether the local adapter has gone bad, or the neighbor
(or something between them) has a problem.
The local node only needs to take action if the local
adapter is the problem; if its own adapter is good,
then we assume it is still reachable by other clients
regardless of the neighbor’s state (the neighbor is
responsible for acting on its local adapters failures).
This decision (local vs remote bad) is made based on

whether any network traffic can be seen on the local
adapter, using the inbound byte count of the interface.
Where VIO is involved, this test becomes unreliable since
there is no way to distinguish whether inbound traffic
came in from the VIO server’s connection to the outside
world, or just from a neighboring VIO client.
(This is a design point of VIO that its virtual adapters
be indistinguishable to the LPAR from a real adapter).
The netmon.cf file can be configured to help avoid false adapter death
events, especially in single adapter networks. RSCT will attempt to ping
the addresses in the file in an attempt to differentiate between a local
adapter failure and a network failure. In prior releases, all addresses
were used, regardless of the adapter which was considered to have failed
or which address responded to the ping request.
This change allows customers to declare that a given adapter

should only to be considered up if it can ping a set of
specified targets.
IMPORTANT: For this fix to be effective, the customer

--------- *must* select targets that are outside the
VIO environment, and not reachable simply
by hopping from one VIO server to another.
NOTE: Neither HACMP nor RSCT will be able
to detect if this restriction is violated.
Keep the single-point-of-failure rule in mind

when selecting targets; do not us targets that

V3.1.0.1
Student Notebook
AP are all on the same physical machine, and do

not make all your targets adapters from the
same HACMP cluster (otherwise any given node
in that cluster cannot keep its adapters up
when it is the only one powered on).
Some good choices for targets are name servers

and gateways, or reliable external IP
addresses that will respond to a ping. These
targets must be maintained through changes in
the enterprise network infrastructure.
How to use this fix:

-------------------
Up to 32 different targets can be provided for each
interface. If *any* given target is pingable, the adapter
will be considered up. Targets are specified using the
existing netmon.cf configuration file located in
/usr/es/sbin/cluster/
using this new format:
!REQD <owner> <target>
Parameters:
----------
!REQD : An explicit string; it *must* be at the
beginning of the line (no leading spaces).
<owner> : The interface this line is intended to be

used by; that is, the code monitoring the
adapter specified here will determine its
own up/down status by whether it can ping
any of the targets (below) specified in
these lines.
The owner can be specified as a hostname, IP
address, or interface name. In the case of
hostname or IP address, it *must* refer to
the boot name/IP (no service aliases).
In the case of a hostname, it must be
resolvable to an IP address or the line will
be ignored.
The string “!ALL” will specify all adapters.
Student Notebook
<target> : The IP address or hostname you want the

owner to try to ping.
As with normal netmon.cf entries, a hostname
target must be resolvable to an IP address
in order to be usable.
The “traditional” format of the netmon.cf file has not

changed -- one hostname or IP address per line.
(Any adapters not matching one of the “!REQD” lines
will still use the traditional lines as they always
have; as extra targets for pinging in addition to
known local or defined adapters, with the intent of
increasing the inbound byte count of the interface.)
Any adapter matching one or more “!REQD” lines (as the

owner) will ignore any traditional lines.
Order from one line to the other is unimportant; you can

mix “!REQD” lines with traditional ones in any way.
However, if using a full 32 traditional lines, do not put
them all at the very beginning of the file -- otherwise
each adapter will read in all the traditional lines
(since those lines apply to any adapter by default), stop
at 32 and quit reading the file there. The same problem
is not true in reverse, as “!REQD” lines which do not
apply to a given adapter will be skipped over and not
count toward the 32 maximum.
Comments (lines beginning with “#”) are allowed on or

between lines and will be ignored.
If more than 32 “!REQD” lines are specified for the same

owner, any extra will simply be ignored (just as with
traditional lines).
Some brief examples just to explain the syntax:

----------------------------------------------
(effective netmon.cf files should not be this small)
9.12.4.11
!REQD en2 100.12.7.9
9.12.4.13
!REQD en2 100.12.7.10

V3.1.0.1
Student Notebook
AP
-- most adapters will use netmon in the traditional
manner, pinging 9.12.4.11 and 9.12.4.13 along with
other local adapters or known remote adapters, and
will only care about the interface’s inbound byte
count for results.
-- interface en2 will only be considered up if it
can ping either 100.12.7.9 or 100.12.7.10.
!REQD host1.ibm 100.12.7.9

!REQD host1.ibm host4.ibm
!REQD 100.12.7.20 100.12.7.10
!REQD 100.12.7.20 host5.ibm
-- The adapter owning “host1.ibm” will only be

considered up if it can ping 100.12.7.9 or
whatever host4.ibm resolves to.
-- The adapter owning 100.12.7.20 will only be
considered up if it can ping 100.12.7.10 or
whatever host5.ibm resolves to.
-- It is possible that 100.12.7.20 is the IP address
for “host1.ibm” (we can’t tell from this example);
if that is true, then all four targets belong to
that adapter.
!REQD !ALL 100.12.7.9

!REQD !ALL 110.12.7.9
!REQD !ALL 111.100.1.10
!REQD en1 9.12.11.10
-- All adapters will be considered up only if they can

ping 100.12.7.9, 110.12.7.9, or 111.100.1.10.
-- en1 has one additional target: 9.12.11.10
-- (In this example having any traditional lines would
be pointless, since all of the adapters have been
defined to use the new method.)
Important notes:
---------------
This APAR will only take effect *if* valid updates in
this new format are made to the netmon.cf file. As long
as you only use the netmon.cf file in the traditional
manner (or do not use it at all), then you can safely
Student Notebook
apply this APAR without changing your cluster’s behavior

in any way.
Similarly, any interfaces which are not included as an

“owner” of one of the “!REQD” netmon.cf lines will
continue to behave in the old manner, even if you are
using this new function for other interfaces.
This fix does *not* change heartbeating behavior itself in

any way; it only changes how the decision is made as to
whether a local adapter is up or down. So this new logic
will be used upon startup (before heartbeating rings are
formed), during heartbeat failure (when contact with a
neighbor is initially lost), or during periods when
heartbeating is not possible (such as when a node is the
only one up in the cluster).
WARNING: It is *not* recommended that any customers use

------- this new function unless they absolutely have
to because of their VIO environment.
Why: Invoking this fix changes the definition of a
--- “good” adapter from:
* Am I able to receive *any* network traffic?
to:
* Can I successfully ping certain addresses?
(regardless of how much traffic I can see)
This fact alone makes it inherently more likely

for an adapter to be falsely considered down,
since the second definition is more restrictive.
For this same reason, customers who find they must take
advantage of this new functionality are encouraged to be
as generous as possible with the number of targets they
provide for each interface (up to the limit).
====================================================================
=
o APAR IZ43939: NON EXISTING SMIT PANELS FOR WPARS REFERENCED IN ADMIN
GUIDE
====================================================================
=
There is WPAR related information in the Administration Guide under the

V3.1.0.1
Student Notebook
AP topic “Using HACMP Workload Partitions” which runs from page 503 to 507.
This information is inaccurate and should be ignored.
Information about WPARs can be found in “Running a resource group in

an AIX WPAR” on page 134.
====================================================================
Multi-node disk heartbeat.
====================================================================
Release Notes for Multi-node disk heartbeat:
o New features
o Prerequisite Software
o MNDHB Configuration and Usage
o MNDHB Limitations
o Best Practices
==========================================
New features
==========================================
o Multi-Node Disk Heartbeat and Disk Fencing
This release provides capabilities for HACMP to detect and react to a

partitioned cluster.
There are 2 concepts:
Multi-node Disk Heartbeat (MNDHB)
Like regular disk heartbeat networks, the disk subsystem
is used as the media for exchanging heartbeat messages.
Multi-node disk heartbeat lets you configure network access
for multiple nodes instead of the simple point to point
network available using regular disk heartbeat. The
following sections describe the requirements and details
for configuring a multi-node disk heartbeat.
Disk Fencing
When a multi-node disk heartbeat network is configured,
HACMP performs additional checks during node_down events.
Each node connected to the network will check its access
to the disk(s) defined for the network. If a node has
Student Notebook
access to less than a quorum (one more than half) of

the disks, it will exercise a configurable policy to
either shut down the node, fence it from the disks,
or simply run a notification event.
====================================
Installation Notes
====================================
In order to configure and use the multi-node disk heartbeat capability

you must first install the required software on all cluster nodes. The
AIX, RSCT and HACMP filesets and PTFs should be installed using the
standard procedures for your installation.
When the requisite software is installed on all nodes you are now ready
to configure the multi-node disk heartbeat network(s). As with any
configuration change made with HACMP, you will do the configuration from a
single cluster node then verify and synchronize the configuration across the
cluster. The new network will become active after the synchronization.
==============================================
MNDHB Configuration and Usage
==============================================
o Overview
The MNDHB capability allows multiple nodes in a cluster to exchange

heartbeats through a single logical volume located on a single physical
disk. HACMP also supports “traditional” disk heartbeat which provide a
point to point network connection between nodes. Traditional and
multi-node disk heartbeat networks are configured and run independently
from each other.
To begin using multi-node disk heartbeat and disk fencing you will
1) Plan the configuration including the disks, volume groups, and
networks.
2) Configure the networks using the MNDHB smit panels.
3) Configure the disk fencing options.
4) Verify and Synchronize the Changes.
5) View the configuration and monitor the network status.
Planning for MNDHB

=====================================

V3.1.0.1
Student Notebook
AP Heartbeat messages are exchanged between nodes by reading and writing

messages from a shared logical volume. You must plan a shared logical volume
for each multi-node disk heartbeat network. Logical volumes used for MNDHB
must:
1) be at least 32 MB and must
2) be contained on a single physical disk
3) not use LVM mirroring, MWC or “write verify”
4) reside on a single physical disk which is accessible from all
cluster nodes.
During the configuration process HACMP will identify all the available
logical volumes which meet these requirements (and tell you which
volumes were found which did not meet the requirements and why).
If you do not have any logical volume available which meet these
requirements, the MNDHB smit options can be used to create them
for you. You can also use the CSPOC smit options to create the logical
volumes, however it is recommended to use the MNDHB smit paths to
ensure the logical volumes are created correctly.
Using the MNDHB smit paths will also create or modify the volume groups
and resource groups which will contain the logical volume used for the
MNDHB networks. You can manage the volume groups and resource groups
using the standard smit paths, however, using the MNDHB smit paths
will ensure all the correct configuration options are used.
Configuring MNDHB networks

==========================
To configure and use MNDHB networks use the SMIT fastpath: cl_manage_mndhb
These menus are also available by following these smit paths:
smit hacmp
HACMP Concurrent Logical Volume Management
Concurrent Volume Groups (smitty cl_convg)
smit hacmp
Student Notebook
Note that multi-node disk heartbeat networks cannot be managed using the
traditional HACMP smit paths for adding or changing networks or adapters -
only the MNDHB smit paths should be used for managing these networks.
The initial smit menu consists of these options which are described
in detail below.
Use an existing Logical Volume and Volume Group for Multi-Node Disk HB
Create a new Volume Group and Logical Volume for Multi-Node Disk HB
Add a Concurrent Logical Volume for Multi-Node Disk Heartbeat
Show Volume Groups in use for Multi-Node Disk Heartbeat
Stop using a Volume Group for Multi-Node Disk Heartbeat
Configure failure action for Multi-Node Disk Heartbeat Volume Groups
Use an existing Logical Volume and Volume Group for Multi-Node Disk HB
SMIT fastpath: cl_enable_mndhb_vg
====================================================================
This menu option allows a user to use an existing Volume Group and
Logical Volume for a new MNDHB network. For example, an Oracle
configuration could have additional shared Logical Volumes in the
Volume Group that contains the Oracle voting disks, which could be used
for MNDHB.
HACMP will check the existing configuration and present a list of resource
groups that have concurrent volume groups associated with them.
+--------------------------------------------------------------------------
+
¦ Select a Volume Group for Multi-Node Disk Heartbeat ¦
¦ ¦
¦ Move cursor to desired item and press Enter. Use arrow keys to scroll.
¦
¦ ¦
¦ #Resource Group Volume Group ¦
¦ shared_storage ccvg ¦
¦ ¦
¦ Esc+8=Image Esc+0=Exit Enter=Do ¦

V3.1.0.1
Student Notebook
AP
+--------------------------------------------------------------------------
+
Choose the Volume Group you wish to add the Logical Volume to.
You will be presented with a picklist containing the logical volumes in
that Volume Group which are candidates for MNDHB. If any volumes are found
which do not meet requirements, they will be listed along with the reason
they are not valid candidates.
+--------------------------------------------------------------------------
+
¦ Select Logical Volume to use for Heartbeat ¦
¦ ¦
¦
¦ ¦
¦ [MORE...39] ¦
¦ # logical volume data_lv1 allowed to span more than one disk (-u
¦
¦ # logical volume data_lv2 allowed to span more than one disk (-u s
¦
¦ # logical volume mnhb_lv2 is already used for multi-node disk hear
¦
¦ # logical volume mnhb_lv3 is already used for multi-node disk hear
¦
¦ mndhb_lv1 ¦
¦ [BOTTOM] ¦
¦ ¦
+--------------------------------------------------------------------------
+
After selecting the logical volume you will proceed to the next menu:
Use an existing Volume Group for Multi-Node Disk Heartbeat

Student Notebook
[Entry Fields]
Existing Logical Volume to reserve for heartbeat [mndhb_lv_02] +
VOLUME GROUP name mndhb_vg_01
Network Name [net_diskhbmulti_02] +
Resource Group Name rg_diskhbmulti_02 +
Node Names racb1,racb2,racb3,rac>
The default names can be changed to suite your naming convention.
After configuring the Logical Volume to use for MNDHB, you must
perform a verification and synchronization operation to activate the new
MNDHB.
Create a new Volume Group and Logical Volume for Multi-Node Disk Heartbeat
SMIT fastpath: cl_add_mndhb_vg
====================================================================
If you do not already have a volume group and shared logical volume for use
with MNDHB, this menu option can used to create the MNDHB network and the
associated volume group and resource group configuration.
The first step is to choose a physical volume from the pick list:
+--------------------------------------------------------------------------
+
¦ Physical Volume to reserve for Heartbeat ¦
¦ ¦
¦ ¦
¦ [TOP] ¦
¦ 00c1890c57280da0 ( hdisk17 on node racc2 ) ¦

¦ 00c1890c57284929 ( hdisk18 on node racc2 ) ¦
¦ [MORE...72] ¦
¦ ¦

V3.1.0.1
Student Notebook
AP ¦ F1=Help F2=Refresh F3=Cancel ¦

+--------------------------------------------------------------------------
+
Note that the same physical disk should appear on all nodes in the cluster.
Pick one instance of the physical disk you would like to use for
MNDHB. You will then be presented with an options page:
Create a new Volume Group for Multi-Node Disk Heartbeat

[Entry Fields]
Volume Group Name [mndhb_vg_06] +
Volume group MAJOR NUMBER [49] +#
PVID for Logical Volume 00c1890c57284929
Logical Volume Name [mndhb_lv_010] +
Physical partition SIZE in megabytes 4 +
Node Names racc1,racc2,racc3>
Resource Group Name [rg_diskhbmulti_04] +
Warning:
Changing the volume group major number may result
in the command being unable to execute
successfully on a node that does not have the
major number currently available. Please check
for a commonly available major number on all nodes
before changing this setting.
In most cases you may keep the default parameters. The Resource Group,
Volume Group and Logical Volume names generated are unique across the
cluster, however you can change them to conform to your naming convention.
Volume Group, Logical Volume, and network names must be unique and
not already in use. You can use an existing concurrent resource group
or let HACMP create a new one for you.
This operation will create a new concurrent Volume Group (and resource
Student Notebook
group, if you did not select an existing group). The Volume Group will
be imported on all nodes and a new Logical Volume will be created within
that Volume Group.
For example, after creating the MNDHB network ‘net_diskhbmulti_02’ as shown

above, you should see the following in the “Change/Show Resources and
Attributes for a Resource Group” smit path.
Change/Show All Resources and Attributes for a Resource Group

[Entry Fields]
Resource Group Name rg_diskhbmulti_04
Participating Nodes (Default Node Priority) racc1 racc2 racc3>
Startup Policy Online On All Available>

Fallover Policy Bring Offline (On Err>
Fallback Policy Never Fallback
Concurrent Volume Groups [mndhb_vg_06] +

Use forced varyon of volume groups, if necessary false +
Automatically Import Volume Groups false +
Note: After making these changes you must perform a verification and
synchronization operation to activate the new MNDHB.

SMIT fastpath: cl_add_mndhb_lv
=========================================================
This menu option is used to add a logical volume and MNDHB network to an
existing Volume Group. The new MNDHB network can be added to a new
physical volume (which is added to the existing Volume Group), or can
be created on an existing physical volume in the Volume Group as long
as no other MNDHB networks are using that physical volume.
A picklist will let you select the Volume Group you wish to use:
+--------------------------------------------------------------------------
+

V3.1.0.1
Student Notebook
AP ¦ Select a Volume Group for Multi-Node Disk Heartbeat ¦

¦ ¦
¦
¦ ¦
¦ rg_diskhbmulti_02 mndhb_vg_01 ¦
¦ ¦
+--------------------------------------------------------------------------
+
Note that there can be multiple Volume Groups associated with a single
Resource Group. After selecting the Volume Group you will be presented
with the following menu:

[Entry Fields]
*Physical Volume name +
Logical Volume Name [] +
Logical Volume Label [] +
Volume Group name mndhb_vg_01
Use F4 to generate a list of candidate physical volumes:
+--------------------------------------------------------------------------
+
¦ Physical Volume name ¦
Student Notebook
¦ ¦
¦
¦ ¦
¦ 00c1890c57284929 ( hdisk18 on all cluster nodes ) ¦
¦ 00c1890c57285a08 ( hdisk19 on all cluster nodes ) ¦
¦ 00c1890c5728836e ( hdisk20 on all cluster nodes ) ¦
¦ ¦

+--------------------------------------------------------------------------
+
Select the physical volume you wish to use, then choose a name for the new
Logical Volume. In this example assume we selected hdisk18, and the new
logical volume name is mndhb_lv_05:

[Entry Fields]
*Physical Volume name 00c1890c57284929 +
Logical Volume Name [mndhb_lv_05] +
Logical Volume Label [] +
Volume Group name mndhb_vg_01
Note: After adding the new Logical Volume for MNDHB you must perform
a verification and synchronization operation to activate the new MNDHB.

SMIT fastpath: cl_disable_mndhb_vg
======================================
This menu option is used to stop using a Logical Volume for MNDHB.

V3.1.0.1
Student Notebook
AP A picklist of existing MNDHB networks, including the associated

Volume Group and Resource Group will be displayed:
+--------------------------------------------------------------------------
+
¦ Select Multi-Node Disk Heartbeat Network to Remove ¦
¦ ¦
¦ ¦
¦ net_diskhbmulti_01 (using mndhb_vg_01 in group rg_diskhbmulti_02)
¦
¦
¦
¦
¦
¦
¦ ¦
+--------------------------------------------------------------------------
+
Select the MNDHB network you wish deactivate and you will be presented
with a confirmation screen:

[Entry Fields]
Node Names racc1 racc2 racc3>
VOLUME GROUP name mndhb_vg_03
Resource Group Name rg_diskhbmulti_02
Network Name net_diskhbmulti_08
Student Notebook
Note that if HACMP cluster services are stopped, there will be an

additional option to remove the underlying Logical Volume associated
with the MNDHB. This option is not available while cluster services are
running.
After successfully removing the MNDHB network, you must perform a

verification and synchronization operation to complete the deactivation
of the MNDHB.

SMIT fastpath: cl_disable_mndhb_vg
=======================================================
When MNDHB networks are configured, HACMP performs additional processing

during node_up and node_down events. A separate “failure action” can be
configured for each Volume Group that contains Logical Volumes that are
used for MNDHB networks.
The failure action determines what action the local HACMP software takes
should it determine that the node has lost access to a quorum (one more
than half) of the volumes in a Volume Group that contains the MNDHB
networks.
For example, if a Volume Group has 30 physical volumes, and 3 of those

volumes contain MNDHB networks, and the node loses access to 2 of those
volumes, the failure action will be executed.
There are 4 possible failure actions:
1) Fence the Node (default action)

Forces the Volume Group off-line through an AIX LVM force off option.
Note that this minimal action is executed as a part of all other
failure actions.
2) Halt the Node
Initiates an immediate halt of AIX on the local node
3) Bring the Resource Group(s) Offline
Stops cluster services on the local node with the ‘graceful’ option
(releases all resources associated with the volume group).
4) Move the Resource Group
Initiates a “rg_move offline” command for the owning resource group.
------------------------------------------------------------------------
** IMPORTANT **

V3.1.0.1
Student Notebook
AP ------------------------------------------------------------------------
The recommended failure action in an Oracle RAC environment is
“Halt the node”. This is not the default action, and must be
explicitly set. See the section on monitoring and displaying
the MNDHB configuration to view the current failure action setting
for each Volume Group.
------------------------------------------------------------------------
When changing the failure action, a picklist of all Volume Groups that
contain Logical Volumes used for MNDHB will be presented:
+--------------------------------------------------------------------------
+
¦ Select Multi-Node Disk Heartbeat Volume Group ¦
¦ ¦
¦ ¦
¦ mndhb_vg_01 ¦
¦ # (Used in network(s) net_diskhbmulti_01 net_diskhbmulti_03 )
¦
¦ mndhb_vg_02 ¦
¦ # (Used in network(s) net_diskhbmulti_07 ) ¦
¦ mndhb_vg_03 ¦
¦ # (Used in network(s) net_diskhbmulti_08 ) ¦
¦ mndhb_vg_04 ¦
¦ # (Used in network(s) net_diskhbmulti_05 net_diskhbmulti_06 )
¦
¦ ¦
+--------------------------------------------------------------------------
+
Select the Volume Group to change the failure action:

[Entry Fields]
Student Notebook
On loss of access Halt the node +

Optional notification method []
Volume Group mndhb_vg_01
The optional notification method is the full path to an executable (script

or binary) to be invoked during the event processing. Notification methods
can be used to perform additional processing (like quiescing an application
or paging out the system administrator).
Note: After changing the failure action for a Volume Group you must
perform a verification and synchronization operation to activate the
changes.
Verify and Synchronize the Changes

SMIT fastpath: cm_ver_and_sync.select
=====================================
After any operation that changes a MNDHB configuration (add, remove,

change MNDHB) the cluster must be verified and synchronized.
If cluster services are active on any cluster nodes, the configuration

change will invoke a Dynamic Automatic Reconfiguration Event or DARE.
With DARE the new MNDHB networks will become active and stable after the
DARE event completes. If you added new MNDHB networks you will see a
number network_up events after the DARE event.
If cluster services are down the MNDHB will be activated the next
time the cluster is started.
See the section on monitoring MNDHB networks for information on how
to display the MNDHB configuration and status.
o Monitoring MNDHB networks
SMIT
====
The primary interface for listing the MNDHB configuration is through the
SMIT path: cl_show_mndhb_dialog

V3.1.0.1
Student Notebook
AP Example output:
The following network is configured for Multi-Node Disk Heartbeat:
Network net_dhbmulti_05
Network net_dhbmulti_05 uses Logical Volume [/dev/mndhb_lv_06] in
Volume Group [mndhb_vg_04] for heartbeat.
Volume Group [mndhb_vg_04] is managed by Resource Group [rg_dhb_03]
The following nodes participate in this network:
racc1
racc2
racc3
On loss of access, HACMP will:
Halt the node
In addition to the SMIT interface, MNDHB configuration information can be

displayed by the following commands:
- clstat
- cldisp
- cltopinfo
========================
MNDHB Limitations
========================
o Number of networks
HACMP and RSCT have limits on the number of networks that can be defined.
The number of “networks” is the number of IP (e.g. ethernet) and non-IP
(e.g. rs232) networks as well as individual network “offsets” as calculated
by RSCT. This number is calculated and checked during the Verification
process.
Adding MNDHB networks will increase the total number of networks in use.
In general, the following table can be used to determine how many MNDHB
networks a given cluster configuration will support. This restriction
will be addressed in a future release of HACMP
====================================================================
Nodes *Existing Non-MNDHB interfaceMNDHB networks
====================================================================
Student Notebook
0-8 Up to 6 per node Up to 6

9-16 Up to 3 per node Up to 3
> 16 Future Release Future Release
* Includes boot addresses and any non-IP adapters per node. Does
not include service or persistent addresses.
o Cannot add or remove nodes in an MNDHB network
Once a MNDHB network has been defined, there is no ability to add or

remove nodes. In order to change node membership to a
given MNDHB network it must be removed and redefined.
This restriction will be removed in a future release of HACMP.
==============================================
o APAR IZ49879: Changes in NFS mount default options
==============================================
Default options for NFS cross mounts are changed in HACMP 5.4.1 and later
releases.
If there are no mount options specified for this filesystem in

/etc/filesystems,
HACMP 5.4.1 and later releases do a hard, interruptible mount
options=hard,intr,vers=$vers
HACMP 5.4.0 does a soft, interruptible mount
options=soft,intr
Please refer to mount man page for a distinction between the two.
====================================================================
o Dealing with an Aynchronous I/O cache logical volume failure
====================================================================
Asynchronous GLVM mirroring uses a logical volume of type aio_cache. An aio

cache failure causes all logical partition copies on the disks at the backup
site to be marked stale.
An aio cache failure can be identified by the following error log entry
71D2CAA4 0506102109 P S LVDD AIO CACHE FAIL NOTIFY RECEIVED
Additionally, the lsmp command will show “ASYNC CACHE VALID: no”

V3.1.0.1
Student Notebook
AP
To minimize the likelihood of an aio cache failure, IBM recommends using
either LVM mirroring of the aio cache logical volume, or placing that
logical volume on a disk subsystem on a device with RAID capabilities.
IBM further recommends that the aio cache logical volume not share the
same disk as data logical volumes.
In the event of a cache device failure, the recovery steps are as follows.
These must be completed prior to any other operations on the volume group.
- Correct the problem that caused the failure. This may involve replacing
a failed disk, relocating volume group contents to a different disk,
moving a disk to active state, or other operations, depending on the
precise failure.
- Use the chmp command to switch from asynchronous to synchronous mirroring.

Assuming the backup site is called “siteB” and the asynchronously mirrored
volume group is called “datav”,
# chmp -S -f -m siteB datavg
- For each remote disk, use the chdev command to tell the RPV client to resume
communication with its RPV server. If ‘hdisk8’ is one of the local disks
in the asychronously mirrored volume group,
# chdev -l hdisk8 -a resume=yes
hdisk8 changed
- Use the varyonvg command to tell LVM to access the disks that it had
previously marked as missing. (Note that you run the varyonvg command
while the volume group is already varied online.)
# varyonvg datavg
This step runs the syncvg command in the background to synchronize the
logical partition copies on the remote disks.
- If the original failure or its recovery resulted in the permanent loss of

the aio_cache logical volume, then create a new aio_cache logical volume.
- Finally, use the chmp command to switch back to asynchronous mirroring.

# chmp -A -m siteB datavg
- The lsmp command should show that the cache device is now valid. Look for
“ASYNC CACHE VALID: yes” in the output for the mirror pool.
Student Notebook
========================
Accessing IBM on the Web
========================
o Access IBM’s home page at:
http://www.ibm.com
The HACMP home page can be found at:
http://www.ibm.com/systems/p/ha/
========
Feedback
========
IBM welcomes your comments. You can send any comments via e-mail to:
hafeedbk@us.ibm.com

V3.1.0.1
Student Notebook
Uempty Appendix C. IPAT via IP Replacement

This unit describes the HACMP IP Address Takeover via IP
replacement function.

• Configure IP Address Takeover (IPAT) via IP replacement

Accountability:
• Checkpoint
References
SC23-4867-05 HACMP for AIX: HACMP Master Glossary
SC23-4864-06 HACMP for AIX: Concepts and Facilities Guide
SC23-4861-06 HACMP for AIX: Planning and Installation Guide
SC23-4862-06 HACMP for AIX: Administration Guide
SC23-5177-00 HACMP for AIX: Troubleshooting Guide
© Copyright IBM Corp. 2007, 2009 Appendix C. IPAT via IP Replacement C-1
Student Notebook
Unit Objectives
•Configure IP Address Takeover (IPAT) via IP replacement
Figure C-1. Unit Objectives QV1203.1
Notes
C-2 PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

V3.1.0.1
Student Notebook
Uempty
IPAT via IP Replacement Configuration
•Define each network’s boot IP addresses in the AIX ODM
– Each interface IP address on a given node must be in a different logical IP subnet* and there must
be a common subnet among the nodes
– Define these address in the /etc/hosts file and configure them in HACMP topology
•Define service IP addresses in /etc/hosts and HACMP resources
– The address must be in the SAME subnet as a common interface subnet
– HACMP configures them to AIX as required
9.47.10.1 (ODM) 9.47.11.1 (ODM) 9.47.10.2 (ODM) 9.47.11.2 (ODM)
* See earlier discussion of heartbeating and failure diagnosis for explanation of why
Figure C-2. IPAT via IP Replacement Configuration QV1203.1
Notes
Requirements
Keep the following items in mind when you configure a network for IPAT via IP
replacement:
- There must be at least one logical IP subnet which has a communication interface
(NIC) on each node. (In HACMP 4.5 terminology, these were called boot adapters.)
- Each service IP address must be in the same logical IP subnet as one of the
non-service addresses. Contrast with IPAT via IP aliasing, where service addresses
are required to NOT be in a boot subnet.
- If you have more than one service IP address, they must all be in the same subnet.
The reason for this will become clear when we discuss what happens during a
takeover, see “IPAT via IP Replacement After a Node Fails” on page C-8.
- None of the other non-service addresses may be in the same subnet as the service
IP address (this is true regardless of whether IPAT via IP replacement is being used
Student Notebook
as the NICs on each node are required to be on different IP subnets in order to

support heartbeating).
- All network interfaces must have the same subnet mask.
IPAT via IP replacement subnet rules example

Each service IP address must be in one, and only one, of the non-service subnets.
All service IP addresses must be in the same subnet.
Each non-service IP address on each node must be in a separate subnet.
For example, in a cluster with one network using IPAT via replacement, where each
node has two communication interfaces and two service IP labels, the network will
require two subnets:
Node name NIC IP Label IP Address
node1 en0 n1-if1 192.168.10.1
node1 en1 n1-if2 192.168.11.1
node2 en0 n2-if1 192.168.10.2
node2 en1 n2-if2 192.168.11.2
Service address appA-svc 192.168.10.22
Service address appB-svc 192.168.10.25
subnet IP labels
n1-if1, n2-if1, appA-svc,
192.168.10/24
appB-svc
192.168.11/24 n1-if2, n2-if2

V3.1.0.1
Student Notebook
Uempty
IPAT via IP Replacement in Operation
•When the resource group comes up on a node, HACMP
replaces a boot (ODM) IP label with the service IP label
–It replaces the boot IP label on the same subnet if the resource group is
on its startup node or if the distribution startup policy is used
–It replaces a boot IP label on a different subnet otherwise
After starting the application resource group
9.47.10.22 (service) 9.47.11.1 (ODM) 9.47.10.2 (ODM) 9.47.11.2 (ODM)
Figure C-3. IPAT via IP Replacement in Operation QV1203.1
Notes
Operation
When the resource group comes up on its home node, the resource group’s service IP
address replaces the interface IP address of the NIC (configured in the AIX ODM) which
is in the same subnet as the service IP label (that is, the boot adapter in HACMP 4.x
terminology). Note that this approach implies that there cannot be two resource groups
in the cluster which both use IPAT via IP replacement and use the same node as their
home node unless their respective service IP addresses are in different subnets (in
other words, associated with different physical networks).
Also, since the service IP address replaces the existing IP address on the NIC, it is not
possible to have two or more service IP addresses in the same resource group which
are in the same IP subnet (as there will not be an adapter to assign the second service
IP address to).
When the resource group comes up on any node other than its home node, the
resource group’s service IP address replaces the interface IP address of one of the
Student Notebook
NICs which is not in the same subnet as the service IP address (this is primarily to allow
some other resource group to use the node as its home node).

V3.1.0.1
Student Notebook
Uempty
IPAT via IP Replacement After an I/F Fails
•If the communication interface being used for the service IP label
fails, HACMP swaps the service IP label with a boot (ODM) IP label
on one of the node's remaining available (that is, currently
functional) communication interfaces
•The IP labels remain swapped when the failed interface recovers
NIC A NIC B
9.47.11.1 (ODM) 9.47.10.22 (service) 9.47.10.2 (ODM) 9.47.11.2 (ODM)
Figure C-4. IPAT via IP Replacement After an I/F Fails QV1203.1
Notes
Interface failure
If a communications interface (NIC A) which is currently assigned an IPAT via IP
replacement service IP address fails, then HACMP moves the service IP address to
one of the other communication interfaces (NIC B) on the same node (to one of the
standby adapters using HACMP 4.x terminology).
If there are no available (that is, functional) NICs left the relevant network then HACMP
initiates a fallover.
Interface swap
The failed communications interface (NIC A) is then reconfigured with the address of
the communication interface (NIC B) as this allows the heartbeat mechanism to watch
for when the failed communication interface (NIC A) recovers.
Student Notebook
IPAT via IP Replacement After a Node Fails

•If the resource group's node fails, HACMP moves the resource
group to a new node and replaces an interface IP label with
the service IP label:
–If the resource group is on its startup node or if the Startup policy is
distribution, it replaces the interface (ODM) IP label in the same subnet
–Else it replaces an interface (ODM) IP label in a different subnet
–Or fails if there isn't an available interface
9.47.10.2 (ODM) 9.47.10.22 (service)
Figure C-5. IPAT via IP Replacement After a Node Fails QV1203.1
Notes
Node failure
If the node currently responsible for an IPAT via IP replacement using resource group
fails then HACMP initiates a fallover. When the resource group comes up on the
takeover node, the service IP addresses are assigned to NICs on the fallover node:
- Home node or Startup policy of Online Using Distribution Policy (rotate in
HACMP 4.x terminology)
If the takeover node is the home node for the resource group or the resource group
has a Startup policy of Online Using Distribution Policy (rotate in HACMP 4.x
terminology), the Service IP addresses replace the IP addresses of a
communications interface (NIC) with an IP address in the same subnet as the
service IP address.
- Not the home node and not Online Using Distribution Policy
If the takeover node is not the home node for the resource group and the resource
group does not have a Startup policy of Online Using Distribution Policy, the

V3.1.0.1
Student Notebook
Uempty service IP addresses replace the IP addresses of a communications interface (NIC)

with an IP address in a different subnet than the subnet of the service IP address (a
standby adapter in HACMP 4.x terminology). This is primarily to allow some other
resource group to use the node as its home node.
Note: This explains why all service IP addresses must be in the same subnet when
using IPAT via replacement.
Home node and startup policy

The home node (or the highest priority node for this resource group) is the first node
that is listed in the participating nodelist for a non-concurrent resource group. The home
node is a node that normally owns the resource group. Note that the takeover node
might actually be the home node since a resource group can be configured to not
always run on the highest priority available node.
Resource groups have three policies that HACMP uses to determine which nodes will
start which resource groups. A Startup policy of Online Using Distribution Policy
(also called a distributed policy) specifies that only one resource group can be active on
a given node. If the first node in the resource group’s list of nodes already has another
resource group started on it then the next node in the list of nodes is tried.
These concepts are discussed in Unit 6.
Student Notebook
IPAT via IP Replacement Summary

•Configure each node with up to eight communication
interfaces (each on a different subnet)
•Assign service IP labels to resource groups as appropriate
–Each node can be the most preferred node for at most one resource
group
–No limit on number of service IP labels per resource group but each
service IP label must be on a different physical network
•HACMP replaces non-service IP labels with service IP labels
on the same subnet as the service IP label when the
resource group is running on its most preferred node or if the
Startup Policy is distributed
•HACMP replaces non-service IP labels with service IP labels
on a different subnet from the service IP label when the
resource group is moved to any other node
•IPAT via IP replacement supports hardware address
takeover
Figure C-6. IPAT via IP Replacement Summary QV1203.1
Notes
Advantages
Probably the most significant advantage of IPAT via IP replacement is that it supports
hardware address takeover (HWAT), which will be discussed in a few pages.
Another advantage is that it requires fewer subnets. If you are limited in the number of
subnets available for your cluster, this may be important.
Note: Another alternative, if you are limited on the number of subnets you have
available, is to use heartbeating via IP aliases. See Heartbeating Over IP Aliases in
Chapter 3 of the HACMP for AIX 5L Planning Guide, Version 5.4.
Disadvantages
Probably the most significant disadvantages are that IPAT via IP replacement dictates
that there is only one service IP label per subnet per resource group on one
communications interface. IPAT via IP replacement also makes it rather expensive (and

V3.1.0.1
Student Notebook
Uempty complex) to support lots of resource groups in a small cluster. In other words, you need
more network adapters to support more applications.
In addition, IPAT via replacement usually takes more time than IPAT via aliasing.
Note that HACMP tries to keep the service IP Labels available by swapping IP
addresses with other communication interfaces (standby adapters in HACMP 4.x
terminology) even if there are no resource groups currently on the node which uses
IPAT via IP replacement.
Student Notebook
Gratuitous ARP Support Issues

•Gratuitous ARP is supported by AIX on the following network
technologies:
–Ethernet (all types and speeds)
–Token-Ring
–FDDI
–SP Switch 1 and SP Switch 2
•Gratuitous ARP is not supported on ATM
•Operating systems are not required to support gratuitous ARP
packets
–Practically every operating system does support gratuitous ARP
–Some systems (for example, certain routers) can be configured to
respect or ignore gratuitous ARP packets
Figure C-7. Gratuitous ARP Support Issues QV1203.1
Notes
Review
When using IPAT via aliasing, you can use AIX’s gratuitous ARP features to update
client and router ARP caches after a takeover. However, there may be issues.
Gratuitous ARP issues

Not all network technologies provide the appropriate capabilities to implement
gratuitous ARP. In addition, operating systems which implement TCP/IP are not
required to respect gratuitous ARP packets (although practically all modern operating
systems do).
Finally, support issues aside, an extremely overloaded network or a network which is
suffering intermittent failures might result in gratuitous ARP packets being lost (a
network which is sufficiently overloaded to be losing gratuitous ARP packets or which is
suffering intermittent failures which result in gratuitous ARP packets being lost is likely

V3.1.0.1
Student Notebook
Uempty to be causing the cluster and the cluster administrator far more serious problems than
the ARP cache issue involves).
Student Notebook
What if Gratuitous ARP is Not Supported?

•If the local network technology doesn't support gratuitous
ARP or there is a client system or router on the local physical
network which must communicate with the cluster and which
does not support gratuitous ARP packets:
–clinfo can used on the client to receive updates of changes
–clinfo can be used on the servers to ping a list of clients, forcing an
update to their ARP caches
–HACMP can be configured to perform hardware address takeover
(HWAT)
•Suggestion:
Do not get involved with using either clinfo or HWAT to deal
with ARP cache issues until you've verified that there actually
are ARP issues.
Figure C-8. What if Gratuitous ARP is Not Supported? QV1203.1
Notes
If gratuitous ARP is not supported

HACMP supports three alternatives to gratuitous ARP. The first two are discussed in
Unit 3. We will discuss the third option here.
Don’t add unnecessary complexity

Cluster configurators should probably not simply assume that gratuitous ARP won’t
provide a satisfactory solution as each of the alternatives introduce additional, possibly
unnecessary complexity into the cluster.
If the cluster administrator or configurator decides that the probability of a gratuitous
ARP update packet being lost is high enough to be relevant, then they should proceed
as though their context does not support gratuitous ARP.

V3.1.0.1
Student Notebook
Uempty
Hardware Address Takeover
•HACMP can be configured to swap a service IP label's
hardware address between network adapters
•HWAT is incompatible with IPAT via IP aliasing because each
service IP address must have its own hardware address and a
NIC can support only one hardware address at any given time
•Cluster implementer designates a Locally Administered
Address (LAA) which HACMP assigns to the NIC which has
the service IP label
Figure C-9. Hardware Address Takeover QV1203.1
Notes

Hardware address takeover (HWAT) is the most robust method of dealing with the ARP
cache issue as it ensures that the hardware address associated with the service IP
address does not change (which avoids the whole issue of whether the client system’s
ARP cache is out-of-date).
The essence of HWAT is that the cluster configurator designates a hardware address
which is to be associated with a particular service IP address. HACMP then ensures
that whichever NIC the service IP address is on also has the designated hardware
address.
Student Notebook
HWAT considerations
There are a few points which must be kept in mind when contemplating HWAT:
- The hardware address which is associated with the service IP address must be
unique within the physical network that the service IP address is configured for.
- HWAT is not supported by IPAT via IP aliasing because each NIC can have more
than one IP address but each NIC can only have one hardware address.
- HWAT is only supported for Ethernet, token ring and FDDI networks (MCA FDDI
network cards do not support HWAT). ATM networks do not support HWAT.
- HWAT increases the takeover time (usually by just a few seconds).
- HWAT is an optional capability which must be configured into the HACMP cluster
(we will see how to do that in a few minutes).
- Cluster nodes using HWAT on token ring networks must be configured to reboot
after a system crash as the token ring card will continue to intercept packets for its
hardware address until the node starts to reboot.

V3.1.0.1
Student Notebook
Uempty
Hardware Address Takeover (1 of 3)
node1-if1 node2-if1
9.47.9.1 9.47.9.2
tr1 255.255.255.0
00:04:ac:48:22:f4
tr1
255.255.255.0
00:04:ac:62:72:61
tr1
Before node2-if2
node1-if2
tr0 9.47.5.3
255.255.255.0
resource 9.47.5.2
255.255.255.0 tr0
00:04:ac:48:22:f6
00:04:ac:62:72:49
group is
started
node1 node2
node2-if1
node1-if1 9.47.9.2
tr1 9.47.9.1 255.255.255.0
255.255.255.0 00:04:ac:62:72:61 tr1
00:04:ac:48:22:f4
After node2-if2
xweb
tr0
9.47.5.1
255.255.255.0
resource 9.47.5.2
255.255.255.0 tr0
00:04:ac:48:22:f6
40:04:ac:62:72:49
group is
started
node1 node2
Figure C-10. Hardware Address Takeover (1 of 3) QV1203.1
Notes
Hardware address takeover (HWAT): boot time

At boot time, the interfaces are assigned their normal hardware addresses.
HWAT: resource group started

When HACMP starts the resource group, the service IP address replaces the
non-service IP address of the interface and the alternate hardware address replaces
the normal hardware address for that NIC.
The alternate hardware address is usually referred to as a Locally Administered
Address or LAA.
Student Notebook

•LAA is moved along with the service IP label
xweb node2-if1
9.47.5.1 9.47.9.2
tr1 255.255.255.0
40:04:ac:62:72:49
tr1
255.255.255.0
00:04:ac:62:72:61
tr1
Interface node2-if2
xweb
tr0 9.47.5.1
255.255.255.0
failure 9.47.5.2
255.255.255.0 tr0
40:04:ac:62:72:49 00:04:ac:48:22:f6
node1 node2
xweb
node1-if1 9.47.5.1
tr1 9.47.9.1 255.255.255.0
255.255.255.0 40:04:ac:62:72:49 tr1
00:04:ac:48:22:f4
Node failure node2-if2
xweb 9.47.5.2
9.47.5.1 255.255.255.0
tr0 255.255.255.0 00:04:ac:48:22:f6
tr0
40:04:ac:62:72:49
node1 node2
Notes
HWAT: interface or node failure

If a NIC (with a service IP address that has an LAA) fails, HACMP moves the IP
address to another available NIC on the node. It also moves the LAA (alternative
hardware address) to the same NIC.
If a node fails, the service IP address, and its associated LAA, are moved to another
node.
The result, in both of these cases, is that the local client’s ARP caches are still up to
date since the HW address associated with the IP address has not changed.

V3.1.0.1
Student Notebook
Uempty
node1-if1 xweb
9.47.9.1 9.47.5.1
tr1 255.255.255.0 255.255.255.0 tr1
40:04:ac:62:72:49
00:04:ac:48:22:f4 When a failed node
comes back to life, node2-if2
node1-if2 the burned in ROM 9.47.5.2
tr0 9.47.5.3 255.255.255.0 tr0
255.255.255.0 Address is used on 00:04:ac:48:22:f6
00:04:ac:62:72:49
the service network
adapter.
node1 node2
node2-if1
node1-if1 9.47.9.2
tr1 9.47.9.1 255.255.255.0 tr1
255.255.255.0 00:04:ac:62:72:61
00:04:ac:48:22:f4
After HACMP is
xweb
started the node node2-if2
reintegrates 9.47.5.2
tr0 9.47.5.1 255.255.255.0 tr0
255.255.255.0 according to its 00:04:ac:48:22:f6
40:04:ac:62:72:49
resource group
parameters
node1 node2
Notes
HWAT: node recovery

When the failed node reboots, AIX must be configured to leave the network card’s
factory-defined hardware address in place. If AIX is configured to set the network card’s
HW address to the alternate hardware address at boot time, then two NICs on the same
network have the same hardware address (weird things happen when you do this).
HWAT: resource moved back to home node

If HACMP ultimately moves the resource group back to the now recovered node, then
the hardware address of the NIC on the backup node is restored to its factory setting,
and the LAA associated with the service IP address moves back to the NIC on the
recovered node.
Student Notebook
Implementing Hardware Address Takeover

•Someone just got a great deal on a dozen used FOOL-97x
computers for the summer students to use
•They run some strange proprietary operating system which refuses
to update its ARP cache in response to either ping or gratuitous ARP
packets
node1 node2
D D
A A
Figure C-13. Implementing Hardware Address Takeover QV1203.1
Notes

In this scenario, we will implement HWAT to support the new computers discussed in
the visual.
Just imagine how much money they have saved once they realize that these new
computers don’t do what the summer students need done!
In the meantime, it looks like we need to implement hardware address takeover in order
to support these FOOL-97X’s.
Reality check
A side note is probably in order: although most TCP/IP-capable systems respect
gratuitous ARP, there are strange devices out there that do not. This scenario is phoney
but it presents a real if rather unlikely problem. For example, the ATM network does not
support gratuitous ARP and so could be a candidate for the use of HWAT.

V3.1.0.1
Student Notebook
Uempty
Our Plan for Implementing HWAT
1.Stop cluster services on both cluster nodes
íUse the graceful shutdown option to bring down the resource groups
and their applications
2.Remove the alias service labels from the Resources
–They are in the wrong subnet for replacement
–They are automatically removed from the RG
3.Convert the net_ether_01 Ethernet network to use IPAT via IP
replacement:
–Disable IPAT via IP aliasing on the Ethernet network.
–Update /etc/hosts on both cluster nodes to describe service IP labels
and addresses on the 192.168.15.0 subnet
–Use the procedure described in the networking to select the (Locally
Administered Addresses (LAA) addresses
–Configure new service IP labels with these LAA addresses in the
HACMP SMIT screens
4.Define resource groups to use the new service IP labels
5.Synchronize the changes
6.Restart HACMP on the two nodes
Figure C-14. Our Plan for Implementing HWAT QV1203.1
Notes
Implementing HWAT
In order to use HWAT, we must use IPAT via replacement.
Stop cluster services

Changing from IPAT via aliasing to IPAT via replacement can not be done dynamically,
we must stop the cluster.
Remove existing service IP labels

The service IP labels used for IPAT via aliasing cannot be used for IPAT via
replacement. They are on the wrong subnet. We will either need to change our service
addresses or change our non-service addresses. In this scenario, we choose to change
the service addresses.
Student Notebook
Convert the network to use IPAT via replacement

In addition to the obvious step of disabling IPAT via aliasing, we also need to update our
name resolution for the new service IP labels and we need to create an alternate
hardware address or Locally Administered Address (LAA) for each service IP label.
Name resolution changes

One slight problem with the above procedure is that it requires the users (or the DNS
administrator) to change the service IP address that they are using. It would arguably
be better if we preserved the service IP address. However, this would require more
network reconfiguration work and it isn’t totally clear that the difference is significant in
the grand scheme of things. Note that either approach requires the cooperation of the
network administrators as we will require IP addresses and probably DNS changes.

V3.1.0.1
Student Notebook
Uempty
Stopping HACMP
# smit clstop
Stop Cluster Services
[Entry Fields]
* Stop now, on system restart or both now
+
Stop Cluster Services on these nodes [node1,node2]
+
BROADCAST cluster shutdown? true
+
* Shutdown mode graceful
+

Figure C-15. Stopping HACMP QV1203.1
Notes
Stop HACMP
Make sure that HACMP is shut down gracefully, as we can’t have the application
running while we are changing service IP addresses.
Student Notebook
Removing a Service IP Label

Press Enter here and you will be prompted to confirm the removal.
Configure HACMP Service IP Labels/Addresses

Add a Service IP Label/Address
Change/Show a Service IP Label/Address
Remove Service IP Label(s)/Address(es)
+--------------------------------------------------------------------------+
¦ Select Service IP Label(s)/Address(es) to Remove ¦
¦ ¦
¦ Move cursor to desired item and press F7. ¦
¦ ¦
¦ xweb ¦
¦ yweb ¦
¦ zweb ¦
F9+--------------------------------------------------------------------------+
Repeat for both service IP labels.

Figure C-16. Removing a Service IP Label QV1203.1
Notes
Remove any service labels configure for IPAT via aliasing

An attempt to convert the network to IPAT via IP replacement fails if there are any
service IP labels that don’t conform to the IPAT via IP replacement rules.

V3.1.0.1
Student Notebook
Uempty
Disable IPAT via Aliases
Set the "Enable IP Address Takeover via IP Aliases" setting to "No"
and press Enter.
Change/Show an IP-Based Network in the HACMP Cluster

[Entry Fields]
* Network Name net_ether_01
New Network Name []
* Network Type [ether] +
* Netmask [255.255.255.0] +
* Enable IP Address Takeover via IP Aliases [No] +
IP Address Offset for Heartbeating over IP Aliases []

Figure C-17. Disable IPAT via Aliases QV1203.1
Notes
Introduction
Here we change the net_ether_01 network to disable IPAT via aliasing.
Student Notebook
The Updated /etc/hosts

•Here's the key portion of the /etc/hosts file with the service IP
labels moved to the 192.168.15.0 subnet:
192.168.5.29 node1 # persistent node IP label on node1

192.168.15.29 node1-if1 # node1's first boot IP label
192.168.16.29 node1-if2 # node1's second boot IP label
192.168.5.31 node2 # persistent node IP label on node2
192.168.15.31 node2-if1 # node2's first boot IP label
192.168.16.31 node2-if2 # node2's second boot IP label
192.168.15.92 xweb # the IP label for the application
normally
# resident on node1
192.168.15.70 yweb # the IP label for the application
normally
# resident on node2
•Note that neither node1 or node2's network configuration (as

defined with the AIX TCP/IP SMIT screens) needs to be changed
•Note that we are not renaming the interface IP labels to something like
node1_boot and node1_standby as changing IP labels in an HACMP
cluster can be quite a bit of work (it is often easier to delete the cluster
definition and start over)
Figure C-18. The Updated /etc/hosts QV1203.1
Notes
IPAT via replacement rules

Remember the rules for IP addresses for IPAT via IP replacement networks (slightly
reworded):
a. The service IP labels must all be on the same subnet
b. There must be one NIC on each host which has an IP address on the same subnet
as the service IP labels (in HACMP 4.x terminology, these NICs are boot adapters)
c. The other NICs on each node must each be in a different subnet than the service IP
labels (in HACMP 4.x terminology, these NICs are standby adapters)
In a cluster with only two NICs per node, NIC IP addresses which conform to the IPAT
via IP aliasing rules also conform to the IPAT via replacement so only the service IP
labels need to be changed.

V3.1.0.1
Student Notebook
Uempty
Creating a
Locally Administered Address (LAA)
•Each service IP label using HWAT will need an LAA
•The LAA must be unique on the cluster's physical network
•The MAC address based technologies (Ethernet, Token ring
and FDDI) use six byte hardware addresses of the form:
xx.xx.xx.xx.xx.xx
•The factory-set MAC address of the NIC will start with 0, 1, 2
or 3
–A MAC address that starts with 0, 1, 2 or 3 is called a Globally
Administered Address (GAA) because it is assigned to the NIC's
vendor by a central authority
•Incrementing this first digit by 4 transforms the GAA into a
Locally Administered Address (LAA) which will be unique
worldwide (unless someone has already used the same GAA
to create an LAA which isn't likely since GAAs are unique
worldwide)
Figure C-19. Creating a Locally Administered Address (LAA) QV1203.1
Notes
Hardware addresses
Hardware addresses must be unique, at a minimum, on the local network to which they
are connected. The factory set hardware address for each network interface card (NIC)
is administered by a central authority and should be unique in the world. These
addresses are called Globally Administered Addresses (GAAs).
Locally administered addresses (LAAs)

Incrementing the first nibble of the GAA by 4 transforms it into an LAA.
Using this method to create an alternate address should provide you with an address
that is also globally unique, as noted in the visual.
Note: According to the IEEE 802 standard for LAN MAC addresses, the second bit
transmitted on the LAN medium (the “4” bit) is the local/global bit. If this bit is zero, the
address is a GAA. Setting this bit to one indicates this address is locally administered.
Student Notebook
Creating Two LAAs for our Cluster

•Here are two Globally Administered Addresses (GAAs)
taken from Ethernet adapters in the cluster:
–0.4.ac.17.19.64
–0.6.29.ac.46.8
•First we make sure that each number is two digits long by
adding leading zeros as necessary:
–00.04.ac.17.19.64
–00.06.29.ac.46.08
•Verify that the first digit is 0, 1, 2 or 3:
–Yep!
•Add 4 to the first digit of each GAA:
–40.04.ac.17.19.64
–40.06.29.ac.46.08
•Done! These two addresses are now LAAs
Figure C-20. Creating Two LAAs for our Cluster QV1203.1
Notes

V3.1.0.1
Student Notebook
Uempty
Hardware Address Takeover Issues
•Do not enable the ALTERNATE hardware address field
in the SMIT devices menu
–Causes the adapter to boot on your chosen LAA rather than the
burned in ROM address
–Causes serious communications problems and puts the cluster in
to an unstable state
–Correct method is to enter your chosen LAA in to the SMIT
HACMP menus (remove the periods or colons before entering it
into the field)
•The Token-Ring documentation states that the LAA must
start with 42
•The FDDI documentation states that the first nibble (digit) of
the first byte of the LAA must be 4, 5, 6 or 7 (which is
compatible with the method for creating LAAs described
earlier)
•Token-Ring adapters do not release the LAA if AIX crashes.
–AIX must be set to reboot automatically after a system crash
(see smitty chgsys)
Figure C-21. Hardware Address Takeover Issues QV1203.1
Notes
Issues
The main thing to remember is that you do NOT configure the ALTERNATE hardware
address field in the SMIT devices panel.
You must leave that blank and configure this using the SMIT HACMP menus.
Student Notebook
Redefining the Service IP Labels for HWAT

Redefine the two service IP labels. Note that the periods are stripped
out before the LAA is entered into the HW Address field.
Add a Service IP Label/Address configurable on Multiple Nodes
(extended)
[Entry Fields]
* IP Label/Address [xweb]
+
* Network Name net_ether_01
Alternate HW Address to accompany IP Label/Address [4004ac171964]
You probably shouldn't use the particular LAAs

shown on these foils in your cluster. Select your
own LAAs using the procedure described
earlier.
Don't forget to specify the second LAA for the second service IP label.
Figure C-22. Redefining the Service IP Labels for HWAT QV1203.1
Notes
Redefining the service IP labels

Define each of the service IP labels making sure to specify a different LAA address for
each one.
The Alternate HW Address to accompany IP Label/Address is specified as a series
of hexadecimal digits without intervening periods or any other punctuation.
If IPAT via IP replacement is specified for the network, which it is in this case, you get an
error or a warning from this screen if you try to define service IP labels which do not
conform to the rules for service IP labels on IPAT via IP replacement networks.

V3.1.0.1
Student Notebook
Uempty
Synchronize the changes and run through the test plan.
HACMP Verification and Synchronization

[Entry Fields]
* Verify, Synchronize or Both [Both] +
Force synchronization if verification fails? [No] +
* Verify changes only? [No] +
* Logging [Standard] +

Figure C-23. Synchronize Your Changes QV1203.1
Notes
Synchronize
Don’t forget to synchronize.
Student Notebook
Review
1. For IPAT via replacement (select all that apply)
a. Each service IP address must be in the same subnet as one of the non-service
addresses
b. Each service IP address must be in the same subnet
c. Each service IP address cannot be in any non-service address subnet
2. True or False?
If the takeover node is not the home node for the resource group and the
resource group does not have a Startup policy of Online Using Distribution
Policy, the service IP address replaces the IP address of a NIC with an IP
address in the same subnet as the subnet of the service IP address.
3. True or False?
In order to use HWAT, you must enable and complete the ALTERNATE
ETHERNET address field in the SMIT devices menu.
4. True or False?
You must stop the cluster in order to change from IPAT via aliasing to IPAT via
replacement.
Figure C-24. Review QV1203.1
Notes

V3.1.0.1
Student Notebook
Uempty
Unit Summary
•IPAT via IP replacement:
–May require fewer subnets than IPAT via aliasing
–May require more NICs than IPAT via aliasing
–Supports hardware address takeover
•HACMP replaces non-service IP labels with service IP labels on the
same subnet as the service IP label when the resource group is started
on its home node or if the Startup Policy is distributed
•HACMP replaces non-service IP labels with service IP labels on a
different subnet from the service IP label when the resource group is
moved to any other node
•IPAT via IP replacement configuration issues
–Service IP address must be the same subnet as one of the non-service
subnets
–All service IP addresses must be in the same subnet
–You must have at least as many NICs on each node as service IP
addresses
•Hardware address takeover (HWAT) issues
–Alternate hardware address (Locally Administered Address or LAA)
must be configured in HACMP. Do NOT use standard SMIT field.
–Alternate hardware address must be unique.
Figure C-25. Unit Summary QV1203.1
Notes
Student Notebook

V3.1.0.1
Student Notebook
Uempty Appendix D. Configuring target mode SSA

This appendix describes steps required to configure Target mode
SSA.

• Configure Target Mode SSA

Accountability:
• Self-guided implementation
References
SC23-5209-00 HACMP for AIX, Version 5.4 Installation Guide
SC23-4864-09 HACMP for AIX, Version 5.4:
Concepts and Facilities Guide
SC23-4861-09 HACMP for AIX, Version 5.4 Planning Guide
SC23-4862-09 HACMP for AIX, Version 5.4 Administration Guide
SC23-5177-03 HACMP for AIX, Version 5.4 Troubleshooting Guide
SC23-4867-08 HACMP for AIX, Version 5.4 Master Glossary
HACMP manuals
© Copyright IBM Corp. 2007, 2009 Appendix D. Configuring target mode SSA D-1
Student Notebook
Unit Objectives
•Configure Target Mode SSA
Figure D-1. Unit Objectives QV1203.1
Notes
D-2 PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

V3.1.0.1
Student Notebook
Uempty
Implementing Target Mode SSA
•The serial cable being used to implement the RS232 non-IP
network has been borrowed by someone and nobody
noticed
•A decision has been made to implement a target mode SSA
(tmssa) non-IP network as it won't fail unless complete
access to the shared SSA disks is lost by one of the nodes
(and someone is likely to notice that)
node1 node2
D D
A A
Figure D-2. Implementing Target Mode SSA QV1203.1
Notes
Target mode SSA or heartbeat over disk networks

Sadly, the premise behind this scenario is all too real. The problem with RS232 non-IP
networks is that if they become disconnected or otherwise disabled, then it is entirely
possible that nobody notices even though HACMP logs the failure of the connection
when it happens and reports it in the logs if it is down at HACMP startup time. In
contrast, a target mode SSA network or heartbeat on disk network won’t fail until all
paths between the two nodes fail. Since such a failure will cause one or both nodes to
lose access to some or all of the shared disks, such a failure is MUCH less likely to go
unnoticed. We focus on SSA in this scenario as we have discussed heartbeat over disk
earlier in the course.
Student Notebook
Setting the SSA Node Number

The first step is to give each node a unique SSA node number.
We'll set node1's ssa node number to 1 and node2's to 2.
Change/Show the SSA Node Number For This System
[Entry Fields]
SSA Node Number [1]
+#

Use the smitty ssaa fastpath to get to AIX's SSA Adapters menu.
Figure D-3. Setting the SSA Node Number QV1203.1
Notes
Required software
Target mode SSA support requires that the devices.ssa.tm.rte file set be installed on
all cluster nodes.
SSA node number and HACMP node ID

The first step in configuring a target mode SSA network is to assign a unique SSA node
number to each node. Earlier versions of HACMP required that the SSA node number
be the same as the node’s HACMP node id. HACMP 5.1 (and above) does not have
this requirement (and does not expose the HACMP node id to the administrator). We
assign 1 as the SSA node number for node1 and 2 as the SSA node number for
node2.

V3.1.0.1
Student Notebook
Uempty Requirements for SSA node numbers

The minimum requirements for HACMP 5.x are that the SSA node numbers be
non-zero and unique for each node within the cluster. Strictly speaking, the SSA node
numbers must also be unique across all systems with shared access to the SSA
subsystem. This is usually not a concern as allowing non-cluster nodes to have any
form of access to a cluster’s shared disks is an unnecessary risk that few cluster
administrators would ever accept.
Student Notebook
Configuring the tmssa Devices

• This is a three-step process for a two-node cluster as each
node needs tmssa devices which refer to the other node:
1.Run cfgmgr on one of the nodes (node1)
• node1 is now ready to respond to tmssa queries
2.Run cfgmgr on the other node (node2)
• node2 is now ready to respond to tmssa queries
• node2 also knows that node1 supports tmssa and has created the
tmssa devices (/dev/tmssa1.im and /dev/tmssa1.tm) which refer to
node1
3.Run cfgmgr again on the first node (node1)
• node1 now also knows that node2 supports tmssa and has created
the tmssa devices (/dev/tmssa2.im and /dev/tmssa2.tm) which refer
to node2
• node1 now has /dev/tmssa2.im /dev/tmssa2.tm devices
which refer to node2
Figure D-4. Configuring the tmssa Devices QV1203.1
Notes
Introduction
Once each node has a unique SSA node number, the AIX configuration manager needs
to be used to define the tmssa devices. Each node must have tmssa devices which
refer to each of the other nodes that they can see via the SSA loops. When cfgmgr is
run on a node, it sets up the node to accept tmssa packets, and it then defines tmssa
devices referring to any other nodes which respond to tmssa packets. In order for this to
all work, the other nodes must all be set up to accept and respond to tmssa packets.
Procedure
The end result is that the following procedure gets all the required tmssa devices
defined:

V3.1.0.1
Student Notebook
Uempty 1. Run cfgmgr on each cluster node in turn. This sets up each node to handle tmssa
packets, and defines the tmssa devices on each node to refer to nodes which have
already been setup for tmssa.
2. Run cfgmgr on each node in turn again (depending upon exactly what order you do this
in, it is actually possible to skip running cfgmgr on one of the nodes, but it is probably
not worth the trouble of being sure that the last cfgmgr run wasn’t required).
3. Verify the tmssa devices exist:
Run
# lsdev -C | grep tmssa
on each node. There should be a tmssa device (which is actually a target mode SSA
router acting as a pseudo device) configured on each node.
4. Verify the tmssa devices exist:
Run
# ls /dev/tmssa*
on each node. Note that each node has target mode SSA devices called
/dev/tmssa#.im and /dev/tmssa#.tm where # refers to the other node’s node number.
5. Test the target mode connection:
Enter the following command on the node with id 1 (make sure you specify the tm suffix
and not the im suffix):
# cat < /dev/tmssa2.tm
(This command should hang)
On the node with ID 2, enter the following command (make sure that you specify the im
suffix and not the tm suffix):
# cat /etc/hosts > /dev/tmssa1.im
(The /etc/hosts file should be displayed on the first node)
This validates that the target mode serial network in functional. Please note that any
text file may be substituted for /etc/hosts and you have to specify different tmssa
device names if you configured different SSA node numbers for each node. This is
simply an example.
Student Notebook
Rediscover the HACMP Information

Next, we need to get HACMP to know about the new communication
devices so we run the auto-discovery procedure again on one of the nodes.
Extended Resource Configuration
Extended Event Configuration
Extended Performance Tuning Parameters Configuration
Security and Users Configuration

Figure D-5. Rediscover the HACMP Information QV1203.1
Notes
HACMP discover
By discovering the new devices, they will appear in SMIT pick lists when we configure
the tmssa non-IP network. Strictly speaking, it is not necessary to rerun the HACMP
discovery as it is possible to configure tmssa networks by entering in the tmssa device
names explicitly. As this is a rather error-prone process, it is probably best to use the
HACMP discovery mechanism to discover the devices for us.

V3.1.0.1
Student Notebook
Uempty
Defining a Non-IP tmssa Network (1 of 3)
This should look very familiar as it is the same procedure that was
used to define the non-IP RS232 network earlier.

Update HACMP Communication Interface with Operating System Settings
+--------------------------------------------------------------------------+
¦ ¦
¦ ¦
¦ Add Discovered Communication Interface and Devices ¦
¦ Add Predefined Communication Interfaces and Devices ¦
¦ ¦
F9+--------------------------------------------------------------------------+
Figure D-6. Defining a Non-IP tmssa Network (1 of 3) QV1203.1
Notes
Defining a non-IP tmssa network

The procedure for defining a non-IP tmssa network is pretty much identical to the
procedure used earlier to define the non-IP RS232 network.
Student Notebook


Update HACMP Communication Interface with Operating System Settings
+--------------------------------------------------------------------------+
¦ ¦
¦ ¦
¦ # Discovery last performed: (Feb 12 18:20) ¦
¦ Communication Interfaces ¦
¦ Communication Devices ¦
¦ ¦
F9+--------------------------------------------------------------------------+
Notes
Communication device
The tmssa devices are communication devices used for non-IP communication, so
select Communication Devices here.

V3.1.0.1
Student Notebook
Uempty
Now, we need to define the tmssa network using a process
+--------------------------------------------------------------------------+
¦ Select Point-to-Point Pair of Discovered Communication Devices to Add ¦
¦ ¦
¦ Move cursor to desired item and press F7. Use arrow keys to scroll. ¦
¦ ¦
¦ # Node Device Device Path Pvid ¦
¦ > node2 tmssa1 /dev/tmssa1 ¦
¦ > node1 tmssa2 /dev/tmssa2 ¦
¦ node1 tty0 /dev/tty0 ¦
¦ ¦
F9+--------------------------------------------------------------------------+
Notes
Final step
Select the tmssa devices on each node and press Enter to define the network.
Refer to Chapter 6 of the HACMP v5.4 Installation Guide for information on configuring
all supported types of non-IP networks.
Student Notebook

Synchronize the changes and run through the test plan
HACMP Verification and Synchronization
[Entry Fields]
* Verify, Synchronize or Both [Both]
+
Force synchronization if verification fails? [No]
+
* Verify changes only? [No]
+
* Logging [Standard]
+

Figure D-9. Synchronize Your Changes QV1203.1
Notes
Synchronize
As when you make any change to the cluster, verify and synchronize.

V3.1.0.1
Student Notebook
Uempty
Unit Summary
•This unit showed the steps necessary to configure Target

Mode SSA
Figure D-10. Unit Summary QV1203.1
Notes
Student Notebook

V3.1.0.1
backpg
Back page
®

QV1203 StudentGuide

Uploaded by

Copyright:

Available Formats

QV1203 StudentGuide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

QV1203 StudentGuide

Uploaded by

Copyright:

Available Formats

V3.1.0.

Front cover

PowerHA for AIX (HACMP) I:

UNIX Software Service Enablement

August 2009 Edition

Unit 1. Introduction to PowerHA for AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1

Unit 2. Configuring Networks for PowerHA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1

© Copyright IBM Corp. 2007, 2009 Contents iii

How PowerHA Uses Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-3

iv PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

TOC What if Gratuitous ARP Is Not Supported? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-51

Unit 3. Configuring Applications for PowerHA . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1

Unit 4. PowerHA installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1

© Copyright IBM Corp. 2007, 2009 Contents v

How Does PowerHA Fit into AIX? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-11

Unit 5. Initial PowerHA Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1

vi PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

TOC Extending the Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-32

Appendix A. Checkpoint solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1

Appendix B. HACMP 5.5 release_notes and README . . . . . . . . . . . . . . . . . . . . . B-1

Appendix C. IPAT via IP Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-1

Appendix D. Configuring target mode SSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-1

© Copyright IBM Corp. 2007, 2009 Contents vii

viii PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

pref Course Description

© Copyright IBM Corp. 2007, 2009 Course Description ix

x PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

pref Text highlighting

© Copyright IBM Corp. 2007, 2009 Course Description xi

xii PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

Uempty Unit 1. Introduction to PowerHA for AIX

What This Unit Is About

What You Should Be Able to Do

How You Will Check Your Progress

1-2 PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

UNIX Software Service Enablement

Figure 1-1. Unit Objectives QV1203.1

IBM's HA Solution for AIX

UNIX Software Service Enablement

Figure 1-2. IBM's HA Solution for AIX QV1203.1

1-4 PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

UNIX Software Service Enablement

Figure 1-3. A Highly Available Cluster QV1203.1

PowerHA's Topology Components

The topology components consist of a cluster, nodes and the

Figure 1-4. PowerHA's Topology Components QV1203.1

1-6 PowerHA-I: Installation © Copyright IBM Corp. 2007, 2009

any combination of nodes within a cluster.

IBM Power 570

p5-570 IBM Power 575

A minimum of four free

IBM Power 520

UNIX Software Service Enablement

Figure 1-5. Nodes QV1203.1

Supported Networking Environments

Server Server Traffic Flow Server

UNIX Software Service Enablement

Figure 1-6. Supported Networking Environments QV1203.1

Supported TCP/IP-based networks (called IP networks)