Powerha Systemmirror For Aix V7.1 Two-Node Quick Configuration Guide

PowerHA SystemMirror for AIX
v7.1 Two-Node Quick

Configuration Guide
Shawn Bodily
Advanced Technical Skills
November 2012
COPYRIGHT IBM CORPORATION, 2012 1

Table of Contents
A. Introduction .................................................................3
B. Configuring the Cluster ..............................................4
Configuring VLAN 3358 for SAN heartbeat.................8
C. Testing the Cluster.....................................................19
E. References ...................................................................22
Appendix A – Cluster Test Tool Log.............................23

A. Introduction
The purpose of this document is to provide the steps to quickly configure a two-node hot-standby PowerHA
v7.1 cluster primarily using SMIT.
My environment during the original writing was comprised of the following:
• One p260 and one p460 Power Compute Nodes with 8GB Fibre Channel Mezzanine Adapters
• Flex System Enterprise Chassis with two 10GB Ethernet Switches and two 8GB QLogic Fibre
Channel Switch Modules
• V7000 Storage
• AIX 7.1, TL 1, SP 3
• PowerHA Version 7.1.1 SP3
• RSCT 3.1.2
If installing PowerHA v7.1.2 or v7.1.3 the following matrix can be referenced for base AIX level
requirements.
http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD101347
Though the following steps are the same for both 7.1.2 and 7.1.3, there is one menu difference in v7.1.3. It is
is the addition of unicast for heartbeating. An updated screen shot has been added to reflect it.

B.Configuring the Cluster
The following prerequisites are required prior to creating a new cluster.
1. CAA specific filesets bos.cluster and bos.ahafs

2. All pre-req filesets and PowerHA installed
3. clcomd daemon must be running on each node (lssrc –ls clcomd)
4. Boot IP addresses must be configured on each interface (smitty chinet)
5. All Boot and Service IPs must be configured in /etc/hosts.
6. Hostname IP addresses added to /etc/cluster/rhosts
7. Application server scripts must exist on each node.
8. PVIDs of the shared disks must be known to both systems.
9. One free shared disk to be specifically used for cluster repository disk.
10. The hostname must resolve to an interface and by default will be the same as the cluster node names.
This name must contain only alpha-numeric and underscores, no dashes.
11. Multicasting must be enabled on your network. The mping command can be used to test if multicasting
is working.
Additional details on installing and configuring PowerHA SystemMirror can be found in the Installing
PowerHA SystemMirror guide.
This cluster was configured utilizing standard SMIT sysmirror menus. There is also the option of utilizing the
PowerHA Systems Director plug-in to configure, monitor and manage the cluster. More information on the
Systems Director plug-in option can be found in the PowerHA v7.1 redbook.
To start creating the cluster, enter smitty sysmirror→ Cluster Nodes and Networks→ Initial Cluster Setup
(Typical) →Setup a Cluster, Nodes and Networks

Fill out the options as desired and press Enter. Upon execution it will perform a discovery to gather both IP
and shared disk information to be used in the cluster configuration.
The next step is to define a cluster repository disk and multicast address. We can use fastpaths in SMIT to
bypass additional menus. Execute smitty cm_setup_menu→ Define Repository Disk and Cluster IP Address:

For the repository disk field, you can press F4 and get a pick list to choose the desired disk. This data is
gathered during the discovery in the first step of creating the cluster. The available disks list is created by
finding all shared disks, with PVIDs, not currently in a volume group.
In PowerHA v7.1.0-v7.1.2 the use of multicasting is required. The Cluster IP address is the multicast address.
Its not required to enter one as PowerHA will choose one for you. It usually creates one by taking the last 3
octets of the hostname IP address from the node in which the cluster is being created on and replacing the first
octet with 228. In our cluster, we created it on the p460 with hostname address of 172.23.17.35. The multicast
address was created automatically of 228.23.17.35.
However, in v7.1.3 unicast was re-introduced making multicast optional. The v7.1.3 menu is shown below:
After these two steps it is recommended to synchronize the cluster. (Execute smitty sysmirror→Cluster Nodes
and Networks→ Verify and Synchronize Cluster Configuration and press Enter twice). The main reason
being, the first time the cluster is synced the CAA cluster is created automatically. That way if a problem is
encountered, it can be addressed before adding all the additional cluster components. The following shows the
disk and CAA volume group information after the synchronization and CAA cluster was created successfully.

Though optional (and not used in this test configuration), its considered a best practice to also configure SAN
based communications. This requires setting the appropriated FC adapter attributes on VIO servers, and
adding virtual Ethernet adapters using a specific VLAN (3358) .
Other than the PowerHA 7.1.1 redbook and APAR that added this support, these steps have never been clearly
documented in an officially publication. However, during the collaboration on another whitepaper involving
PowerHA v7.1 and v7000 (http://tinyurl.com/c3vksk7), we took the opportunity to document it. The
following is a snippet from that whitepaper.
Login to each VIOS partition and change the fibre channel attributes using the chdev command. Depending
on how disks are configured in the system, you may have to use the rmdev command to put the device in the
defined state. After changing the attributes, use the cfgdev command to configure the device or reboot the
partition.
# > rmdev -dev fcs0 –ucfg -recursive
fcnet0 Defined
sfwcomm1 Defined
fscsi0 Defined
fcs0 Defined
# > chdev -dev fcs0 -perm -attr tme=yes
fcs0 changed
# > chdev -dev fscsi0 -perm -attr dyntrk=yes
fscsi0 changed
# > chdev -dev fscsi0 -perm -attr fc_err_recov=fast_fail
fscsi0 changed
# > lsdev -dev fcs0 -attr |grep tme
tme yes Target Mode Enabled True
# > lsdev -dev fscsi0 -attr |grep dyntrk
dyntrk yes Dynamic Tracking of FC Devices True
# > lsdev -dev fscsi0 -attr |grep fc_err_recov

fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True
# > cfgdev
Configuring VLAN 3358 for SAN heartbeat

Since the fibre channel devices are not owned directly by the client LPARs, CAA uses a special VLAN to
communicate the SAN heartbeat. Create a virtual ethernet adapter using the HMC with the VLAN id = 3358,
on the client LPAR and each VIOS partition. Then activate the new profiles.
Another valuable source of documentation for this topic can be found at:
http://www.ibm.com/developerworks/aix/library/au-aix-powerha-heartbeat/index.html

Overall, you should never have to manually administer the CAA cluster. Additional information on the CAA
cluster can be seen by using the lscluster command as follows:
We now need to create our resources (application controller, service address, and shared volume group) and a
resource group to add them into.
In our scenario we have no real application to utilize. So we created a dummy application controller by simply
having it execute a banner command. We can add it by executing smitty sysmirror→ Cluster Applications and
Resources→Resources→ Configure User Applications (Scripts and Monitors) → Application Controller
Scripts

Now we need to add a service IP address. To do so execute the fastpath of smitty cm_resource_menu→
Configure Service IP Labels/Addresses→ Add a Service IP Label/Address (choose net_ether0 from pop-up)
After adding the service IP we can see it has been added to the cluster topology as shown from the cllsif output
as follows:

Now we need to create a resource group and add these resources to it. To create a new resource group, execute
the fastpath of smitty cm_add_resource_group.
To add the resources to the resource group, execute the same fastpath of smitty
cm_resource_groups→Change/Show Resources and Attributes for a Resource Group and choose the
previously created resource group. Then for the fields of Service IP Labels/Addresses and Application
Controllers, press F4 and a pop-up will appear with the ones previous created. Choose them, and press Enter.
The last thing to configure is the shared data volume group, logical volume(s) and filsystem(s). This can be
accomplished by using the Cluster Single Point of Control facility (C-SPOC). Enter smitty
cspoc→Storage→Volume Groups→Create a Volume Group (choose both nodes). Then choose the desired
disk as shown below:

Then choose the desired volume group type. In most cases, and our example, a scalable volume group is
appropriate.
In the final menu, fill out the fields as desired. Also note that you can choose to added the volume group into
the previously created resource group as shown.

Though logical volumes and filesystems need to be created, the cluster can be synchronized at this time as the
resources technically will not change.
Even when using fileystems, we always recommend creating the underlying logical volume and log logical
volume in order to control the naming convention. That way it ensures the names are unique among the cluster
nodes.
To create the logical volumes and filesystems we continue to utilize C-SPOC to create a new logical volume
and log logical volume. Execute smitty cspoc→Storage→Logical Volumes→Add a Logical Volume

(choose previously created volume group)
Choose the desired disk to create the logical volume on as shown above. Then on the final menu, specify the
desired size, name and type. In our case the type is “jfs2”

After creation, we repeat the previous steps to create a jfs2log device. The only difference is specifying
“jfs2log” in the “Logical volume Type” field as shown below.
Lastly we will now create a JFS2 filesystem on top of our original previously created logical volume. We
execute smitty cspoc→Storage→File Systems→Add a File System (choose previously created volume group)
Notice: In the volume group pop-up list the CAA specific volume group of caavg_private may appear.
NEVER chose that volume group. It should be considered a bug that needs to be fixed.
Then choose the specific filesystem, in our case it is “Enhanced Journaled File System” as shown below.
Choose the previously created logical volume as shown below.

Then complete the final menu as desired and press Enter. In this case we created a filesystem with a
mountpoint of /phaflexfs.

For your environment, repeat the previous steps as needed. Once completed, just to be sure, go ahead and
synchronize the cluster.
We now have a two-node “hot-standby” cluster created consisting of the following:
• Two nodes (nodenames are same as hostnames p260_PowerHA, p460_PowerHA)

• One IP-Network (defaults to net_ether_01)
• One repository disk
• One Resource Group (p460 is primary, p260 is backup)
• One Application Server (haflextest)
• One Service Address (flexhasvc via IP Aliasing)
• One shared vg (flexhavg)
The cluster configuration details can be seen in the following screenshots:
Cluster, Nodes, Topology, and Resource Group

Resource Group and Resources (edited to show only relevant resources)

C. Testing the Cluster
Testing was completed by utilizing the Automated Procedure of the Cluster Test Tool as shown below. To
execute this test plan the cluster nodes must not be active in the cluster.
The execute the cluster test tool, enter smitty hacmp_testtool_menu, then choose “Execute Automated Test
Procedure” as shown below:
Once pressing enter, the final menu is displayed as shown below. The detailed results of each test are
displayed in the SMIT window during execution and are also saved in /var/hacmp/log/cl_testtool.log. Our
actual test results were added into the Appendix.

The overall test time was 7 minutes and the following events were executed successfully:
1. NODE_UP -- Each node one at a time.

2. NODE_DOWN_GRACEFUL – Same as above.
3. NODE_UP – Same as above.
4. NODE_DOWN_TAKEOVER – Graceful down and moves resource group from p460 to p260.
5. NODE_UP – Restart services on previously down node (p460)
6. NODE_DOWN_FORCED – On p460
7. NODE_UP – Restart services on previously down node (p460)
8. VG_DOWN – Simulates volume group loss (rg_move runs from p260 to p460)
9. CLSTRMGR_KILL – Creates hard fallover via halt on p460
While this testing does cover the core basic functionality of the cluster, additional granular level testing via the
Custom Test Procedure is often desired to include such common events as:
• FAIL_LABEL – (Both IP and Non-IP)

• NETWORK_DOWN_LOCAL – (Both IP and Non-IP)
• JOIN_LABEL – (Both IP and Non-IP)
• NETWORK_UP_LOCAL – (Both IP and Non-IP)
• SERVER_DOWN – (nice test when application monitoring is being used)

There are several specific events related to additional configuration options within PowerHA (i.e. sites, global
networks, etc). Manually creating failures for testing is also encouraged. (i.e. disabling ports, pull cables, etc).

E. References
PowerHA SystemMirror v7.1.1 for AIX www.redbooks.ibm.com/redpieces/abstracts/sg248030.html
PowerHA for AIX Version Compatibility Matrix

PowerHA for AIX Hardware Support Matrix

Implementing PowerHA and IBM i Cookbook http://www.redbooks.ibm.com/abstracts/sg247405.html?Open
PowerHA System Mirror for IBM i http://www.redbooks.ibm.com/abstracts/sg247994.html?Open

Appendix A – Cluster Test Tool Log
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: | Initializing Variable Table
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: Using Process Environment for Variable Table
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: | Reading Static Configuration Data
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: Cluster Name: PHAFlexCluster
08/08/2012_14:13:11: Cluster Version: 13
08/08/2012_14:13:11: Local Node Name: p260_PowerHA
08/08/2012_14:13:11: Cluster Nodes: p260_PowerHA p460_PowerHA
08/08/2012_14:13:11: Found 1 Cluster Networks
08/08/2012_14:13:11: Found 3 Cluster Interfaces/Device/Labels
08/08/2012_14:13:11: Found 1 Cluster Resource Groups
08/08/2012_14:13:11: Found 10 Cluster Resources
08/08/2012_14:13:11: Event Timeout Value: 720
08/08/2012_14:13:11: Maximum Timeout Value: 2880
08/08/2012_14:13:11: Found 0 Cluster Sites
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: | Building Test Queue
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: Test Plan: /usr/es/sbin/cluster/cl_testtool/auto_topology
08/08/2012_14:13:11: Event 1: NODE_UP: NODE_UP,ALL,Start cluster services on all
available nodes
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: | Validate NODE_UP
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: Event node: ALL
08/08/2012_14:13:11: Configured nodes: p260_PowerHA p460_PowerHA
08/08/2012_14:13:11: Event 2: NODE_DOWN_GRACEFUL: NODE_DOWN_GRACEFUL,node1,Stop
cluster services gracefully on a node
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: | Validate NODE_DOWN_GRACEFUL
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: Event node: p260_PowerHA
08/08/2012_14:13:11: Event 3: NODE_UP: NODE_UP,node1,Restart cluster services on the
node that was stopped
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: | Validate NODE_UP
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: Event 4: NODE_DOWN_TAKEOVER: NODE_DOWN_TAKEOVER,node2,Stop
cluster services with takeover on a node
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: | Validate NODE_DOWN_TAKEOVER
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: | Validate NODE_UP
08/08/2012_14:13:11: -------------------------------------------------------

08/08/2012_14:13:11: Event 6: NODE_DOWN_FORCED: NODE_DOWN_FORCED,node3,Stop cluster
services forced on a node
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: | Validate NODE_DOWN_FORCED
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: | Validate NODE_UP
08/08/2012_14:13:11: -------------------------------------------------------
08/08/2012_14:13:11: ###########################################################################
08/08/2012_14:13:11: ##
## Starting Cluster Test Tool: -e /usr/es/sbin/cluster/cl_testtool/auto_topology
##
08/08/2012_14:13:11: ###########################################################################
08/08/2012_14:13:11: ===========================================================================
08/08/2012_14:13:11: ||
|| Starting Test 1 - NODE_UP,ALL,Start cluster services on all available nodes
||
08/08/2012_14:13:11: ===========================================================================
08/08/2012_14:13:12: -------------------------------------------------------
08/08/2012_14:13:12: | is_rational NODE_UP
08/08/2012_14:13:12: -------------------------------------------------------
08/08/2012_14:13:12: Checking cluster stability
08/08/2012_14:13:12: p260_PowerHA: ST_INIT
08/08/2012_14:13:12: p460_PowerHA: ST_INIT
08/08/2012_14:13:12: Cluster is stable
08/08/2012_14:13:12: Active Nodes:
08/08/2012_14:13:12: -------------------------------------------------------
08/08/2012_14:13:12: | Executing Command for NODE_UP
08/08/2012_14:13:12: -------------------------------------------------------
08/08/2012_14:13:12: /usr/es/sbin/cluster/cl_testtool/cl_testtool_ctrl -e NODE_UP -m execute
'p260_PowerHA,p460_PowerHA'
08/08/2012_14:13:40: -------------------------------------------------------
08/08/2012_14:13:40: | Entering wait_for_stable
08/08/2012_14:13:40: -------------------------------------------------------
08/08/2012_14:13:40: Waiting 30 seconds for cluster to stabilize.
08/08/2012_14:14:10: Checking Node States:
08/08/2012_14:14:10: Node p260_PowerHA: ST_UNSTABLE
08/08/2012_14:14:10: Node p460_PowerHA: ST_UNSTABLE
08/08/2012_14:14:10: Cluster is not yet stable.
08/08/2012_14:14:41: Node p260_PowerHA: ST_STABLE
08/08/2012_14:14:41: Active Timers: None
08/08/2012_14:14:41: Checking if node p260_PowerHA is active
08/08/2012_14:14:41: prevstate = ST_UNSTABLE, curstate = ST_STABLE
08/08/2012_14:14:41: Checking if node p460_PowerHA is active
08/08/2012_14:14:41: prevstate = ST_UNSTABLE, curstate = ST_STABLE
08/08/2012_14:14:41: -------------------------------------------------------
08/08/2012_14:14:41: | NODE_UP: Checking post-event status
08/08/2012_14:14:41: -------------------------------------------------------
08/08/2012_14:14:41: Event Nodes: p260_PowerHA p460_PowerHA
08/08/2012_14:14:41: pre-event online nodes:
08/08/2012_14:14:41: post-event online nodes: p260_PowerHA p460_PowerHA

08/08/2012_14:14:41: Checking node states
08/08/2012_14:14:41: p260_PowerHA: Preevent state: ST_INIT, Postevent state: ST_STABLE
08/08/2012_14:14:41: Checking RG states
08/08/2012_14:14:41: Resource Group: FlexRG
08/08/2012_14:14:41: Node: p260_PowerHA Pre Event State: OFFLINE, Post Event State:
OFFLINE
ONLINE
08/08/2012_14:14:41: Checking event history
08/08/2012_14:14:41: Begin Event History records:
08/08/2012_14:14:41: NODE: p260_PowerHA
Aug 8 14:13:57 EVENT COMPLETED: node_up p260_PowerHA 0
Aug 8 14:13:59 EVENT COMPLETED: node_up_complete p260_PowerHA 0
Aug 8 14:14:12 EVENT COMPLETED: rg_move_fence p260_PowerHA 1 0
Aug 8 14:14:12 EVENT COMPLETED: rg_move p260_PowerHA 1 ACQUIRE 0
Aug 8 14:14:12 EVENT COMPLETED: rg_move_acquire p260_PowerHA 1 0
Aug 8 14:14:15 EVENT COMPLETED: rg_move_complete p260_PowerHA 1 0
08/08/2012_14:14:41: NODE: p460_PowerHA
Aug 8 14:14:14 EVENT COMPLETED: acquire_aconn_service en0 net_ether_01 0
Aug 8 14:14:14 EVENT COMPLETED: acquire_service_addr 0
Aug 8 14:14:16 EVENT COMPLETED: start_server haflextest 0
08/08/2012_14:14:41: End Event History records
08/08/2012_14:14:41: ===========================================================================
08/08/2012_14:14:41: ||
|| Test 1 Complete - NODE_UP: Start cluster services on all available nodes
||
08/08/2012_14:14:41: || Test Completion Status: PASSED
||
08/08/2012_14:14:41: ===========================================================================
08/08/2012_14:14:41: ===========================================================================
08/08/2012_14:14:41: ||
|| Starting Test 2 - NODE_DOWN_GRACEFUL,p260_PowerHA,Stop cluster services gracefully on a node
||
08/08/2012_14:14:41: ===========================================================================
08/08/2012_14:14:42: -------------------------------------------------------
08/08/2012_14:14:42: | is_rational NODE_DOWN_GRACEFUL
08/08/2012_14:14:42: -------------------------------------------------------
08/08/2012_14:14:42: p260_PowerHA: ST_STABLE
08/08/2012_14:14:42: Node: p260_PowerHA, State: ST_STABLE
08/08/2012_14:14:42: -------------------------------------------------------
08/08/2012_14:14:42: | Executing Command for NODE_DOWN_GRACEFUL
08/08/2012_14:14:42: -------------------------------------------------------
08/08/2012_14:14:42: /usr/es/sbin/cluster/cl_testtool/cl_testtool_ctrl -e NODE_DOWN_GRACEFUL
-m execute 'p260_PowerHA'
08/08/2012_14:14:52: -------------------------------------------------------
08/08/2012_14:14:52: -------------------------------------------------------
08/08/2012_14:15:23: Node p260_PowerHA: ST_INIT

08/08/2012_14:15:23: -------------------------------------------------------
08/08/2012_14:15:23: | NODE_DOWN_GRACEFUL: Checking post-event status
08/08/2012_14:15:23: -------------------------------------------------------
08/08/2012_14:15:23: Event Nodes: p260_PowerHA
08/08/2012_14:15:23: pre-event online nodes: p260_PowerHA p460_PowerHA
08/08/2012_14:15:23: post-event online nodes: p460_PowerHA
08/08/2012_14:15:23: p260_PowerHA: Preevent state: ST_STABLE, Postevent state: ST_INIT
08/08/2012_14:15:23: p460_PowerHA: Preevent state: ST_STABLE, Postevent state: ST_STABLE
OFFLINE
08/08/2012_14:15:23: Node: p460_PowerHA Pre Event State: ONLINE, Post Event State:
ONLINE
08/08/2012_14:15:23: NODE: p260_PowerHA
Aug 8 14:14:44 EVENT COMPLETED: node_down p260_PowerHA graceful 0
Aug 8 14:14:47 EVENT COMPLETED: node_down_complete p260_PowerHA 0
08/08/2012_14:15:23: NODE: p460_PowerHA
Aug 8 14:14:46 EVENT COMPLETED: node_down p260_PowerHA graceful 0
08/08/2012_14:15:23: ===========================================================================
08/08/2012_14:15:23: ||
|| Test 2 Complete - NODE_DOWN_GRACEFUL: Stop cluster services gracefully on a node
||
||
08/08/2012_14:15:23: ===========================================================================
08/08/2012_14:15:23: ===========================================================================
08/08/2012_14:15:23: ||
|| Starting Test 3 - NODE_UP,p260_PowerHA,Restart cluster services on the node that was stopped
||
08/08/2012_14:15:23: ===========================================================================
08/08/2012_14:15:24: -------------------------------------------------------
08/08/2012_14:15:24: -------------------------------------------------------
08/08/2012_14:15:24: p260_PowerHA: ST_INIT
08/08/2012_14:15:24: Node: p260_PowerHA, State: ST_INIT
08/08/2012_14:15:24: -------------------------------------------------------
08/08/2012_14:15:24: -------------------------------------------------------
'p260_PowerHA'
08/08/2012_14:15:49: -------------------------------------------------------
08/08/2012_14:15:49: -------------------------------------------------------
08/08/2012_14:16:20: -------------------------------------------------------
08/08/2012_14:16:20: -------------------------------------------------------

08/08/2012_14:16:20: pre-event online nodes: p460_PowerHA
OFFLINE
ONLINE
08/08/2012_14:16:20: NODE: p260_PowerHA
08/08/2012_14:16:20: NODE: p460_PowerHA
Aug 8 14:16:14 EVENT COMPLETED: node_up p260_PowerHA
08/08/2012_14:16:20: ===========================================================================
08/08/2012_14:16:20: ||
|| Test 3 Complete - NODE_UP: Restart cluster services on the node that was stopped
||
||
08/08/2012_14:16:20: ===========================================================================
08/08/2012_14:16:20: ===========================================================================
08/08/2012_14:16:20: ||
|| Starting Test 4 - NODE_DOWN_TAKEOVER,p460_PowerHA,Stop cluster services with takeover on a node
||
08/08/2012_14:16:20: ===========================================================================
08/08/2012_14:16:21: -------------------------------------------------------
08/08/2012_14:16:21: | is_rational NODE_DOWN_TAKEOVER
08/08/2012_14:16:21: -------------------------------------------------------
08/08/2012_14:16:21: -------------------------------------------------------
08/08/2012_14:16:21: | Executing Command for NODE_DOWN_TAKEOVER
08/08/2012_14:16:21: -------------------------------------------------------
08/08/2012_14:16:21: /usr/es/sbin/cluster/utilities/cl_rsh -n p460_PowerHA
/usr/es/sbin/cluster/cl_testtool/cl_testtool_ctrl -e NODE_DOWN_TAKEOVER -m execute 'p460_PowerHA'
08/08/2012_14:16:34: -------------------------------------------------------
08/08/2012_14:16:34: -------------------------------------------------------
08/08/2012_14:17:04: Node p460_PowerHA: ST_INIT
08/08/2012_14:17:04: -------------------------------------------------------
08/08/2012_14:17:04: | NODE_DOWN_TAKEOVER: Checking post-event status
08/08/2012_14:17:04: -------------------------------------------------------
08/08/2012_14:17:05: p460_PowerHA: Preevent state: ST_STABLE, Postevent state: ST_INIT

ONLINE
OFFLINE
08/08/2012_14:17:05: NODE: p260_PowerHA
Aug 8 14:16:23 EVENT COMPLETED: node_down p460_PowerHA 0
Aug 8 14:16:25 EVENT COMPLETED: rg_move p260_PowerHA 1 RELEASE 0
Aug 8 14:16:25 EVENT COMPLETED: rg_move_release p260_PowerHA 1 0
Aug 8 14:16:31 EVENT COMPLETED: acquire_takeover_addr 0
08/08/2012_14:17:05: NODE: p460_PowerHA
Aug 8 14:16:27 EVENT COMPLETED: stop_server haflextest 0
Aug 8 14:16:28 EVENT COMPLETED: release_service_addr 0
08/08/2012_14:17:05: ===========================================================================
08/08/2012_14:17:05: ||
|| Test 4 Complete - NODE_DOWN_TAKEOVER: Stop cluster services with takeover on a node
||
||
08/08/2012_14:17:05: ===========================================================================
08/08/2012_14:17:05: ===========================================================================
08/08/2012_14:17:05: ||
||
08/08/2012_14:17:05: ===========================================================================
08/08/2012_14:17:06: -------------------------------------------------------
08/08/2012_14:17:06: -------------------------------------------------------
08/08/2012_14:17:06: p460_PowerHA: ST_INIT
08/08/2012_14:17:06: Node: p460_PowerHA, State: ST_INIT
08/08/2012_14:17:06: -------------------------------------------------------
08/08/2012_14:17:06: -------------------------------------------------------
'p460_PowerHA'
08/08/2012_14:17:33: -------------------------------------------------------
08/08/2012_14:17:33: -------------------------------------------------------

08/08/2012_14:18:03: -------------------------------------------------------
08/08/2012_14:18:03: -------------------------------------------------------
08/08/2012_14:18:04: pre-event online nodes: p260_PowerHA
ONLINE
OFFLINE
08/08/2012_14:18:04: NODE: p260_PowerHA
08/08/2012_14:18:04: NODE: p460_PowerHA
08/08/2012_14:18:04: ===========================================================================
08/08/2012_14:18:04: ||
||
||
08/08/2012_14:18:04: ===========================================================================
08/08/2012_14:18:04: ===========================================================================
08/08/2012_14:18:04: ||
|| Starting Test 6 - NODE_DOWN_FORCED,p460_PowerHA,Stop cluster services forced on a node
||
08/08/2012_14:18:04: ===========================================================================
08/08/2012_14:18:04: -------------------------------------------------------
08/08/2012_14:18:04: | is_rational NODE_DOWN_FORCED
08/08/2012_14:18:04: -------------------------------------------------------
08/08/2012_14:18:04: Node: p260_PowerHA, Force Down:
08/08/2012_14:18:04: Node: p460_PowerHA, Force Down:
08/08/2012_14:18:04: -------------------------------------------------------
08/08/2012_14:18:04: | Executing Command for NODE_DOWN_FORCED
08/08/2012_14:18:04: -------------------------------------------------------
/usr/es/sbin/cluster/cl_testtool/cl_testtool_ctrl -e NODE_DOWN_FORCED -m execute 'p460_PowerHA'
08/08/2012_14:18:06: -------------------------------------------------------
08/08/2012_14:18:06: -------------------------------------------------------
08/08/2012_14:18:37: -------------------------------------------------------
08/08/2012_14:18:37: | NODE_DOWN_FORCED: Checking post-event status

08/08/2012_14:18:37: -------------------------------------------------------
08/08/2012_14:18:37: Checking forced down node lists
08/08/2012_14:18:37: Node: p260_PowerHA, Force Down: p460_PowerHA
ONLINE
OFFLINE
08/08/2012_14:18:37: NODE: p260_PowerHA
Aug 8 14:18:07 EVENT COMPLETED: node_down p460_PowerHA forced 0
Aug 8 14:18:09 EVENT COMPLETED: node_down_complete p460_PowerHA forced 0
08/08/2012_14:18:37: NODE: p460_PowerHA
Aug 8 14:18:08 EVENT COMPLETED: node_down p460_PowerHA forced 0
Aug 8 14:18:10 EVENT COMPLETED: node_down_complete p460_PowerHA forced 0
08/08/2012_14:18:37: ===========================================================================
08/08/2012_14:18:37: ||
|| Test 6 Complete - NODE_DOWN_FORCED: Stop cluster services forced on a node
||
||
08/08/2012_14:18:37: ===========================================================================
08/08/2012_14:18:37: ===========================================================================
08/08/2012_14:18:37: ||
||
08/08/2012_14:18:37: ===========================================================================
08/08/2012_14:18:38: -------------------------------------------------------
08/08/2012_14:18:38: -------------------------------------------------------
08/08/2012_14:18:38: Node: p460_PowerHA, State: ST_STABLE
08/08/2012_14:18:38: Node: p260_PowerHA, Force Down: p460_PowerHA
08/08/2012_14:18:38: -------------------------------------------------------
08/08/2012_14:18:38: -------------------------------------------------------
'p460_PowerHA'
08/08/2012_14:18:49: -------------------------------------------------------
08/08/2012_14:18:49: -------------------------------------------------------
08/08/2012_14:19:19: -------------------------------------------------------
08/08/2012_14:19:19: -------------------------------------------------------

ONLINE
OFFLINE
08/08/2012_14:19:20: NODE: p260_PowerHA
08/08/2012_14:19:20: NODE: p460_PowerHA
08/08/2012_14:19:20: ===========================================================================
08/08/2012_14:19:20: ||
||
||
08/08/2012_14:19:20: ===========================================================================
08/08/2012_14:19:20: ###########################################################################
08/08/2012_14:19:20: ##
## Cluster Testing Complete: Exit Code 0
##
08/08/2012_14:19:20: ###########################################################################
08/08/2012_14:19:20: -------------------------------------------------------
08/08/2012_14:19:20: -------------------------------------------------------
08/08/2012_14:19:20: -------------------------------------------------------
08/08/2012_14:19:20: -------------------------------------------------------
08/08/2012_14:19:20: Cluster Version: 13
08/08/2012_14:19:20: -------------------------------------------------------
08/08/2012_14:19:20: -------------------------------------------------------
08/08/2012_14:19:20: Test Plan: /usr/es/sbin/cluster/cl_testtool/auto_vg
08/08/2012_14:19:20: Event 1: VG_DOWN: VG_DOWN,vg1,ANY,Bring down volume group
08/08/2012_14:19:20: -------------------------------------------------------
08/08/2012_14:19:20: | Validate VG_DOWN
08/08/2012_14:19:20: -------------------------------------------------------
08/08/2012_14:19:20: Event node: ANY
08/08/2012_14:19:20: VG: flexhavg, RG Name: FlexRG
08/08/2012_14:19:20: ###########################################################################
08/08/2012_14:19:20: ##
## Starting Cluster Test Tool: -c -e /usr/es/sbin/cluster/cl_testtool/auto_vg

##
08/08/2012_14:19:20: ###########################################################################
08/08/2012_14:19:20: ===========================================================================
08/08/2012_14:19:20: ||
|| Starting Test 1 - VG_DOWN,ANY,flexhavg
||
08/08/2012_14:19:20: ===========================================================================
08/08/2012_14:19:21: -------------------------------------------------------
08/08/2012_14:19:21: | is_rational VG_DOWN
08/08/2012_14:19:21: -------------------------------------------------------
08/08/2012_14:19:21: VG: flexhavg, RG: FlexRG, ONLINE NODES: p260_PowerHA
08/08/2012_14:19:21: -------------------------------------------------------
08/08/2012_14:19:21: | Executing Command for VG_DOWN
08/08/2012_14:19:21: -------------------------------------------------------
08/08/2012_14:19:21: /usr/es/sbin/cluster/cl_testtool/cl_testtool_ctrl -e VG_DOWN -m execute
'flexhavg'
08/08/2012_14:19:22: -------------------------------------------------------
08/08/2012_14:19:22: -------------------------------------------------------
08/08/2012_14:19:52: -------------------------------------------------------
08/08/2012_14:19:52: | VG_DOWN: Checking post-event status
08/08/2012_14:19:52: -------------------------------------------------------
08/08/2012_14:19:53: RESID: 2, RG: FlexRG, RGID: 1, TYPE: 0
08/08/2012_14:19:53: Volume Group: flexhavg Failure Action: fallover
OFFLINE
ONLINE
08/08/2012_14:19:53: NODE: p260_PowerHA
Aug 8 14:19:22 EVENT COMPLETED: resource_state_change p260_PowerHA 0
Aug 8 14:19:23 EVENT COMPLETED: stop_server haflextest 0
Aug 8 14:19:25 EVENT COMPLETED: release_takeover_addr 0
Aug 8 14:19:28 EVENT COMPLETED: rg_move_fence p260_PowerHA 1
Aug 8 14:19:33 EVENT COMPLETED: resource_state_change_complete p260_PowerHA 0
08/08/2012_14:19:53: NODE: p460_PowerHA
Aug 8 14:19:23 EVENT COMPLETED: resource_state_change p260_PowerHA 0

Aug 8 14:19:30 EVENT COMPLETED: acquire_aconn_service en0 net_ether_01 0
Aug 8 14:19:30 EVENT COMPLETED: acquire_service_addr 0
Aug 8 14:19:34 EVENT COMPLETED: resource_state_change_complete p260_PowerHA 0
08/08/2012_14:19:53: ===========================================================================
08/08/2012_14:19:53: ||
|| Test 1 Complete - VG_DOWN: Bring down volume group
||
||
08/08/2012_14:19:53: ===========================================================================
08/08/2012_14:19:53: ###########################################################################
08/08/2012_14:19:53: ##
##
08/08/2012_14:19:53: ###########################################################################
08/08/2012_14:19:53: -------------------------------------------------------
08/08/2012_14:19:53: -------------------------------------------------------
08/08/2012_14:19:53: -------------------------------------------------------
08/08/2012_14:19:53: -------------------------------------------------------
08/08/2012_14:19:53: Cluster Version: 13
08/08/2012_14:19:53: -------------------------------------------------------
08/08/2012_14:19:53: -------------------------------------------------------
08/08/2012_14:19:53: Test Plan: /usr/es/sbin/cluster/cl_testtool/auto_cluster_kill
08/08/2012_14:19:53: Event 1: CLSTRMGR_KILL: CLSTRMGR_KILL,node1,Kill the cluster
manager on a node
08/08/2012_14:19:53: -------------------------------------------------------
08/08/2012_14:19:53: | Validate CLSTRMGR_KILL
08/08/2012_14:19:53: -------------------------------------------------------
08/08/2012_14:19:53: ###########################################################################
08/08/2012_14:19:53: ##
## Starting Cluster Test Tool: -c -e /usr/es/sbin/cluster/cl_testtool/auto_cluster_kill
##
08/08/2012_14:19:53: ###########################################################################
08/08/2012_14:19:53: ===========================================================================
08/08/2012_14:19:53: ||
|| Starting Test 1 - CLSTRMGR_KILL,p460_PowerHA,Kill the cluster manager on a node
||
08/08/2012_14:19:53: ===========================================================================
08/08/2012_14:19:54: -------------------------------------------------------
08/08/2012_14:19:54: | is_rational CLSTRMGR_KILL
08/08/2012_14:19:54: -------------------------------------------------------

08/08/2012_14:19:54: -------------------------------------------------------
08/08/2012_14:19:54: | Executing Command for CLSTRMGR_KILL
08/08/2012_14:19:54: -------------------------------------------------------
/usr/es/sbin/cluster/cl_testtool/cl_testtool_ctrl -e CLSTRMGR_KILL -m execute 'p460_PowerHA'
08/08/2012_14:19:55: -------------------------------------------------------
08/08/2012_14:19:55: -------------------------------------------------------
08/08/2012_14:20:31: Node p460_PowerHA:
08/08/2012_14:20:31: -------------------------------------------------------
08/08/2012_14:20:31: | CLSTRMGR_KILL: Checking post-event status
08/08/2012_14:20:31: -------------------------------------------------------
08/08/2012_14:20:38: p460_PowerHA: Preevent state: ST_STABLE, Postevent state:
ONLINE
OFFLINE
08/08/2012_14:20:38: NODE: p260_PowerHA
Aug 8 14:19:59 EVENT COMPLETED: acquire_takeover_addr 0
Aug 8 14:20:03 EVENT COMPLETED: start_server haflextest 0Aug 8 14:20:03 EVENT COMPLETED:
rg_move_complete p260_PowerHA 1 0
08/08/2012_14:20:38: ===========================================================================
08/08/2012_14:20:38: ||
|| Test 1 Complete - CLSTRMGR_KILL: Kill the cluster manager on a node
||
||
08/08/2012_14:20:38: ===========================================================================
08/08/2012_14:20:38: ###########################################################################
08/08/2012_14:20:38: ##
##
08/08/2012_14:20:38: ###########################################################################

Powerha Systemmirror For Aix V7.1 Two-Node Quick Configuration Guide

Uploaded by

Copyright:

Available Formats

Powerha Systemmirror For Aix V7.1 Two-Node Quick Configuration Guide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Powerha Systemmirror For Aix V7.1 Two-Node Quick Configuration Guide

Uploaded by

Copyright:

Available Formats

PowerHA SystemMirror for AIX

v7.1 Two-Node Quick

COPYRIGHT IBM CORPORATION, 2012 1

COPYRIGHT IBM CORPORATION, 2012 2

My environment during the original writing was comprised of the following:

COPYRIGHT IBM CORPORATION, 2012 3

1. CAA specific filesets bos.cluster and bos.ahafs

COPYRIGHT IBM CORPORATION, 2012 4

COPYRIGHT IBM CORPORATION, 2012 5

COPYRIGHT IBM CORPORATION, 2012 6

COPYRIGHT IBM CORPORATION, 2012 7

Configuring VLAN 3358 for SAN heartbeat

COPYRIGHT IBM CORPORATION, 2012 8

COPYRIGHT IBM CORPORATION, 2012 9

COPYRIGHT IBM CORPORATION, 2012 10

COPYRIGHT IBM CORPORATION, 2012 11

COPYRIGHT IBM CORPORATION, 2012 12

COPYRIGHT IBM CORPORATION, 2012 13

COPYRIGHT IBM CORPORATION, 2012 14

Choose the previously created logical volume as shown below.

COPYRIGHT IBM CORPORATION, 2012 15

COPYRIGHT IBM CORPORATION, 2012 16

We now have a two-node “hot-standby” cluster created consisting of the following:

• Two nodes (nodenames are same as hostnames p260_PowerHA, p460_PowerHA)

The cluster configuration details can be seen in the following screenshots:

Cluster, Nodes, Topology, and Resource Group

COPYRIGHT IBM CORPORATION, 2012 17

COPYRIGHT IBM CORPORATION, 2012 18

COPYRIGHT IBM CORPORATION, 2012 19

1. NODE_UP -- Each node one at a time.

• FAIL_LABEL – (Both IP and Non-IP)

COPYRIGHT IBM CORPORATION, 2012 20

COPYRIGHT IBM CORPORATION, 2012 21

PowerHA SystemMirror v7.1.1 for AIX www.redbooks.ibm.com/redpieces/abstracts/sg248030.html

PowerHA for AIX Version Compatibility Matrix

PowerHA for AIX Hardware Support Matrix

Implementing PowerHA and IBM i Cookbook http://www.redbooks.ibm.com/abstracts/sg247405.html?Open

PowerHA System Mirror for IBM i http://www.redbooks.ibm.com/abstracts/sg247994.html?Open

COPYRIGHT IBM CORPORATION, 2012 22

COPYRIGHT IBM CORPORATION, 2012 23

COPYRIGHT IBM CORPORATION, 2012 24

COPYRIGHT IBM CORPORATION, 2012 25

COPYRIGHT IBM CORPORATION, 2012 26

COPYRIGHT IBM CORPORATION, 2012 27

COPYRIGHT IBM CORPORATION, 2012 28

COPYRIGHT IBM CORPORATION, 2012 29

COPYRIGHT IBM CORPORATION, 2012 30

COPYRIGHT IBM CORPORATION, 2012 31

COPYRIGHT IBM CORPORATION, 2012 32

COPYRIGHT IBM CORPORATION, 2012 33

COPYRIGHT IBM CORPORATION, 2012 34

You might also like