Red Hat Enterprise Virtualization 3.0 and Netapp Storage Deployment Guide
Red Hat Enterprise Virtualization 3.0 and Netapp Storage Deployment Guide
Red Hat Enterprise Virtualization 3.0 and Netapp Storage Deployment Guide
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
Jon Benedict, NetApp March 2012 | TR-3940
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
8.1
11 Deploying and Configuring Disaster Recovery for RHEV 3 ........................................................... 18 12 Site Failover for RHEL 6 KVM and NetApp Storage ........................................................................ 18
12.1 Deploying SnapMirror Async for Site Failover...............................................................................................18 12.2 Deploying MetroCluster for Site Failover ......................................................................................................21
Appendix: Ports to Allow Through Firewall ........................................................................................... 24 References ................................................................................................................................................. 25 Version History ......................................................................................................................................... 26
LIST OF TABLES
Table 1) Ports to allow through firewall. .......................................................................................................................24 Table 2) Ports for LDAP-based authentication. ............................................................................................................24
LIST OF FIGURES
Figure 1) Thick and thin hypervisors. .............................................................................................................................5 Figure 2) Example of the secondary site layout. ...........................................................................................................19 Figure 3) Example of an RHEV 3 layout with MetroCluster. .........................................................................................22
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
NetApp technology enables data centers to extend their virtual infrastructures to include the benefits of advanced storage virtualization. NetApp unified storage platforms offer industry-leading technologies in the areas of storage efficiencies, instantaneous virtual machine (VM) and datastore cloning for virtual servers, and virtual data center backup and business continuance solutions. Red Hat Enterprise Linux (RHEL) also offers the benefits of flexible deployment: It can be deployed as a bare-metal operating system, as a hypervisor, or as a virtual guest operating system. From a storage perspective, RHEL KVM supports both storage area network (SAN: iSCSI, Fibre Channel [FC], Fibre Channel over Ethernet [FCoE]) and network-attached storage (NAS: Network File System [NFS]) for shared virtual machine storage.
NetApp and Red Hat maintain a long-term strategic alliance that includes end-to-end solution testing between Red Hat products and NetApp storage. As a result of this testing, NetApp has developed operational guidelines and best practices for storage arrays running the NetApp Data ONTAP operating system in support of Red Hat Enterprise Linux. These guidelines have been extended to include RHELbased KVM virtualization. The following sections provide the steps necessary to deploy Red Hat Enterprise Virtualization 3.0 and NetApp storage together in accordance with TR-3914: Red Hat Enterprise Virtualization 3 and NetApp Storage: Best Practices Guide.
1.2
Intended Audience
This document is intended for system architects, system administrators, and storage administrators who are deploying Red Hat Enterprise Virtualization on NetApp storage.
1.3
Scope
The steps provided are specific to where Red Hat and NetApp technologies intersect. That is to say that because Red Hat and NetApp provide excellent product documentation, this document cover topics that are different or not covered by existing documents.
1.4
To avoid unnecessary duplication in the areas of RHEL 6 KVM hosts and virtual machines, some sections of this deployment guide have been truncated. In these cases, there are references made to existing documents such as: TR-3848: RHEL 6, KVM, and NetApp Storage: Best Practices Guide TR-4034: Red Hat Enterprise Linux 6, KVM, and NetApp Storage Deployment Guide TR-3914: Red Hat Enterprise Virtualization 3 and NetApp Storage: Best Practices Guide Red Hat Enterprise Linux 6 Product Guides Red Hat Enterprise Virtualization Product Guides
These documents should be reviewed thoroughly prior to planning or deploying RHEV 3.0 with NetApp storage. Additionally, this document does not attempt to recreate Red Hats product documentation. Instead, this document attempts to highlight where Red Hat Enterprise Virtualization and NetApp storage intersect or where deployment steps for one technology affect the other.
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
1.5
Deployment steps associated with IP and Fibre Channel networks are not covered in this document. However, a good understanding of these topics is necessary for properly configuring items such as VLANs, switched fabrics, and other related technologies.
1.6
Following the steps in this document is fully supported by both NetApp and Red Hat for their respective products. However, the ways and means described here cannot possibly cover every scenario or business need. Rather, this deployment guide should serve as a starting point. Should your environment require topics not covered in this document, technical resources from Red Hat and NetApp are available to provide guidance that is compliant with support.
1.7
As described in TR-3914: Red Hat Enterprise Virtualization 3 and NetApp Storage: Best Practices Guide, Red Hat supports both thick and thin hypervisors. This document focuses mostly on the thin or RHEV-H hypervisor available with Red Hat Enterprise Virtualization. There are use cases for using both thick and thin hypervisors in an RHEV environment.
Figure 1) Thick and thin hypervisors.
RHEL 6
KVM
KVM
Various Drivers
2.1
Prerequisites
Data ONTAP installed on its own root FlexVol volume and aggregate
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
Basic networking to include a primary interface, management interface, and host name that is fully resolvable in DNS A second NetApp FAS controller configured in active-active configuration (or MetroCluster configuration) At least one large disk aggregate that is separate from the root aggregate to be used for data storage
2.2
3. Log in with the new administrator account to verify that it works. 4. Disable the root account from the new administrator account:
options security.passwd.rootaccess.enable off
6. Secure Shell (SSH) protocol must be configured and enabled. Use the following one-time command to generate host keys:
secureadmin setup -q ssh 768 512 768
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
2.3
If this is a new NetApp FAS controller, specific licenses must be obtained and installed prior to deploying storage. The most relevant licenses to this deployment guide include, but are not limited to: FCP (includes FCoE) iSCSI NFS Deduplication SnapRestore SnapMirror SyncMirror local (if using MetroCluster) Cluster remote (if using MetroCluster)
One or more licenses can be installed as follows from the NetApp command line:
license add <first_license> <second_license> <third_license>
2.4
NFSv3
1. Add a license for NFS:
license add <LICENSE_KEY>
3. Enable NFS:
nfs on
Note:
If this is part of an active-active pair, run these steps against both controllers.
3. Record the WWPN or FC port name for later use by entering the following command:
fcp show adapters
4. Check whether the Fibre Channel ports are targets or initiators by entering the following command:
fcadmin config
5. Make a Fibre Channel port into a target. Note: Only Fibre Channel ports that are configured as targets can be used to connect to initiator hosts on the SAN.
For example, make a port called 0b into a target port run by entering the following command:
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
Note:
If an initiator port is made into a target port, a reboot will be required. NetApp recommends rebooting after completing the entire configuration, because other configuration steps might also require a reboot. If this is part of an active-active pair, run these steps against both controllers.
Note:
iSCSI
The following steps configure the iSCSI service on a storage system. These steps do not include any host-specific configuration tasks. 1. From the storage controller CLI, license the iSCSI protocol:
license add <LICENSE_KEY>
Note:
iscsi start
3. Enable or disable the appropriate storage system interfaces for use with iSCSI. This example enables the iSCSI protocol on interfaces e0a and e0c and disables iSCSI on interfaces e0b, e0d, and e0M. Note: By default, all storage system interfaces are enabled for iSCSI when the service is started.
iscsi interface enable e0a e0c iscsi interface disable e0b e0d e0M
Note:
If this is part of an active-active pair, run these steps against both controllers.
2.5
Prior to creating NFS exports or LUNs, one or more FlexVol volumes must be created. This in turn requires that at least one disk aggregate be created based on NetApp best practices.
Creating a LUN
LUNs are created within a FlexVol volume. 1. To create a 2GB LUN in volume rhev_vol called rhev_lun, run the following command:
lun create s 2g t linux /vol/rhev_vol01/rhev_lun01
2. Create an igroup:
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
The -t option specifies the OS type, and f specifies FC. Change the WWPN to reflect your environment. For iSCSI, an igroup named rhev_igroup02 is created:
Note: Note:
The -t option specifies the OS type, and i specifies iSCSI. Change the IQN to reflect your environment. The IQN can be found on the HBA BIOS (if using a hardware-based initiator) or in the /etc/iscsi/initiatorname.iscsi file (if using the native RHEL software-based initiator).
3. Map the LUN used to boot the igroup (in this example to the iSCSI igroup):
lun map /vol/rhev_vol01/rhev_lun01 rhev_igroup02
4. Mount and format the LUN if using for data, or install the operating system if it is a boot LUN. Note: Depending on the operating system, the concept of file system alignment might need to be addressed. Refer to TR-3747: Best Practices for File System Alignment in Virtual Environments.
Note:
The autodelete setting differs from the one presented in RA-0007: Storage Efficiency Every Day: How to Achieve and Manage Best-in-Class Storage Use.http://media.netapp.com/documents/RA-0007.pdf In RA-0007, autodelete is set to off. Here it is set to on to allow the storage to be imported into Provisioning Manager. Setting the autosize option to two times the original size of the volume allows the volume to double its original size by using more space from the aggregate. Be certain to understand the consequences of this setting and to monitor free space in all aggregates.
Note:
3. Export a new volume read/write to a host called 192.168.27.40 and put the entry permanently into the /etc/exports file:
exportfs p rw=192.168.27.40 /vol/rhev_vol02
The volume should now be accessible from the host. 4. The /etc/exports file can also be modified manually. When this modification is complete, run the following command to export all existing entries in the /etc/exports file:
exportfs a
Note:
The exportfs command has many options. For more information, refer to Data ONTAP 7.3.3 Documentation.
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
Note:
The autodelete setting differs from the one presented in RA-0007: Storage Efficiency Every Day: How to Achieve and Manage Best-in-Class Storage Use. In RA-0007, autodelete is set to off. Here it is set to on to allow the storage to be imported into Provisioning Manager. Setting the autosize option to two times the original size of the volume allows the volume to double its original size by using more space from the aggregate. Be certain to understand the consequences of this setting and to monitor free space in all aggregates.
Note:
Enabling Deduplication
1. Enable deduplication on volume /vol/rhev_vol01 by running:
sis on /vol/rhev_vol01
2. Start the initial scan (which should be run only once). Note: The initial scan of the data is the most resource intensive, so make certain this is an acceptable time to run it.
3. Answer yes to the question. 4. Change the default deduplication schedule by running:
sis config s SCHEDULE /vol/rhev_vol01
Where SCHEDULE can be in one of these formats: Note: [day_list][@hour_list] [hour_list][@day_list] auto The auto setting runs only when 20% of the data in the volume has changed.
3.1
1. Install the MultiStore license. 2. Create an IP space for a vFiler unit or multiple vFiler units:
ipspace create var_ipspace01
3. Assign an interface that will be used for a vFiler unit to the newly created IP space:
10
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
The following procedure provisions one nondefault vFiler unit with a dedicated IP space, and one additional data volume is also assigned to the vFiler unit. 4. Create a primary storage unit for the vFiler unit:
vol create var_vfiler01_rootvol var_aggr01 256m
5. Create a vFiler unit and specify the root volume and IP space that will be used. Note: Note: The IP address used by the vFiler unit must not be configured when the vFiler unit is created. Quotas must be turned off before assigning a qtree or volume to a vFiler unit; they may be turned back on after the resource is assigned.
6. After assigning an IP space to a vFiler unit, the IP space cannot be changed without destroying the vFiler unit.
vfiler create var_vfiler01 -n -s var_ipspace01 -i var_interface01 /vol/var_vfiler01_rootvol
7. Disallow rsh and any other protocols not needed by the newly created vFiler unit by using the vfiler disallow command. The following command disables rsh, cifs, iscsi, http, and ftp on the vFiler unit:
vfiler disallow var_vfiler01 proto=rsh proto=cifs proto=iscsi proto=http proto=ftp
8. Use the ifconfig command on the hosting storage system to configure the interface as Up with the IP address specified during the creation of the vFiler unit. Note: The IP address used here must be the same address used in the vfiler create command:
9. Modify the routing table of the IP space the vFiler unit is using with the vfiler run command. This command adds a default route for the newly created IP space that the Vfiler unit is using.
vfiler run var_vfiler01 route add default var_dgateway01 1
The following deployment examples assume that redundant switches are set up and that Link Aggregate Control Protocol (LACP) has been enabled and configured on the relevant switch ports.
4.2
IFGRP LACP
Since this type of interface group requires two or more Ethernet interfaces and a switch that supports LACP, make sure that the switch is configured properly. The commands are the same for Gigabit Ethernet (GbE) or 10 Gigabit Ethernet (10GbE). Run the following command on the command line and also add it to the /etc/rc file, so it is activated upon boot. This example assumes that there are two network interfaces called e0a and e0b and that an interface group called vif01 is being created:
11
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
ifgrp create lacp vif01 b ip e0a e0b wrfile -a /etc/rc "ifgrp create lacp vif01 b ip e0a e0b"
Note:
All interfaces must be in down status before being added to an interface group.
4.3
VLAN
1. Run the following commands to create tagged VLANs. This example assumes that there is a network ifgrp called vif01 and that VLANs are being created that are tagged with numbers 10, 20, and 30. For failover to work properly, the commands must also be run on the other controller in an activeactive pair.
vlan create vif01 10 20 30 wrfile -a /etc/rc "vlan create e0a 10 20 30"
Note:
Although this example uses an ifgrp network interface, a non-ifgrp (e0a, e0b) can be used instead.
4.4
Note:
IP Config
This example assumes that the user has a network interface called vif01-10 with IP address 192.168.10.10 and default router 192.168.10.1. If routing is not needed, skip the commands starting with route add.
192.168.10.10 netmask 255.255.255.0 mtusize 1500 flowcontrol none "ifconfig vif01-10 192.168.10.10 netmask 255.255.255.0 mtusize 1500 flowcontrol 192.168.10.1 1 "route add default 192.168.10.1 1" "routed on"
ifconfig vif01-10 wrfile a /etc/rc none" route add default wrfile a /etc/rc routed on wrfile a /etc/rc
Note:
The interface can be either physical (for example, e0a), an ifgrp (VIF), or a VLAN. Run this routine once for every physical or virtual interface for which an IP address is needed. If MultiStore is used, routing is common to every vFiler unit in an IP space. If a 10Gb per second interface is not used, remove the flowcontrol none option. If jumbo frames are needed, change the mtusize option to 9000.
12
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
5.1
The steps to deploy RHEV-M do not differ from Red Hats product guides, with one exception. It should be deployed on a virtual machine within an infrastructure cluster, as noted earlier. The following steps describe the overall work flow for deploying RHEV-M. The following steps assume that NetApp storage has been provisioned for an infrastructure cluster as well as storage for ISO images and virtual machine storage. 1. Deploy an infrastructure cluster of at least two RHEL 6 KVM hosts, as described in TR-3848: RHEL 6, KVM, and NetApp Storage: Best Practices. 2. Deploy an RHEL 6 virtual machine on the infrastructure cluster. It must have a fully resolvable host name. See the appendix for the ports that should be allowed through the firewall. 3. Register the virtual machine to Red Hat Network (RHN) and subscribe it to the RHEV-M-specific software channels. For example:
rhn_register
4. Follow the prompts to complete the registration, then subscribe to the required channels:
rhn-channel --add --channel=rhel-x86_64-server-6-rhevm-3 rhn-channel --add --channel=jbappplatform-5-x86_64-server-6-rpm rhn-channel --add --channel=rhel-x86_64-server-supplementary-6
5. Update the system, install the RHEV-M package, and configure RHEV-M:
yum y upgrade yum y install rhevm rhevm-setup
Note:
Accept all of the defaults for ports. Do not set up a default storage domain. Do not allow the configuration tool to rewrite the IPtables firewall.
6. Configure the firewall on the RHEV-M virtual machine. The list of required ports is included in the appendix. 7. After the basic install and configuration are complete, at least one RHEV data center will need to be created. This will include the following items: At least one RHEV cluster At least one logical network (with VLAN tagging) At least one hypervisor At least one storage domain
1. Download the RHEV-H ISO image from Red Hat Network (RHN). Depending on how it is to be deployed, it can be burned to CD, written to a USB drive, or converted for used with a PXE network. 2. Configure the physical server to boot from the NetApp controller using FC, FCoE, or iSCSI. 3. Boot to the RHEV-H image and follow the prompts (if installing interactively) or allow PXE to automatically deploy the image. If deploying interactively, the installer will also prompt for the boot device. Note: When the initial install is complete, the RHEV-H host will reboot. Log in to the host with the user admin and the password that was chosen during the installation. No root user is available.
13
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
If deploying interactively, the RHEV-H configuration will require at least a host name, network interface, DNS server, NTP server, and host name for RHEV-M. These items are configured through a text user interface (TUI). When the configuration is complete, the RHEV-H host will show up in RHEV-M with a status of Pending Approval.
6.2
These steps assume that the RHEL 6 KVM host is fully installed and configured per TR-3848: RHEL 6, KVM, and NetApp Storage: Best Practices. This includes but is not limited to security configuration, storage configuration, and package selection. In addition, the RHEL 6 KVM host must have a fully resolvable host name. It is also assumed that the RHEL 6 KVM host boots from a SAN device on the NetApp controller. 1. Register the RHEL 6 KVM host to Red Hat Network. 2. Subscribe the system to the Red Hat Enterprise Virt Management software channel. 3. Make sure that there is a complete host name listing in the /etc/hosts file. 4. Configure IPtables to allow required ports as listed in the appendix. Note: Configuring virtual networks or SSH host keys is not necessary (RHEV-M will take care of this).
5. Log in to RHEV-M, and click the Hosts tab. 6. Click the New button. Provide the host name, IP address, and root password, then click OK. 7. RHEV-M will then install a number of packages and reboot the host.
7 Configuring RHEV-M
7.1 Configuring an RHEV Data Center
1. Log in to the RHEV-M Web portal and click the New button on the Data Centers tab. 2. Enter a name and description of the data center to be created. 3. Select the storage type of the data center. Select the storage protocol to be used for storing virtual machines in the data center from the drop-down menu to your data center. 4. Select the compatibility level of the data center, 2.2 or 3.0, and click OK. 5. When the Guide Me dialog box is displayed, click Configure Later.
7.2
1. Log in to the RHEV-M Web portal and select the Clusters tab, then click the New button. 2. When the New Cluster dialog box is displayed, select the General tab. 3. Select an existing data center from the drop-down menu and then enter a cluster name and description. 4. Select the CPU name for hosts in this cluster and then select the compatibility level. 5. Select the memory optimization to be utilized. 6. Select the Resilience Policy tab to define if high availability is to be a consideration for any virtual guest in the RHEV cluster. 7. Click OK to create the cluster. 8. When the Guide Me dialog box is displayed, click Configure Later.
14
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
7.3
1. Log in to the RHEV-M Web portal and select the Data Centers tab, then the Logical Networks tab. 2. Click the New button. Enter a name and description. Check the Enable VLAN tagging box and also enter a VLAN number. This should match the VLAN configured on the network. 3. Select the RHEV cluster(s) that will have access to this VLAN, then click OK. 4. The remaining steps are completed while configuring network interfaces on the hypervisors. a. Select the Hosts tab, then the hypervisor that will have an interface and VLAN configured, then the Network Interfaces tab. b. Select the physical interface to be configured, then click Add/Edit. c. Select the network from the drop-down menu and then select either DHCP or Static for the configuration. If Static is selected, an IP address and network mask are also required. These steps can be repeated several times (especially on 10GbE devices) to maintain and operate multiple logical networks and VLANs on the same physical device.
7.4
Although there are three types of storage domains in RHEV, this guide will only discuss the use of a data domain and an ISO domain. The Red Hat Enterprise Virtualization product guides should be followed for specific details. The following sections describe the workflow required.
15
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
Note:
Prior to performing any actions in RHEV-M, the NFS export must be mounted manually to configure proper ownership for RHEV. Failure to perform this initial step will result in the inability to mount NFS storage:
This section describes how to provision virtual machines within RHEV using the RHEV-M portal.
The configuration of virtual machines is covered in detail in TR-3848: RHEL 6, KVM, and NetApp Storage: Best Practices and TR-4034: Red Hat Enterprise Linux 6, KVM, and NetApp Storage: Deployment Guide. Procedure for provisioning virtual machines within RHEV using the RHEV-M portal: 1. Log in to the RHEV-M Web portal and select the Virtual Machines tab. 2. Click the New Server button. 3. In the New Server Virtual Machine dialog box, select Data Center and Cluster where the virtual machine is operating, and then enter a name and description. Adjust the memory size, number of CPU cores, and number of CPU sockets, and select the specific operating system from the dropdown menu. 4. Select the Console tab and select VNC from the drop-down menu. Click OK. 5. Click the Configure Network Interfaces button. Select the logical network to which the network interface will attach. Click OK.
16
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
6. Click Configure Virtual Disks. Enter the size of the disk and check the Wipe after delete option. Click OK. 7. Right-click the newly created virtual machine and select Run Once. Check the Attach CD option and select the installation ISO image from the drop-down menu. Click OK. 8. Right-click the newly created virtual machine and select Console to interact with the installation.
9.1
RHEV-M
To launch the RHEV-M Web portal, open Internet Explorer and go to the URL https://hostname.domain.com:8080, where host name and domain are replaced with the fully qualified domain name for your RHEV-M server. Select Administrator Portal.
RESTful API
To view the RESTful API guide, open Internet Explorer and go to the URL https://hostname.domain.com:8080, where host name and domain are replaced with the fully qualified domain name for your RHEV-M server. Select REST API Guide.
9.2
Red Hat Network Satellite can be deployed as a virtual machine within the RHEV infrastructure cluster. For detailed information on Red Hat Network, see Red Hat Network Documentation. For detailed information, including deployment procedures, for Red Hat Network Satellite server, see the documents available at Red Hat Network Satellite.
9.3
NetApp Operations Manager can be deployed as a virtual machine within the RHEV infrastructure cluster. For detailed deployment instructions and best practices, see the Operation Manager, Provisioning Manager, and Protection Manager documents referenced in the appendix.
9.4
Kickstart Server
The Kickstart server can be deployed as a virtual machine within the RHEV infrastructure cluster. See the Red Hat Enterprise Linux 6 Deployment Guide for information on setting up and using Kickstart.
17
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
18
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
RHEV Data Centre RHEV Cluster Secondary (maintenance mode) Shared IP Space and VLANs
SC Data
5. Configure zoning and igroups. Unlike IP networking, FC zoning is not shared between the sites. Hypervisors at the primary site are to remain zoned for the primary NetApp FAS controller. Hypervisors at the secondary site should only be zoned for the secondary NetApp FAS controller. The same rules apply to igroups for FC and iSCSI. However, be sure to properly map the secondary igroups to the right LUNs. 6. Query and log all LUN serial numbers on the primary and secondary controllers. When a volume is copied with SnapMirror, the LUNs at the secondary site will have a different serial number. In a site failover situation, they will need to be changed. Prior to giveback, the original serial numbers will need to be returned. Run the following command to query the serial number for a given LUN:
lun serial /vol/InfraVMVol/vmlun
Note:
This is a critical step in the process. Log the serial numbers for all LUNs and keep copies at the secondary site.
7. The secondary hypervisor nodes must have the same IP connectivity (bridges, VLANs, and so on) as the ones at the primary site.
4. If using LUNs, bring the LUN(s) offline and make note of the original serial number. Change serial numbers on LUNs to the serial numbers from the LUNs at the primary site. Finally, bring it back online:
19
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
lun offline /vol/InfraVMVol/vmlun lun serial /vol/InfraVMVol/vmlun Serial#: P3OoG4WVLFoa lun serial /vol/InfraVMVol/vmvol <new_serial_#> lun online /vol/InfraVMVol/vmlun
5. Bring up failover ifgrps on the secondary NetApp controller. 6. Bring up the RHEL 6 KVM hosts (infrastructure cluster) at the secondary site, if they are not running. 7. Start the virtual machine that is running RHEV-M, and then verify that RHEV-M is accessible by logging into the RHEV-M Web portal. 8. Bring up the hypervisors at the secondary site and bring them out of maintenance mode. One of the hypervisors should automatically attach to the storage. When RHEV-M comes up, the secondary hypervisor nodes will not be able to take over storage (attain SPM) until the primary hypervisor nodes are confirmed shut down or rebooted within RHEV-M. Confirm this, and then activate the hypervisors. 9. Move the critical virtual machines from the primary cluster to the secondary cluster (this can be scripted using the RESTful API). 10. Start the critical virtual machines in RHEV. 11. Failover is complete.
6. Next, resync the volumes between sites. From the NetApp controller at the primary site, run the following command from the primary controller:
snapmirror resync -S ice3170-3b:InfraVMVol ice3170-3a:InfraVMVol
From the NetApp controller at the secondary site, run the following command:
snapmirror resync -S ice3170-3a:InfraVMVol ice3170-3b:InfraVMVol
7. Put the full network back online. 8. Start up the RHEL 6 KVM hosts (infrastructure cluster) at the primary site. 9. Move the virtual machines back to the primary cluster. 10. Start the virtual machine that is running RHEV-M, then verify that RHEV-M is accessible by logging into the RHEV-M Web portal. 11. Start the virtual machines in RHEV. Giveback is complete.
20
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
Configuring MetroCluster
1. Configure the network. All Layer 2 and Layer 3 networking between the sites must be shared. That is to say that IP spaces, VLANs, and VIFs should be accessible between the sites. 2. Configure the hardware. MetroCluster has specific hardware requirements and specific hardware configuration that are outside of the scope of this document. It is likely that a NetApp professional services consultant will implement the hardware portion of the configuration. 3. Configure partner IPs on each NetApp FAS controller for each network interface, VLAN, and ifgrp. 4. Configure zoning and igroups. Zoning and igroups do not need special configuration. Storage zoning and NetApp igroups should be site specific. 5. Create a mirrored aggregate:
aggr create aggr1 -m 12
This will create a mirrored aggregate named aggr1 with 12 disks. Six disks will be on the primary controller, and six disks will be on the secondary controller. 6. Create FlexVol volumes, NFS exports, and LUNs using standard NetApp best practices. 7. Disable the changing of LUN serial numbers on forced takeover:
options cf.takeover.change_fsid off
Note:
This will alleviate the need to change the LUN serial numbers during failover or failback.
8. Create RHEL 6 KVM hosts as prescribed in this document. 9. Configure RHEV-M as a VM, as prescribed in this document. When creating the RHEV cluster for the secondary hypervisor nodes, be sure that the cluster is part of the same RHEV data center as the primary hypervisor nodes. This is a critical step in the process. 10. When setting up the secondary hypervisors, configure them to be a separate RHEV cluster in the same RHEV data center as the primary hypervisors. Then place them in maintenance mode or power them down. Note: Placing the secondary hypervisor nodes in the same RHEV data center, but different RHEV cluster, is a crucial part of successfully bringing RHEV up in the secondary site.
21
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
KS
Red Hat
Primary Infrastructure Nodes & Infrastructure VMs (Nodes Clustered to provide HA to VMs)
Boot LUNs
Production VMs
Boot LUNs
Infra VMs
Primary Storage
SC Data
Secondary Storage
Note:
If the RHEL 6 KVM hosts are still operational, then they can still reach the secondary storage. However, if the distance is great, then latency might be an issue. For this reason, the following procedures assume that everything is being failed over.
4. Bring up the secondary infrastructure nodes and start the infrastructure VMs (including RHEV-M). When RHEV-M comes up, the secondary hypervisor nodes will not be able to take over storage (attain SPM) until the primary hypervisor nodes are confirmed shut down or rebooted within RHEV-M. Confirm this, and then activate the hypervisors. 5. Move the critical virtual machines from the primary cluster to the secondary cluster. 6. Start the critical virtual machines. The failover is complete.
22
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
c.
From within partner mode, make a determination as to which aggregates are at what site and which aggregates are out of sync:
aggr status -r
Note:
4. The primary controller will pause its own boot procedure until a giveback is initiated from the secondary controller. Caution Do not initiate the giveback until the following command is run from the secondary NetApp FAS controller: cf giveback. Note: The secondary NetApp FAS controller will reboot.
5. Bring up the infrastructure cluster at the primary site and start the virtual machine that is hosting RHEV-M. 6. Move the virtual machines back to the primary cluster. 7. Start the RHEL 6 KVM guests. Giveback is complete.
23
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
Port
22 80, 443 111 123 16514 3260 53 5353 54321 5900-5910 (range can be increased) 32803, 662 4915249216 67, 68 8080 8088 8443 8488 9090 N/A
Protocol
TCP TCP TCP, UDP TCP TCP TCP, UDP TCP, UDP TCP, UDP TCP TCP TCP TCP TCP, UDP TCP TCP TCP TCP TCP N/A
Description
SSH HTTP, HTTPS Portmap NTP libvirt iSCSI (optional) DNS mDNS KVM interhost communication VNC consoles (optional) NFS client KVM migration DHCP RHEV 3, Snap Creator, and Operations Manager portals NetApp Management Console Secure Operations Manager Console Secure NetApp Management Console Snap Creator agent ICMP
Table 2 lists ports for LDAP-based authentication. Ports that are specific to LDAP based on Active Directory are noted as such.
Table 2) Ports for LDAP-based authentication.
Port
135 1025, 1026 389 636 445
Protocol
TCP TCP TCP TCP TCP
Description
MS RPC (Active Directory) Login and replication (Active Directory) LDAP Secure LDAP Microsoft DS (Active Directory)
24
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
Port
139 137 and 138 88
Protocol
TCP UDP UDP
Description
SMB (Active Directory) NetBIOS (Active Directory) Kerberos v5 (Active Directory)
References
Home page for KVM www.linux-kvm.org Red Hat and Microsoft Virtualization Interoperability http://www.redhat.com/promo/svvp/ KVM: Kernel-Based Virtual Machine www.redhat.com/f/pdf/rhev/DOC-KVM.pdf Red Hat Enterprise Linux 6 Virtualization Guide http://docs.redhat.com/docs/enUS/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/index.html Red Hat Enterprise Virtualization Administration Guide http://docs.redhat.com/docs/enUS/Red_Hat_Enterprise_Virtualization/3.0/html/Administration_Guide/index.html Red Hat Enterprise Virtualization Installation Guide http://docs.redhat.com/docs/enUS/Red_Hat_Enterprise_Virtualization/3.0/html/Installation_Guide/index.html Red Hat Enterprise Linux 6 Deployment Guide http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/index.html Red Hat Enterprise Linux 6 Installation Guide http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/index.html Red Hat Enterprise Linux 6 Security-Enhanced Linux User Guide http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/SecurityEnhanced_Linux/index.html Best Practices for File System Alignment in Virtual Environments http://www.netapp.com/us/library/technical-reports/tr-3747.html Storage Best Practices and Resiliency Guide http://media.netapp.com/documents/tr-3437.pdf NetApp Deduplication for FAS and V-Series Deployment and Implementation Guide http://www.netapp.com/us/library/technical-reports/tr-3505.html SnapMirror Async Overview and Best Practices Guide http://www.netapp.com/us/library/technical-reports/tr-3446.html Operation Manager, Protection Manager, and Provisioning Manager Sizing Guide http://www.netapp.com/us/library/technical-reports/tr-3440.html Operations Manager, Provisioning Manager, and Protection Manager Best Practices Guide http://www.netapp.com/us/library/technical-reports/tr-3710.html RHEL 6, KVM, and NetApp Storage: Best Practices http://media.netapp.com/documents/tr-3848.pdf RHEL 6, KVM, and NetApp Storage: Deployment Guide http://media.netapp.com/documents/tr-4034.pdf Red Hat Enterprise Virtualization 3 and NetApp Storage: Best Practices http://media.netapp.com/documents/tr-3914.pdf
25
Red Hat Enterprise Virtualization 3.0 and NetApp Storage Deployment Guide
Version History
Version
Version 2.0
Date
March 2012
NetApp provides no representations or warranties regarding the accuracy, reliability, or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS, and the use of this information or the implementation of any recommendations or techniques herein is a customers responsibility and depends on the customers ability to evaluate and integrate them into the customers operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.
26
2012 NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Go further, faster, Data ONTAP, FlexVol, MetroCluster, MultiStore, Snap Creator, SnapMirror, SnapRestore, Snapshot, SyncMirror, and vFiler are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. Active Directory and Microsoft are registered trademarks of Microsoft Corporation. Linux Virtualization trademark of LInus Torvalds. All other brands or Red Hat Enterprise is a registered 3.0 and NetApp Storage Deployment Guide products are trademarks or registered trademarks of their respective holders and should be treated as such. TR-3940-0312