SAN Client Deployment - Best Practices and Performance Metrics

Issue Date: 10th March 2010

Version number: 1.2

Applies to: NetBackup 6.5.x and 7.0

Note: This is a living document and will be subject to periodic updates. Please
check the data and version number to ensure you are referencing the latest
version.

This document is provided for informational purposes only and is intended for distribution only by Symantec employees to selected
partners and customers. All warranties relating to the information in this document, either express or implied, are disclaimed to the
maximum extent allowed by law. The information in this document is subject to change without notice. Copyright © 2010 Symantec
Corporation. All rights reserved. Symantec, the Symantec logo and NetBackup are trademarks or registered trademarks of Symantec
Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.
Contents

INTRODUCTION............................................................................................................................1
1.1 ADDITIONAL READING...................................................................................................................1
PERFORMANCE PARAMETERS FOR CONFIGURING THE SAN CLIENT................................2
1.2 SAN INTERCONNECT...................................................................................................................2
1.3 SAN CLIENT SCALABILITY AND ZONING BEST PRACTICES .....................................................................2
1.4 MEDIA SERVER PROFILE..............................................................................................................3
1.5 CLIENT PROFILE..........................................................................................................................3
BEST PRACTICES.........................................................................................................................5
1.6 SEGREGATED FT MEDIA SERVERS.................................................................................................5
1.7 COMBINED FT/LAN MEDIA SERVERS.............................................................................................6
1.8 MULTI-STREAMING BACKUP...........................................................................................................7
1.8.1 Stream distribution..........................................................................................................7
1.8.2 Connection restrictions on the FT Media Server.............................................................7
1.8.3 Increasing the maximum target ports available to a SAN Client.....................................8
1.8.4 Increasing the number of SAN clients per target port.....................................................8

SAN Client Deployment - Best Practices and Performance Metrics 10th March 2010
Page i
 Introduction
In today’s 24/7 business environment, corporations are looking for ways to protect their data
without disrupting business operations. This eliminates the options of using the LAN during of
peak hours to backup and protect the data. To address this issue we have introduced a new
feature in the 6.5 release of NetBackup called “SAN Client”.
The SAN Client takes advantage of Fiber Channel Storage Area Network (SAN) connections to
perform a fast backup of a client machine directly to a Media Server, moving data transfers off the
LAN and away from conflicting traffic. The SAN Client provides a high performance data transfer
for the client and offloads the LAN with the backup traffic.
The primary purpose of the SAN Client is to back up large applications and file systems faster
and more efficiently than is possible using LAN based backup or locally attached backup devices
(a SAN Media Server). However the SAN Client architecture does not allow the same degrees of
parallel backup operation as the alternative approaches and is therefore less suited to handling
large numbers of small backup jobs. As a general rule SAN Client should be deployed selectively
on servers where the data volumes are such that LAN backup within a reasonable time period is
impractical – it should not be deployed with a very large number of clients with a wide range
of backup sizes and a small number of FT Media Servers target ports (grouped together) as this
may lead to excessive resource contention on the FT Media Servers when there is a wide range
of job sizes.
This document describes the NetBackup SAN Client feature and provides some information on
performance considerations and best practices when deploy the SAN Client. The document is
intended to provide backup administrators with the necessary information to help our customers
take full advantage of this advanced feature and maximize their ROI.

1.1 Additional Reading

The following documents provide more information on the subjects covered in this paper:
Section “SAN Client and Fibre Transport” in Shared Storage Guide (Introduction and
Configuration steps) http://seer.entsupport.symantec.com/docs/290238.htm
Device Configuration Guide (OS specific configuration steps for SAN Clients)
http://seer.entsupport.symantec.com/docs/290200.htm
SAN Client and Fibre Transport Troubleshooting Guide
http://seer.entsupport.symantec.com/docs/288437.htm
NBU 6.5.4 Documentation Updates.
ftp://exftpp.symantec.com/pub/support/products/NetBackup_Enterprise_Server/318350.pdf

SAN Client Deployment - Best Practices and Performance Metrics 10th March 2010
Page 1
Performance Parameters for configuring the SAN Client
To implement the SAN Client, the user needs to deploy a Fiber Channel based SAN between the
client and the Media Server. The Server and Clients can each have multiple SAN ports zoned
together.
A Media Server that can receive data from a SAN Client is referred to as a Fiber Transport or FT
Media Server. The HBAs on the Media Server that receive the data are referred to as Fiber
Transport Targets and must be of a supported type (the HBAs used on the SAN Client itself can
be of any type).
If configured correctly the feature allows a client transfer rate exceeding two Terabytes of data in
an hour. Obtaining this type of performance requires using the right hardware not only for the
connectivity between the client and the Media Server but also for the front end client data and the
back end storage disks. The whole system must be configured and tuned to achieve the desired
performance requirement.
The rest of this document lists the profile of the appropriate hardware and how to configure the
system to deliver the optimum performance.

Figure 1 - SAN Client Components

This section provides some information about the type of hardware and the main parts that affect
the performance of the system.

1.2 SAN Interconnect

When configuring the SAN Client, the first step is to deploy a Fiber Channel SAN between the
client and the Media Server.
The architecture of the SAN should be such that the throughput between SAN Clients and Fiber
Transport Media Servers is maximized and the latency is minimized. For example, one important
requirement is a SAN interconnect design that has no points of congestion. One way of
achieving no congestion is to avoid using Inter Switch Links (ISL) and trunking. It is therefore
recommended the connection between SAN Clients and the Fiber Transport Servers be on the
same switch and switch blade wherever possible.
If you have to use ISLs, we recommend that you under subscribe them. At any time, the trunking
between switches should be able to handle the throughput required for your environment's
Quality of Service requirements.

1.3 San Client Scalability and Zoning Best Practices

The larger amount of client initiators with overlapping zones the more likely there will be a
widespread reconfiguration notice to all initiators. This can result in all initiators trying to scan all
visible Fibre Transport ports at the same time. Because of this potential, there is a technical
limitation to how many SAN client initiator ports should be used per Fibre Transport Media Server
target port.

SAN Client Deployment - Best Practices and Performance Metrics 10th March 2010
Page 2
Symantec recommends a maximum of 30 SAN Client initiator ports zoned with a Fibre Transport
target port because of the effective queue depth of about 60 for a QLogic hardware/firmware
target port. Because the NetBackup Fibre Transport media server will simulate two "pseudo tape
devices" on each Fibre Transport target port, the number of commands arriving will be two times
the number of initiators. Keeping the number of SAN Clients per port at 30 or below prevents
each target port from having to queue more than 60 low-level fibre channel commands at any
point in time.
Limiting the number of initiator ports for each target port to 30 will prevent the queue from being
overflowed. With more than 30 clients per port there is risk that the Fibre Transport Media Server
will not respond to a SCSI inquiry. This could have undesirable results on the client operating
system during boot or device discovery.
While all inclusive zoning is supported for SAN Client configurations, it is not recommended. An
all inclusive zone runs a risk of a single miss-configured client or server affecting all the other
SAN Clients in the zone.

1.4 Media Server Profile

The FT Media Server is only supported on a limited number of platforms and selected Qlogic2xxx
cards and their OEM variants as Fiber Transport Targets on the Media Server. For SAN Clients
and for the backup storage devices on the FT Media Server we support QLogic, Emulex, and
most OEM Fiber Channel cards. The range of platforms and HBAs supported is constantly being
expanded. Refer to the Hardware Compatibility List (HCL) for more information on the
supported types of hardware.
Below is a list of some important parameters for tuning the system for high performance.
1) Use PCI-x slots with a speed of 133MHz. Avoid oversubscribing your PCI-x and PCI-
e buses. If possible use PCI-e slots to connect the backend storage devices. If PCI-
e slots are not available make certain that your Media Server hardware has different
dedicated PCI-x buses for the backend storage devices and the Fiber Transport
targets.
2) Make sure the total aggregated IO throughput is not more than the total memory
bandwidth.
3) In general, we recommend that you don’t use more than 4 HBA Fiber Transport ports
per Media Server, as 4 Fiber Transport ports provide a maximum actual transfer rate
of 600MB/second. However don’t use more than 8 ports in any circumstance. Since
the backend storage must match or exceed the Fiber Transport Data Rate, the
bandwidth requirement on a system with 8 Fiber Transport ports would be 2.4
GB/second which is near the maximum I/O load on most systems. Up to 8 Fiber
Transport ports are possible; however, there are diminishing returns as the available
bandwidth is spread over the ports.
4) Use a system test tool (like dd) to make sure the backend disk can write as fast as
the expected FT transfer speed. For example, if you expect to transfer an aggregate
of 600 MB/s over 8 streams, verify that your disk can sustain at least 600 MB/s rate
using 8 streams with 256k blocks.
5) Finally, a multi-processor system is required. While the actual CPU utilization for
Fiber Transport streams is small, keeping the interrupt latency low is required to
maintain high transfer rates. This requires a minimum of two CPU. An x86_64
based system that has a constant 600 MB/s of Fiber Transport data will utilize
about 50% of a dual core 3 GHz processor. A very general rule of thumb would be
that there should be at least 5 MHz of CPU speed for every 1 MB/s of data transfer.

1.5 Client profile

On the client side three parameters are to be considered:
1) Operating System – The SAN Client is supported on the following operating systems:
Solaris, Windows, AIX, HP-UX and Linux. Please refer to the release notes and
SAN Client Deployment - Best Practices and Performance Metrics 10th March 2010
Page 3
 HCLs for details of the operating system versions supported. Note that the SAN
Client is generally supported on the same operating system versions as a Media
Server, not a LAN Client.
2) CPU - As with the Media Servers, the rule we use is 1MB/s for every 5 MHz of CPU
cycle. This means that the CPU use for your application server may be reduced by
up to 5GHz during a particularly heavy I/O load.
3) HBA - If the data is stored on a SAN attached disk, do not use the same HBA for the
disk and backup SANs. This allows you to better tune your environment including
ISL links and HBA parameters. While the same HBA can be successfully used for
both SANs, this practice is discouraged from a performance standpoint.
4) Disk Speed - The speed at which the client can read the data is a fundamental factor.
Again you can use dd to test the disk performance and tune it. Keep in mind that the
Fiber Transport uses 256k block sizes, so use this factor when profiling the
performance of your disk arrays.
5) Data Streams - The minimum optimal number of data streams between a SAN Client
and a Fiber Transport Media server is two streams per available Fiber Transport
target port. For example, a SAN Client backup to a Media Server with 4 available
ports should utilize 8 data streams, writing 1 stream to each LUN available between
the Media Server and the SAN Client. Writing more data streams will create
contention for the LUNs when the data sources are capable of supplying over half of
the maximum data rate for the port, and using less streams of data will mean that the
ports on the Media Server are not fully utilized.
In common with all types of backup operation care should be taken when planning backup
policies and schedules to avoid situations in which large numbers of small backup jobs
monopolize too many resources on all the FT Media Servers and delay the start of large backup
jobs such that one or more of them fail to start early enough to allow them to complete within the
backup window.

SAN Client Deployment - Best Practices and Performance Metrics 10th March 2010
Page 4
 Best practices
One question that usually comes to mind when designing and implementing SAN Client in a
backup environment is should I mix regular clients with SAN Clients on the same Media Servers
or should I segregate them. The answer to this question depends on your environment and
performance requirements.

1.6 Segregated FT Media Servers

In this scenario the FT Media Servers are dedicated for SAN Client backup only. This case is
recommended when you have different tiers of backend storage. The high end, high
performance disk storages are connected to the FT Media Servers. This provides enough traffic
to optimize the usage of the backend storage.
Another benefit of this architecture is that it’s easy to distribute the clients to the different Media
Servers. As they all have the same speed, an equal distribution of the clients to the Media
Servers is usually enough.
In a dedicated Fiber Transport backup pool, you must monitor the environment so that you
always know that the load on the Media Servers is at an acceptable level. If the load is such that
the Media Servers can't handle all the backup requests from SAN Clients in an acceptable
amount of time, the pool will have to be grown, and likewise you may consider shrinking the pool
of dedicated Media Servers if you find that the servers sit idle or underutilize the available Disk
I/O.
The advantage of a segregated FT Media Server pool is that you are better able to guarantee the
class of service necessary is available to SAN Clients since you don't have to allow for LAN
based transfers. Disadvantages are that more Media Servers are required overall and/or the pool
of Media Servers available to the SAN Clients is smaller than it would be in a mixed environment.
This means that the failure of a single Media Server will have a larger impact on the overall SAN
Client backup window than it would in a shared environment.

Figure 2 - Segregated FT Servers

SAN Client Deployment - Best Practices and Performance Metrics 10th March 2010
Page 5
1.7 Combined FT/LAN Media Servers
In this scenario the same FT Media Servers are used to backup both SAN and LAN based
clients. Using Media Servers in this way addresses the principle disadvantage of the segregated
model by making more Media Servers available for both FT and LAN backup, thus reducing the
number of Media Servers required and/or increasing the number available for SAN and LAN
backups. This is particularly useful in environments where some client platforms are not
supported with the SAN Media Server.
However it should be noted that LAN jobs utilize more CPU per MB of data transferred and the
LAN protocol interrupt load can adversely affect interrupt latency and reduce FT data rates.
Additional CPU's may or may not help depending on the affinity of the interrupt processing for one
CPU on some operating systems.

Figure 3 - Combined FT/LAN Media Servers

A major advantage to this model is that it utilizes a larger group of servers with lower performance
requirements for a cost savings on a per-server basis. Since the pool of FT servers is more
distributed, it is expected that each server will have a smaller data transfer load while still
maintaining the same overall throughput performance due to the cumulative effect of having more
active Media Servers.
The downside to this model is that it makes it harder to guarantee a higher Quality of Service to
individual large application servers. An individual application server may support 600 MB/s of
backup I/O, but if the selected Media Server is already writing many streams of LAN based
backups from multiple clients there will be a performance impact to both the SAN Clients and
LAN Clients.

SAN Client Deployment - Best Practices and Performance Metrics 10th March 2010
Page 6
1.8 Multi-Streaming backup
The typical maximum practical transfer rate for a single 2 gigabit Fiber Channel port is around
160 MB/sec and this is the maximum transfer rate you can expect to see between the SAN Client
and FT Media Server over a single port.
The read speed of a single backup stream from the disk system on the client may be significantly
less than this maximum figure, resulting in apparently poor performance. However it is possible
to run multiple backup streams over the same port between the client and server and thus better
backup performance may be achieved by using multi-streamed backups.
The performance of individual backup streams will vary depending on the read speed of the
source data. This can be established (on UNIX and Linux clients) by running a performance test
using a dd command of the form:
time dd if=<path> of=/dev/zero bs=256k count=8192
Specifying the path to be backed up should be specified in place of <path>. This will return a
value for the elapsed backup time which, in turn, will enable you to calculate the read speed of
that stream. Once you have a feel for the read speed of each stream you should be able to
configure a policy with sufficient parallel streams to achieve the maximum throughput on the Fiber
Channel port.
Note that in some cases multiple streams may contend with each other for disk controller
resources resulting in further performance degradation when streams are run in parallel and
further tuning may be required.
If transfer rates greater than 160 MB/s are required it is possible to use multiple HBA Ports for the
client and the server in conjunction with multiple streams. The ratio of client ports to server ports
depends on the speed of the HBA cards in the client. For example, if you have to transfer 300
Megabytes per second from a SAN Client to an FT Media Server, you will need to have 2 x two
gigabit client ports and the Media Server will require 2 x two gigabit target ports.

1.8.1 Stream distribution

When multiple streams are used from a client with more than 1 HBA port, the jobs are distributed
in a round robin fashion with one stream being assigned to each port in turn. E.g. if 4 jobs are run
using 2 ports then job1 will use port 1, job 2 will use port 2, job 3 will share port 1 with job 1 and
job 4 will share port 2 with job 2. This means that as job streams complete and new job streams
do not take their place the ports will be under-utilized in the same way that tape drives using
multiplexing may be under utilized as the number of concurrent jobs decreases.

1.8.2 Connection restrictions on the FT Media Server

On important limiting factor to bear in mind when running multiple streams to multiple ports on a
the same FT Media Server is that the maximum number of connections allowed to a single FT
Media Server is 16. This means that the total number of concurrent backup streams across all
the available ports on the FT Media Server cannot exceed 16. This figure is increased to 32
connections in NetBackup 6.5.6 and 7.0.
It is also important to remember that each port used by a specific client impacts the performance
of other SAN Clients utilizing the same Media Server. If two SAN Clients utilize the same Media
Server with only four target ports, and each client claims all four of those ports, the clients will
contend for those ports. The effect of this contention is that the client with a lower latency will
'win' and transfer data faster than the higher latency client, sometimes as much as twice as fast.
If each client had only claimed two ports, they would have been able to share the available ports
and not run into contention issues.
The key to tuning is therefore to limit the number of ports used by a SAN Client while maximizing
the number of streams on each port but ensuring that the maximum number of connections
across all the available ports does not exceed 16 (32 for NetBackup 6.5.6 and above).

SAN Client Deployment - Best Practices and Performance Metrics 10th March 2010
Page 7
1.8.3 Increasing the maximum target ports available to a SAN Client
By default, SAN Clients supports a maximum of two Fiber Transport ports at any given time; this
allows FT Media Servers to balance an I/O load fairly among multiple SAN Clients.
If a particular client requires a higher Quality of Services than is provided by two Fiber Channel
ports then it is possible to increase the number of ports per server the client is allowed to use by
executing the following command.
nbftconfig -changeclient -np 4 -C <clientName>
The above command changes the number of ports for the specific client <clientName>. To
change it for all clients the setconfig option is used:
nbftconfig –setconfig –np 4
Note that setting –np 4 will have no effect on a client that only has a 2 port HBA.

1.8.4 Increasing the number of SAN clients per target port

By default, one Fibre Transport port can only be used by up to two different SAN Clients at any
given time; this prevents oversubscribing a FT Media sever port to multiple clients.
In environments with a greater number of clients per Media Server target port, this value can be
increased. This may reduce the Quality of Service and should be done only if necessary. This
value is set on a Master server and applies to all the Fibre Transport Media Servers in that
domain. The value can be configured by executing the following command
nbftconfig –setconfig –ncp 4
The recommended value is less than or equal to 4.

SAN Client Deployment - Best Practices and Performance Metrics 10th March 2010
Page 8