Dell Equallogic Configuration Guide
Dell Equallogic Configuration Guide
Dell Equallogic Configuration Guide
Configuration Guide
Dell EqualLogic
Storage Infrastructure and Solutions
v11.2 July 2010 Quarterly version cycle update: content additions and edits.
V11.3 Oct 2010 Quarterly version cycle update: content additions and edits.
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL
ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR
IMPLIED WARRANTIES OF ANY KIND.
© 2010 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without
the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.
Dell, the DELL logo, and the DELL badge, PowerConnect™, Dell™ EqualLogic™, PowerEdge™ and
PowerVault™ are trademarks of Dell Inc. Broadcom™ is a registered trademark of Broadcom
Corporation. Intel™ is a registered trademark of Intel Corporation in the U.S. and other countries.
Microsoft™, Windows™, Windows Server™, and Active Directory™ are either trademarks or registered
trademarks of Microsoft Corporation in the United States and/or other countries.
To be able to leverage the advanced features provided by an EqualLogic array, a robust, standards-
compliant iSCSI storage area network (SAN) infrastructure must be created. While almost any industry
standard, enterprise class Ethernet switch can be used to connect hosts to the EqualLogic-based virtual
storage volumes, some switches, network interface controllers(NICs), iSCSI Host Bus Adapters (HBAs),
and iSCSI initiators can directly impact the overall performance of the SAN storage solution. The Dell
EqualLogic Configuration Guide is provided as an aid to help storage administrators determine how
best to build an iSCSI infrastructure for use within an EqualLogic SAN solution. This document focuses
on network configuration, host integration and other topics that help to ensure a smooth deployment
with maximum SAN performance.
This document should be used strictly as a guide in planning an EqualLogic SAN solution. It should not
be considered a statement of support for any specific configuration. Actual viability of any
configuration will depend on the capabilities of the individual components (switches, network
interface controllers, initiators, etc.) that make up the SAN infrastructure.
1.1 Audience
This configuration guide is intended for storage administrators, SAN designers, storage consultants, or
anyone who is considering purchasing or has purchased an EqualLogic PS Series Array(s) for use in a
production storage area network. It is assumed that all readers have experience in designing and/or
administering a shared storage solution. Also, there are some assumptions made in terms of familiarity
with all current and possibly future Ethernet standards as defined by the Institute of Electrical and
Electronic Engineers (IEEE) as well as all TCP/IP and iSCSI standards as defined by the Internet
Engineering Task Force (IETF).
The four digit model number indicates the major feature sets and generational information. The first
digit is the "family" and is typically referred to as the "x-thousand" family where "x" is (currently) either
5 or 6. The second digit designates the chassis form-factor (16 drives vs. 48 drives). The last two are
digits designate the controller type (―00‖ indicates a controller with 1GbE ports, ―10‖ indicates a
controller with 10GbE ports).
The letter designation at the end of the model name indicates the array drive technology being used
within the array: "E" indicates SATA technology, "X" indicates 10,000 RPM SAS technology, "XV" indicates
15,000 RPM SAS technology and "S" indicates SSD technology.
1
In a mixed group of PS6000, PS6500 and PS4000 the higher limits normally prevail
2
In a mixed group of PS6000, PS6500 and PS4000 arrays the maximum number of PS4000 arrays in the group is two
The Active Controller is the controller which is processing all disk and network I/O operations for the
array.
A second controller in dual controller configurations will always be in a ―passive‖ operating mode. In
this mode, the secondary controller will exhibit the following characteristics:
Each of its Ethernet ports are electrically inactive (active lights are off)
The passive controller cache mirrors the cache of the active controller.
The backup controller automatically enables each of the corresponding Ethernet ports that
were enabled on the failed primary controller
The IP addresses that were assigned to each of the failed controller Ethernet ports are
reassigned to the corresponding ports on the second controller.
A link failure on one or more of the Ethernet ports on the active controller does not constitute a
controller failure. For this reason, it is important to ensure that ports from each controller are
connected to at least two different switches. This will prevent a switch failure from also disabling all
paths between a host and its connected volumes.
It is critical that port connections for both controllers are configured so that the corresponding ports
on each controller are connected to the SAN. If port 0 on the active controller is the only port
connected to the SAN, then port 0 on the passive controller must also be connected to the SAN. This is
shown in the partial and fully connected controller failover scenarios illustrated in Figure 1 and Figure
2 below. Note how IP addresses are reassigned to the ports during the failover process.
2.3.4 Firmware
Each EqualLogic PS Series array runs a core operating system in firmware that provides all of the PS
Series features and functionality. The firmware version is defined using a version number and will be
updated from time to time as new features are added or for general maintenance improvements.
"X" is used to identify the "major" release number. This changes when there are very significant
changes in the features and functionality.
"Y" is the "minor" release number. This changes as new minor features or improvements in
existing features are made for a given major release.
"Z" denotes maintenance releases. This changes whenever a maintenance release level is
released to resolve any identified technical or usability issues.
In addition to the Release Notes, the process for updating controller firmware is described in detail in
the following document (Support ID required for login access):
Supported firmware upgrade paths (up to version 5.0.x) are shown in Table 4 below:
4.2.* 4.3.*
4.0.* 4.1.*
3.0.* 3.1.*
RAID level
hot-spare configuration
Each array implements a default RAID policy that includes a hot-spare. To configure a RAID policy that
does not include a hot-spare you must manually initialize the array using the Command Line Interface.
Note: Configuring a RAID policy that does not include a hot-spare will increase the risk
potential for data loss in the event of multiple drive failure.
2.4.1 RAID 5
RAID 5 (striped disks with distributed parity) will combine N disks in an arrangement where each stripe
consists of N-1 disks that contain data blocks plus 1 disk that contains a parity block. For each stripe,
the parity block will be placed on a different disk ensuring that the parity blocks are not located on a
single disk in the RAID set. RAID 5 implementations can tolerate a single drive failure without data
loss.
For EqualLogic arrays using a RAID 5 policy, Table 5 shows the drive layouts that are enforced based on
the number of drives in each array and the hot spare configuration.
For EqualLogic arrays using a RAID 5 policy, the total quantity of raw storage available can be
computed based on number of drives and the hot spare configuration. The Table 6 below shows the
amount of raw storage available using the hot spare drive layouts specified in Table 5 above.
8(c) - - - - 1500 - - - - - - - -
14 (d)
- - 888 - 3000 - 4800 - 6000 - - - -
16 700 1400 - 2044 3500 4200 5600 6300 7000 8400 10500 14000 -
2.4.2 RAID 6
RAID6 (striped set with dual distributed parity) combines N disks in an arrangement where each stripe
consists of N-2 disks that contains data blocks and 2 disks that contain parity blocks. Each parity block
generates parity using a different ―view‖ of the data blocks depending on the RAID 6 implementation.
RAID 6 implementations can tolerate up to 2 drive failures per RAID set at the same time without data
loss. RAID 6 is not recommended for workloads consisting mainly of random writes. For EqualLogic
arrays using a RAID 6 policy, Table 7 shows the drive layouts that are enforced based on the number of
drives in each array and the hot spare configuration.
Table 8 shows the amount of raw storage available using the hot spare drive layouts specified in Table
7 above.
8(c) - - - - 1250 - - - - - - - -
14 (d)
- - 740 - 2500 - 4000 - 5000 - - - -
16 650 1300 - 1898 3250 3900 5200 5850 6500 7800 9750 13000 -
8 / 8(e) - - - - - - - 2850 - - - - -
2.4.3 RAID 10
RAID 10 (mirrored sets in a striped set) combines two high performance RAID types: RAID 0 and RAID 1.
A RAID 10 is created by first building a series of two disk RAID 1 mirrored sets, and then distributing
data over those mirrors. RAID 10 implementations can tolerate one drive failure per mirror pair.
For EqualLogic arrays using a RAID 10 policy, Table 9 shows the drive layouts that are enforced based
on the number of drives in each array and the hot spare configuration.
For EqualLogic arrays using a RAID 10 policy, the total quantity of raw storage available can be
computed based on number of drives and the hot spare configuration. The Table 10 below shows the
amount of raw storage available using the hot spare drive layouts specified in Table 9 above.
8(c) - - - - 750 - - - - - - - -
(d)
14 - - 444 - 1500 - 2400 - 3000 - - - -
16 350 700 - 1022 1750 2100 2800 3150 3500 4200 5250 7000 -
2.4.4 RAID 50
RAID 50 (RAID 5 sets in a striped set) is created by first creating two or more RAID 5 sets, and then
striping data over those RAID5 sets. RAID 50 implementations can tolerate a single drive failure per
RAID5 set.
For EqualLogic arrays using a RAID 50 policy, Table 11 shows the drive layouts that are enforced based
on the number of drives in each array and the hot spare configuration.
For EqualLogic arrays using a RAID 50 policy, the total quantity of raw storage available can be
computed based on number of drives and the hot spare configuration. The Table 12 below shows the
amount of raw storage available using the hot spare drive layouts specified in Table 11 above.
8(c) - - - - 1000 - - - - - - - -
(d)
14 - - 740 - 2500 - 4000 - 5000 - - - -
16 600 1200 - 1752 3000 3600 4800 5400 6000 7200 9000 12000 -
3.1 Groups
A PS Series SAN Group is a Storage Area Network (SAN) comprised of one or more PS Series arrays
connected to an IP network. Each array in a group is called a group member. Each member is assigned
to a storage pool. There can be up to 4 pools within the group.
A group can consist of up to 16 arrays of any family or model as long as all arrays in the group are
running firmware with the same major and minor release number. For example, it is supported to have
different arrays in the same group running different maintenance release levels, as long as their major
and minor revision levels match. An example of this would be running version 4.0.1 and 4.0.3 on
different arrays in the same group. An example of an unsupported configuration would be different
arrays in the same group running 4.0.1 and 4.1.0 (the minor versions differ). The only exception to this
rule is for short term time periods when array firmware versions may be out of sync while upgrading
the firmware on each array within a group. Features available for use within the group are determined
by the lowest version of FW running within the group.
Note: It is recommended that all arrays run the exact same version of FW at all times,
except during a FW upgrade process. (See Section 2.3.4 on page 7 for more information.)
3.2 Pools
A pool is a container that each member is assigned after being added to the group. A pool can have
between 1 to 8 members. There is always at least one pool in any group and it is called the ―default
pool‖ unless the name is changed. Regardless of the name of this pool, it is always considered the
default storage pool. All newly added members are automatically assigned to the default pool. The
default pool cannot be deleted.
Pools can be used as part of an overall tiered storage strategy for the SAN. Tiered storage is storage
that is differentiated and optimized for the type of applications and data being hosted. Instead of
optimizing all of the storage in the group for maximum storage utilization, a tiered storage system
allows for the administrator to optimize different arrays for a variety of requirements such as
application performance requirements or cost efficiency.
Pools are the containers that provide scope for all automated storage optimization features of the
group. Pools with multiple arrays can implement different RAID policies within the pool. The
EqualLogic automated optimization features will automatically move a volume from an array or set of
arrays with one RAID policy to another array or set of arrays with a different RAID policy if it is
determined that the application needs more (or less) performance.
Figure 3 shows a PS Series group with the maximum of 4 pools. Note the use of Pool 3 for containing
PS5500/PS6500 series arrays only. Also note that Pool 3 contains arrays that implement both 1GbE and
10GbE controllers.
Do not mix arrays with different drive speeds within a single pool unless they are running a
unique RAID policy
Do not mix arrays with different drive technologies (SATA, SAS, SSD) within a single pool unless
they are running a unique RAID policy.
Do not mix arrays with different controller speeds (1GbE, 10GbE) within a single pool unless
they are each running unique RAID policies.
To override the automated performance decisions for a specific volume, indicate a ―preferred‖
RAID type for that volume. If that RAID type exists within the pool, the volume will reside on
those arrays that match the preferred RAID policy for the volume.
3.3 Volumes
Volumes provide the storage allocation structure within an EqualLogic SAN. Volumes are seen on the
network as iSCSI targets by hosts and are presented to the user as disk drives once the iSCSI initiator
Volumes are identified within the Group Manager with a unique volume name. The volume name is
created by the administrator during volume creation and can be configured with several additional
characteristics and capabilities. The following sections provide additional details.
1 to 64 alpha-numeric characters
A-Z, a-z, 0-9, ―.‖, ―-―, ―:‖ are legal characters
Volumes are assigned a unique iSCSI Qualified Name (iqn) that consists of the following parts:
The following is an example of an iSCSI target name for a volume named db3:
iqn.2001-05.com.equallogic:7-8b0900-6d0000000-001ebbc5d80sf0k0-db3
Volumes are visible to iSCSI initiators through one or more of three mechanisms during volume creation:
Note: IQN names are assigned to volumes automatically when they are created. They
cannot be changed for the life of the volume. If a volume name is changed, the IQN name
associated with the volume will remain unchanged.
A volume can be provisioned as a thin provisioned volume at creation or after creation. The following
rules apply to thin provisioned volumes:
The following best practices should be considered when using thin provisioned volumes:
Use Pool Free Space, not Group Free Space when making all determinations of thin provisioned
volume physical capacity allocation.
Create regular volumes before creating thin provisioned volumes. This provides the
administrator with a better view of the remaining available free space in the pool.
Set each thin provisioned volume’s ―In use Volume Reserve Warning Level‖ to a level that
allows the administrator to ensure that additional physical storage can be deployed before the
volume uses the remainder of current pool free space.
Ensure that the Sum of all ―In use Volume Reserve Warning Levels‖ for all thin provisioned
volumes does not exceed the current pool free space capacity minus 10%.
The storage administrator and application owner should reach an agreement on storage use and
procedures in determining the best automatic growth policy, reducing the frequency of
monitoring and intervention.
3.3.4 Snapshots
Snapshots are point in time copies of volumes. Snapshots have some features and properties similar to
volumes as well as some unique capabilities. Like volumes, snapshots can be assigned an iqn and
presented as volumes to a host. This allows hosts to mount a snapshot, potentially change the data in
the snapshot, or convert it to a real volume (clone) that has dedicated space allocated to it from the
free storage pool.
Snapshots require that space be reserved during volume creation (or after volume is created) to store
the snapshot(s) created from that volume. All snapshot reserve space for a specific volume always
resides in the same storage pool as the volume. By default, Group Manager allocates snapshot reserve
space equal to 100% of the host volume’s allocated space. This ensures that a 100% data change in the
volume can be protected by a single snapshot. This value can be set to a lower value based on the
application’s data change rate, snapshot/backup plan, or role that the snapshot will be used. The
following items should be considered when determine the size of a volume’s snapshot reserve:
Data change rate for the application(s) who is using the volume
Defragmentation policy of the OS or application using the volume
The role or purpose for creating the snapshot(s).
They are identified using names that consist of the parent volume name plus a date/time
stamp indicating when the snapshot was created as a default name.
If name of parent volume changes, existing snapshots iqn names do NOT change accordingly,
but retain their original iqn name.
Deleting a snapshot’s parent volume deletes all associated snapshots for that volume.
Snapshots of volumes with a high data change rate will require a larger snapshot reserve space.
Snapshots have access control lists that are inherited from the parent volume by default.
Snapshot reserve space for any volume can be decreased at any time. The minimum size
allowed will be based on the current space usage consumed by existing snapshots using the
snapshot reserve.
Snapshot reserved space for any volume can be increased at any time assuming there is
available free space in the storage pool hosting the volume.
3.3.5 Cloning
Cloning creates a new volume by copying an existing volume. The new volume has the same reported
size, contents and thin-provision settings as the original volume. You can clone a regular volume, a
specific replica of a volume, or a specific snapshot of a volume.
(a) Replica clones are created in the secondary group pool and are immediately available at
the secondary group IP address.
Table 13 Cloning Modes and Properties
3.3.6 Replication
Replication is a powerful feature that can help you manage and implement a disaster recovery strategy
for your business applications and data. By replicating business-critical volumes, you ensure that your
business operations can resume quickly on a partner group in the event of a disaster on the primary
group. You also have the ability to restore the configuration to its original state if the problem on the
original group can be corrected. The replication process does not require placing volumes offline. Thus
you can take advantage of the benefits of replication without causing interruption in application or
data availability.
In EqualLogic PS Series storage, replication is the process of copying volume data from one PS Series
group to another PS Series group. Physical distance between the replication groups is not a concern as
long as they are connected by a reliable TCP/IP network path.
You need to consider the following limits when designing a volume replication topology:
Warning: Any type of application or operating system level process that causes block level
data changes will affect replication data size and time to complete. For example, file
system defragmentation of a large partition will significantly increase the amount of space
consumed in replica reserve, delegated space on the secondary group and time to complete
the next scheduled replication.
Figure 4 shows the basic replication paths, with one partner group hosting the primary copy of
the volume and a second partner group hosting the replica copy of the same volume. We also
show the reverse direction of the path for the Fast Failback replication feature, if it is enabled
for the replica set.
Figure 5 shows how a single group can support replication of multiple primary volumes to
different secondary replication partner groups.
Figure 6 shows how you can create bi-directional replication paths between partner groups.
Figure 7 shows a common scenario in which multiple primary groups replicate their volumes to
a single secondary group partner in a different location.
Delegated space for a replication partnership must be assigned to a pool (the ―Default‖ pool, by
default). Delegated space for different replication partnerships can be assigned to different pools on
the secondary group.
Delegated Space Sum of all replica reserve sizes Monitor CR, adjust to lower
(Secondary, for all replica sets for all volumes replicating to than recommended value and
coming from single group) that group continue monitoring.
(a) CR: ―Change Rate‖. For details on how volume changes affect replication space, see the Centralized
Replication section under Replication Configuration Options in the Group Manager Help documentation.
(b) Start with 105%, then add to that the maximum number of replicas expected to be stored in the replica set
minus 1, multiplied by the expected Change Rate.
3.3.7 Clustering
To support a shared storage environment, EqualLogic allows concurrent access to any volume.
Concurrent volume access is enabled on a per volume basis within the Group Manager or via CLI. The
EqualLogic array will not manage concurrent access to these volumes. Control of access to a shared
volume must be provided by means of access control software on each host that has access to the
volume.
Warning: Failure to provide access control to a shared volume can result in data loss.
IETF Standards
o IETF RFC1122 ―Requirements for Internet Hosts – Communications Layers‖
o IETF RFC1123 ―Requirements for Internet Hosts – Application and Support‖
o IETF RFC3270 ―Internet Small Computer Systems Interface (iSCSI)‖
IEEE Standards
o 802.1
o 802.3
iSNS Support
An Internet Storage Name Service3 (iSNS) server can support discovery, management and configuration
of group members by dynamically updating information about the iSCSI target names for group
volumes. Once the IP address of an iSNS server is entered in an iSCSI initiator’s configuration utility,
the setting is persistent across initiator sessions. A PS Series group can be configured to register with
up to three iSNS servers.
Note: Starting with Firmware V4.1.4, volume and snapshot identifiers are no longer
automatically published to iSNS servers. This applies to new volumes and snapshots as well
as volumes and snapshots that existed before the group was upgraded to V4.1.4.
For all members (arrays) in a given SAN Group all ports should be connected to the same
subnet. This allows the arrays to communicate with each other as a group of peer members.
The arrays must be in the same subnet as the group’s ―well known‖ IP address.
Note: Hosts can be in a different subnet as long as those hosts have layer 3 routing available
to the subnet containing the arrays and the group’s well known IP address.
Rapid Spanning Tree Protocol must be enabled if the SAN infrastructure has more than two
switches in a non-stacked configuration.
3
The Internet Storage Name Service (iSNS) specification: http://tools.ietf.org/html/rfc4171
For EqualLogic PS Arrays, the following general SAN design recommendations apply:
Take advantage of your switch’s VLAN capabilities. You should create a VLAN dedicated to iSCSI
traffic (even on dedicated switches). If necessary, create a second VLAN for management
traffic. The actual VLAN configuration of your iSCSI SAN will be dictated by your SAN network
design requirements and the features of the iSCSI SAN switches being used.
Jumbo frames should be enabled. If you choose to use jumbo frames then all nodes in the SAN
fabric must have jumbo frames enabled.
For best performance and reliability, we recommend that all interconnection paths between
non-stacking switches (LAGs) use a dynamic link aggregation protocol such as LACP
The ability to provide different priority levels to different applications, users, or data flows, or
to guarantee a certain level of performance to a data flow.
A network function implemented in some routers and switches that provides high priority for
certain types of latency-sensitive traffic (for example, VoIP) and lower priority for other types
of traffic (for example, web or http).
PS Series arrays are designed to provide I/O as fast as your network can support it. Therefore, using
QoS with iSCSI traffic does not produce expected or desirable results on the SAN. Also, QoS rules can
affect how well–or even whether–replication succeeds between PS Series groups. If you plan to use QoS,
Dell recommends that you use it only on VLANs that do not carry iSCSI traffic, or on WANs, where
bandwidth is shared with other applications and the PS Series array uses it for time-insensitive
replication traffic.
Low latency
Switches with relatively high latency may cause SAN throughput performance to degrade, and
under high load conditions they could increase the risk of dropped connections.
Non-blocking backplane design
SAN Switches should be able to provide the same amount of backplane bandwidth to support
full duplex communication on ALL ports simultaneously.
Adequate buffer space per switch port
In addition to supporting data transfers between the hosts and the SAN, Dell EqualLogic arrays
also use the SAN to support inter-array communication and data load balancing. For this
reason, the more buffer space per port that a switch can provide the better. Due to the
multitude of buffer implementations used by switch vendors, Dell cannot provide definitive
guidelines as to how much is enough, but as a general guideline switches should have at least
256KB per port. Port buffers should be designed such that data is not lost when traffic reaches
extreme levels.
Support for IEEE 802.3x flow control (passive and/or active) on ALL ports.
Switches and network interface controllers used in an EqualLogic SAN must be able to passively
respond to any ―pause‖ frames received. If possible, you should use switches that have the
ability to transmit ―pause‖ frames to external devices in the event that the device cannot
adequately forward traffic in a timely fashion.
Support for Jumbo Frames
This is not a requirement. But, the use of jumbo frames may yield desirable results. Most iSCSI
SAN implementations should benefit from using jumbo frames. The actual impact on SAN
throughput when using jumbo frames will depend on your workload’s I/O characteristics.
Support for Rapid Spanning Tree protocol (IEEE 802.1w), or Cisco “portfast” functionality if
the SAN infrastructure will consist of more than two switches
For SAN infrastructures consisting of more than 2 non-stacking switches, R-STP must be enabled
on all ports used for inter-switch trunks. All non-inter-switch trunk ports should be marked as
―edge‖ ports or set to ―portfast‖.
Support for unicast storm control
iSCSI in general, and Dell EqualLogic SANs in particular can send packets in a very ―bursty‖
profile that many switches could misdiagnose as a virally induced packet storm. Since the SAN
should be isolated from general Ethernet traffic, the possibility of actual viral packet storms
occurring is non-existent. In an EqualLogic SAN, the switches must always pass Ethernet
packets regardless of traffic patterns.
Support for Inter-Switch Trunking (IST) or Stacking
IST support is required to link all switches in SAN infrastructure together. For stacking capable
switches, the use of stacking ports for IST is assumed. A good rule of thumb for stacking link
bandwidth would be a minimum 20 Gbps full-duplex.
Support for vLAN functionality if SAN traffic is to share the same physical switch resources
with other (non-iSCSI SAN) network traffic.
Note: For 1GbE SANs, using non-stacking switches to connect three or more EqualLogic
arrays into a single group may negatively impact SAN I/O throughput performance.
Stacking Switches
Stacking switches provide the preferred method for creating an inter-switch connection within a Layer
2 network infrastructure. Stacking is typically accomplished using a vendor proprietary, high-bandwidth,
low-latency interconnect that allows two or more switches to be connected in such a way that each
switch becomes part of a larger, virtual switch. A stackable switch will provide a set of dedicated
stacking ports. Installation of an optional stacking module may be required. Considerations for stacking
link bandwidth:
1Gb Switches The stacking link bandwidth should be at least 10Gb/s in each direction on
each wire (full-duplex) to provide adequate throughput to support an
EqualLogic SAN consisting of 1Gb arrays.
10Gb Switches The stacking link bandwidth should be at least 40Gb/s in each direction on
each wire (full-duplex) to provide adequate throughput to support an
EqualLogic SAN consisting of 10Gb arrays or a mix of 1Gb and 10Gb arrays.
Note: For best performance and reliability, we recommend that all interconnection paths
between non-stacking switches (LAGs) use a dynamic link aggregation protocol such as LACP.
If possible, use 10Gb connections for all links between each stack.
Distribute the links across multiple switches within each stack (this is known as a ―cross-
stack‖ link aggregation group).
Use LACP or another type of dynamic link aggregation protocol.
Perform tests to determine the best hashing algorithm to use within the link aggregation
group (port channel).
1GbE switches connecting to Actual requirements will vary. For more details see Section
10GbE switches in a mixed speed SAN 4.6, Integrating 1GbE and 10GbE SANs
Using the Dell PowerEdge R610 server as an example, you configure redundant connection paths to the
SAN switches as shown in Figure 11 below. The R610 server shown in Figure 11 has one additional dual-
port PCI-E NIC installed. This configuration leaves two unused ports on the LOM controller for
connecting to the server LAN.
Figure 11 Redundant NIC Connections from Server to SAN using one installed PCI-E NIC and LOM
An R610 server with two additional dual-port PCI-E NICs installed is shown in Figure 12 below. This
configuration leaves all four ports on the LOM controller available for other connections.
Note: As a best practice, we recommend using the same NIC devices in your server for all
connection paths to the iSCSI SAN. This will minimize the complexity of server configuration
management.
For maximum performance, ensure that the PCI express slot hosting the network controller has
the same specifications as the network controller. For example, if the network controller has a
PCIe x8 interface then ensure that it is installed into a PCIe slot that can support 8 or more PCIe
lanes.
Configure volume access controls to use standard iSCSI IQN names (See Section 3.3.1 for
details). For a more secure configuration you can use the IQN name plus the CHAP security ID.
On each array enable at least two(2) ports for host connectivity.
Install the Dell provided MPIO extension features if available for the host operating system.
For Microsoft Windows, install the Device Specific Module (DSM) found in the Host Integration
Toolkit for Windows.
For VMware vSphere 4.1, install the EqualLogic Multipathing Extension Module.
For other operating systems, use the native MPIO functionality.
Change the ―Subnets included‖ field to include ONLY the subnet(s) dedicated to the SAN
network infrastructure.
Change the ―Subnets excluded‖ field to include all other subnets.
The ―Load balancing policy‖ should remain set to the default value of ―Least queue depth‖.
―Max Sessions per volume slice‖ should be set to the number of network ports dedicated to SAN
subnet (maximum of 4).
―Max sessions per entire volume‖ should be set to three (3) times the value of ―Max sessions
per volume slice‖ (maximum of 12).
―Use MPIO for snapshots‖ should remain at the default setting.
―Use IPv6 or IPv4‖ should be set to IPv4 unless your network is configured to use IPv6 as the
default communications protocol.
Change the ―Subnets included‖ field to include ONLY the subnet(s) dedicated to the SAN
network infrastructure.
―membersessions‖ should be set to the number of network ports dedicated to SAN subnet
(maximum of 4).
―volumesessions‖ should be left at the default value of 6.
―totalsessions‖ should be left at the default value of 512.
The information provided here will not be able to address all of the possible variations in a customer
network environment. All information is presented using a set of basic reference designs that make
the following assumptions:
The SAN network is physically isolated from all other network traffic
All best practices recommendations will be used to derive all examples
Unless otherwise stated, all reference designs will promote end-to-end host to volume
redundant paths
A minimal number of switches will be illustrated that allows the design concept to be
understood. Actual implementations will vary depending on customer’s actual network
infrastructure.
If sharing physical switches with other, non-SAN traffic, assume all switches can represent
a VLAN rather than physical switches.
Note: For a production environment, this configuration shown in Figure 13 will protect your access to
data and is the ONLY type of configuration recommended by Dell.
Single Controller
Single Switch
Single NIC
Single Controller
Single Switch
Dual NIC
Single Controller
Dual Switch
Dual NIC
Dual Controller
Single Switch
Single NIC
Dual Controller
Dual Switch
Single NIC
Dual Controller
Single Switch
Dual NIC
Not all of the application workloads on a SAN will require storage I/O performance that the
10Gb arrays provide. Thus, SAN administrators will have additional storage tiering flexibility
based on array I/O performance.
The PS Series Group Manager will allow the SAN administrator to still manage both types of
arrays within the same SAN group.
There are many ways to design an iSCSI SAN that integrates 1GbE and 10GbE networks together. A
significant factor affecting the design of a mixed speed iSCSI SAN is the feature set and port densities
provided by the Ethernet switches you will be using in your SAN. To create a redundant, mixed speed
iSCSI SAN, at a minimum we recommend that you start with dual 1GbE and dual 10GbE switches.
Figure 14 shows an example SAN design, where two switches of each type are used. The design in
Figure 14 is based on the features provided by the Dell PowerConnect 6248 1Gb Ethernet switch with
10GbE uplink modules, and the Dell PowerConnect 8024 10Gb Ethernet switch. The Dell PowerConnect
8024 is not a stackable switch, so a link aggregation group (LAG) is used to create the inter-switch
trunk paths. Two additional LAG groups are created to connect to the 10GbE uplink ports on the 1Gb
switches to the 10Gb switches.
Note: See the following Dell EqualLogic whitepaper for more detailed analysis and guidelines for
integrating 1GbE and 10GbE iSCSI SANs:
Note: See the following Dell EqualLogic whitepaper for more information on integrating blade
solutions into the EqualLogic SAN infrastructures:
Integrating Blade Solutions with the Dell EqualLogic PS Series, available at:
http://www.equallogic.com/resourcecenter/assetview.aspx?id=7173
Note: All configurations documented in this section assume that Fabric A is dedicated to client LAN
traffic and Fabric B and/or C will be used for SAN infrastructure connectivity
Blade I/O modules and external switches should be from the same vendor if possible.
As a best practice, you should consider using Blade I/O modules as host aggregation switches
only. The limited number of external ports on the blade I/O modules drastically reduces the
scalability of a SAN if arrays are directly connected them.
Stack blade I/O modules into one of two stacking architectures as described in Section 5.2 and
Section 5.3 when possible. Other configurations are possible and may be used if desired.
Stack any external switches into a single stack as described in Section 4.3.
Interconnect blade I/O stack(s) with external switch stacks via aggregated links as described in
Section 4.3.1.
Use the switch vendor preferred method for link aggregation (if all switches from same
vendor).
Use LACP for link aggregation when I/O modules and external switches are from different
vendors
Using 10GbE port options where available (See Figure 18):
If blade I/O modules are stacked:
o Aggregate at least two 10GbE ports from each blade I/O module stack for redundancy
o Distribute 10GbE links among multiple switches within each I/O module stack
o Distribute 10GbE links evenly between all switches within the external switch stack
If blade I/O modules are NOT stacked:
o Aggregate at least two 10GbE ports from each switch to one external switch if external
switches are not stacked.
o Aggregate at least two 10GbE ports from each switch to two different external switches if
external switches are stacked.
Using 1GbE port options only if 10GbE is not available for up-linking:
If blade I/O modules are stacked:
Aggregate at least two 10GbE ports from each blade I/O module stack for redundancy.
If possible, any Link Aggregation Group used for connecting the I/O modules to the external
switch stack should contain at least 2 links for redundancy. The actual number will depend on
both the number of 10GbE ports available on each stack (spread across multiple modules within
the stack) and the number of 10GbE ports available on the external switch stack.
For example, each Cisco 3130X blade I/O module can have up to two 10GbE X2 ports and the
Cisco 3750-E can have at most two 10GbE X2 ports. In most configurations, your external stack
will be comprised of two switches – and a maximum of four 10GbE ports. Regardless of the
number of M1000e enclosures, there are only four available ports to use for uplinking.
Therefore, only four 10GbE ports total can be used for uplinking: two ports per redundant stack
in the dual stack blade configuration, or four ports for the single stack blade configuration.
Distribute 10GbE links among multiple switches within each I/O module stack if supported
When the blade solution consists of more than one M1000e enclosure, the 10GbE ports used for
up-linking should be distributed between multiple enclosures to protect against a complete
enclosure failure. Doing this requires that the M1000e I/O Modules be stacked.
Distribute 10GbE links evenly between all switches within the external switch stack if
supported
When the external switch stack consists of more than one switch, the 10GbE ports used for up-
linking should be distributed between all switches in the stack to protect against a switch
failure.
Use Switch vendor preferred method for link aggregation (if all switches from same vendor)
Use LACP as method for link aggregation when I/O modules and external switches are from
different vendors
The industry standard link aggregation protocol is IEEE 802.3ad (802.1AX), also known as the
Link Aggregation Control Protocol (LACP). LACP is typically the only method for creating link
aggregation groups that can link switches from multiple vendors.
Figure 18Error! Reference source not found. shows how to link the M1000e enclosures using I/O
modules connecting to an external switch stack consisting of Dell PowerConnect 62XX model switches.
Aggregate at least two 1GbE ports from each blade I/O module stack for redundancy
o Configurations will be limited to one active link aggregation group per I/O module stack
due to rapid spanning tree protocol restrictions.
o Link aggregation groups will typically contain a maximum size permitted by switch and/or
I/O module vendor (typically eight 1GbE links).
Distribute links amongst multiple switches within each I/O module stack if supported
When the blade solution consists of more than one M1000e enclosure, the 1GbE ports used for
up-linking should be distributed between multiple enclosures to protect against a complete
enclosure failure.
Distribute links evenly between all switches within the external switch stack if supported
When the external switch stack consists of more than one switch, the 1GbE ports used for up-
linking should be distributed between all switches in the external switch stack to protect
against a switch failure.
Use Switch vendor preferred method for link aggregation (if all switches from same vendor)
Some switch vendors have proprietary or enhanced link aggregation protocols. For example,
Cisco supports Port Aggregation Protocol (PAgP) that contains advanced management features.
Use LACP as method for link aggregation when I/O modules and external switches are from
different vendors
The industry standard link aggregation protocol is IEEE 802.3ad, also known as the Link
Aggregation Control Protocol (LACP). LACP is typically the only method for creating link
aggregation groups that can link switches from multiple vendors.
Figure 19 shows how to link the M1000e enclosures using I/O modules connecting to an external switch
stack consisting of Dell PowerConnect 62XX model switches.
iSCSI
TCP 3260 iSCSI To the group IP address and all individual member IP addresses
EqualLogic Internal
UDP 161 SNMP Management operations
TCP 9876 Internal iSCSI intra-system control
TCP 25555 Internal Group communication
TCP 20002 Internal Event logging
Table 21 Required Ports and Protocols
CLI Management
TCP 23 Telnet To group IP address
TCP 22 SSH To group IP address
Web Based Management
TCP 80 HTTP To group IP address
TCP 3002 GUI communication To group IP address
TCP 3003 GUI communication To group IP address
(encrypted)
SNMP
UDP 161 SNMP To and from group IP address
Syslog
UDP 514 Syslog From group IP address
The choice of switches available for Ethernet networking is too broad to allow Dell to test every device
for use within an EqualLogic SAN. Dell’s goal is not to restrict which infrastructure components a
customer can integrate with the EqualLogic SAN, but to provide a clear statement of how Dell will
support 3rd Party devices that have not been tested and validated by the switch vendors. We will
provide support for these products to the best of our ability, but we make no guarantee that the device
will be compatible with EqualLogic storage solutions.
In general, Dell will allow any infrastructure component to be used within a Dell EqualLogic SAN
solution as long as it meets minimum standards (defined in Section 4.3) required for use within a Dell
EqualLogic SAN infrastructure. Dell makes no guarantee that these switches will meet the customer’s
expectations for performance or scalability.
Dell Labs has developed a set of tests to help determine if an infrastructure component will operate
well within an EqualLogic SAN. These tests provide a level of confidence in the component’s ability to
be successfully integrated into an EqualLogic iSCSI SAN, and to provide adequate performance for use
in a production storage solution.
Initiator Vendor Type (HBA, iSCSI Offload) Product Operating Systems Supported
Vendor OS Version
Note: for optimum performance and stability you should insure that all 10GbE NICs are installed in PCI-
E slots that support x8 or x16 modes.
console> enable
console#
Note: You may be prompted for a password after submitting the enable command
The PowerConnect 54xx Family of switches must be place into configuration mode before any
configuration steps can be performed. To enter configuration mode, the following command must be
entered:
console# configure
console(config)#
PowerConnect 54xx Family of switches are non-stacking switches and must be configured
independently using either the web-based Switch manager or the Command Line Interface (CLI). The
instructions in this Appendix provide CLI commands for configuration. Please go to Dell’s support
website for the latest documentation if the web-interface is preferred.
Port references for the PowerConnect 54xx switch must use the interface command and the port
references are in the form of ―g‖+<port#>.
To reference a range of ports, the interface range command must be used with the port reference in
the form of ―g(―+<begport#> + ―-― + <endport#>+ ―)‖.
For example, to reference all ports between port g1 and port g15 would be referenced as
To disable the iSCSI Optimization settings for the PowerConnect 54xx family of switches, perform the
following steps:
console(config)# exit
console# exit
console# configure
console(config-if)# exit
console(config)# exit
console# exit
console# configure
console(config-if)# flowcontrol on
console(config)# exit
console# exit
The following steps are an example of using this command to disable storm control on a single port of a
PowerConnect 5448 switch:
console# configure
console(config-if)# exit
console(config)# exit
console# exit
The following steps are an example of how to disable storm control on all ports of a PowerConnect
5448 switch:
console# configure
console(config-if)# exit
console(config)# exit
console# exit
console# configure
console(config)# exit
console# exit
console> enable
console#
Note: You may be prompted for a password after submitting the enable command
The PowerConnect 62xx Family of switches must be place into configuration mode before any
configuration steps can be performed. To enter configuration mode, the following command must be
entered:
console# configure
console(config)#
PowerConnect 62xx Family of switches are stacking switches and must be configured as a stack using
either the web-based Switch manager or the Command Line Interface (CLI). The instructions in this
Appendix provide CLI commands for configuration. Please go to Dell’s support website for the latest
documentation if the web-interface is preferred.
Unit#/Interface ID — each interface is identified by the Unit# followed by a / symbol and then
the Interface ID (see below). For example, 2/g10 identifies gigabit port 10 within the second
unit of a stack.
Unit# — the unit number is used only in a stacking solution where a number of switches are
stacked to form a virtual device. In this case, the unit number identifies the physical device
identifier within the stack.
Interface ID — is formed by the interface type followed by the interface number. There is
currently a predefined list of interface types (see below). If additional interface types are to
be defined, they must be registered with Dell. For example, 2/g10 identifies the gigabit port
10 on the second unit.
Interface Types — the following interface types are defined in the 6200 series switches:
o g — gigabit Ethernet port (for example, 1/g2 is the gigabit Ethernet port 2).
o g — 10 Gigabit Ethernet port (for example, 1/xg2 is the 10 gigabit Ethernet port 2).
console# configure
console(config-if)# exit
console(config)# exit
console# exit
To enable flow control on all ports of a PowerConnect 6248, enter the following commands:
console# configure
console(config)# flowcontrol
console(config)# exit
console# exit
console# configure
console(config)# exit
console# exit
The following steps are an example of how to disable unicast storm control on all ports in a
PowerConnect 62xx switch stack:
console# configure
console(config-if)# exit
console(config)# exit
console# exit
console# configure
console(config-if)# exit
console(config)# exit
console# exit
Switch> enable
Switch#
Additionally, you must be in configuration mode before any configuration steps can be performed. To
enter configuration mode, the following command must be entered:
To configure a port on non-chassis based Cisco IOS switch, the interface type, stack member number,
module number, and switch port number must be provided (once you have entered interface
configuration mode).
Interface Type: Gigabit Ethernet and small form-factor pluggable (SFP) modules
(gigabitethernet or gi), 10-Gigabit Ethernet (tengigabitethernet or te).
Stack member number: Identifies the switch within the stack. The switch number range is 1 to
9 and is assigned the first time the switch initializes. All standalone switches have stack
member number equal to 1. When a switch is added to an existing stack it will receive a new
stack member number and it keeps that number until another is assigned to it. Non-Stackable
switches have a stack member number of 1. The switch port LEDs can be configured in Stack
mode to identify the stack member number of a switch.
Module number: The module or slot number on the switch is always 0.
Port number: Reflects the actual port number on the switch. Port numbers always begin at 1,
starting with the far left port when facing the front of the switch.
For switches that have Cisco TwinGig Converter Modules in 10-Gigabit Ethernet module slots, the
interface type is tengigabitethernet, and the port numbers restart at 1. For example, the first
port on the first TwinGig Converter Module is referenced as tengigabitethernet1/0/1 and the
first port on the second TwinGig Converter Module would be referenced as
tengigabitethernet1/0/3.
For switches that are using Cisco dual SFP X2 converter modules in the 10-Gigabit Ethernet module
slots, the SFP module ports are numbered consecutively following the fixed port interfaces. For
example, if the switch has 24 fixed ports, the SFP module ports are gigabitethernet1/0/25
through gigabitethernet1/0/28.
For example, on a Cisco Catalyst® 3750-E, port #4 is identified by entering the following command:
The following example shows how to enable PortFast on Gigabit Ethernet interface 0/1 on switch 1:
Switch> enable
Switch(config-if)# exit
Switch(config)# exit
To view or confirm Port Fast status on a port, use the following command.
Note: Use the spanning-tree portfast default global configuration command to globally enable the
PortFast feature on all non-trunking ports.
This example shows how to globally enable PortFast by default on all access ports:
Switch(config)# end
For additional information on understanding and configuring Spanning-Tree Protocol on Cisco Catalyst
switches, see:
http://www.cisco.com/en/US/tech/tk389/tk621/technologies_configuration_example09186a00800946
7c.shtml
Note: Cisco Catalyst switch ports are capable of receiving, but not sending, pause frames.
Switch> enable
Switch(config-if)# exit
Switch(config)# exit
To view or confirm Flow Control status on a port, use the following command:
Switch> enable
Switch(config-if)# exit
Switch(config)# exit
To view or confirm storm control status on a port, use the following command:
Use the following commands to configure Jumbo Frames, which sets the maximum packet size to 9000
bytes:
Switch> enable
Switch(config)# exit
Switch# reload
The following example shows the output when you try to set Gigabit Ethernet interfaces to an out of
range value:
Once the switch completes reload, the configuration can be verified by entering the following
commands:
Switch> enable
Switch# configure t
Switch(config)# exit
The commands shown next are used to enable an individual VLAN to use Jumbo Frames. Note that
VLAN1 cannot have Jumbo Frames enabled. VLAN 2 must be used if Jumbo Frames are required.
Switch(vlan)# exit
To view or confirm MTU size on port 7 of the switch, use the following command.
http://www.cisco.com/en/US/products/hw/switches/ps700/products_configuration_example09186a00
8010edab.shtml
In future versions of this document we will continue to extract and include more information from the
various white papers and technical reports that are referenced here. All documents listed in Table 28
below are available for internet download, unless noted otherwise.
Published in 2010
Sizing and Best Practices for Deploying Microsoft Exchange Server 2010 on VMware vSphere and
Dell EqualLogic Storage (EqualLogic support site – registration required)
Deploying Microsoft SQL Server 2008 based Decision Support Systems using Dell EqualLogic 10GbE
iSCSI Storage (EqualLogic support site – registration required)
Published in 2009
Benefits and Best Practices for Deploying SSDs in an OLTP Environment Using Dell EqualLogic PS
Series (EqualLogic support site – registration required)
Configuring and Deploying the Dell EqualLogic Multipath I/O Device Specific Module (DSM)
Configuring VMware vSphere Software iSCSI with Dell EqualLogic PS Series Storage
Dell Reference Architecture Implementation Guide for Microsoft Exchange Server 2007 in a
Virtualized Environment using EqualLogic PS Series Storage
Dell Reference Architecture Sizing and Best Practices for Microsoft Exchange Server 2007 in a
VMware ESX Server Environment using EqualLogic PS Series Storage
Red Hat Linux v5.x Software iSCSI Initiator Configuration, MPIO and tuning Guide
Published in 2008
Creating and Applying IP Access Control Lists on Dell PowerConnect 62XX Series
Published in 2006
Published in 2005
Using Cisco Catalyst 3750 and 2970 Switches with a PS Series Group
Published in 2004