Status Code: 41

network connection timed out

The server did not receive any information from the client for too long a period of time. This can occur when a network connection is broken during a backup or restore. This can also occur when a NetBackup timeout is exceeded. Status Code 41

During Backup

When does the problem occur?

Start of Backup

How many clients are affected?

One

Network Issue Go to Section 6

Go to Section 3 Many

Media Server Issues: - Check NBU configuration and timeout values. - Check OS network settings or filesystem access. Client Issues: - Check Network configuration - Check OS network settings or filesystem access.

Go to Section 1

Status Code 41: network connection timed out

Page 1 of 17

Table of Contents
1 2 3 4 NetBackup Status Code: 41 - Defined................................................................................................... 3 NetBackup Status Code: 41 - Overview ................................................................................................ 3 General Troubleshooting Steps: ............................................................................................................ 5 Resolving Network Naming & Communication Problems...................................................................... 5 4.1 UNIX Clients..................................................................................................................................... 6 4.2 Windows Clients............................................................................................................................... 8 4.3 Verifying Host Names and Services Entries .................................................................................. 10 4.4 Using bpclntcmd to confirm name resolution ................................................................................. 12 Suggested NetBackup Logging ........................................................................................................... 13 5.1 NetBackup logs Windows ........................................................................................................... 13 5.2 NetBackup logs UNIX ................................................................................................................. 14 The Network......................................................................................................................................... 15 6.1 Physical Network problems............................................................................................................ 15 6.2 Heavy Network Load...................................................................................................................... 15 Hardware related Issues ...................................................................................................................... 16 Firewalls and VNETD/VxPBX .............................................................................................................. 16 Links..................................................................................................................................................... 17

5 6 7 8 9

Status Code 41: network connection timed out

Page 2 of 17

1 NetBackup Status Code: 41 - Defined

A Status code 41 is primarily related to two different system dependencies. These are network component addressing and network data communications. Network component addressing is the design for naming every network device (computer, array, router, switch, etc.) using a unique identifier so each network device can be located by any other network device without confusion. Network data communication is dependant on the supporting hardware, cabling and software that handles the transmission and reception of data between each network device. If either of these two network components fail during a backup or restore then NetBackup will likely exit with a status code 41. NetBackup relies on the rules that govern the TCP/IP network protocol for successful communication. The concepts of this network protocol as it relates to network communication are further defined with the OSI model where ISO outlines the seven network layers. NetBackup operates primarily at the application layer (layer 7) but depends on stability in at least layers 1 4 for dependable operation. The following link can be used for additional information on the ISO/OSI model: http://www.webopedia.com/quick_ref/OSI_Layers.asp

2 NetBackup Status Code: 41 - Overview

The NetBackup exit status code 41 will generate the message network connection timed out. This indicates the server did not receive any information from the client for too long a period of time. The following Recommended Actions can be taken for this error code: 1. On UNIX or Windows clients, check for the following problems with the bpbkar client process. On Windows clients: The bpbkar client process may not be hung, but due to the files and directories it is scanning, it has not replied to the server within the Client Read Timeout or Client Connect Timeout period. This has been known to occur during incremental backups when directories have thousands of unmodified files. For this case, launch the GUI and go to Host Properties > Master Servers section to change Client connect timeout or Client read timeout. These settings are in the Timeouts > Universal Settings tab. The default for these timeouts is 300 seconds. On UNIX clients: The bpbkar client process may be hung on a specific file or directory. For this case, add VERBOSE to the following to the clients /usr/openv/netbackup/bp.conf file. Then as root on the client execute:
# touch /usr/openv/netbackup/bpbkar_path_tr # mkdir /usr/openv/netbackup/logs/bpbkar

Then retry the operation. The names /usr/openv/netbackup/logs/bpbkar/log.<date> file will contain the names of the files before bpbkar processes them. The last file SelectFile: entry in the log will be the file name that is causing problems. Note: This procedure can be used to isolate other, unknown bpbkar hangs.

Status Code 41: network connection timed out

Page 3 of 17

When NetBackup hangs on a file due to mandatory file locking NetBackup can be configured to skip the locked files by setting LOCKED_FILE_ACTION = SKIP in the /usr/openv/netbackup/bp.conf file on the client. The bpbkar client process may not be hung on a file with mandatory locking, but due to the number of files and directories it is scanning, it has not replied to the server within the configured timeout. This can occur during backups when directories have thousands of unmodified files, or during restores of sparse files that have thousands of holes. It has also been seen when backing up file systems or directories that reside on optical disk, which is considerably slower than magnetic disk. For this case, try adding or modifying the CLIENT_READ_TIMEOUT and CLIENT_CONNECT_TIMEOUT values in the servers /usr/openv/netbackup/bp.conf file. The default for the CLIENT_READ_TIMEOUT and CLIENT_CONNECT_TIMEOUT is 300 seconds. After investigating the problem, delete the /usr/openv/netbackup/logs/bpbkar directory, since the log files can become quite large and are not deleted automatically. Also delete /usr/openv/netbackup/bpbkar_path_tr so you do not generate larger log files than needed the next time you create directory /usr/openv/netbackup/logs/bpbkar. 2. On Windows systems, disable the following file: <install_path>\VERITAS\NetBackup\bin\tracker.exe Repair hard drive fragmentation. You could try an application called Diskeeper Lite, which is part of the Windows NT Resource Kit. Make sure there is enough space available in \temp. 3. If the server cannot connect to the client, create bpcd or bpbkar debug log directories on the client. Then retry the operation and check the resulting logs. If these logs do not provide a clue, create a bpbrm debug log on the media server. Then retry the operation again and check the resulting debug log. If the UNIX /usr/openv/netbackup/logs/bpbrm/log.<date> file has entries similar to the following then the problem is in the routing configuration on the server.
<2> bpbrm hookup_timeout: timed out waiting during the client hookup <2> bpbrm Exit: client backup EXIT STATUS 41: network connection timed out

Verify that the client IP address is correct in the name service that is being used. On UNIX, if both NIS and DNS files are used, verify that they match. Also, see Resolving Network Naming & Communication Problems Section 4. 4. When using an AIX token ring adapter and when the routed daemon is running, a timeout can occur because of the token ring adapter. The token ring adapter creates dynamic routes which can cause the routed daemon to crash. 5. For a FlashBackup client, this can happen if the file system being backed up is very large and has a very large number of files. It can also occur if a large number of concurrent data streams are active at the same time. The corrective action is to add CLIENT_READ_TIMEOUT to the /usr/openv/netbackup/bp.conf file and set it to increase the timeout interval. 6. Make sure all recommended NetBackup patches have been installed. Check the VERITAS support web site for current patch information. To download the latest version of NetBackup 4.5, visit the Support Web site: http://support.veritas.com/menu_ddProduct_NETBACKUPDC_view_DOWNLOAD.htm

Status Code 41: network connection timed out

Page 4 of 17

To download the latest version of NetBackup 5.x, visit the Support Web site: http://support.veritas.com/menu_ddProduct_NBUESVR_view_DOWNLOAD.htm Run the NetBackup Configuration Validation Utility (NCVU) for the associated NetBackup nodes. Section two of the NCVU output will show which patches are applied. 7. Add the CLIENT_READ_TIMEOUT values to the master server, media server and client when a NetBackup database extension product is installed. The values should all be the same for each server. The value set is dependent on the size of the database being backed up. See the NetBackup System Administrators Guide for more information on CLIENT_READ_TIMEOUT. 8. If enhanced authentication is being used, ensure that its configured correctly. See the Configuring the NetBackup-Java console chapter in the NetBackup System Administrator's Guide for details on enhanced authentication. For example: If host A is configured to use enhanced authentication with host B, but host B is not configured to use enhanced authentication with host A. Connections from host B to host A are likely to fail with status code 41. Connections from host A to host B are likely to fail with authentication errors (status code 160).

3 General Troubleshooting Steps:

An important factor in determining the most likely cause of a Status 41 is to note at what point during the backup or restore the error occurs. If the Status 41 occurs immediately at the onset of the backup the problem is generally related to Network problems. If the Status 41 occurs after a significant amount of data has been transferred, or occurs at the end of the backup then client-side resource issues, NetBackup Configuration issues, or Hardware issues should be further explored. In most cases it will be necessary to review logs from a time in which the error is being returned to determine where in the backup the failure is occurring. Refer to the NetBackup Troubleshooting Guide for information on how to enable NetBackup logging. The steps listed below include the logs that will be most useful in troubleshooting a status code 41. Troubleshooting Steps: Enable and review the NetBackup bptm, bpbrm and bpcd logs on the NetBackup Media Server with the verbosity set to 5. Enable and review the NetBackup bpcd, bpbkar (for backups) and tar (for restores) logs on the NetBackup client with the verbosity set to 5. Review the Operating Systems system logs for any hardware related error messages. On the Windows platform see System & Application Event Viewer logs.

4 Resolving Network Naming & Communication Problems

The following procedure is for resolving NetBackup communications problems, such as those associated with NetBackup status code 41. There are two variations of this procedure: one for UNIX clients and another for PC clients. Note: In all cases, ensure that your network configuration is working correctly outside of NetBackup before trying to resolve NetBackup problems.

Status Code 41: network connection timed out

Page 5 of 17

4.1 UNIX Clients

For UNIX clients, perform the following steps. Before starting this procedure, add the VERBOSE = 5 option to the /usr/openv/netbackup/bp.conf file. Also, create a /usr/openv/netbackup/logs/bpcd debug log directory on your server and clients and a /usr/openv/netbackup/logs/bprd log directory on the server. During subsequent retries, the debug logs will provide detailed debug information that will be useful in analyzing the problem. To resolve network communication problems with UNIX clients: 1. For new or modified configurations: Check any recent modifications to ensure that they did not introduce the problem. Ensure that the client software was installed properly. Ensure that the client operating system is supported by the client software. Check the client names, server names, and service entries in your NetBackup configuration as explained in Verifying Host Names and Services Entries section 4.3. Check the verbose bprd debug log on the server to determine what occurred when the server received the request. Pay special attention to NIS or DNS updates that are required. Failing to properly update these services is a common source of network problems with NetBackup. Verify basic network connectivity between client and server by trying to ping the client from the server.
# ping <client_name>

2.

For example, to ping a client named ant:

# ping ant ant.nul.nul.com: 64 byte packets 64 bytes from 199.199.199.24: icmp_seq=0. time=1. ms ----ant.nul.nul.com PING Statistics---2 packets transmitted, 2 packets received, 0% packet loss round-trip (ms) min/avg/max = 1/1/1

Then try ping from the server from the client. If ping succeeds in both instances, it verifies basic connectivity between the server and client. If ping fails, a network problem outside of NetBackup exists that must be resolved before proceeding. 3. Check that the client is listening on the correct port for connections to bpcd. This can be done by running each of the following commands. The output will vary depending on platform and operating system.
# netstat -a | grep bpcd # netstat -a | grep 13782

For example, assume the client is a Solaris system and you execute:
# netstat -a | grep bpcd

If the bpcd daemon is listening on the port, the results will be similar to:
*.bpcd *.* 0 0 49152 0 LISTEN

The LISTEN indicates that the client is listening for connections on this port. If there is a problem, this line does not appear and one of the following three conditions exists: The /etc/services or NIS services map does not have the correct bpcd entry. The correct /etc services entry is:

Status Code 41: network connection timed out

Page 6 of 17

bpcd 13782/tcp bpcd

The /etc/inetd.conf does not have the correct bpcd entry. The correct /etc/inetd.conf entry is:
bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd

The /etc/inetd.conf was changed but was not reread. Correct this condition by executing one of the following depending on the operating system:
# /bin/ps -ef | grep inetd # kill -HUP <the_inetd_pid>

or
# /bin/ps -aux | grep inetd # kill -HUP <the_inetd_pid>

Note: On a Hewlett-Packard platform, use inetd -c to send a SIGHUP to inetd. For an AIX client, there are several ways to make this update. Use SMIT to verify that the InetServ object policy has been updated with the correct /etc/inetd.conf and /etc/services information about the bpcd process. After modifying the InetServ object policy, the inetexp command is automatically invoked. Edit the InetServ object policy using an ODM editor. Then run the inetexp command to export the InetServ object policy to the /etc/inetd.conf and /etc/services files. This keeps these files in sync with the InetServ object policy. Modifiy the files directly and then run the refresh -s inetd or kill -1 <the_inetd_pid> command to inform the inetd daemon of the changes to its configuration file. 4. Attempt to telnet to bpcd on the client. If it succeeds, keep the connection open until step 5 is performed.
# telnet <client_name> 13782

To telnet to the bpcd port on a client named ant do:

# telnet ant bpcd Trying x.x.x.x ... Connected to ant.nul.nul.com. Escape character is ^].

In this example, telnet can establish a connection to the client ant. If the telnet succeeds, then inetd on the client is configured correctly and is able to pass its connection to bpcd and NetBackup should also be able to establish a connection. If telnet doesnt work, ensure that the /etc/inetd.conf file and /etc/services files on both the server and client have correct and matching entries. Also, verify and update the applicable NIS maps. If all these files are correct and the connection is not successful to the client, a network problem outside of NetBackup exists that must be resolved before proceeding. 5. Check that the client is listening on the correct port for the telnet connection to bpcd. This can be done by running one of the following commands. These will vary depending on platform and operating system.
# netstat -a | grep bpcd # netstat -a | grep 13782

For example, assume the client in step 4 is a Solaris system named ant and the telnet is from a NetBackup server named whale:
# netstat -a | grep bpcd

Status Code 41: network connection timed out

Page 7 of 17

If the bpcd daemon is listening on the port, the results will be similar to:
*.bpcd ant.bpcd *.* 0 whale.33132 17520 0 0 49152 49640 0 LISTEN 0 ESTABLISHED

The LISTEN in the first line indicates that the client is listening for further connections on this port. The second line of the result, ESTABLISHED indicates that the telnet connection was established to bpcd through port 13782 on the client. Note: It is not recommended to change the port number for bpcd or other NetBackup services. Do so only if there is no alternative; and then, remember that all NetBackup servers and clients in the configuration must be updated use this new port assignment. 6. If there is a process other than bpcd using the port, try a reboot of the client. If the problem is still not fixed, the other process may be using the same port number as bpcd. It might be necessary to change one of the service numbers, preferably for this other service. This other process will need to be identified before attempting any changes to port numbers. 7. To verify basic client to master server communications, use the bpclntcmd utility. When run on a NetBackup client, the -pn and -sv options initiate inquiries to the NetBackup master server as configured in the bp.conf file on the client. The master server then returns information to the requesting client. For more information, see Using bpclntcmd Section 4.4.

4.2 Windows Clients

To resolve network communication problems with Windows clients: 1. Enable logging before retrying the failed operation: Increase the Verbose logging level to five and General logging level to two on the client. To change these values start the Backup, Archive, and Restore interface and go to File > NetBackup Client Properties > Troubleshooting. On the NetBackup master server, create a bprd directory and on the clients create a <install_path>\VERITAS\NetBackup\logs\bpcd directory. 2. If this is a new client, verify the client and server names in the NetBackup configuration as explained in Verifying Host Names and Services Entries section 4.3. 3. Verify basic network connectivity between client and server by pinging from the server to the client and from the client to the server. Use the following command:
% ping <hostname>

The <hostname> is the name of the host as configured in the NetBackup policy configuration. This name should also match the WINS and DNS configuration or the hosts file in the system directory: %SystemRoot%\system32\drivers\etc\hosts (Windows NT/2000, XP, 2003) C:\Windows\hosts (default on Windows 98 and 95) If ping succeeds this verifies basic connectivity between the server and client. If ping fails, a network problem exists outside of NetBackup that must be resolved before proceeding. 4. On Microsoft Windows or NetWare clients, check the NetBackup Client service is working properly:

Status Code 41: network connection timed out

Page 8 of 17

a. Ensure that the service is active, either by checking the logs or as follows: On Windows NT/2000, XP or Windows Server 2003 clients, use the Services application in the Control Panel to verify that the NetBackup Client service is running and start it if necessary. On Windows 98 or 95 clients, check the system tray on the taskbar for the NetBackup client icon. If the icon is not there, run the NetBackup Client Job Tracker program from the NetBackup Program folder or the Start menu. When the icon is present, rightclick on the icon to start the NetBackup client daemon. On NetWare clients, enter load bpcd from the NetWare server console to start the NetBackup client daemon. b. Check the bpcd debug logs for problems or errors. See Using Logs and Reports chapter for instructions on enabling and using these logs. c. Verify that the same NetBackup client Service (bpcd) port number is specified on both the NetBackup client and server. To display this dialog, start the Backup, Archive, and Restore interface on the client and click File > NetBackup Client Properties > Network and check the NetBackup Client Service Port number. Verify that the setting on the Network tab matches the one in the services file. The services file is located in: %SystemRoot%\system32\drivers\etc\services (Windows NT/2000, XP or 2003) C:\Windows\services (Windows 98 and 95) On NetWare clients, see the BPCD setting in the openv\netback\bp.ini file. On UNIX NetBackup servers, the bpcd port number is in the /etc/services file. On Windows NetBackup servers, see the Client Properties dialog box in the Host Properties. If necessary, correct the port number and then, on Windows clients and servers, stop and restart the NetBackup Client service. On Microsoft Windows 98 or 95 and NetWare clients, stop and restart the NetBackup client daemon (bpcd). Note: Do not change NetBackup port assignments unless it is absolutely necessary in order to resolve conflicts with other applications. If these must be changed, do so on all NetBackup clients and servers. These numbers must be the same throughout your NetBackup configuration. 5. Verify that the NetBackup Request Service (bprd) Port number on Microsoft Windows and NetWare clients is the same as on the server (by default, 13720). 6. Verify that the hosts file or its equivalent contains the NetBackup server name. The hosts files for different operating systems are located in: %SystemRoot%\system32\drivers\etc\hosts (Windows NT/2000, XP or 2003) C:\Windows\hosts (Windows 98 or 95) SYS:etc\hosts (NetWare) /etc/hosts (UNIX) 7. Verify client-to-server connectivity by using ping, telnet or ftp from the client back to the server. 8. For a NetWare client, ensure that the server is not trying to connect when a backup or restore is already in progress on the client. Attempting more than one job at a time on these clients, results in a cant connect or similar error.

Status Code 41: network connection timed out

Page 9 of 17

9. Use the bpclntcmd utility to verify basic client to master server communications. When run on a NetBackup client, the -pn and -sv options initiate inquiries to the NetBackup master server configured in the server list on the client. The master server then returns information to the requesting client. For more information, see Using bpclntcmd section 4.5. 10. Verify that the client operating system is supported by the client software.

4.3 Verifying Host Names and Services Entries

This procedure is useful if you encounter problems with host names or network connections and want to verify that the NetBackup configuration is correct. Note: For more information on host names, refer to the Networks and Hostnames section in this document and to the Rules for Using Host Names in NetBackup appendix in the NetBackup System Administrators Guide. To verify the client and server host names in NetBackup: 1. Verify that the correct client and server host names are configured in NetBackup. On Windows servers, Windows clients and Non Target NetWare clients, check the General tab in the NetBackup Client Properties dialog and the Servers tab in the Specify NetBackup Machines and Policy Type dialog box. To display these dialog boxes start the Backup, Archive, and Restore interface on the client. For the General tab, click NetBackup Client Properties on the File menu. For the Servers tab, click Specify NetBackup Machines and Policy Type on the File menu. On the Servers tab, ensure that there is a server entry for the master server and each media server. On Windows systems, the correct server must be designated as the current master server in the list. Stop and restart the NetBackup daemons if these names are changed. On the General tab, verify that the client name setting is correct and matches what is in the policy client list on the master server. On a master or media server, ensure there is a server entry for each Windows administrative client that can be used to administer that server. If a host name is misspelled in the bp.conf file (UNIX) or via the servers list (Windows) on the master server, or if a host name cannot be resolved via gethostbyname, the following error messages will be logged in the NetBackup error log:
Gethostbyname failed for <hostname>:<h_errno_string> (<h_errno>) One or more servers was excluded from the server list because gethostbyname() failed.

Make the above changes on the appropriate tabs in the properties dialog boxes on a Windows NetBackup server. On UNIX NetBackup servers and clients, and Macintosh clients, check the server and client name entries in the bp.conf file: Ensure there is a SERVER entry for the master server and each media server in the configuration. The master server must be the first name in the list. Remember, if you add or modify SERVER entries on the master server, you must stop and restart bprd and bpdbm before the changes take effect. Ensure that the CLIENT_NAME option (if included) is correct and matches what is in the policy client list on the master server. The bp.conf file is in the /usr/openv/netbackup directory on UNIX clients and it is in the Preferences:NetBackup folder on Macintosh clients. Users on UNIX clients can

Status Code 41: network connection timed out

Page 10 of 17

also have a personal bp.conf file in their home directory. A CLIENT_NAME option in $HOME/bp.conf overrides the one in /usr/openv/netbackup/bp.conf. On NetWare clients, check the openv\netback\bp.ini file to ensure that: There is a SERVER entry for the master server and each media server in the configuration. The master server must be the first name in the list. The ClientName entry and the entries in the [clients] section are correct and match what is in the policy client list on the master server. On the master server, see if someone has created any required /usr/openv/netbackup/db/altnames files (UNIX) <install_path>\NetBackup\db\altnames files (Windows). Pay particular attention to requirements for host.xlate file entries. 2. Verify that each server and client has the required entries for NetBackup reserved port numbers. a. On NetBackup servers, check the services files to ensure that they have entries for: bpcd and bprd vmd bpdbm Processes for configured robots (for example, tl8cd). See the Media Manager System Administrators Guide for a list of these processes. b. On UNIX, Windows, and NetWare clients, verify the NetBackup client daemon or service number, and the request daemon or service port number. On UNIX clients, check the bprd and bpcd entries in the /etc/services file. On Microsoft Windows clients, verify that the NetBackup Client Service Port number and NetBackup Request Service Port number on the Network tab in the NetBackup Client Properties dialog match the settings in the services file. To display this dialog, start the Backup, Archive, and Restore interface on the client and click NetBackup Client Properties on the File menu. The values on the Network tab are written to the services file when the NetBackup Client service starts. The services file is located in: %SystemRoot%\system32\drivers\etc\services (Windows NT/2000, XP or 2003) C:\Windows\services (Windows 98 and 95) openv\netbackup\bp.ini (NetWare clients, check the BPCD and BPRD entries) /etc/services (UNIX) 3. On UNIX servers and clients, verify the /etc/inetd.conf file to has the following entry:
bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd

4. On Linux servers and clients, verify that the file /etc/xinetd.d/bpcd is correct. The file should look similar to the following:
{ disable socket_type protocol wait user server } = = = = = = no stream tcp no root /usr/openv/netbackup/bin/bpcd

5. On Windows servers and clients, verify that the NetBackup Client service is running.

Status Code 41: network connection timed out

Page 11 of 17

6. If you are using NIS in your network, update those services to include the NetBackup information that is added to the /etc/services file. 7. The hostname information in NIS, DNS and hosts files must correspond to what is in the policy configuration and the entries in the following: On Windows NetBackup servers, Microsoft Windows clients, and Non Target NetWare clients, check the General tab in the NetBackup Client Properties dialog and the Servers tab in the Specify NetBackup Machines and Policy Type dialog. To display these dialogs, start the Backup, Archive, and Restore interface on the client. For the General tab, click NetBackup Client Properties on the File menu; for Servers tab, click Specify NetBackup Machines and Policy Type on the File menu. The bp.conf file on UNIX servers and clients and Macintosh clients. The openv\netback\bp.ini file on NetWare clients. Also, verify that reverse DNS addressing is configured.

4.4 Using bpclntcmd to confirm name resolution

The bpclntcmd utility resolves IP addresses into host names and host names into IP addresses by using the same system calls as the NetBackup application software. The bpclntcmd command can be found at: For Windows: <install_path>\NetBackup\bin\bpclntcmd.exe For UNIX: /usr/openv/netbackup/bin/bpclntcmd On Windows, run this command in an MS-DOS command window. The bpclntcmd options that are useful for testing hostname and IP address resolution are -ip, -hn, -sv and -pn.
# bpclntcmd -ip <IP_Address>

The -ip option looks up a specified IP address. The bpclntcmd uses gethostbyaddr() on the IP address and returns the hostname. This result is based on how the IP address is defined in the systems DNS, WINS, NIS, or local hosts file entries. No connection is established with the NetBackup server. This should return the same hostname and IP information as the hn option.
# bpclntcmd -hn <hostname>

The -hn option looks up a specific hostname. The bpclntcmd uses gethostbyname() on the hostname to obtain the IP address. This result is based on how the hostname is defined in the systems DNS, WINS, NIS, or local hosts file entries. No connection is established with the NetBackup server. This should return the same hostname and IP information as the ip option.
# bpclntcmd sv

The -sv option displays the NetBackup version number.

# bpclntcmd pn

Run on the client, the pn option verifies the ability of the NetBackup client to resolve and connect to the server, and at the same time verifies the servers ability to resolve the clients name and connect back to it. The pn first identifies the configured master in the bp.conf on the client, then resolves its IP address. Next, it sends connects to that IP address asking what system the query came from. The master then resolves the requesting clients hostname from the IP address, and sends the name it resolved back to the requesting address. For example:
bpclntcmd -pn expecting response from server rabbit.friendlyanimals.com dove.friendlyanimals.com dove 123.145.167.3 57141

Status Code 41: network connection timed out

Page 12 of 17

Where:

expecting response from server rabbit.friendlyanimals.com is the master server entry from the SERVER list on the client. dove.friendlyanimals.com is the hostname returned by the master server. The master server obtained this name through gethostbyaddr(). dove is the client name configured in the NetBackup policy client list. 123.145.167.3 is the IP address of the client connection at the master server. 57141 is the port number of the connection on the client.

5 Suggested NetBackup Logging

In this section the minimum logging selections are outlined based on the role of the specific NetBackup host. Determine the most likely hosts involved with the problem and use this section to determine the necessary logs needed to do in-depth Status 41 troubleshooting.

5.1 NetBackup logs Windows

NetBackup logs on a Windows host are located at: <install_path>\VERITAS\NetBackup\logs For all Windows NetBackup Master Servers enable bpcd and bprd logs. For all Windows NetBackup Media Servers enable bpcd and bpbrm logs. Based on the operation being performed for a Windows NetBackup client enable the following logs: File System Backups Enable bpcd and bpbkar logging. File System Restores: Enable bpcd and tar logging. IBM DB2 Database agent Backups and Restores Enable bphdb, bpdb2 and bpcd logging. Lotus Notes, MS Exchange and MS Sharepoint Portal Server Backups and Restores Enable bphdb, bpcd and bpbkar logging for backups. Enable bphdb, bpcd and tar and logging for restores. MS SQL Server Backups and Restores Enable bphdb, bpcd and dbclient logging for backups and restores. Collect progress logs: <install_path>\NetBackup\logs\user_ops\MsSql\Logs \*.prg Oracle Backups and Restores Enable bphdb, bpcd and dbclient logging for backups and restores. Collect Oracle specific RMAN output file <scriptname>.out. SAP Backups and Restores Enable bphdb, backint, bpbackup and bpbkar logging for backups. Enable bphdb, backint, bprestore and tar logging for restores. Collect SAP script, Progress log, and sapdba backup log. Sybase Backups and Restores Enable bphdb, bpcd and sybackup logging for backups and restores. Teradata Backups and Restores Enable bphdb, bpcd, dbclient and dbbackup logging for backups and restores. Collect progress logs: <install_path>\NetBackup\logs\user_ops\teradata\Logs \*.prg

Status Code 41: network connection timed out

Page 13 of 17

5.2 NetBackup logs UNIX

NetBackup logs on a UNIX host are located at: /usr/openv/netbackup/logs For all UNIX NetBackup Master Servers enable bpcd and bprd logs. For all UNIX NetBackup Media Servers enable bpcd and bpbrm logs. For a UNIX NetBackup client enable the following logs based on the operation being performed. File System Backups Enable bpcd and bpbkar logging. File System Restores: Enable bpcd and tar logging. IBM DB2 Database agent Backups and Restores Enable bphdb, bpdb2 and bpcd logging. Informix Backups and Restores Enable bphdb, bpcd and infxbsa logging for backups and restores. Collect Informix specific - BAR_ACT_LOG, BAR_DEBUG_LOG & MSGPATH logs. Lotus Notes Backups and Restores Enable bphdb, bpcd and bpbkar logging for backups. Enable bphdb, bpcd and tar logging for restores. Oracle RMAN Backups and Restores Enable bphdb, bpcd and dbclient logging for backups and restores. Collect Oracle specific - RMAN output file: <scriptname>.out Oracle (Script-based BLI) Backups and Restores Enable bphdb, bpcd and bpbkar logging for backups. Enable bphdb, bpcd and tar logging for restores. Oracle (Proxy Copy/RMAN BLI) Backups and Restores Enable bphdb, bpcd, bpbkar, dbclient and bpdbsbora logging for backups. Enable bphdb, bpcd, tar, dbclient and bpdbsbora logging for restores. SAP Backups and Restores Enable bphdb, bpcd, backint, bpbackup and bpbkar logging for backups. Enable bphdb, bpcd, backint, bprestore and tar logging for restores. Collect progress log, sapdba restore log, brrestore output (.anf / .pnf logs) Sybase Backups and Restores Enable bphdb, bpcd, sybackup logging for backups and restores. Teradata Backups and Restores Enable bphdb, bpcd, dbclient and dbbackup logging for backups and restores. Collect the Teradata specific Arcmain log.

Status Code 41: network connection timed out

Page 14 of 17

6 The Network
A Status 41 can be triggered by a servers network card configuration. This would include incorrect link negotiation, outdated driver or firmware version or issues with vendor specific software such as network teaming. NIC Troubleshooting Overview: Check the NIC vendors web site for the latest drivers and firmware and update if necessary. Confirm that the NIC is hard-coded to the appropriate duplex setting. (Example: 100 megabyte Full Duplex) The default duplex setting is typically to auto negotiate which has been known to cause some problems and may trigger a status 41. If NIC teaming is being used confirm that the latest teaming software is being used. Network teaming software is used to team multiple network cards for either fail-over or load balancing. Network teaming software has been known to cause unexpected behavior with NetBackup. Network related errors such as a status code 41 have been seen as a result of using network teaming. If the server generating the status code 41 is using network teaming software, ensure that it is running the latest network teaming software. In troubleshooting network problems, it would also be a good idea to disable network teaming as a test and see if the problem persists.

6.1 Physical Network problems

The physical network includes cables, routers, switches, bridges or any other physical medium connecting the NetBackup servers. There are several different network tools that can be used to diagnose problems with the physical network. Possible Tools for Troubleshooting the Physical Network: VERITAS Support uses a tool called AppareNet which is a very useful tool for troubleshooting network problems. Operating System utilities such as ping and traceroute can be used to help isolate physical network problems. There are also various third party utilities that can be used to help isolate network problems. Use a cable tester to confirm that the network cables are working correctly. Check the switch and routers management console which should log any errors that are occurring. Output of the OS command netstat can be used to see error conditions on the network interface

6.2 Heavy Network Load

NetBackup is designed to backup the data as fast as possible and can put a heavy load on the network. If the network can not handle the load being generated by the backup, network related problems will be seen. Recommendations: Configure the backup windows to run during non-peak operating hours. Make the larger, more powerful servers NetBackup Media Servers so the data is written directly to tape rather than over the network.

Status Code 41: network connection timed out

Page 15 of 17

7 Hardware related Issues

Hardware related issues that would cause the disk I/O to be degraded or fail. This can occur if the backup is unable to read from the files from the disk efficiently. Likely causes would be a bad or faulty drive, SCSI card or HBA if the disk is attached to a SAN. A highly fragmented disk could also cause a status code 41. This has been seen when the disk is fragmented greater than 20%. Defragmenting the disk may help correct the problem. Note: Do not run a disk defragmenter utility during a backup. The backup process attempts to read the data from the disk as fast as possible which will cause heavy disk I/O normally not seen during regular day to day user operations.

8 Firewalls and VNETD/VxPBX

Firewalls create a unique set of circumstances that have the potential to cause Exit Status 41 problems by directing communication through an independent device or application that has the ability to limit port functionality. This in-turn can take control away from NetBackup processes causing timeouts or delays in data transmission. The VERITAS Network Daemon, or vnetd, was introduced in NetBackup 4.5GA. The primary goal of vnetd was to provide the ability to configure communication in NetBackup environments protected by firewalls. Since NetBackup connectivity relies primarily on TCP connections the goal of vnetd in the NetBackup 4.5 GA release was to resolve the "call-back" or "connect-back" issue. This is where connections made back to a NetBackup Server by client processes (bpbkar, bpcd, etc.) were made to destination TCP ports randomly chosen from a predefined range. Instead, with vnetd, we can now configure these connect-backs to NetBackup Servers to use a single port - vnetd port 13724. In addition, starting in NetBackup 4.5 FP6, configuring "Reduced Firewall Usage" provides the capability where vnetd can now make connections to the standalone daemons bprd, bpdbm, bpjobd, vmd and robotic control daemons. Benefits of using "vnetd" instead of Standalone Daemon Port Numbers: Applicable to standalone daemons such as bprd, bpdbm, bpjobd, vmd, and robotic daemons. Forces connections made to the standalone daemons to connect to the destination server's vnetd port instead, and vnetd then passes the socket to the applicable standalone daemon. Using vnetd be configured in the GUI under both Master and Media Server Host properties using the Firewall tab, as explained above. You would enter the hostname(s) of the servers you want the current server to connect to using Automatic, vnetd only, or daemon only. This can also be configured manually in the bp.conf file if the GUI is not available. If the bp.conf file on a server shows "CONNECT_OPTIONS = <servername> 0 1 1" this indicates vnetd only instead of the usual daemon ports when connecting to the specified server. Note: The last digit in the entry enables the use of vnetd. In order to force connections made to vmd and robotic daemons in a SAN environment to use vnetd, the "CONNECT_OPTIONS = <servername> 0 1 1" must be added to all the vm.conf files of the Master Server and all Media Servers. VxPBX - VERITAS Private Branch Exchange Starting in NetBackup 6.0GA, VxPBX goes beyond vnetd to perform many of the same socket streamlining functions but with greater performance. It acts as a one time communication broker connecting authorized client services with server services.

Status Code 41: network connection timed out

Page 16 of 17

From the NBU 6.0 Manual: NetBackup 6.0 contains features that are dependent on a new Common Services Framework (CSF) called, VERITAS Private Branch Exchange (VxPBX). VxPBX helps limit the number of TCP/IP ports used by many new features in NetBackup, such as IRM and EMM. VxPBX is required for all NetBackup uses. It is installed with NetBackup 6.0GA

9 Links
Click here to Search for other documents on Status 41
Also, you may click below to perform a search on the following relevant items: Status Code 41 network connection timed out Windows NT Client backups very slow, on the order of kb /sec rather than mb /sec., or exits with status 41. http://support.veritas.com/docs/203981 All backups fail when using the Microsoft (Internet Security and Acceleration server) firewall client http://library.veritas.com/docs/268872 When a Firewall is installed between Master/Media Server and Client, the network connection to the client times out, although no port restrictions are implemented on the Firewall. http://support.veritas.com/docs/245355

Status Code 41: network connection timed out

Page 17 of 17