Troubleshooting TCPIP
Troubleshooting TCPIP
Troubleshooting TCPIP
Troubleshooting TCP/IP
This chapter presents protocol-related troubleshooting information for Transmission Control
Protocol/Internet Protocol (TCP/IP) connectivity and performance problems.
The sections in this chapter focus on general TCP/IP problems and on routing problems related to
the Routing Information Protocol (RIP), the Interior Gateway Routing Protocol (IGRP), Enhanced
IGRP, Open Shortest Path First (OSPF), the Border Gateway Protocol (BGP), and the Hot Standby
Router Protocol (HSRP). Each section describes a specific symptom, the problems that are likely to
cause each symptom, and the solutions to those problems.
• TCP/IP: Local Host Cannot Access Remote Host
• TCP/IP: Routes Learned from Wrong Interface or Protocol
• TCP/IP: Routing Not Functioning Properly on New Interface
• TCP/IP: Host Connections Fail Using Certain Applications
• TCP/IP: Problems Forwarding BOOTP and Other UDP Broadcasts
• TCP/IP: Poor Performance
• RIP/IGRP: Routes Missing from Routing Table
• OSPF: Routers Not Establishing Neighbors
• OSPF: Routes Missing from Routing Table
• IP Enhanced IGRP: Routers Not Establishing Neighbors
• IP Enhanced IGRP: Routes Missing from Routing Table
• IP Enhanced IGRP: Router Stuck in Active Mode
• BGP: Routes Missing from Routing Table
• BGP: Routers Not Advertising Routes
• HSRP: Hosts Cannot Reach Remote Networks
The symptoms described in the following sections are generic in nature and pertain to general
TCP/IP internetwork problems. However, when host configuration problems are discussed, they are
addressed assuming UNIX end systems. Similar types of actions might be applicable for non-UNIX
hosts, but the discussion does not specifically address non-UNIX end-station problems.
where address is the IP address of the default gateway (the router local to the host).
The value 1 indicates that the specified gateway is one hop away.
You might need to reboot the host for this change to take effect.
Step 3 It is recommended that you specify a default gateway as part of the boot process.
Specify the IP address of the gateway in the /etc/defaultrouter UNIX host file. This
filename might be different on your UNIX system.
If you are working with a PC or a Macintosh, consult the corresponding
documentation to determine how to set the default gateway.
Misconfigured or missing routed default Step 1 If the host is running routed, use the netstat -rn UNIX command to view the host’s
routes routing table. The entry with Destination “default” denotes the default route.
Step 2 The default route entry should point to the router which has the route to the remote
host. If there is no default route entry, use the route UNIX command to manually
configure the default gateway.
DNS1 host table is incomplete If the DNS host table is incomplete, the DNS cannot reply to some lookup requests. If the DNS
receives a lookup request for a hostname that is not in its cache, it cannot reply to the request,
and the client cannot establish a connection.
Step 1 At the UNIX prompt, enter the following command:
unix-host% host address
Step 5 If routing is not enabled on the router (or routers), enable the proper routing protocol
using the router global configuration command.
Step 6 In router configuration mode, enter the appropriate network commands to associate
networks with the routing process, as applicable.
For example, to enable IGRP routing for networks 193.166.66.0 and 193.168.25.0,
enter the following configuration commands:
Router(config)# router igrp 109
Router(config-router)# network 193.166.66.0
Router(config-router)# network 193.168.25.0
For complete information on configuring specific IP routing protocols, see the Cisco IOS
Network Protocols Configuration Guide, Part 1 and Network Protocols Command Reference,
Part 1.
Routing is misconfigured on one or more Narrow the specific symptoms down and troubleshoot the problem using the procedures
routers outlined later in this chapter.
For example, check the routing tables on various routers using the show ip route privileged
EXEC command. If you are running IGRP and there are routes missing from the routing table
(that is, you see no routes to certain networks that you know are connected), refer to the section
“RIP/IGRP: Routes Missing from Routing Table” later in this chapter.
1. DNS=Domain Name Service
Step 3 If split horizon is not enabled, enter the ip split-horizon interface configuration
command on the remote router interface.
For example, to enable split horizon on serial interface 0, enter the following
commands:
C4500(config)#interface s0
C4500(config-if)#ip split-horizon
Note: The default split-horizon setting for all LAN interfaces is enabled. However, for WAN
multipoint interfaces configured with X.25, Frame Relay, or SMDS encapsulation, the default
split-horizon setting is disabled.
Redistribution
S0 E1
Router 2
Network WAN
168.170.69.0 Router 1
255.255.255.0 Network
193.10.1.0
255.255.255.0
S0
Router 3
S4918
IGRP RIP
RIP routing information learned by Router 2 from Router 1 is redistributed into the IGRP domain.
IGRP routing updates are sent to Router 3 from Router 2. If split horizon is disabled on Router 3,
Router 3’s updates to Router 2 will include information about network 193.10.1.0 (which was
originally learned from RIP updates sent from Router 1 to Router 2).
Because IGRP routes by default are given a lower (better) administrative distance than RIP routes,
Router 2 will route traffic to network 193.10.1.0 out serial interface 0 (towards Router 3) rather than
out Ethernet interface 1 (towards Router 1).
Enabling split horizon on Router 3’s serial interface prevents the router from advertising any of the
RIP routes it has learned. However, in some cases, enabling split horizon is not desirable (for
example, in a hub-and-spoke environment). In such a situation, route filtering using an input
distribution list can be configured on Router 2’s serial interface 0, as the following example shows:
Router_2(config)#router igrp 100
Router_2(config-router)#distribute-list 5 in
Router_2(config)#access-list 5 deny 193.10.1.0 255.255.255.0
Router_2(config)#access-list 5 permit 168.170.69.0 255.255.255.0
This distribution list specifically denies routing updates from Router 3 that advertise network
193.10.1.0, thus preventing Router 2 from learning information about this network from the wrong
protocol and the wrong interface. Be sure to configure explicit permit statements for any traffic that
you do want Router 2 to accept.
Make sure that process IDs, addresses, and other variables are properly specified for the routing
protocol you are using. For more information, refer to the Cisco IOS configuration guides and
command references.
No active interfaces are configured with an OSPF uses an IP address on the router as its router ID. Therefore, to configure the OSPF
IP address (OSPF only) protocol on a router, you need at least one active interface configured with an IP address. If
there is no active interface with an IP address, the router will return the following error:
2509(config)#router ospf 100
2509(config)#
OSPF: Could not allocate router id
Step 1 Use the show ip interfaces privileged EXEC command on the router to make sure
there is a router interface that is up and configured with an IP address.
Step 2 If there is no active interface with an IP address, configure an interface with the ip
address interface configuration command. If necessary, use the no shutdown
interface configuration command to bring an interface up.
Step 3 After disabling all of the access lists on the router, determine if the application in
question operates normally.
Step 4 If the application operates normally, an access list is probably blocking traffic.
Step 5 To isolate the problem list, enable access lists one at a time until the application no
longer functions. Check the problem access list to see if it is filtering traffic from any
TCP or UDP ports.
Step 6 If the access list denies specific TCP or UDP ports, make sure that it does not deny the
port used by the application in question (such as TCP port 23 for Telnet).
Enter explicit permit statements for those ports used by applications you want to have
functional.
Step 7 If you altered an access list, enable the list to see if the application can still operate
normally.
Step 8 If the application operates normally, perform the preceding steps to isolate any other
problem access lists until the application operates correctly with all access lists
enabled.
For more information about misconfigured access lists, see the section “Misconfigured Access
List Example” later in this chapter. For more information on configuring access lists, see the
Cisco IOS configuration guides and command references.
S0
S5046
When examining the configuration of Router Y, the network administrator finds the following
extended access list configured on the router:
C4500#show ip access-lists
Extended IP access list 101
permit tcp any any eq telnet
permit icmp any any
C4500#show running-config
[...]
interface Serial0
ip address 192.168.54.92 255.255.255.0
ip access-group 101 out
[...]
The access list permits only ICMP (used by the ping application) and TCP (used by the Telnet
application) traffic to pass serial interface 0. Any traffic destined for UDP ports, including the default
ports used by the trace application (UDP ports 33434 and above), is implicitly denied.
To allow trace traffic to pass through Router Y, the network administrator makes the following
change to the access list:
C4500#configure terminal
C4500(config)#access-list 101 permit udp any any gt 33433
C4500(config)#^Z
C4500#
%SYS-5-CONFIG_I: Configured from console by console
C4500#show ip access-lists
Extended IP access list 101
permit tcp any any eq telnet
permit icmp any any
permit udp any any gt 33433
C4500#
Table 5-5 TCP/IP: Problems Forwarding BOOTP and Other UDP Broadcasts
Step 2 If UDP broadcasts are disabled at specific UDP ports, enter the ip forward-protocol
udp port global configuration command (you can also specify a keyword, such as
domain, rather than the port number).
For example, to reenable DNS broadcasts, enter the following command:
C4500(config)#ip forward-protocol udp domain
Step 3 After disabling all access lists, determine if the BOOTP or other UDP broadcasts are
forwarded normally.
Step 4 If broadcasts are forwarded normally, an access list is probably blocking traffic.
Step 5 To isolate the problem access list, enable access lists one at a time until broadcasts are
no longer forwarded.
Step 6 Check the problem access list to see if it is filtering traffic from any UDP ports. If an
access list denies specific UDP ports, make sure that it does not deny ports used to
forward the broadcast traffic in question (such as UDP port 67 for BOOTP or port 68
for BOOTP replies).
Enter explicit permit statements for those ports used to forward broadcasts that you
want to have forwarded.
Step 7 If you altered an access list, enable the list to see if broadcasts are still forwarded
normally.
Step 8 If problems persist, perform the preceding steps on routers in the path until broadcast
traffic is forwarded correctly.
For more information about misconfigured access lists, see the section “Misconfigured Access
List Example” earlier in this chapter. For more information on configuring access lists, see the
Cisco IOS configuration guides and command references.
Make sure the proper process IDs, addresses, and other variables are properly specified for the
routing protocol you are using. For more information, consult the Cisco IOS configuration
guides and command references.
Step 4 After disabling all distribution lists on the router, use the clear ip route privileged
EXEC command to clear the routing table.
Step 5 Determine if the routes appear in the routing table by using the show ip route
privileged EXEC command.
Step 6 If routes appear properly in the routing table, the access list referenced by the
distribute-list command is probably configured to deny certain updates.
Step 7 To isolate the problem list, enable distribution lists until routes stop appearing in the
routing table. (You might have to use the clear ip route command after enabling each
list.)
Step 8 Use the show running-config command and make sure that the problem list does not
deny updates inappropriately. If the access list denies updates from specific addresses,
make sure that it does not deny the address of a router from which routing updates
should be received.
Change the access list to allow the router to receive updates from the proper addresses.
Remember that an implicit deny any ends every access-list.
Configure explicit permit statements for those addresses from which the router should
receive updates.
Step 9 If you altered an access list, enable the distribution list using the distribute-list
command. Use the clear ip route command and check to see if the missing routing
information appears in the routing table.
Step 10 If the routes appear, perform the preceding steps on all routers in the path until the
routing information appears properly with all distribution lists enabled.
For more information on configuring access lists, see the Cisco IOS configuration guides and
command references.
For more information about subnet masks, see the section “Host and Router Subnet Mask
Mismatch Example” later in this chapter.
Missing default-metric command This problem is restricted to environments in which route redistribution is being performed
between autonomous systems or between multiple routing protocols.
Step 1 Use the show running-config privileged EXEC command on suspect routers. Look
for default-metric router configuration command entries. This command assigns
default metric values to redistributed routes.
Step 2 IGRP requires a default-metric parameter to redistribute routes. If you are running
IGRP, define the default metrics for redistributed routes using the default-metric
router configuration command.
The following example shows a configuration that redistributes RIP routes and assigns
them IGRP metrics with values as follows: bandwidth = 1000, delay = 100,
reliability = 250, loading = 100, and mtu = 1500.
router igrp 109
network 131.108.0.0
redistribute rip
default-metric 1000 100 250 100 1500
Step 3 If you are running RIP, you do not have to configure a default metric in order to
redistribute routes. By default, the metric assigned to all routes redistributed into RIP
is 1. However, this value can be changed using the default-metric command.
If a default-metric statement that is applied to RIP appears in the configuration, make
sure that the metric value it assigns will not adversely affect network performance. If
you are unsure, restore the default value for the routing metric using the no
default-metric router configuration command.
For more information on the default-metric router configuration command, see the Cisco IOS
configuration guides and command references.
Step 4 If the router is running multiple routing protocols, look for a redistribute router
configuration command entry. Make sure that routing information is being properly
exchanged between protocols.
For example, to redistribute routes between RIP (running in network 15.0.0.0) and
IGRP autonomous system 109 (network 128.1.0.0), enter the following commands:
C7010(config)#router igrp 109
C7010(config-router)#network 128.1.0.0
C7010(config-router)#redistribute rip
C7010(config-router)#default-metric 10000 100 255 1 1500
C7010(config-router)#distribute-list 10 out rip
C7010(config-router)#access-list 10 permit 15.0.0.0
Step 5 If you want static routes to be redistributed between autonomous systems or between
two different routing protocols, use the redistribute static router configuration
command.
For example, to redistribute static routes IGRP autonomous systems, add the following
command to the configuration:
C7010(config-router)#redistribute static
For more information on using the redistribute router configuration command, see the
Cisco IOS configuration guides and command references.
The host interprets the IP address 192.31.7.49 as being Host 1 on the third subnet (subnet
address 48). However, because it is using a different subnet mask, the router interprets the address
as being to Host 17 on the first subnet (subnet address 32). Depending on the network topology and
the router configuration, packets destined for IP address 192.31.7.49 might be sent to the wrong
destination host, sent out the wrong interface, or dropped altogether.
Make sure the proper process IDs, addresses, wildcard masks, and other variables are
properly specified.
Note: There is no correlation between OSPF wildcard masks (used in OSPF network
commands) and the subnet mask configured as part of an interface IP address.
Step 4 Check other OSPF routers on the network using the preceding steps. Make sure that
OSPF is configured properly on all neighboring routers so that neighbor relationships
can be established.
Step 3 Compare the values configured for the timers on each router. If there is a mismatch,
reconfigure the timer values so that they are the same on the router and its neighbor.
For example, to change the Hello timer interval to 10 on Ethernet interface 0/1, enter
the following commands:
C7010(config)#interface e0/1
C7010(config-if)#ip ospf hello-interval 10
Step 4 Use the debug ip ospf adj privileged EXEC command. Check the output for
mismatched values.
In the following example, there is a network mask mismatch. The mask received from
router 141.108.10.3 is 255.255.255.0, and the mask configured on the router C4500 is
255.255.255.252:
C4500#debug ip ospf adj
OSPF: Mismatched hello parameters from 141.108.10.3
Dead R 40 C 40, Hello R 10 C 10 Mask R 255.255.255.0 C 255.255.255.252
Step 5 If mismatches are indicated in the debug output, try to resolve the mismatch. For
detailed information about configuring OSPF, see the Cisco IOS Network Protocols
Configuration Guide, Part 1.
Step 6 Perform the same types of steps for all of these parameters. Check that all routers in an
area have the same area ID, whether all routers in the area are configured as stub
routers, whether the same authentication type is configured for all routers, and so
forth. For information on configuring these parameters, consult the Cisco IOS Network
Protocols Configuration Guide, Part 1.
Note: Timer values are extremely important when Cisco routers interoperate with routers from
other vendors.
Step 3 After disabling all access lists on the router, determine if the router is able to establish
neighbor relationships normally. Use the show ip ospf neighbor privileged EXEC
command. If the proper neighbor relationships have been established, an access list is
probably filtering OSPF hello packets.
Step 4 To isolate the problem access list, enable access lists one at a time until the router
cannot establish neighbors (use the clear ip ospf neighbors privileged EXEC
command to force the router to clear the neighbor table).
Step 5 Check the access list to see if it is filtering traffic from port 89, the port used by OSPF.
Remember that every access list ends with an implicit deny any statement. If an
access list denies OSPF traffic, enter an explicit permit statement for port 89 to ensure
that neighbor relationships can be established properly. (You can also use the ospf
keyword when configuring the access list.)
For example, to configure input access list 101 to allow OSPF traffic to pass, enter the
following commands on the router:
C4500(config)#access-list 101 permit ospf any any
Step 6 If you altered an access list, enable the list and enter the clear ip ospf neighbors
privileged EXEC command. Then enter the show ip ospf neighbor command to see if
neighbor relationships are established normally.
Step 7 If the router is establishing neighbors, perform the preceding steps for other routers in
the path until all access lists are enabled and the router can still establish neighbors
normally.
For more information on configuring access lists, see the Cisco IOS configuration guides.
Virtual link and stub area configuration Step 1 A virtual link cannot be configured across a stub area. Check router configurations for
mismatch routers configured both as part of a stub area and as an ABR1 that is part of a virtual
link. Use the show running-config privileged EXEC command and look for command
entries that are similar to the following:
area 2 stub
area 2 virtual-link 192.169.100.10
Step 2 If both of these commands are present, there is a misconfiguration. Remove one of the
commands (using the no form of the command) to resolve the misconfiguration.
1. ABR=area border router
This disables the no-summary keyword and keeps the router configured as a stub.
Step 3 To advertise external routes into the area, you must configure the area as a non-stub.
Make certain that all routers in the area are reconfigured as non-stub routers.
Step 4 After disabling all distribution lists, use the clear ip route privileged EXEC command
to clear the routing table.
Step 5 Determine if the routes appear in the routing table by using the show ip route
privileged EXEC command. If routes appear properly in the routing table, the access
list referenced by the distribute-list command is probably configured to deny certain
updates.
Step 6 To isolate the problem list, enable distribution lists one at a time until the routes no
longer appear in the table.
Step 7 Use the show running-config command and check the access list to make sure it does
not deny updates inappropriately. If the access list denies updates from specific
addresses, make sure that it does not deny the address of a router from which routing
updates should be received. Change the access list to allow the router to receive
updates from the proper addresses. Remember that an implicit deny any ends every
access-list.
Configure explicit permit statements for those addresses from which the router should
receive updates.
Step 8 If you altered an access list, enable the distribution list using the distribute-list
command. Use the clear ip route command and check to see if the missing routing
information appears in the routing table.
Step 9 If the routes appear in the routing table, perform the preceding steps on every router in
the path until all distribution lists are enabled and routing information appears properly
in the routing table.
For more information on configuring access lists, see the Cisco IOS configuration guides.
Interface E0 Interface E0
121.10.1.1 121.10.100.46
Interface E1 Interface E1
169.192.56.10 108.31.1.1
Step 3 If you want static routes to be redistributed, you must use the redistribute static router
configuration command.
For more information on using the redistribute router configuration command, see the
Cisco IOS configuration guides and command references.
Routes are not being redistributed between Step 1 Use the show running-config privileged EXEC command on routers that border
different routing protocols networks running different routing protocols.
Step 2 Look for a redistribute router configuration command entry. Make sure that routing
information is being properly exchanged between protocols.
For example, to redistribute routes between IGRP autonomous system 500 and
Enhanced IGRP autonomous system 200, enter the following commands:
C2509(config)#router igrp 500
C2509(config-router)#redistribute eigrp 200
C2509(config-router)#exit
C2509(config)#router eigrp 200
C2509(config-router)#redistribute igrp 500
Step 3 To redistribute static routes, you must use the redistribute static router configuration
command.
For more information on using the redistribute router configuration command, see the
Cisco IOS configuration guides and command references.
For a more detailed explanation of Enhanced IGRP Active mode, see the section “Enhanced IGRP
and Active/Passive Modes” later in this chapter.
Note Occasional messages of this type are not a cause for concern. This is how an Enhanced IGRP
router recovers if it does not receive replies to its queries from all of its neighbors. However, if these
error messages occur frequently, you should investigate the problem.
Table 5-13 outlines the problems that might cause this symptom and describes solutions to those
problems.
Step 3 After disabling all access lists on the router, determine if the missing routing
information is now appearing in routing tables.
Step 4 If the information is now appearing, it is likely that an access list is filtering traffic. To
isolate the problem access list, enable access lists one at a time until the routing
information is no longer appearing in the routing table.
Step 5 Check the access list to see if it is filtering traffic from specific TCP ports. If an access
list denies specific TCP ports, make sure that it does not deny TCP port 179, the port
used by BGP.
Enter an explicit permit statement for port 179 to ensure that BGP traffic is forwarded
normally.
Step 6 If you altered an access list, enable the list to see if routing information can still pass
normally.
Step 7 If routing information is no longer missing, perform the preceding steps on any other
routers in the path until all access lists are enabled and routing information appears in
the appropriate routing tables.
For more information on configuring access lists, see the Cisco IOS configuration guides.
where address is the IP address of the default gateway (the router local to the host).
The value 1 indicates that the specified gateway is one hop away.
You might need to reboot the host for this change to take effect.
Step 3 It is recommended that you specify a default gateway as part of the boot process.
Specify the IP address of the gateway in the following UNIX host file:
/etc/defaultrouter
This filename might be different on your UNIX system. If you are working with a PC
or a Macintosh, consult the accompanying documentation to determine how to set the
default gateway.
To configure a router as the backup hot standby router, enter the following commands:
C4500(config)#interface e0
C4500(config-if)#standby ip 192.192.192.3
Step 4 If the backup hot standby router is misconfigured and the active router fails, the
backup router might not go active.
One potential misconfiguration is a missing hot standby address in the backup router.
A router can be configured successfully as a hot standby router simply by entering the
following commands:
C4500(config)#interface e0
C4500(config-if)#standby ip
That is, you do not have to include the hot standby IP address in the standby ip
command. As long as one hot standby router has the hot standby IP address in its
configuration, every other hot standby router will learn the address from that router.
However, if only one router has the hot standby address configured, and that router
fails, other hot standby routers will not know the hot standby address and HSRP will
not work.
Be sure that at least two hot standby routers have the hot standby address in their
configuration.
No routes in active hot standby router If HSRP appears to be configured correctly, but connectivity fails, make sure that your other
routing protocols are working correctly. If your other routing protocols are not advertising
routes correctly, hot standby routers will have incomplete or empty routing tables and traffic
will not be forwarded correctly.
Follow the troubleshooting procedures outlined in this chapter to ensure that your other routing
protocols work correctly.