C9800 Module 3 Verify The Config Troubleshooting Basics and Tools

Verify the config : Troubleshooting
Basics and Tools

Module 3
Nicolas Darchis, CX Technical Leader

June 2020
Nicolas, Darchis
• Joined Cisco 13 years ago, Wireless (and

AAA) TAC forever
• Video games, singing very poorly rock
songs in my car without knowing the lyrics
except the last word of every sentence,
Star Wars
• @DarchisNicolas on Twitter. Only wifi
tweets
• Failed 4 times before passing CCIE
Wireless v1 on 5th attempt.
• Did my University thesis with Cisco, failed Countryside outside
it but still got hired Brussels, BELGIUM
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
Basic AP join and
client troubleshooting
Agenda Always-on logs and

how to get debug logs ?
Valuable
troubleshooting tools
My APs are not joining !
Did you define a wireless management interface, is it correct and did
you define the proper trustpoint for it?
myc9800-CL#show wireless interface summary
Wireless Interface Summary
Interface Name Interface Type VLAN ID IP Address IP Netmask NAT-IP Address MAC Address
--------------------------------------------------------------------------------------------------
GigabitEthernet1 Management 0 172.31.46.79 255.255.240.0 52.29.98.144 061a.4aa6.625c
myc9800-CL#show wireless management trustpoint 9800L#show wireless management trustpoint

Trustpoint Name : myc9800-CL_WLC_TP Trustpoint Name : CISCO_IDEVID_SUDI
Certificate Info : Available
Certificate Info : Available
Certificate Type : SSC
VM/Cloud Certificate Type : MIC
Certificate Hash : 58c7cf70878015224a9564d52c237fec271e0051 Private key Info : Available
Private key Info : Available FIPS suitability : Not Applicable
FIPS suitability : Not Applicable
appliance 9800L#show run | i management trustpoint
myc9800-CL#show run | i management trustpoint 9800L
wireless management trustpoint myc9800-CL_WLC_TP
myc9800-CL#wireless config vwlc-ssc key-size 3072 signature-algo sha256 password 0 <password>

My APs are not joining !
Dashboard shows you number of APs joined and number of AP trying
to join but failing (or that were joined and dropped)
Demo
later on
Equivalent to #show wireless stats ap discovery

#show wireless stats ap join summary
#show wireless stats ap history
New config model
Verifying applied configuration – WLC CLI
9800#show ap name APa80c.0dd2.1fa8 tag detail
AP Name : APa80c.0dd2.1fa8
AP Mac : a80c.0dd2.1fa8
Tag Type Tag Name

-----------------------------
Policy Tag default-policy-tag
RF Tag default-rf-tag
Site Tag default-site-tag
Policy tag mapping

------------------
WLAN Profile Name Policy Name VLAN Central Switching IPv4 ACL IPv6 ACL
----------------------------------------------------------------------------------------------------------------------
dot1x-test default-policy-profile VLAN0711 ENABLED Not Configured Not Configured
Site tag mapping

----------------
Flex Profile : default-flex-profile
AP Profile : default-ap-profile
Local-site : Yes
New config model
Verifying applied configuration – AP CLI
Verify the mappings on the AP as well !
LAB_3802#show capwap client config

AdminState : ADMIN_ENABLED(1)
Name : Lab_3802
<snip>
AP Policy Tag : ewlc_policy_tag

AP RF Tag : default-rf-tag
AP Site Tag : elwc_site_tag
New config model
Verifying applied configuration – Web UI
My clients are acting weird!
The client monitoring page is a good place to check the current state
of clients as well as if client is getting excluded
My clients are acting weird!
The client detail page has more tabs, but take the time to dig to verify
all the ACLs and policies applied to the client to validate they are
what you want
9800 Always-on Logging
• ERROR level represent abnormal situations. We want to raise the user attention to these
• WARNING represent an incident that could potentially lead to an error (or not…)
• NOTICE is the default logging level for binos daemons. It captures significant events if they are
normal working conditions. (client connect, failover)
• INFO contains details about state machines and the communication flow
DEBUG contains traces needed to root cause failure conditions
•
2-Critical
• VERBOSE : 3-Error
4-Warning
5-Notice
6-Info
7-Debug
8-Verbose
• INTERNAL is not a level but a flag on any log line when it is not meant to be understood by mere
mortals but only by developers
9800 Always-on Logging
• This means you can collect logs at notice level for any client or any
AP or any event even after the facts, and for days or weeks !
• You can also activate debugging (called RadioActive tracing) for AP
or client mac to get debug level logs.
• Here’s an example for troubleshooting an AP join :
9800 Logging
9800 RadioActive Tracing
• Add mac addresses, start debugging, generate log files when ready
9800 RadioActive Tracing
• Logs on the box persist as long as there is space, but when clicking
Generate you decide for what time period you want to compile
them
9800 Logs : CLI
There are 2 unrelated activities one can do :
• Get the logs : #show logging profile wireless (filter mac <mac>)
start last <x> minutes/hours to-file bootflash:decodedlogs.txt
• Enable radioactive trace (debug level) : #debug platform condition
feature wireless mac <mac of client or ap> + #debug platform
condition start
• Or use the macro “debug wireless mac <mac>” which will run
debugs for 30 minutes and produce the output file automatically
after that or when you “no” the command.
9800 Troubleshooting page
9800 Troubleshooting page
“show tech-
support”
“show tech-
support wireless”
# show tech wireless # show tech wireless client #show tech wireless qos
# show tech memory. # show tech wireless multicast #show tech wireless datapath
9800 Troubleshooting tools
https://developer.cisco.com/docs/wireless-troubleshooting-tools/
9800 Sniffing
• Get packets sent from or to and through the controller
• Export to Wireshark
• No need for switch capture
• Accessible either from GUI or CLI
9800 Sniffing
• Web interface to the existing EPC

CLI “monitor capture …”
• One click start/stop/download
• Physical and VLAN interfaces can

be selected
9800 Sniffing
• Although EPC can be collected easily from Web UI and most of the time options provided there are
sufficient for certain troubleshooting tasks, CLI provides more granular settings for EPC configuration.
It can be configured to match inner identity (currently mac-address only) which allows to focus on
traffic related to specific client event when CAPWAP encapsulated.
• monitor capture client_inner_mac inner mac f0c1.f10b.8ac1 interface vlan39 both control-plane
both
• monitor capture client_inner_mac match any
• monitor capture client_inner_mac start
• monitor capture client_inner_mac stop
• monitor capture client_inner_mac export bootflash:inner-mac.pcap
9800 Sniffing
Collected captures can be either uploaded to some file server in the
network or downloaded from WLC web interface directly.
9800 AP Sniffing
• When using DNAC, Intelligent Capture takes a full wireless sniffer trace on
the client serving radio without disrupting operation. It also automatically
takes capture when a anomaly is detected. This is the best way to go
• The AP Packet capture in the troubleshooting page refers to an older radio
tracing feature of IOS APs. Not the way forward
• Putting an AP in sniffer mode is the simplest way to go if you only own a
9800
• CLI : APs can take wired sniffer traces as well as control-plane trace on the
radio. https://www.cisco.com/c/en/us/support/docs/wireless/aironet-2800-
series-access-points/214560-troubleshoot-wave-2-aps.html
9800 AP Sniffing
9800 AP Sniffing
9800 AP Sniffing
• ”decode as …”
PEEKREMOTE
• Wireshark 3.4 should

decode wifi6 frames just
fine
Conclusion : troubleshooting recap
Step 1 : Health Monitoring
# show wireless stats ap join summary

# show wireless stats ap history
# show wireless stats client detail
Step 2 : Basic logging tracking
# show log
Dec 18 13:38:18.228: %LINEPROTO-5-UPDOWN: Line protocol on Interface Capwap1, changed state to down
Dec 18 13:38:18.205: %CAPWAPAC_SMGR_TRACE_MESSAGE-3-EWLC_GEN_ERR: Chassis 1 R0/0: wncd: Error in Session-
IP: 192.168.16.134[5264] Mac: 7069.5a51.46e0 Heartbeat timer expiry for AP. Close CAPWAP DTLS session
Dec 18 13:38:18.231: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP
Name: 4802paolo, MAC: 4c77.6d9e.60e4 Disjoined
Dec 21 06:19:45.425: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the
maximum limit(500)as specified by the platform.
..Dec 21 06:20:00.748: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the
.Dec 21 06:20:00.785: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the
.Dec 21 06:20:15.616: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the
Step 3 : Pull always on data for a client/AP
# show logging profile wireless filter-mac <mac> start last <minutes>

to-file <filename>
• Notice level data
• Or in Web UI Radioactive Trace page, add the mac and generate the logs immediately
without clicking start.
Step 4 : More information needed? RA Traces
# debug wireless mac aaaa.bbbb.cccc monitor-time 10
Use the Web UI for it !
Step 5 : TAC case
• RA-trace output (internal level, while we’re at it) or show logging profile wireless
of always-on output filtered for the problematic mac or timestamp
• Relevant show techs (at least show tech + show tech wireless)
• Your observations from “show logging”
• Core dump files from the web UI troubleshooting page (if the problem is a crash)
References
If you want more troubleshooting info
• BRKEWN-3013 Cisco Live Barcelona 2020
• March TAC workshop on SalesConnect
• https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-
wireless-controllers/213949-wireless-debugging-and-log-collection-on.html
• https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-
wireless-controllers/213970-catalyst-9800-wireless-controllers-commo.html

C9800 Module 3 Verify The Config Troubleshooting Basics and Tools

Uploaded by

Copyright:

Available Formats

C9800 Module 3 Verify The Config Troubleshooting Basics and Tools

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

C9800 Module 3 Verify The Config Troubleshooting Basics and Tools

Uploaded by

Copyright:

Available Formats

Verify the config : Troubleshooting

Basics and Tools

Nicolas Darchis, CX Technical Leader

• Joined Cisco 13 years ago, Wireless (and

Agenda Always-on logs and

Wireless Interface Summary

myc9800-CL#show wireless management trustpoint 9800L#show wireless management trustpoint

myc9800-CL#wireless config vwlc-ssc key-size 3072 signature-algo sha256 password 0 <password>

Equivalent to #show wireless stats ap discovery

Tag Type Tag Name

Policy tag mapping

Site tag mapping

Verify the mappings on the AP as well !

LAB_3802#show capwap client config

AP Policy Tag : ewlc_policy_tag

• Get packets sent from or to and through the controller

• No need for switch capture

• Accessible either from GUI or CLI

• Web interface to the existing EPC

• Physical and VLAN interfaces can

• Wireshark 3.4 should

# show wireless stats ap join summary

# show logging profile wireless filter-mac <mac> start last <minutes>

• Notice level data

# debug wireless mac aaaa.bbbb.cccc monitor-time 10

Use the Web UI for it !

• Your observations from “show logging”

You might also like