Brkewn 3011 PDF

Troubleshooting Wireless
LANs
Javier Contreras Albesa
Wireless Group - Escalation
BRKEWN-3011
Agenda
• Software and Support
• Troubleshooting Methodology
• Client Troubleshooting – Local
• Client Troubleshooting – Flex
• AP Troubleshooting
• Mobility
• Fast Roaming
• Q&A
3
Software and Support
4
Cisco Support Model - Expectations
 What to expect from TAC  What not to expect from TAC

 Configuration assistance  Design and deployment
 Problem analysis / bug isolation  Complete configuration
 Workarounds or fixes  Sales related information
 Action plan to resolve SR  RF Tuning
 Hardware replacement
 Engage BU when appropriate
5
Opening a TAC Service Request
 What should I have ready?

• Clear problem description
• Always: Show run-config
• If client involved, always: debug client <mac address>
• Your analysis of any data provided
• Set clear expectation of timeline and severity
6
Cisco Support Model - Escalation
 TAC Escalation Process

• Multi-Tier support resources within a technology
• TAC to engage resources (TAC/BU) when appropriate
• SR ownership might not change hands
 Customer Escalation Process

• Raise SR priority (S1/S2)
• Engage account team
• Your satisfaction is important to the Cisco TAC. If you have concerns about the
progress of your case, please contact your regional TAC Duty Manager.
7
WLC Software Trains - CCO
 ED/MD
• ED tag represents latest releases typically with latest features and HW support
• MD tag represents stable releases for mass adoption
• MD tag will be considered on CCO after AssureWave release validation, 10 weeks in
field and TAC/Escalation signoff
 Escalation builds
• Used through TAC to deliver urgent fixes before next CCO
• Supported by TAC
• “Copy” of CCO plus pointed fixes
8
WLC Software Trains - CCO
 CCO - Cisco.com release

• 8.0.121.0, 8.1.131.0, 8.2.100.0…
• Full test cycle
• Classified as ED when posted
 TAC WLC Software Recommendations:

Code recommendations based on feedback from TAC, WNBU Escalation & Advanced
Services, reviewed once a week. Requirements are field exposure & critical bug fixes.
https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-TAC-
Recommended-AireOS.html
9
Takeaways
 Have at hand:
• Show run-config
• Clear problem description
• Problem reproduction if known
 Client issues
• Debug client <mac-address>
 Crash AP
• Show tech (from the AP)
• Crash file (transfer upload…)
• Crash WLC
• Crash file and if possible , Core dump
10
Troubleshooting
Methodology
11
Where do we start?
EAP
Chan. 1
IP RADIUS ISE
driver
supp.
radio
802.11
CAPWAP
EOIP
802.11 Management
IP
IP
CAPWAP
WLC
DHCP
802.11 Management
A wireless connection is like a complex multivariable equation. So how do we

solve the equation?
 Isolate and remove the variables
12
Troubleshooting Basics
 Troubleshooting 101
• Clearly define the problem
Problem
• Understand any possible triggers Definition
• Know the expected behavior
• Reproducibility Questions
• Do not jump to conclusions
Tests
Analysis
Solution(s)
13
 Troubleshooting is an art with no right or wrong procedure, but best with a

logical methodology.
 Step 1: Define the problem
• Bad description: “Client slow to connect”
• Good description: “Client associations are rejected with Status17 several times before
they associate successfully.”
• Reduce Scope!
• Isolate multiple possible problems over same setup
14
 Step 2: Understand any possible triggers

• If something previously worked but no longer works, there should be an identifiable
trigger
• Understanding any and all configuration or environmental changes could help pinpoint a
trigger
• Finding a pattern may help with root cause isolation
 Step 3: Know the expected behavior

• If you know the order of expected behavior that is failing, defining where the behavior
breaks down (Problem Description) is better than defining the end result.
• Example: “One way audio between Phone A and B, because Phone A does not get an
ARP Response for Phone B”
15
 Step 4: Reproducibility
• Any problem that has a known procedure to reproduce (or frequently randomly occurs)
should be easier to diagnose
• Being able to easily validate or disprove a potential solution saves time by being able to
quickly move on to the next theory
• If the problem can be reproduced, it makes things much easier to work with
development, test the fix and deliver with lower impact to the end customer
• Tests will be conducted to isolate the root cause
 Step 5: Fix
• Validate Root Cause Analysis
• Develop Fix
• Test for solution, intersection
16
Useful Troubleshooting Tools
 Wireless Sniffer  Wired Packet Capture

• Example: Linksys USB600N with • Example: Wireshark
Omnipeek • Use for spanned switch ports of AP/WLC or
• TAC can publish Omnipeek - Remote client side data
Assistant if you have compatible HW
• Windows 7 with Netmon 3.4  Spectrum Analyzer
https://supportforums.cisco.com/docs/D • Spectrum Expert with Card or Clean-Air
OC-16398 AP
• Mac OS X 10.6+
https://supportforums.cisco.com/docs/D  Debug client
OC-19212
• NOTE: Wireshark running on a PC is  AP Packet Capture
not able to capture a promiscuous
Wireless trace
• AP on Sniffer mode
17
Client Troubleshooting -
Local
18
Understanding the Client State
Name Description
8021X_REQD 802.1x (L2) Authentication Pending
DHCP_REQD IP Learning State
WEBAUTH_REQD Web (L3) Authentication Pending
RUN Client Traffic Forwarding
(Cisco Controller) >show client detail 00:16:ea:b2:04:36

Client MAC Address............................... 00:16:ea:b2:04:36
…..
Policy Manager State............................. WEBAUTH_REQD
00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)
19
Steps to Building an 802.11 Connection
1. Listen for Beacons
State 1: 2. Probe Request
Unauthenticated,
Unassociated 3. Probe Response AP
4. Authentication Request
5. Authentication Response
State 2:
Authenticated, 6. Association Request
Unassociated
7. Association Response
State 3: 8. (Optional: EAPOL Authentication) WLC
Authenticated,
Associated 9. (Optional: Encrypt Data)
10. Forward User Data 20
The Client Debug
A multi-debug macro that covers over all the main client states
• (Cisco Controller) >debug client 00:16:EA:B2:04:36
• (Cisco Controller) >show debug
• MAC address ................................ 00:16:ea:b2:04:36
dhcp packet enabled
• Up to 3 addresses in 7.2 dot11 mobile enabled
• Up to 10 in 7.3 and higher dot11 state enabled
dot11 mobile enabled dot1x events enabled
dot11 state enabled dot1x states enabled
dot1x events enabled pem events enabled
pem state enabled
dot1x states enabled
CCKM client debug enabled
pem events enabled
pem state enabled
CCKM client debug enabled
21
Client Debugs
• On the WLC: debug client <MAC Address>
debug aaa events/errors enable
debug dot1x all enable
debug web-auth redirect enable mac <client mac address> (~Webauth)
debug mdns all enable (~Bonjour)
22
Client Debugs
• On the AP: debug dot11 <do0/do1> monitor addr <client mac address>
debug dot11 <d0/d1> trace print client mgmt keys rxev txev rcv xmt
debug dot11 wpa-cckm-km-dot1x
debug dot11 events
debug capwap client mgmt
23
Client Flow
The Route Toward the RUN State:
WEBAUTH_
Assoc
Associate 8021X_REQD DHCP_REQD RUN
REQD
24
Association
*apfMsConnTask_4: Dec 16 11:30:42.058: 00:1c:58:8e:a5:84 Association received from mobile on BSSID
00:3a:9a:a8:ac:d2..
Applying Local Bridging Interface Policy for station 00:1c:58:8e:a5:84 - vlan 50, interface id 14, interface 'vlan50'
processSsidIE statusCode is 0 and status is 0
processSsidIE ssid_done_flag is 0 finish_flag is 0
STA - rates (8): 130 132 139 12 18 150 24 36 0 0 0 0 0 0 0 0
suppRates statusCode is 0 and gotSuppRatesElement is 1
STA - rates (12): 130 132 139 12 18 150 24 36 48 72 96 108 0 0 0 0
extSuppRates statusCode is 0 and gotExtSuppRatesElement is 0.0.0.0 START (0) Change state to AUTHCHECK
(2) last state START (0)
0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state AUTHCHECK (2)
*apfMsConnTask_4: Dec 16 11:30:42.060: 00:1c:58:8e:a5:84 apfPemAddUser2 (apf_policy.c:333) Changing state for

mobile 00:1c:58:8e:a5:84 on AP 00:3a:9a:a8:ac:d0 from Idle to Associate
*apfMsConnTask_4: Dec 16 11:30:42.060: 00:1c:58:8e:a5:84 Sending Assoc Response to station on BSSID

00:3a:9a:a8:ac:d2 (status 0) ApVapId 3 Slot 0
*apfMsConnTask_4: Dec 16 11:30:42.060: 00:1c:58:8e:a5:84 apfProcessAssocReq (apf_80211.c:7975) Changing state for

mobile 00:1c:58:8e:a5:84 on AP 00:3a:9a:a8:ac:d0 from Associated to Associated
25
Association - Roaming
*apfMsConnTask_1: Dec 16 14:42:18.472: 00:1e:be:25:d6:ec Reassociation received from mobile on BSSID
f8:4f:57:a1:d8:a2
..
*apfMsConnTask_1: Dec 16 14:42:18.473: 00:1e:be:25:d6:ec Applying Local Bridging Interface Policy for station
00:1e:be:25:d6:ec - vlan 50, interface id 14, interface 'vlan50'
processSsidIE statusCode is 0 and status is 0
processSsidIE ssid_done_flag is 0 finish_flag is 0
STA - rates (8): 130 132 139 12 18 150 24 36 48 72 96 108 0 0 0 0
suppRates statusCode is 0 and gotSuppRatesElement is 1
STA - rates (12): 130 132 139 12 18 150 24 36 48 72 96 108 0 0 0 0
extSuppRates statusCode is 0 and gotExtSuppRatesElement is 1
*apfMsConnTask_1: Dec 16 14:42:18.473: 00:1e:be:25:d6:ec 192.168.50.100 RUN (20) Deleted mobile LWAPP rule on AP
[04:da:d2:28:94:c0]
*apfMsConnTask_1: Dec 16 14:42:18.473: 00:1e:be:25:d6:ec Updated location for station old AP 04:da:d2:28:94:c0-0,
new AP f8:4f:57:a1:d8:a0-0
26
Association – AAA filter failed
*apfMsConnTask_0: Oct 11 15:11:33.604: cc:52:af:fc:89:26 Association received from mobile on AP 00:17:0e:aa:46:30
0.0.0.0 START (0) Changing ACL 'none' (ACL ID 255) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:1626)
STA - rates (7): 22 24 36 48 72 96 108 0 0 0 0 0 0 0 0 0
Processing RSN IE type 48, length 20 for mobile cc:52:af:fc:89:26
Received RSN IE with 0 PMKIDs from mobile cc:52:af:fc:89:26
*apfMsConnTask_0: Oct 11 15:11:33.604: cc:52:af:fc:89:26 apfProcessAssocReq (apf_80211.c:5118) Changing state for
mobile cc:52:af:fc:89:26 on AP 00:17:0e:aa:46:30 from Authenticated to AAA Pending
*apfMsConnTask_0: Oct 11 15:11:33.604: cc:52:af:fc:89:26 Scheduling deletion of Mobile Station: (callerId: 20) in 10
seconds
*radiusTransportThread: Oct 11 15:11:33.610: cc:52:af:fc:89:26 Access-Reject received from RADIUS server

10.100.76.10 for mobile cc:52:af:fc:89:26 receiveId = 0
*radiusTransportThread: Oct 11 15:11:33.611: cc:52:af:fc:89:26 Returning AAA Error 'Authentication Failed' (-4) for
mobile
*apfReceiveTask: Oct 11 15:11:33.611: cc:52:af:fc:89:26 Sending Assoc Response to station on BSSID

00:17:0e:aa:46:30 (status 1) ApVapId 4 Slot 0
27
Association – CCKM failed
*apfMsConnTask_1: Mar 01 11:03:36.686: 64:00:f1:79:a9:39 Reassociation received from mobile on AP a0:cf:5b:fa:df:60
*apfMsConnTask_1: Mar 01 11:03:36.686: 64:00:f1:79:a9:39 172.25.3.179 RUN (20) Changing ACL 'none' (ACL ID 255)
===> 'none' (ACL ID 255) --- (caller apf_policy.c:1621)
*apfMsConnTask_1: Mar 01 11:03:36.686: 64:00:f1:79:a9:39 Applying site-specific IPv6 override for station
64:00:f1:79:a9:39 - vapId 1, site 'default-group', interface 'voip'
*apfMsConnTask_1: Mar 01 11:03:36.687: 64:00:f1:79:a9:39 Applying IPv6 Interface Policy for station 64:00:f1:79:a9:39
- vlan 25, interface id 11, interface 'voip'
*apfMsConnTask_1: Mar 01 11:03:36.687: 64:00:f1:79:a9:39 STA - rates (0): 152 36 48 72 96 108 0 0 0 0 0 0 0 0 0 0
*apfMsConnTask_1: Mar 01 11:03:36.687: 64:00:f1:79:a9:39 STA - rates (6): 152 36 48 72 96 108 0 0 0 0 0 0 0 0 0 0
*apfMsConnTask_1: Mar 01 11:03:36.687: 64:00:f1:79:a9:39 Processing RSN IE type 48, length 22 for mobile
64:00:f1:79:a9:39
*apfMsConnTask_1: Mar 01 11:03:36.687: 64:00:f1:79:a9:39 Received RSN IE with 0 PMKIDs from mobile
64:00:f1:79:a9:39
*apfMsConnTask_1: Mar 01 11:03:36.687: 64:00:f1:79:a9:39 CCKM: Processing REASSOC REQ IE
*apfMsConnTask_1: Mar 01 11:03:36.687: 64:00:f1:79:a9:39 CCKM: Failed to validate REASSOC REQ IE
*apfMsConnTask_1: Mar 01 11:03:36.687: 64:00:f1:79:a9:39 Sending Assoc Response to station on BSSID
a0:cf:5b:fa:df:60 (status 1) ApVapId 1 Slot 0
28
Association – Blacklisted
*apfMsConnTask_0: Dec 16 15:29:40.487: 00:40:96:b5:db:d7 Ignoring assoc request due to mobile in exclusion list or
marked for deletion
00:40:96:b5:db:d7 *apfMsConnTask_0: Dec 16 15:29:41.494: 00:40:96:b5:db:d7 Ignoring assoc request due to mobile in
exclusion list or marked for deletion
marked for deletion
marked for deletion
29
Got a Client, but no logs…
 Hey, I have a client trying to connect, but nothing is showing up!

 After 7.0: Client probing activity is aggregated, will not show up in the
logs
 Deb client will not show anything for “just probing” client
>debug dot11 probe event enable
>*apfProbeThread: Jan 03 07:59:30.738: Received aggregated probe, dataLen = 127
*apfProbeThread: Jan 03 07:59:30.738: 39:c4:eb:dd:1b:00 aggregated probe IE elmId=221, elm_len=9, dataLen=127
*apfProbeThread: Jan 03 07:59:30.738: aggregated probe IE: TIMESTAMP
*apfProbeThread: Jan 03 07:59:30.738: 00:1a:70:35:84:d6 aggregated probe IE elmId=221, elm_len=27, dataLen=116
*apfProbeThread: Jan 03 07:59:30.738: 00:1a:70:35:84:d6 aggregated probe IE: AGGR PROBE
*apfProbeThread: Jan 03 07:59:30.738: 00:1a:70:35:84:d6 probing client, ver=1, slot=0, wlan=0, snr=23, tx_pwr=0, chan=11, reg_class=0, ts_diff=346ms, seq_num=12303, ant_cnt=2, rssi[0]=214, rssi[1]=205
 Typical reasons:
• Misconfigured SSID/security settings
• IE on response not handled properly by client
30
Client Flow
WEBAUTH_
Assoc
REQD
32
PSK authentication
Probe Request AP WLC Radius
Probe Response
Auth Request
Auth Response
Association Request
Association Response
EAPoL 4 way Exchange
DATA
33
PSK – Successful
*apfMsConnTask_1: Dec 16 15:30:14.920: 00:40:96:b5:db:d7 Association received from mobile on BSSID f8:4f:57:a1:d8:aa
*apfMsConnTask_1: Dec 16 15:30:14.921: 00:40:96:b5:db:d7 Sending Assoc Response to station on BSSID f8:4f:57:a1:d8:aa (status 0)
*spamApTask3: Dec 16 15:30:14.923: 00:40:96:b5:db:d7 Sent 1x initiate message to multi thread task for mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.924: 00:40:96:b5:db:d7 Initiating RSN PSK to mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.924: 00:40:96:b5:db:d7 dot1x - moving mobile 00:40:96:b5:db:d7 into Force Auth state
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.924: 00:40:96:b5:db:d7 Starting key exchange to mobile 00:40:96:b5:db:d7, data packets will be
dropped
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.924: 00:40:96:b5:db:d7 Sending EAPOL-Key Message to mobile 00:40:96:b5:db:d7
state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.929: 00:40:96:b5:db:d7 Received EAPOL-Key from mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.929: 00:40:96:b5:db:d7 Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile
00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.929: 00:40:96:b5:db:d7 Received EAPOL-key in PTK_START state (message 2) from mobile
00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.929: 00:40:96:b5:db:d7 Stopping retransmission timer for mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.929: 00:40:96:b5:db:d7 Sending EAPOL-Key Message to mobile 00:40:96:b5:db:d7
state PTKINITNEGOTIATING (message 3), replay counter 00.00.00.00.00.00.00.01
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.934: 00:40:96:b5:db:d7 Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile
00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.934: 00:40:96:b5:db:d7 Received EAPOL-key in PTKINITNEGOTIATING state (message 4) from
mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.934: 00:40:96:b5:db:d7 Stopping retransmission timer for mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:30:14.934: 00:40:96:b5:db:d7 0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4) last state
8021X_REQD (3)
34
AP Flexconnect debugs
REAP Mgmt: FR_SM: dot11_mgmt_smact_lwapp_fast_roam(): a088.b41f.cb24 :: INIT (FAST_ROAM, 0x0) ->
PMKID_VALIDATED
REAP Mgmt: dot11_lwapp_fr_sm_before_4way_add_to_drv(): a088.b41f.cb24 : PMK Validated, before 4way

handshake adding client to driver, reap_flags 0x800
REAP Mgmt: dot11_lwapp_fr_sm_start_4way_hs():client a088.b41f.cb24 : Starting WPAv2 4way handshake
dot11_mgr_sm_send_wpav2_ptk_msg1: [1] Sent PTK msg 1 to a088.b41f.cb24, no timer set
dot11_dot1x_verify_ptk_handshake: verifying PTK msg 2 from a088.b41f.cb24
dot11_dot1x_build_ptk_handshake: building PTK msg 3 for a088.b41f.cb24
dot11_mgr_disp_client_send_eapol: sending eapol to client a088.b41f.cb24 on BSSID a89d.2131.be2f
dot11_mgr_sm_send_wpav2_ptk_msg3: [1] Sent PTK msg 3 to a088.b41f.cb24, no timer set
dot11_dot1x_verify_ptk_handshake: verifying PTK msg 4 from a088.b41f.cb24
dot11_mgr_sm_handshake_pass: Handshake pass for a088.b41f.cb24
dot11_aaa_send_response: Sending response: 2 for: a088.b41f.cb24
REAP Mgmt: FR_SM: dot11_mgmt_lwapp_aaa_resp(): a088.b41f.cb24 :: 4WAY_HS_STARTED (AAA_SUCCESS_RSP,

0x0) -> AUTHENTICATED
35
PSK – Wrong secret
*apfMsConnTask_1: Dec 16 15:25:28.923: 00:40:96:b5:db:d7 Association received from mobile on BSSID f8:4f:57:a1:d8:aa
..
*apfMsConnTask_1: Dec 16 15:25:28.925: 00:40:96:b5:db:d7 Sending Assoc Response to station on BSSID f8:4f:57:a1:d8:aa (status 0)
ApVapId 6 Slot 1
*spamApTask3: Dec 16 15:25:28.927: 00:40:96:b5:db:d7 Sent 1x initiate message to multi thread task for mobile 00:40:96:b5:db:d7
..
*Dot1x_NW_MsgTask_7: Dec 16 15:25:28.927: 00:40:96:b5:db:d7 Starting key exchange to mobile 00:40:96:b5:db:d7, data packets will
be dropped
config cl;d*Dot1x_NW_MsgTask_7: Dec 16 15:25:28.933: 00:40:96:b5:db:d7 Ignoring invalid EAPOL version (1) in EAPOL-key
message from mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:25:28.933: 00:40:96:b5:db:d7 Received EAPOL-key in PTK_START state (message 2) from mobile
00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:25:28.933: 00:40:96:b5:db:d7 Received EAPOL-key M2 with invalid MIC from mobile
00:40:96:b5:db:d7 version 2
*osapiBsnTimer: Dec 16 15:25:30.019: 00:40:96:b5:db:d7 802.1x 'timeoutEvt' Timer expired for station 00:40:96:b5:db:d7 and for
message = M2
*dot1xMsgTask: Dec 16 15:25:32.019: 00:40:96:b5:db:d7 Retransmit failure for EAPOL-Key M1 to mobile 00:40:96:b5:db:d7, retransmit
count 3, mscb deauth count 2
..
*dot1xMsgTask: Dec 16 15:25:32.020: 00:40:96:b5:db:d7 Sent Deauthenticate to mobile on BSSID f8:4f:57:a1:d8:a0 slot 1(caller
1x_ptsm.c:570)
*dot1xMsgTask: Dec 16 15:25:32.020: 00:40:96:b5:db:d7 Scheduling deletion of Mobile Station: (callerId: 57) in 10 seconds
36
PSK – Wrong secret - excluded
*dot1xMsgTask: Jan 02 11:19:56.190: 68:7f:74:75:f1:cd Blacklisting (if enabled) mobile 68:7f:74:75:f1:cd
*dot1xMsgTask: Jan 02 11:19:56.190: 68:7f:74:75:f1:cd apfBlacklistMobileStationEntry2 (apf_ms.c:5850) Changing state for mobile
68:7f:74:75:f1:cd on AP 04:da:d2:4f:f0:50 from Associated to Exclusion-list (1)
*dot1xMsgTask: Jan 02 11:19:56.190: 68:7f:74:75:f1:cd Scheduling deletion of Mobile Station: (callerId: 44) in 10 seconds
*dot1xMsgTask: Jan 02 11:19:56.190: 68:7f:74:75:f1:cd 0.0.0.0 8021X_REQD (3) Change state to START (0) last state 8021X_REQD (3)
*dot1xMsgTask: Jan 02 11:19:56.190: 68:7f:74:75:f1:cd 0.0.0.0 START (0) Reached FAILURE: from line 5274
*dot1xMsgTask: Jan 02 11:19:56.190: 68:7f:74:75:f1:cd Scheduling deletion of Mobile Station: (callerId: 9) in 10 seconds
37
802.1X Authentication
Supplicant Authenticator Server
EAPOL-START
EAP-ID-Request
EAP-ID-Response
RADIUS (EAP-ID_Response)
Rest of the EAP Conversation
Radius-Access-Accept
EAP-Success
The Supplicant Derives the

Session Key from the EAP process or
Certificate and Authentication
Exchange
39
802.1X Authentication
Probe Request AP WLC Radius
Probe Response
Auth Request
Auth Response
Association Request
EAP Start
EAP ID Request
EAP ID Response
EAP Method
Between 4 and
20+ frames EAP Success
EAPoL 4 way Exchange
DATA
40
802.1x - Successful
*apfMsConnTask_0: Dec 16 15:36:07.557: 00:40:96:b5:db:d7 Sending Assoc Response to station on BSSID 04:da:d2:28:94:ce (status 0)
ApVapId 2 Slot 1
Dot1x_NW_MsgTask_7: Dec 16 15:36:07.559: 00:40:96:b5:db:d7 dot1x - moving mobile 00:40:96:b5:db:d7 into Connecting state
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.560: 00:40:96:b5:db:d7 Sending EAP-Request/Identity to mobile 00:40:96:b5:db:d7 (EAP Id 1)
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.566: 00:40:96:b5:db:d7 Received EAPOL START from mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.566: 00:40:96:b5:db:d7 dot1x - moving mobile 00:40:96:b5:db:d7 into Connecting state
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.566: 00:40:96:b5:db:d7 Sending EAP-Request/Identity to mobile 00:40:96:b5:db:d7 (EAP Id 2)
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.567: 00:40:96:b5:db:d7 Received EAPOL EAPPKT from mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.567: 00:40:96:b5:db:d7 Received EAP Response packet with mismatching id (currentid=2,
eapid=1) from mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.569: 00:40:96:b5:db:d7 Received Identity Response (count=2) from mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.569: 00:40:96:b5:db:d7 EAP State update from Connecting to Authenticating for mobile
00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.569: 00:40:96:b5:db:d7 dot1x - moving mobile 00:40:96:b5:db:d7 into Authenticating state
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.569: 00:40:96:b5:db:d7 Entering Backend Auth Response state for mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.571: 00:40:96:b5:db:d7 Processing Access-Challenge for mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.571: 00:40:96:b5:db:d7 Entering Backend Auth Req state (id=220) for mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.571: 00:40:96:b5:db:d7 WARNING: updated EAP-Identifier 2 ===> 220 for STA
00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.571: 00:40:96:b5:db:d7 Sending EAP Request from AAA to mobile 00:40:96:b5:db:d7 (EAP Id
220)
41
802.1x - Successful
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.575: 00:40:96:b5:db:d7 Received EAP Response from mobile 00:40:96:b5:db:d7 (EAP Id
220, EAP Type 3)
..
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.718: 00:40:96:b5:db:d7 Entering Backend Auth Response state for mobile
00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.719: 00:40:96:b5:db:d7 Processing Access-Accept for mobile 00:40:96:b5:db:d7
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.719: 00:40:96:b5:db:d7 Resetting web IPv4 acl from 255 to 255
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.719: 00:40:96:b5:db:d7 Resetting web IPv4 Flex acl from 65535 to 65535
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.720: 00:40:96:b5:db:d7 Username entry (cisco) already exists in name table, length = 253
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.720: 00:40:96:b5:db:d7 Username entry (cisco) created in mscb for mobile, length = 253
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.720: 00:40:96:b5:db:d7 Setting re-auth timeout to 1800 seconds, got from WLAN config.
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.720: 00:40:96:b5:db:d7 Station 00:40:96:b5:db:d7 setting dot1x reauth timeout = 1800
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.720: 00:40:96:b5:db:d7 Creating a PKC PMKID Cache entry for station 00:40:96:b5:db:d7
(RSN 2)
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.721: 00:40:96:b5:db:d7 Sending EAP-Success to mobile 00:40:96:b5:db:d7 (EAP Id 228)
*Dot1x_NW_MsgTask_7: Dec 16 15:36:07.721: 00:40:96:b5:db:d7 Freeing AAACB from Dot1xCB as AAA auth is done for mobile
00:40:96:b5:db:d7
42
Layer 2 Authentication
AP Radio Debugs
*Jan 15 02:50:07.804: A6504097 r 1 3 - B008 2800 2FB698 6F9E11 6F9E11 CFC0 auth l 6
*Jan 15 02:50:07.807: A6504BC0 t 1 69/67 14- B008 13A 6F9E11 2FB698 6F9E11 65C0 auth l 6
*Jan 15 02:50:07.809: A6505313 r 1 69/67 19- 0000 13A 6F9E11 2FB698 6F9E11 65D0 assreq l 139
*Jan 15 02:50:07.827: A6509A92 t 1 2 - 1008 000 2FB698 6F9E11 6F9E11 CFE0 assrsp l 151
*Jan 15 02:50:07.829: A650A056 t 1 0 - 8802 000 2FB698 6F9E11 6F9E11 0290 q7 l87
EAPOL3 EAP id 93 req ident 0 "networkid=peapradius,nasid=SURBG-5508,portid=0"
*Jan 15 02:50:07.879: A6516524 r 1 68/67 19- 8801 13A 6F9E11 2FB698 6F9E11 0010 q7 l22
EAP id 93 resp ident "surbg"
Rest of the EAP Transaction
|
*Jan 15 02:50:08.247: A6570622 t 1 0 - 8802 000 2FB698 6F9E11 6F9E11 0330 q7 l54
EAPOL3 EAP id 93 success
43
Client Flow
WEBAUTH_
Assoc
REQD
45
Client DHCP
Client State = “DHCP_REQD“
Client is in DHCP_REQD state
DHCP Proxy Enabled DHCP Proxy Disabled
• Proxy Enabled:
DHCP Relay/Proxy
Client DHCP Discover Client DHCP Discover Is
Between WLC and Server Unicast to DHCP Servers Bridged to DS
Required for Internal DHCP
• Proxy Disabled: DHCP Offer from Server
Between Client and Server

Client DHCP Request
DHCP is forwarded as a broadcast on VLAN
IP helper or other means required Address
DHCP ACK from Server Learned!
IP Address Learned
46
Client DHCP
*apfReceiveTask: Jan 02 10:45:27.476: 68:7f:74:75:f1:cd 0.0.0.0 DHCP_REQD (7) State Update from Mobility-Incomplete to
Mobility-Complete, mobility role=Local, client state=APF_MS_STATE_ASSOCIATED
*apfReceiveTask: Jan 02 10:45:27.476: 68:7f:74:75:f1:cd 0.0.0.0 DHCP_REQD (7) pemAdvanceState2 5752, Adding TMP rule
*apfReceiveTask: Jan 02 10:45:27.476: 68:7f:74:75:f1:cd 0.0.0.0 DHCP_REQD (7) Adding Fast Path rule
type = Airespace AP - Learn IP address
on AP 04:da:d2:4f:f0:50, slot 0, interface = 1, QOS = 0
IPv4 ACL ID = 255, IPv
*apfReceiveTask: Jan 02 10:45:27.476: 68:7f:74:75:f1:cd 0.0.0.0 DHCP_REQD (7) Fast Path rule (contd...) 802.1P = 0, DSCP = 0,
TokenID = 15206 Local Bridging Vlan = 50, Local Bridging intf id = 12
*apfReceiveTask: Jan 02 10:45:27.476: 68:7f:74:75:f1:cd 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (IPv4 ACL
ID 255, IPv6 ACL ID 255, L2 ACL ID 255)
*pemReceiveTask: Jan 02 10:45:27.476: 68:7f:74:75:f1:cd 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0
47
Client DHCP – Process Start
DHCP received op BOOTREQUEST (1) (len 308,vlan 5, port 1, encap 0xec03)

DHCP (encap type 0xec03) mstype 0ff:ff:ff:ff:ff:ff
DHCP selected relay 1 - 192.168.50.1 (local address 192.168.50.15, gateway 192.168.50.1, VLAN 50, port 1)
DHCP transmitting DHCP DISCOVER (1)
DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 1
DHCP xid: 0xa504e3 (10814691), secs: 0, flags: 0
DHCP chaddr: 68:7f:74:75:f1:cd
DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0
DHCP siaddr: 0.0.0.0, giaddr: 192.168.50.15
DHCP sending REQUEST to 192.168.50.1 (len 350, port 1, vlan 50)
48
Client DHCP – Offer
DHCP received op BOOTREPLY (2) (len 308,vlan 50, port 1, encap 0xec00)
DHCP setting server from OFFER (server 192.168.0.21, yiaddr 192.168.50.101)
DHCP sending REPLY to STA (len 418, port 1, vlan 5)
DHCP transmitting DHCP OFFER (2)
DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0
DHCP server id: 1.1.1.1 rcvd server id: 192.168.0.21
DHCP received op BOOTREQUEST (1) (len 335,vlan 5, port 1, encap 0xec03)
DHCP (encap type 0xec03) mstype 0ff:ff:ff:ff:ff:ff
49
DHCP – Request - ACK
DHCP selected relay 1 - 192.168.0.21 (local address 192.168.50.15, gateway 192.168.50.1, VLAN 50, port 1)
DHCP transmitting DHCP REQUEST (3)
DHCP requested ip: 192.168.50.101
192.168.50.101 DHCP_REQD (7) Change state to WEBAUTH_REQD (8) last state DHCP_REQD (7)
192.168.50.101 WEBAUTH_REQD (8) pemAdvanceState2 6662, Adding TMP rule

192.168.50.101 WEBAUTH_REQD (8) Replacing Fast Path rule
type = Airespace AP Client - ACL passthru
on AP 04:da:d2:4f:f0:50, slot 0, interface = 1, QOS = 0
IPv4 A
Plumbing web-auth redirect rule due to user logout
Assigning Address 192.168.50.101 to mobile
50
DHCP – Rejected
DHCP transmitting DHCP REQUEST (3)
DHCP xid: 0xf3a2fca6 (4087544998), secs: 3, flags: 0
DHCP chaddr: d0:b3:3f:33:1c:88
DHCP requested ip: 10.65.8.177
DHCP selecting relay 2 - control block settings:
dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0,
dhcpGateway: 0.0.0.0, dhcpRelay: 10.87.193.2 VLAN: 703
DHCP selected relay 2 - NONE
DHCP sending REPLY to STA (len 402, port 1, vlan 701)
DHCP transmitting DHCP NAK (6)
DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0
DHCP xid: 0xf3a2fca6 (4087544998), secs: 0, flags: 8000
DHCP chaddr: d0:b3:3f:33:1c:88
51
Learning IP without DHCP
*Orphan Packet from 10.99.76.147 on mobile
*0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)
*Installing Orphan Pkt IP address 10.99.76.147 for station
*10.99.76.147 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)
 Multiple mechanisms to learn Client IP address:

• Mobility
• ARP/GARP from client
• Traffic from/to client
• DHCP
 Non-DHCP: Seen with mobile devices that attempt to send data before
validating DHCP
 Up to client to realize their address is not valid for the subnet
 DHCP Required enabled on WLAN mitigates this client behavior
52
DHCP Required - Caveats
DTL-1-ARP_POISON_DETECTED: STA [00:0b:7d:0e:33:33, 0.0.0.0] ARP (op 1) received with invalid SPA 192.168.1.152/TPA
192.168.0.206
 Modifies Address learning

• Limits to only DHCP and mobility
 Good for security
 It can cause problems if client is deleted
• On new association client must do DHCP renew
• Client may hold until DHCP half-lease time
53
Client Flow
WEBAUTH_
Assoc
REQD
55
Webauth- Walkthrough
Client Controller Radius Captive Portal
Association
1
2
DHCP
3
HTTP Request
4
Redirection
To Captive portal: ap_mac, switch_url (controller auth url), redirect(original url),

statusCode (result code from wlc), wlan (SSID user is connected), user_mac
HTTP Request
5
56
Webauth- Walkthrough
Client Controller Radius Captive Portal
Final Customized page to client
Sign up page (web user)

URL points to controller
Username/password pre-populated with expected values
User Form Submit

6
Authenticate
7
Username/pass
Access Accept
Redirection URL
8 Redirect to splash page
Portal/Network Access
57
Webauth Redirect
*pemReceiveTask: Jan 02 10:45:30.824: 68:7f:74:75:f1:cd 192.168.50.101 Added NPU entry of type 2, dtlFlags 0x0
captive-bypass detection disabled, Not checking for wispr in HTTP GET, client mac=68:7f:74:75:f1:cd
Preparing redirect URL according to configured Web-Auth type
Checking custom-web config for WLAN ID:2
unable to get the hostName for virtual IP, using virtual IP =1.1.1.1
Global status is enabled, checking on web-auth type
Web-auth type Internal, no further redirection needed. Presenting default login page to user
http_response_msg_body1 is <HTML><HEAD><TITLE> Web Authentication Redirect</TITLE><META http-equiv="Cache-
control" content="no-cache"><META http-equiv="Pragma" content=“
http_response_msg_body2 is "></HEAD></HTML>
parser host is 192.168.0.45

- parser path is /
added redirect=, URL is now https://1.1.1.1/login.html?
str1 is now https://1.1.1.1/login.html?redirect=192.168.0.45/
clen string is Content-Length: 302
Message to be sent is
HTTP/1.1 200 OK
Location: https://1.1.1.1/login.html?redirect=192.168.0.45/
Content-Type: text/html
Content-Length: 302
<HTML><HEAD><TITLE>
send data length=428
58
Webauth Redirect – IPv6
webauthRedirect: Jan 02 14:57:23.734: 28:37:37:7f:5c:7- str1 is now
https://[::FFFF:1.1.1.1]/login.html?redirect=www.apple.com/library/test/success.html
*webauthRedirect: Jan 02 14:57:23.734: 28:37:37:7f:5c:7- clen string is Content-Length: 337
*webauthRedirect: Jan 02 14:57:23.734: 28:37:37:7f:5c:7- Message to be sent is

HTTP/1.1 200 OK
Location: https://[::FFFF:1.1.1.1]/login.html?redirect=www.apple.com/library/test/success.html
Content-Type: text/html
Content-L
*webauthRedirect: Jan 02 14:57:23.734: 28:37:37:7f:5c:7- send data length=498
59
Webauth Success
*emWeb: Jan 02 10:46:42.904:

ewaURLHook: Entering:url=/login.html, virtIp = 1.1.1.1, ssl_connection=1, secureweb=1
*ewmwebWebauth1: Jan 02 10:46:42.905: 68:7f:74:75:f1:cd Username entry (cisco) created for mobile, length = 5
*ewmwebWebauth1: Jan 02 10:46:42.905: 68:7f:74:75:f1:cd Username entry (cisco) created in mscb for mobile,
length = 5
*ewmwebWebauth1: Jan 02 10:46:42.906: 68:7f:74:75:f1:cd 192.168.50.101 WEBAUTH_REQD (8) Change state to
WEBAUTH_NOL3SEC (14) last state WEBAUTH_REQD (8)
*ewmwebWebauth1: Jan 02 10:46:42.906: 68:7f:74:75:f1:cd apfMsRunStateInc

*ewmwebWebauth1: Jan 02 10:46:42.906: 68:7f:74:75:f1:cd 192.168.50.101 WEBAUTH_NOL3SEC (14) Change state to
RUN (20) last state WEBAUTH_NOL3SEC (14)
*ewmwebWebauth1: Jan 02 10:46:42.906: 68:7f:74:75:f1:cd Session Timeout is 1800 - starting session timer for
the mobile
*ewmwebWebauth1: Jan 02 10:46:42.906: 68:7f:74:75:f1:cd 192.168.50.101 RUN (20) Reached PLUMBFASTPATH: from
line 6550
*ewmwebWebauth1: Jan 02 10:46:42.906: 68:7f:74:75:f1:cd 192.168.50.101 RUN (20) Replacing Fast Path rule
60
Webauth Typical problems
 No DNS resolution
 No default GW
 Client doing request on different port

• No HTTPS, or using 8000, etc.
61
No Preauth-ACL for External Webauth
• Server IP must be allowed on the preauth ACL… otherwise, we get a loop!
*webauthRedirect: Jan 02 12:27:08.254: 68:7f:74:75:f1:cd- Web-auth type External, using
URL:http://192.168.0.21/login.htm
..
*webauthRedirect: Jan 02 12:27:08.255: 68:7f:74:75:f1:cd- parser host is 192.168.0.21
*webauthRedirect: Jan 02 12:27:08.255: 68:7f:74:75:f1:cd- parser path is /
*webauthRedirect: Jan 02 12:27:08.255: 68:7f:74:75:f1:cd- added redirect=, URL is now
http://192.168.0.21/login.htm?switch_url=https://1.1.1.1/login.html&ap_mac=04:da:d2:4f:f0:50&client_mac=68:7f:
74:75:f1:cd&wlan=webauth&
NEXT:
*webauthRedirect: Jan 02 12:27:08.332: 68:7f:74:75:f1:cd- parser host is 192.168.0.21
*webauthRedirect: Jan 02 12:27:08.255: 68:7f:74:75:f1:cd- parser path is /
*webauthRedirect: Jan 02 12:27:08.332: 68:7f:74:75:f1:cd- added redirect=, URL is now
…
*webauthRedirect: Jan 02 12:27:08.332: 68:7f:74:75:f1:cd- str1 is now
http://192.168.0.21/login.htm?switch_url=https://1.1.1.1/login.html&ap_mac=04:da:d2:4f:f0:50&client_mac=68:7f:
74:75:f1:cd&wlan=webauth&redirect=192.168.0.21/
62
Untrusted Cert
• Specially important when using ISE or any other external web server
• Depending on client type/version:
• External server not displayed
• Authentication form not posted -> wlc sends internal page
• Nothing is sent -> “client hangs”
Session Timeout too low

• Users may need to reauthenticate often
63
Webauth Take away
 If using external webauth

• Certificate trust is critical (both WLC and external server). If suspected test with https
disabled
• Preauth ACL
 ARP/DNS must work before you can do anything

 Additional debug needed
• debug web-auth redirect enable mac <mac addr>
 Client side capture/logs may be needed
64
Client Flow
WEBAUTH_
Assoc
REQD
65
RUN status
 RUN means: client has completed all required policy states

 “NPU entry of Type 1” is the goal
*dot1xMsgTask: Nov 05 14:35:11.838: 2c:54:2d:ea:e7:aa 10.253.42.45 RUN (20) Reached PLUMBFASTPATH: from line
6076Nov 5 *dot1xMsgTask: Nov 05 14:35:11.838: 2c:54:2d:ea:e7:aa 10.253.42.45 RUN (20) Adding Fast Path rule
*dot1xMsgTask: Nov 05 14:35:11.838: 2c:54:2d:ea:e7:aa 10.253.42.45 RUN (20) Fast Path rule (contd...) 802.1P = 5,
DSCP = 0, TokenID = 15206 Local Bridging Vlan = 101, Local Bridging intf id = 18
*dot1xMsgTask: Nov 05 14:35:11.841: 2c:54:2d:ea:e7:aa 10.253.42.45 RUN (20) Successfully plumbed mobile rule
(IPv4 ACL ID 255, IPv6 ACL ID 255)Nov 5 14:35:13 btwlc01 BTWLC01 *pemReceiveTask:
Nov 05 14:35:11.842: 2c:54:2d:ea:e7:aa 10.253.42.45 Added NPU entry of type 1, dtlFlags 0x0
66
RUN status - Typical Problems
 Random Disconnections – Radio Reset
• There are normal radio resets: Channel changes, etc
emWeb: Jan 03 08:56:14.809: 00:1a:70:35:84:d6 Cleaning up state for STA 00:1a:70:35:84:d6 due to
event for AP 04:da:d2:4f:f0:50(0)
*apfReceiveTask: Jan 03 08:56:14.810: 00:1a:70:35:84:d6 Scheduling deletion of Mobile Station:
(callerId: 45) in 10 seconds
• Watch out for anomalous reset counts in short uptime

>sh cont d0 | b Reset
Last radio reset code: 62
Radio resets - total:113 retries:0 failed:0
Reset Stats: Start Cnt: 94, Recovery: Cnt 0, Last Ret: 0, Fails: 0, Recvry Status: Stalled NO, In Prog
NO
Code/Count: 37/00010 84D7 51/00021 F25E 52/00012 F25E 54/00002 84D6
Code/Count: 62/00067 F25F 67/00001 0
67
Environmental trigger
• Typical high channel utilization
ap2600-sw1-0-31#sh cont d0 | b QBS

QBSS Load: 0xFE Tx 0 Rx 0 AP 0
*Nov 21 10:59:06.244: %DOT11-3-NO_BEACONING: Error on Dot11Radio0 - Not Beaconing for too long -
Current 2887074 Last 2887074*Nov 21 10:59:06.274: %LINK-5-CHANGED: Interface Dot11Radio0, changed
state to reset
*Nov 21 10:59:07.693: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to
down
*Nov 21 10:59:08.485: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Nov 21 10:59:09.485: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
68
Poor Performance
• RF issues
• Client side bugs
69
RUN status - RF Analysis WLCCA
WLCCA
• New tool for quick RF analysis
• RF Health - > simplified quick view on RF, per Band, AP, AP Group, Flex Group
70
71
72
Deauthenticated Client
 Idle Timeout
Occurs after no traffic received from Client at AP
Default Duration is 300 seconds
Received Idle-Timeout from AP 00:26:cb:94:44:c0, slot 0 for STA 00:1e:8c:0f:a4:57
apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 4, reasonCode 4
Scheduling deletion of Mobile Station: (callerId: 30) in 1 seconds
apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!
Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)
 Session Timeout
Occurs at scheduled duration (default 1800 seconds)
apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on
AP 00:26:cb:94:44:c0 from Associated to Disassociated
73
 WLAN Change
• Modifying a WLAN in anyway Disables and Re-enables WLAN
apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile
00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated
Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)
 Manual Deauthentication
• From GUI: Remove Client
• From CLI: config client deauthenticate <mac address>
apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 6, reasonCode 1
apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on
AP 00:26:cb:94:44:c0 from Associated to Disassociated
74
 Authentication Timeout
Auth or Key Exchange max-retransmissions reached
Retransmit failure for EAPOL-Key M3 to mobile 00:1e:8c:0f:a4:57, retransmit count 3, mscb deauth
count 0
Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller 1x_ptsm.c:534)
 AP Radio Reset (Power/Channel)

AP disasassociates clients but WLC does not delete entry
Cleaning up state for STA 00:1e:8c:0f:a4:57 due to event for AP 00:26:cb:94:44:c0(0)
apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile
00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated
Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)
75
 Failed Broadcast key rotation
*dot1xMsgTask: Oct 22 15:32:49.863: 24:77:03:c2:8a:20 Key exchange done, data packets from mobile
24:77:03:c2:8a:20 should be forwarded shortly
*dot1xMsgTask: Oct 22 15:32:49.863: 24:77:03:c2:8a:20 Sending EAPOL-Key Message to mobile
24:77:03:c2:8a:20
*osapiBsnTimer: Oct 22 15:32:51.056: 24:77:03:c2:8a:20 802.1x 'timeoutEvt' Timer expired for station
24:77:03:c2:8a:20 and for message = M5*dot1xMsgTask: Oct 22 15:32:51.056: 24:77:03:c2:8a:20
Retransmit 1 of EAPOL-Key M5 (length 131) for mobile 24:77:03:c2:8a:20*osapiBsnTimer: Oct 22
..
*dot1xMsgTask: Oct 22 15:32:53.056: 24:77:03:c2:8a:20 Retransmit failure for EAPOL-Key M5 to mobile
24:77:03:c2:8a:20, retransmit count 3, mscb deauth count 0
*dot1xMsgTask: Oct 22 15:32:53.056: 24:77:03:c2:8a:20 Sent Deauthenticate to mobile on BSSID

20:3a:07:e4:c8:f0 slot 0(caller 1x_ptsm.c:570)
76
Client Issues - Takeaway
 Client can be removed for numerous reasons

 WLAN change, AP change, configured interval
 Start with Client Debug to see if there is a reason for a client’s deauthentication
 Further Troubleshooting
 Client debug should give some indication of what kind of deauth is happening
 Packet capture or client logs may be required to see the exact reason
 Never forget Radio status and RF conditions
77
Client Troubleshooting -
Flex
78
Client Debugs
• On the FlexConnect AP: debug capwap flex
debug capwap client config
debug capwap flexconnect mgmt
debug capwap flexconnect pmk
debug capwap flexconnect cckm
debug capwap flexconnect dot11r
79
Debugs on Flex APs – AireOS 8.1 new features
 Client based debugging exist on WLC but such ability was lacking on the AP side.
 Lack of filtering capabilities on debugging information in AP
Specific Flexconnect AP :
debug flexconnect client ap <ap-name> add/delete <addr1> {<addr2> <addr3> <addr4>}

debug flexconnect client ap <ap-name> syslog <server-ip/disable>
For a Flex group :
debug flexconnect client group <group-name>add/delete <addr1> { <addr2> | <addr3> | <addr4>}

debug flexconnect client group <group-name> syslog <serverip/disable>
80
Debugs on Flex Aps – Example
(5500-4-82) >debug flexconnect client group ciscolive add 00:40:96:b5:db:d7
Warning! Flex group client debugs will not be enabled on AP where AP specific client debugs are
already enabled.
(5500-4-82) >show deb
MAC debugging .............................. disabled
Debug Flags Enabled:
Flex-AP Client Debugging ................... disabled
Flex-Group Client Debugging ................ enabled
Group Name Syslog IP Address Mac Addresses
------------------------- ----------------------------- -----------------
ciscolive 0.0.0.0 00:40:96:b5:db:d7
81
Client debugs on Flex – 802.11 Auth
*Jan 26 15:27:28.263: (0040.96b5.dbd7): CAPWAP: Central auth client, Not sending delete mobile to
controller
*Jan 26 15:27:28.263: (0040.96b5.dbd7): dot11_mgmt:Setting AID 0 for station
*Jan 26 15:27:28.263: (0040.96b5.dbd7): Reap_Mgmt: Open Auth Client, auth_type 0, auth_algorithm 0

fsm_type 4
*Jan 26 15:27:28.263: (0040.96b5.dbd7): Reap_Mgmt; Created MN , wlan 2, association in progress
*Jan 26 15:27:28.263: (0040.96b5.dbd7): SM: ---REAP Open Authentication 0x6E7888C: AuthReq (0)SM:
Assoc (2) --> DONT CHANGE STATE (255)
*Jan 26 15:27:28.263: (0040.96b5.dbd7): dot11_mgmt:FR_SM: dot11_mgmt_smact_lwapp_reauth_req():

0040.96b5.dbd7 :: INIT (STOP_FR, 0x0) -> INIT
82
Client debugs on Flex – 802.11 Association
(0040.96b5.dbd7): dot11_driver: Dot11Radio1: Rx AssocReq for the client
(0040.96b5.dbd7): SM: ---REAP Open Authentication 0x6E7888C: AssocReq (1)SM: Assoc (2) --> DONT
CHANGE STATE (255)
(0040.96b5.dbd7): dot11_mgmt: found a valid rsnie with key_mgmt FAC02 and encrypt_type 512
(0040.96b5.dbd7): dot11_mgmt: WLAN MFP=1 WPA2=yes
(0040.96b5.dbd7): dot11_mgmt: Allocated AID 1 for client
(0040.96b5.dbd7): dot11_mgmt: [7E818AA6] send assoc resp, status[0] to dst=0040.96b5.dbd7, aid[1] on

Dot11Radio1
(0040.96b5.dbd7): dot11_driver: Dot11Radio1: Tx AssocResp to client 0040.96b5.dbd7
(0040.96b5.dbd7): dot11_mgmt: forwarding assoc req to controller
(0040.96b5.dbd7): dot11_mgmt: start assoc timer
(0040.96b5.dbd7): Reap_Mgmt: Stopping assoc timer
83
Client debugs on Flex – 802.11 First Capwap Add
(0040.96b5.dbd7): Reap_Mgmt: Assoc Resp for this station accepted by controller
(0040.96b5.dbd7): capwap: MN FSM cur state = CAPWAP_MN_ST_ADDED, evt = CAPWAP_MN_EV_ASSOC_RSP
(0040.96b5.dbd7): capwap: MN FSM new state = CAPWAP_MN_ST_ADDED
(0040.96b5.dbd7): CAPWAP_ADD_MN: Reap Flags: 0x0, FlexAclName: , webAclName:
(0040.96b5.dbd7): CAPWAP_ADD_MN: slot 1, wlan 2, vlanId -1, AID 1, encrypt policy 0x1 encrypt_type
0x0000, parent 0000.0000.0000
(0040.96b5.dbd7): CAPWAP_ADD_MN: GatewayIp = 0.0.0.0 GateWay Mask 0.0.0.0 Client IP 4.0.16.11 IP-
learn-type = 3
(0040.96b5.dbd7): CAPWAP_ADD_MN: Other Flags 0x0, session_timeout 0, Local Switch 0
(0040.96b5.dbd7): CAPWAP_ADD_MN: It is a Non WGB Client
(0040.96b5.dbd7): capwap: MN FSM cur state = CAPWAP_MN_ST_ADDED, evt = CAPWAP_MN_EV_ADD
(0040.96b5.dbd7): capwap: MN FSM cur state = CAPWAP_MN_ST_ADDED, evt = CAPWAP_MN_EV_ADD_KEY

84
Client debugs on Flex – 802.11 2nd Capwap Add
(0040.96b5.dbd7): capwap: MN FSM cur state = CAPWAP_MN_ST_ADDED, evt = CAPWAP_MN_EV_ADD
(0040.96b5.dbd7): capwap: MN FSM cur state = CAPWAP_MN_ST_ADDED, evt = CAPWAP_MN_EV_ADD_KEY
(0040.96b5.dbd7): capwap: Not a fast-roaming client, plumbing keys with encrypt policy 4, encrypt
type 0x200
(0040.96b5.dbd7): Dot11_Driver: setting client key with encrypt type 0x200
85
Client debugs on Flex – Address Learning
(0040.96b5.dbd7): capwap: MN FSM cur state = CAPWAP_MN_ST_ADDED, evt = CAPWAP_MN_EV_ADD_DONE
(0040.96b5.dbd7): DHCP: 'BOOT REPLY' message type: DHCP_OFFER, MAC da: 0000.0004.0200, MAC sa:
1009.8020.1900, IP da: 192.168.5.250, IP sa: 1.1.1.1, DHCP ciaddr: 0.0.0.0, DHCP yiaddr:
192.168.5.250, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 0040.96b5.dbd7
*Jan 26 15:27:34.783: (0040.96b5.dbd7): DHCP: 'BOOT REPLY' message type: DHCP_ACK, MAC da:
0100.0000.c562, MAC sa: 1d00.c562.1500, IP da: 192.168.5.250, IP sa: 1.1.1.1, DHCP ciaddr: 0.0.0.0,
DHCP yiaddr: 192.168.5.250, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 0040.96b5.dbd7
86
Client debugs on Flex – Authentication
*Jan 26 16:09:01.527: (0040.96b5.dbd7): dot11_aaa: Received EAPOL packet from client
*Jan 26 16:09:01.527: (0040.96b5.dbd7): dot11_aaa: eapol ver 1 type 3 posting event 0x9
*Jan 26 16:09:01.527: (0040.96b5.dbd7): DOT1X_SM: Executing Action [state: WPAV2_PTK_MSG2_WAIT, event:

RECV_EAPOL_KEY_RSP] for client
*Jan 26 16:09:01.527: (0040.96b5.dbd7): dot11_dot1x: Received wpav2 ptk msg2
*Jan 26 16:09:01.527: (0040.96b5.dbd7): dot11_dot1x: verifying PTK msg 2 from client
..
*Jan 26 16:09:01.527: (0040.96b5.dbd7): dot11_dot1x: Starting wpav2 ptk msg 3 to supplicant
*..
*Jan 26 16:09:01.527: (0040.96b5.dbd7): dot11_dot1x: mcst_key_len 16 index 1 vlan 0
87
AP Troubleshooting
88
AP Join Process
 WLC Discovery
 DTLS/Join
 Image Download
 Configuration Check
 REG
More information:
http://www.cisco.com/c/en/us/support/docs/wireless-mobility/wireless-lan-wlan/70333-lap-registration.html
89
L3 WLC Discovery
Discover
AP tries to send discover messages to all the WLC addresses

that its hunting process has turned up
90
AP Discover/Join
 AP Discovery Request sent to known  Configured (nvram)

and learned WLCs • High Availability WLCs –
Pri/Sec/Ter/Backup
 Broadcast
• Last WLC
• Reaches WLCs with MGMT Interface in
• All WLCs in same mobility group as
local subnet of AP
last WLC
• Use “ip helper-address <ip>” with “ip
• Manual from AP - “capwap ap
forward-protocol udp 5246”
controller ip address <ip>”
 Dynamic
• DNS: cisco-capwap-controller
• DHCP: Option 43
91
AP Discover/Join – AP Side
*Jan 2 15:41:42.035: %CAPWAP-3-EVENTLOG: Starting Discovery. Initializing discovery latency in

discovery responses.
*Jan 2 15:41:42.035: %CAPWAP-3-EVENTLOG: CAPWAP State: Discovery.
*Jan 2 15:41:42.035: CAPWAP Control mesg Sent to 192.168.70.10, Port 5246
*Jan 2 15:41:42.039: Msg Type : CAPWAP_DISCOVERY_REQUEST
*Jan 2 15:41:42.039: CAPWAP Control mesg Recd from 192.168.5.54, Port 5246
*Jan 2 15:41:42.039: HLEN 2, Radio ID 0, WBID 1
*Jan 2 15:41:42.039: Msg Type : CAPWAP_DISCOVERY_RESPONSE
*Jan 2 15:41:42.055: CAPWAP Control mesg Recd from 192.168.5.55, Port 5246
*Jan 2 15:41:42.055: HLEN 2, Radio ID 0, WBID 1
*Jan 2 15:41:42.055: Msg Type : CAPWAP_DISCOVERY_RESPONSE
92
AP Discover/Join – AP Side
*Jan 2 15:41:52.039: %CAPWAP-3-EVENTLOG: Calling wtpGetAcToJoin from timer expiry.

*Jan 2 15:41:52.039: %CAPWAP-3-ERRORLOG: Selected MWAR '5500-5'(index 0).
*Jan 2 15:41:52.039: %CAPWAP-3-EVENTLOG: Selected MWAR '5500-5' (index 2).
*Jan 2 15:41:52.039: %CAPWAP-3-EVENTLOG: Ap mgr count=1
*Jan 2 15:41:52.039: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Jan 2 15:41:52.039: %CAPWAP-3-EVENTLOG: Adding Ipv4 AP manager 192.168.5.55 to least load
*Jan 2 15:41:52.039: %CAPWAP-3-EVENTLOG: Choosing AP Mgr with index 0, IP = 192.168.5.55, load =
3..
*Jan 2 15:41:52.039: %CAPWAP-3-EVENTLOG: Synchronizing time with AC time.
*Jan 2 15:41:52.000: %CAPWAP-3-EVENTLOG: Setting time to 15:41:52 UTC Jan 2 2014
*Jan 2 15:41:52.467: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip:

192.168.5.55 peer_port: 5246
93
AP Discover/Join – WLC Side
*spamApTask7: Jan 02 15:35:57.295: 04:da:d2:4f:f0:50 Discovery Request from 192.168.5.156:7411
*spamApTask7: Jan 02 15:35:57.296: 04:da:d2:4f:f0:50 ApModel: AIR-CAP2602I-E-K9
*spamApTask7: Jan 02 15:35:57.296: apModel: AIR-CAP2602I-E-K9
*spamApTask7: Jan 02 15:35:57.296: apType = 27 apModel: AIR-CAP2602I-E-K9
*spamApTask7: Jan 02 15:35:57.296: apType: Ox1b bundleApImageVer: 7.6.100.0
*spamApTask7: Jan 02 15:35:57.296: version:7 release:6 maint:100 build:0
*spamApTask7: Jan 02 15:35:57.296: 04:da:d2:4f:f0:50 Discovery Response sent to 192.168.5.156 port
7411
*spamApTask7: Jan 02 15:36:07.762: 44:03:a7:f1:cf:1c DTLS keys for Control Plane are plumbed
successfully for AP 192.168.5.156. Index 7
*spamApTask6: Jan 02 15:36:07.762: 44:03:a7:f1:cf:1c DTLS Session established server
(192.168.5.55:5246), client (192.168.5.156:7411)
*spamApTask6: Jan 02 15:36:07.762: 44:03:a7:f1:cf:1c Starting wait join timer for AP:
192.168.5.156:7411
*spamApTask7: Jan 02 15:36:07.764: 04:da:d2:4f:f0:50 Join Request from 192.168.5.156:7411
*spamApTask7: Jan 02 15:36:07.765: 04:da:d2:4f:f0:50 Join resp: CAPWAP Maximum Msg element len = 83
*spamApTask7: Jan 02 15:36:07.765: 04:da:d2:4f:f0:50 Join Response sent to 192.168.5.156:7411
*spamApTask7: Jan 02 15:36:07.765: 04:da:d2:4f:f0:50 CAPWAP State: Join
94
AP Join – Country Mismatch - AP
*Jan 3 07:48:36.603: %CAPWAP-3-ERRORLOG: Selected MWAR '5500-4'(index 0).

*Jan 3 07:48:36.603: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Jan 3 07:48:37.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.5.54
peer_port: 5246
*Jan 3 07:48:37.467: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip:
192.168.5.54 peer_port: 5246
*Jan 3 07:48:37.467: %CAPWAP-5-SENDJOIN: sending Join Request to 192.168.5.54
*Jan 3 07:48:37.467: %CAPWAP-3-ERRORLOG: Invalid event 10 & state 5 combination.
*Jan 3 07:48:37.467: %CAPWAP-3-ERRORLOG: CAPWAP SM handler: Failed to process message type 10 state
5.
*Jan 3 07:48:37.467: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller
*Jan 3 07:48:37.467: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from
192.168.5.54
95
AP Join – Country Mismatch - WLC
*spamApTask5: Jan 03 07:49:16.571: #CAPWAP-3-POST_DECODE_ERR: capwap_ac_sm.c:5660 Post decode

processing failed for Config status from AP 04:da:d2:28:94:c0
*spamApTask5: Jan 03 07:49:16.563: #LWAPP-3-RD_ERR4: capwap_ac_sm.c:3085 The system detects an

invalid regulatory domain 802.11bg:-A 802.11a:-A for AP 04:da:d2:28:94:c0
*spamApTask5: Jan 03 07:49:16.563: #LOG-3-Q_IND: spam_lrad.c:10946 Country code (ES ) not configured
for AP 04:da:d2:28:94:c0[...It occurred 2 times.!]
96
Troubleshooting Lightweight APs
Can the AP and the WLC communicate?
• Make sure the AP is getting an address from DHCP (check the DHCP server
leases for the AP’s MAC address)
• If the AP’s address is statically set, ensure it is correctly configured
• Try pinging from AP to controller and vice versa
• If pings are successful, ensure the AP has at least one method to discover the
WLC
• Console or telnet/ssh into the controller to run debugs
• If you do not have access to APs, use “show cdp neighbors port <x/y> detail” on
connected switch to verify if the AP has an IP address
97
Show Commands
• On the WLC:
show msglog
show traplog
• On the AP:
show tech
show log
show capwap client rcb
show capwap client config
show capwap reap status
98
Debugs to be enabled
• On the WLC:
debug mac addr <AP Ethernet/Radio mac>
debug capwap events enable
debug capwap errors enable
debug dtls all enable
debug pm pki enable
• On the AP:
debug dhcp detail
debug capwap client detail
99
AP Supportability
 Filtering per AP
debug mac addr 04:da:d2:4f:f0:50
debug capwap events enable
Use radio mac for mac-addr filters
 For large deployments: CSCul27717

Improvements made to dtls/capwap debug output for large AP counts
Available on latest 7.0/7.4/7.6
100
AP Supportability
APs have a “flight recorder”

cap2600i-sw1-033#dir
Directory of flash:/
2 -rwx 64279 Jan 2 2014 10:32:24 +00:00 event.log

3 drwx 128 Jan 2 2014 10:26:12 +00:00 configs
4 -rwx 280 Jan 2 2014 15:41:58 +00:00 lwapp_officeextend.cfg
10 -rwx 352 Jan 2 2014 15:41:52 +00:00 env_vars
5 -rwx 49168 Jan 3 2014 06:26:30 +00:00 lwapp_non_apspecific_reap.cfg
7 -rwx 965 Nov 21 2013 15:22:52 +00:00 lwapp_mm_mwar_hash.cfg
6 -rwx 95008 Jan 2 2014 15:42:08 +00:00 lwapp_reap.cfg
17 -rwx 125501 Nov 20 2013 17:22:48 +00:00 event.r0
13 drwx 192 Dec 20 2012 21:37:52 +00:00 ap3g2-rcvk9w8-mx
11 -rwx 265 May 28 2013 09:58:54 +00:00 lwapp_mobileconcierge.cfg
12 -rwx 66319 Jan 2 2014 10:08:04 +00:00 event.capwap
25 -rwx 7192 Jan 3 2014 06:26:27 +00:00 private-multiple-fs
18 drwx 1792 Jan 2 2014 10:22:48 +00:00 ap3g2-k9w8-mx.152-4.JB3
101
AP Supportability
AP Remote Commands (WLC CLI)

 Debug AP enable <AP name>
Enables AP Remote Debug
AP Must be associated to WLC
Redirects AP Console output to WLC session
 Debug AP command “<command>” <AP name>

Output is redirected to WLC session
AP runs IOS, numerous generic IOS commands available
102
AP Supportability
 Methods of Accessing the AP
• Console
• Telnet / SSH
• No GUI support
• AP Remote Commands
 Enabling Telnet/SSH
• WLC CLI: config ap [telnet/ssh] enable <ap name>
• WLC GUI: Wireless > All APs > Select AP > Advanced > Select [telnet/ssh] > Apply
103
AP Supportability
Show Commands (AP CLI or WLC Remote Cmd)

 Show controller Do[0/1] (or Show Tech)
Must have! Before/During/After event
 Show log
 WLC: show ap eventlog <ap name>
 Show capwap client <?>
 CLI Tips
Debug capwap console cli
Debug capwap client no-reload
104
Mobility
105
Mobility—Types
 Legacy – Flat
• Old style
• Discriminator is mobility group name
 New – Hierarchical
• For 7.3, 7.5+ and Converged access
• Supports large setups, multiple device roles
• Covered on Converged access troubleshooting
106
Mobility - Messaging Flow
 When a client connects to a WLC for the first time,

the following happens:
• New WLC sends MOBILE_ANNOUNCE to all controllers in the mobility group when
client connects (note: if possible, configure multicast mobility to lower CPU load and
handoff times)
• Old WLC sends HANDOFF_REQUEST, telling the new WLC
I have an entry for this client, here is the client status
• New WLC sends HANDOFF_REPLY, telling the old WLC OK
107
Mobility - Intra-Controller
• Client roams between two APs on the same controller
108
Mobility - Inter-Controller (Layer 2)
109
Layer 2 roaming
 Layer 2 roams occur when

you move between WLCs
and both WLCs have
connectivity to the same
client subnets. In this case,
the client database entry is
simply moved to the new
WLC.
110
Mobility— Layer 2 Inter WLC
Debug Client <Mac Address>
Debug Mobility Handoff Enable
MobileAnnounce
MobileHandoff
111
Mobility - Inter-Controller (Layer 3)
 Layer 3 roaming (a.k.a. anchor/foreign)
• Dual client ownership
• Foreign owns “L2”: 802.1x, encryption, AP
• Anchor owns “L3”: IP address, webauth
 Two main types

• Auto anchor
• Dynamic
 Symmetric
traffic path
112
Layer 3 Roaming
 Layer 3 roams occur when

the controllers do not have
matching vlans, so we
have to tunnel the traffic
back to the original
controller, so the client
session is not interrupted.
This tunnel is an Ethernet-
over-IP tunnel (EoIP), and
in 7.3 and later WLC code
it can be configured to be
a CAPWAP tunnel.
113
Mobility— Layer 3 Inter WLC
Debug Client <Mac Address>
Debug Mobility Handoff Enable
MobileAnnounce
MobileHandoff
114
Mobility Group vs. Mobility Domain
 Mobility Group - WLCs with the same group name

• L2/L3 Handoff
• Auto Anchoring
• Fast Secure Roaming
• APs get all of these as a Discover candidate
 Mobility Domain - WLCs in the mobility list

• L2/L3 Handoff
• Auto Anchoring
115
Mobility – Typical Problems
 Misconfiguration
• Wrong policy set
*mmListen: Jan 03 12:03:36.613: 68:7f:74:75:f1:cd Adding mobile on Remote AP

00:00:00:00:00:00(0)
*mmListen: Jan 03 12:03:36.613: 68:7f:74:75:f1:cd mmAnchorExportRcv:, Mobility role is Unassoc
*mmListen: Jan 03 12:03:36.614: 68:7f:74:75:f1:cd mmAnchorExportRcv Ssid=webauth Security
Policy=0x2050
*mmListen: Jan 03 12:03:36.614: 68:7f:74:75:f1:cd mmAnchorExportRcv: WLAN webauth policy
mismatch between controllers, WLAN webauth not found, or WLAN disabled. Ignore ExportAnchor
mobility msg. Delete client.
• Wrong IP/MAC/Mobility name
116
Fast Roaming
117
Fast Secure Roaming – PMKID Caching
 Opportunistic Key caching

• Default Key caching, always on in WLC for WPA2 wlans
• Client support varies
• Scalable
• One full auth, global roaming
 Sticky Key caching

• Another caching mechanism
• Caching is per AP, not global
• Not Scalable, at least one full auth per AP
• Typical of Apple IOS devices (now they are moving to 11r)
• WLC optional support, up to 8 Aps, no flex support
118
Fast Secure Roaming – CCKM
119
Fast Secure Roaming – 802.11r Fast Transition
• The fast-secure roaming technique based on the 802.11r
amendment (officially named Fast BSS Transition by the
802.11 standard, and known as FT) is the first method
officially ratified (on 2008) by the IEEE for the 802.11
standard as the solution to perform fast transitions
between APs (Basic Service Sets or BSSs), which clearly
defines the key hierarchy that is used when you handle
and cache keys on a WLAN
• Fast BSS Transition Over-the-DS
• Fast BSS Transition Over-the-Air
120
Fast BSS
Transition
Over-the-
DS
121
Fast BSS
Transition
Over-the-
Air
122
Fast Secure Roaming – Reference
See the following Cisco TechNote for more detailed

info on Fast Secure Roaming:
http://www.cisco.com/c/en/us/support/docs/wireles
s-mobility/wireless-lan-wlan/116493-technote-
technology-00.html
123
Key things to remember
124
Key “Take Aways”
• Troubleshooting
• Proper problem description
• Divide and conquer: during a problem isolate
on which client state is happening
• How to reproduce the problem
• Understanding is key: working vs non
working scenarios
• No stone throwing!
125
Call to Action
• Visit the World of Solutions for
• Cisco Campus
• Walk in Labs
• Technical Solution Clinics
• Meet the Engineer

• Lunch and Learn Topics
• DevNet zone related sessions
Complete Your Online Session Evaluation
• Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.
• All surveys can be completed via

the Cisco Live Mobile App or the
Communication Stations
Wireless Cisco Education Offerings
Course Description Cisco Certification
• Conducting Cisco Unified Wireless Site Survey Professional level instructor led trainings to prepare candidates to conduct CCNP® Wireless
• Implementing Cisco Unified Wireless Voice site surveys, implement, configure and support APs and controllers in
Networks converged Enterprise networks. Focused on 802.11 and related
• Implementing Cisco Unified Wireless Mobility technologies to deploy voice networks, mobility services, and wireless
Services security.
• Implementing Cisco Unified Wireless Security
Services
Implementing Cisco Unified Wireless Network Prepares candidates to design, install, configure, monitor and conduct CCNA® Wireless
Essential basic troubleshooting tasks of a Cisco WLAN in Enterprise installations.
For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com
128
Thank you
129

Brkewn 3011 PDF

Uploaded by

Copyright:

Available Formats

Brkewn 3011 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brkewn 3011 PDF

Uploaded by

Copyright:

Available Formats

Troubleshooting Wireless

 What to expect from TAC  What not to expect from TAC

 What should I have ready?

 TAC Escalation Process

 Customer Escalation Process

 CCO - Cisco.com release

 TAC WLC Software Recommendations:

A wireless connection is like a complex multivariable equation. So how do we

 Troubleshooting is an art with no right or wrong procedure, but best with a

 Step 2: Understand any possible triggers

 Step 3: Know the expected behavior

 Wireless Sniffer  Wired Packet Capture

(Cisco Controller) >show client detail 00:16:ea:b2:04:36

• On the WLC: debug client <MAC Address>

debug aaa events/errors enable

debug dot1x all enable

debug web-auth redirect enable mac <client mac address> (~Webauth)

debug mdns all enable (~Bonjour)

debug dot11 wpa-cckm-km-dot1x

debug dot11 events

debug capwap client mgmt

*apfMsConnTask_4: Dec 16 11:30:42.060: 00:1c:58:8e:a5:84 apfPemAddUser2 (apf_policy.c:333) Changing state for

*apfMsConnTask_4: Dec 16 11:30:42.060: 00:1c:58:8e:a5:84 Sending Assoc Response to station on BSSID

*apfMsConnTask_4: Dec 16 11:30:42.060: 00:1c:58:8e:a5:84 apfProcessAssocReq (apf_80211.c:7975) Changing state for

*radiusTransportThread: Oct 11 15:11:33.610: cc:52:af:fc:89:26 Access-Reject received from RADIUS server

*apfReceiveTask: Oct 11 15:11:33.611: cc:52:af:fc:89:26 Sending Assoc Response to station on BSSID

 Hey, I have a client trying to connect, but nothing is showing up!

REAP Mgmt: dot11_lwapp_fr_sm_before_4way_add_to_drv(): a088.b41f.cb24 : PMK Validated, before 4way

REAP Mgmt: FR_SM: dot11_mgmt_lwapp_aaa_resp(): a088.b41f.cb24 :: 4WAY_HS_STARTED (AAA_SUCCESS_RSP,

Rest of the EAP Conversation

The Supplicant Derives the

Rest of the EAP Transaction

Between Client and Server

DHCP received op BOOTREQUEST (1) (len 308,vlan 5, port 1, encap 0xec03)

192.168.50.101 WEBAUTH_REQD (8) pemAdvanceState2 6662, Adding TMP rule

 Multiple mechanisms to learn Client IP address:

 Modifies Address learning

To Captive portal: ap_mac, switch_url (controller auth url), redirect(original url),

Final Customized page to client

Sign up page (web user)

User Form Submit

parser host is 192.168.0.45

*webauthRedirect: Jan 02 14:57:23.734: 28:37:37:7f:5c:7- Message to be sent is

*emWeb: Jan 02 10:46:42.904:

*ewmwebWebauth1: Jan 02 10:46:42.906: 68:7f:74:75:f1:cd apfMsRunStateInc

 Client doing request on different port

Session Timeout too low

 If using external webauth

 ARP/DNS must work before you can do anything

 RUN means: client has completed all required policy states

• Watch out for anomalous reset counts in short uptime

ap2600-sw1-0-31#sh cont d0 | b QBS

 AP Radio Reset (Power/Channel)

*dot1xMsgTask: Oct 22 15:32:53.056: 24:77:03:c2:8a:20 Sent Deauthenticate to mobile on BSSID

 Client can be removed for numerous reasons

• On the FlexConnect AP: debug capwap flex

debug capwap client config

debug capwap flexconnect mgmt

debug capwap flexconnect pmk