BRKCRS 2813

#CLMEL
Cisco SD-Access
Monitoring and
Troubleshooting
BRKCRS-2813
Parthiv Shah, Technical Leader, Escalation
Derek Huckaby, Technical Marketing Engineer
#CLMEL
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Open the Cisco Events Mobile App
2 Find your desired session in the “Session Scheduler”
3 Click “Join the Discussion”
4 Install Webex Teams or go directly to the team space
5 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKCRS-2813
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• DNA Architecture Overview
• DNA Center Troubleshooting
• Install / Services Debugging / HA
• Log Collection
• ISE and DNA Center Integration
• Device Discovery
• Provisioning
• DHCP
• SD-Access Fabric Troubleshooting

• Host Onboarding
• External Connectivity
• Host Mobility
#CLMEL BRKCRS-2813 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Objectives and Assumptions
Objectives
After completing this module you will:
• Understand the DNA Center Server Troubleshooting
• Understand SD-Access Fabric Deployment and Troubleshooting
• Understand SD-Access Host Onboarding and Troubleshooting
Assumptions
• Audience must be familiar with ISE deployment scenarios, pxGrid and Cisco TrustSec.
• Working knowledge of APIC-EM and PKI.
• Working knowledge of Routing/Switching and Cisco Fabric architecture.
• This session will not cover CLI based Cisco Fabric or ISE troubleshooting.
6
DNA Architecture
Overview
The DNA Center Appliance
Fully Integrated Automation & Assurance
• Centralised Deployment - Cloud Tethered
• Built-In Telemetry Collectors (FNF, SNMP, Syslog, etc)
• Built-In Contextual Connectors (ISE/pxGrid, IPAM, etc)
• Multi-Node High Availability (3 Node, Automation)
• RBAC, Backup & Restore, Scheduler, APIs
DNA Center Platform
1RU Server (Small form factor)
DN2-HW-APL •
•
UCS 220 M5S: 64-bit x86
vCPU: 44 core (2.2GHz)
• RAM: 256GB DDR4
• Control Disks: 2 x 480GB SSD RAID1
DNAC 1.2 Scale: Per Node • System Disks: 6 x 1.9TB SSD M-RAID
• 5,000 Nodes (1K Devices + 4K APs) • Network: 2 x 10GE SFP+
• 25,000 Clients (Concurrent Hosts) • Power: 2 x 770W AC PSU
Single Appliance for DNAC (Automation + Assurance)

#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
DNA Solution DNA Center
Cisco Enterprise Portfolio Simple Workflows
DESIGN PROVISION POLICY ASSURANCE
DNA Center
Identity Services Engine Network Control Platform Network Data Platform
Routers Switches Wireless Controllers Wireless APs
Cisco DNA Center
Cisco SD-Access – Key Components
ISE Appliance Cisco DNA Center

Appliance
API Cisco DNA Center API
Design | Policy | Provision | Assurance
API
Cisco& ISE
Identity Policy Automation
NCP Assurance
NDP
API API
Identity Services Engine Network Control Platform Network Data Platform
NETCONF
SNMP
SSH
AAA
RADIUS
EAPoL
Fabric HTTPS
NetFlow
Syslogs
Cisco Switches | Cisco Routers | Cisco Wireless
DNA Center and ISE integration
ISE node roles in SD-Access
Admin/Operate
DNA Center
Devices REST pxGrid
Things
Config Sync Context
ISE-PSN ISE-PAN ISE-PXG

Users
Authorisation Policy Exchange Topics
TrustSecMetaData
If Employee then VN/SGT-10
SGT Name: Employee = SGT-10
Network
Users SGT Name: Contractor = SGT-20
Devices If Contractor then VN/SGT-20 ...
SessionDirectory*
If Things then VN/SGT-30
Bob with Win10 on CorpSSID
ISE-MNT
DNA Center Solution Basic Pre-requisite
• Hardware
• Supported DNA Center Appliance (DN2-HW-APL)
• Supported switch/router/WLC/AP models
• Software
• Check various platform for recommended IOS-XE software version
• Check License for planned platforms
• Recommended ISE and DNA Center software
• Underlay/Overlay
• IP address plan for DNA Center and ISE
• Check for underlay network / routing configured correctly and devices are reachable
• Reachability to Internet – Direct or Proxy connection
• Access to an NTP server
• Make sure DNA Center appliance is close to real time using CIMC
Cisco DNA Center
Troubleshooting
Cisco DNA Center
Cisco SD-Access 4 Step Workflow
Design Provision Policy Assurance

Assure
Provision Assure
Assure Assure
• Global Settings • Fabric Domains • Virtual Networks • Health Dashboard

• Site Profiles • CP, Border, Edge • ISE, AAA, Radius • 360o Views
• DDI, SWIM, PNP • FEW, OTT WLAN • Endpoint Groups • FD, Node, Client
• User Access • External Connect • Group Policies • Path Traces
Planning & Preparation

Installation & Integration
DNA Center – Maglev Logical Architecture
App Stack 1 App Stack 2 App Stack N
APIs, SDK & Packaging APIs, SDK & Packaging

Standards Standards
Maglev Services
IaaS
(Baremetal, ESXi, AWS, OpenStack etc)
Cisco SD-Access (Fusion) Package Services
Trap events, host discovery we leverage ipam-service IP Address manager
apic-em-event-service
snmp traps so they are handled here.
Critical during Provisioning
apic-em-inventory- Provides communication service between network-orchestration-service
orchestation.
manager-service inventory and discovery service
Certificate authority and enables controller orchestration-engine-service Orchestration Service
apic-em-jboss-ejbca
authority on the DNAC. pnp-service PNP Tasks
apic-em-network- Configure devices. Critical service to check
programmer-service during provisioning. policy-analysis-service Policy related
apic-em-pki-broker-
PKI Certificate authority policy-manager-service Policy related
service
command-runner- Responsible for Command Runner related Core database management
postgres
service task system
distributed-cache- rbac-broker-service RBAC
Infrastructure
service
sensor-manager Sensor Related
dna-common-service DNAC-ISE integration task
site-profile-service Site Profiling
dna-maps-service Maps Related services Core service during Provisioning
spf-device-manager-service
phase
dna-wireless-service Wireless Core service during Provisioning
spf-service-manager-service
phase
identity-manager-pxgrid-
DNAC-ISE integration task swim-service SWIM
service
Assurance Services Base Services
cassandra Database
cassandra Core Database
collector-agent Collector Agents
catalogserver Local Catalog Server for update
collector-manager Collector Manager
elasticsearch Search elasticsearch Elastic Search Container
ise ISE data collector glusterfs-server Core Filesystem
kafka Communication service
identitymgmt Identity Managenent container
mibs-container SNMP MIBs
netflow-go Netflow data collector influxdb Database
pipelineadmin kibana-logging Kibana Logging collector
pipelineruntime-jobmgr
kong Infrastructure service
pipelineruntime-taskmgr
pipelineruntime-taskmgr maglevserver Infrastructure
pipelineruntime-taskmgr- mongodb Database
data
pipelineruntime-taskmgr- rabbitmq Communication service
timeseries Various Pipelines and Task nanager
workflow-server
snmp SNMP Colelctor
syslog Syslog Collector workflow-ui
trap Trap Collector workflow-worker Various Update workflow task
Most Commonly Used Maglev CLI
$ maglev $ magctl
Usage: maglev [OPTIONS] COMMAND [ARGS]... Usage: magctl [OPTIONS] COMMAND [ARGS]...
Tool to manage a Maglev deployment Tool to manage a Maglev deployment
Options:
Options:
--version Show the version and exit.
--version Show the version and exit. -d, --debug Enable debug logging
-d, --debug Enable debug logging --help Show this message and exit.
-c, --context TEXT Override default CLI context
--help Show this message and exit.
Commands: Commands:
backup Cluster backup operations api API related operations
appstack AppStack related operations
catalog Catalog Server-related management operations completion Install shell completion
completion Install shell completion disk Disk related operations
context Command line context-related operations glusterfs GlusterFS related operations
cronjob Cluster cronjob operations iam Identitymgmt related operations
job Cluster job operations job Job related operations
login Log into the specified CLUSTER logs Log related operations
logout Log out of the cluster maglev Maglev related commands
node Node related operations
maintenance Cluster maintenance mode operations service Service related operations
managed_service Managed-Service related runtime operations tenant Tenant related operations
node Node management operations token Token related operations
package Package-related runtime operations user User related operations
restore Cluster restore operations workflow Workflow related operations
service Service-related runtime operations
system System-related management operations
system_update_addon System update related runtime operations
system_update_package System update related runtime operations
Collecting Integrating
Logs ISE
Bring-up
Issues
Provisioning Discovery
Issues Issues
Cisco DNA Center Services are not coming up
Have Patience
120 to 180 minutes bring-up time
• Check network connectivity

• Check NTP/DNS server reachability
• Check any specific service not coming up
• During install or update use GUI
Avoid console login or don’t run

any system related commands
Install Failure
If you are unable to run maglev/magctl commands after install:
• Check RAID configuration and install error messages
• USB 3.0 is recommended for installation.
• Avoid KVM and/or USB 2.0 or NFS mount method for installation
• Use Windows 10 or Linux/Mac based system to build burn ISO image.
• Check for Error or Exception in following log files:
• /var/log/syslog
• /var/log/maglev_config_wizard.log
Package Status – GUI / CLI
How to Check Package Status from GUI How to Check Package Status from CLI
System Settings  App Management: Packages & Updates maglev package status
System Settings  Software Updates  Installed Apps
Check for any status
not “DEPLOYED”
Check for
“Failed”
Verify H/W profile complies with requirements
Verify sufficient disk and memory available
Verify number of CPUs to be minimum 88

and minimum memory is 256 GB.
Check Health Status of Cisco DNAC Cluster
Should show
Result as
SUCCESS
(Continued)
Troubleshooting – Kubernetes & Docker
Docker health check

The "Active" line should
show as "running".
Package Update
Package Update Troubleshooting
Fail to Download Packages:
• Check connectivity to Internet
• During update download internet connectivity is mandatory
Fail to install packages:
• During install internet connectivity is mandatory
• Check if there is any failure displayed in GUI
• Check the status from CLI if there is any error
Package Update Ordering

https://www.cisco.com/c/en/us/td/docs/cloud-systems-management/network-automation-and-
management/dna-center/1-1/rn_release_1_1_2_2/b_dnac_release_notes_1_1_2_2.html#task_nj3_nww_qcb
Proxy Setting check If Proxy server
configured then
check for Proxy
server
Check Parent
Catalog server and
Repository
Package Mapping – GUI v/s CLI
CLI Package Name GUI Display Name GUI Display Name CLI Package Name
application-policy Automation - Application Policy Automation - Application Policy application-policy
assurance Assurance - Base Assurance - Base assurance
automation-core NCP - Services Assurance - Path Trace path-trace
base-provisioning-core Automation - Base Assurance - Sensor sensor-automation
command-runner Command Runner Automation - Base base-provisioning-core
core-network-visibility Network Controller Platform Automation - Device Onboarding device-onboarding
device-onboarding Automation - Device Onboarding Automation - Image Management image-management
image-management Automation - Image Management Automation - SD Access sd-access
iwan IWAN Automation - Sensor sensor-automation
migration-support Automation - WAAS waas
ncp-system NCP - Base Command Runner command-runner
ndp Network Data Platform IWAN iwan
ndp-base-analytics Network Data Platform - Base Analytics NCP - Base ncp-system
ndp-platform Network Data Platform - Core NCP - Services automation-core
Ndp-ui Network Data Platform - Manager Network Controller Platform core-network-visibility
Network-visibility Network Controller Platform Network Controller Platform Network-visibility
path-trace Assurance - Path Trace Network Data Platform ndp
sd-access Automation - SD Access Network Data Platform - Base Analytics ndp-base-analytics
system System Or Infrastructure Network Data Platform - Core ndp-platform
waas Automation - WAAS Network Data Platform - Manager Ndp-ui
sensor-automation Automation - Sensor System Or Infrastructure system
sensor-automation Assurance - Sensor migration-support
Package Deploy Failure and Recovery
$ maglev package status
maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]
NAME DEPLOYED AVAILABLE STATUS
-----------------------------------------------------------------------------------
network-visibility 2.1.1.60067 - UPGRADE_ERROR - maglev_workflow.workflow.exceptions.TaskCallableExecutionError:
(1516326117.1073043, 1516327147.0490577, 'TimeoutError', 'Timeout of 1020 seconds has expired while watching for k8s changes for apic-em-jboss-
ejbca ')
$ maglev catalog package display network-visibility | grep fq Find the package name
fqn: network-visibility:2.1.1.60067
$ maglev catalog package delete network-visibility:2.1.1.60067 Delete the package

Ok
$ maglev package undeploy network-visibility. Undeploy failed package using the

Undeploying packages 'network-visibility:2.1.1.60067'
Package will start getting undeployed momentarily
–force option only with DE help.
$ maglev catalog package pull network-visibility:2.1.1.60067

Package pull initiated
Pull the package again
Use "maglev catalog package status network-visibility:2.1.1.60067" to monitor the progress of the operation
Once above steps completed, go to GUI and download the package again and install it.
Or you can use “maglev package deploy <>”
High Availability
High Availability(HA) Overview
• Minimize Downtime for Cisco DNAC Clsuter
• HA cluster consists of multiple nodes that communicate and share/replicate information to
ensure high system availability, reliability, and scalability
• Cisco DNAC HA is limited to 3 nodes (active  active).
• Can Handle maximum one node failure
• Components scaled as part of HA :
• Managed Service Addons: Rabbitmq, Kong, Cassandra DB, Mongo DB, Postgres DB, Glusterfs, Elastic search,
Minio
• Maglev Core Service Addons: Maglevserver, Identity Management, agent, fluent-es, keepalived, platform-ui
• K8S Components: kube-apiserver , etcd , calico, kube-controller-manager , kube-dns , kube-proxy , kube-
scheduler
Creation of 3 node cluster
Switch 1 Switch 2 Switch 3

Switch 1
Cisco Cisco Cisco

Cisco
DNAC1 DNAC2 DNAC3
DNAC1
Cluster nodes MUST be on the same version
To Configure node-2 point to first node-1 as first step of software install
Repeat the same for node-3 after node-2 completes installation
Redistribute services through System 360 enables the cluster to act as a single unit
Install Initial DNA Center Node
Kong
Fusion Services NDP Services
CatalogServer
MaglevServer DockerRegistry
WorkflowServer WorkflowWorker
GlusterFS MongoDB Kubernetes
RabbitMQ Cassandra Docker
Install Additional DNA Center Nodes
Kong Kong Kong
Fusion Services NDP Services
CatalogServer
MaglevServer DockerRegistry
WorkflowServer WorkflowWorker
GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes
RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker
Distribute Services
Kong Kong Kong
Fusion Services NDP Services Fusion Services NDP Services Fusion Services NDP Services
CatalogServer MaglevServer WorkflowServer DockerRegistry WorkflowWorker
GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes
RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker
Bringing up Cisco DNA Center 3 node cluster
• Always ensure the seed Cisco DNA Center node is up and running before
adding other cluster nodes
• After forming the cluster, make sure that all the nodes are in READY state
when you run ‘kubectl get nodes’ command from CLI.
• Enabling HA should only be done after confirming that the 3-node cluster
is successfully formed and operational with full stack deployed.
• DO NOT try to add two nodes in parallel i.e. add nodes sequentially.
Cisco DNA Center settings after second node install
Enable Service Distribution Not

showing up after the second node
is installed as HA requires 3
nodes.
Cisco DNA Center settings after third node install
Enable Service Distribution shows

up after the third node is installed
as HA requires 3 nodes.
Enabling HA using CLI

$ maglev service nodescale refresh
Scheduled update of service scale (task_id=afeca07f-5a87-410a-be48-3eef76b08db6)
Enable Service Distribution
Service Distribution happened
Check services on each node
Automation Behavior on node failure
Switch 1 Switch 2 Switch 3 Switch 1 Switch 2 Switch 3
Cisco Cisco Cisco Cisco Cisco Cisco

DNAC1 DNAC2 DNAC3 DNAC1 DNAC2 DNAC3
Node fails, automation services are automatically distributed
Current re-distribution takes 25 minutes (unplanned)
Node failure restore (RMA) will require re-distribution of services needs (25 minutes – can be planned outage)
Link failure - no significant delay in redistribution of services when link comes back up
Failure of two nodes will bring the cluster down
UI Notification on HA failure
Persistent notification of failure:

1. Node
2. Services
3. Interfaces
Node Failure UI Notifications
Node down notification
 2nd and 3rd node will form a quorum

 UI won’t be available till services are
distributed
Service Failure UI Notifications
Nodes are up but one or more

services are down
Service Failure UI Notifications
Some services are pending and

not ready
Cluster Link Failure Notifications
Node down Some services

showing status
as NodeLost
Cluster Link Came Up
Node down Banner changed from Node Lost to Services

temporarily Disrupted. When all the services
are up, this banner should go away also.
Cluster Link Came Up
Node Up Fully restored so banner gone
Remove a node from cluster (RMA use case)
• If a node in a one of the node in cluster is in failed state and is not recovering after
several hours, users should remove it from the cluster
by running CLI : $ maglev node remove <node_ip>
Gracefully removing a node

• If for any reason, customer want to remove one of the active nodes in cluster, use
the following steps:
• Move services on the given host another node by issuing:
$ maglev node drain <node_ip>
• Once all services are up and running, power down the node and remove it from the
cluster: $ maglev node remove <node_ip>
HA Commands Cheat Sheet
HA commands:
• maglev service nodescale status
• maglev service nodescale refresh
• maglev service nodescale progress
maglev service nodescale history
•
Check All 3 nodes available
• maglev node remove <node_ip>
• maglev node allow <node_ip>
• maglev cluster node display
Logs ISE
Bring-up
Issues
Issues Issues
UI Debugging from Browser
• Use Browser Debugging mode to find out API or GUI related Errors
For Chrome/Firefox Browsers

• Enable Debugging mode by going to
Menu  More Tools  Developer mode
• Select Console from top menu
• For clarity clear existing log.
• Run the task from DNA Center GUI
• Capture the console screenshot to

identify API/Error details.
UI Debugging from Browser
Firebug is another Tool for debugging mode.
• Install Firebug add-on in Firefox Browser
• Enable Firebug add-on
• Launch Firebug and Go to Console
• Run the task and it will capture detailed API information and related operation
Post/Get Operation and API name Task Success / Fail Code
Live Log - Service
Log Files:
• To follow/tail the current log of any service:

magctl service logs –r -f <service-name>
EX: magctl service logs -r -f spf-service-manager-service
Note: remove -f to display the current logs to the terminal

• To get the complete logs of any service:
• Get the container_id using:
docker ps | grep <service-name> | grep -v pause | cut -d' ' -f1
• Get logs using: docker logs <container_id>
Check Service Log in GUI
Click on Kibana Icon
Click on Service Counts

#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring / Log Explorer / Workflow
System Settings  System360:  Tools
https://<dnacenter_ip>/dna/systemSettings
Check Service Log using Log Explorer
Log Messages
Changing DNA Center Logging Levels
How to Change the Logging Level
• Navigate to the Settings Page:  System Settings  Settings  Debugging Levels
• Select the service of interest
• Select the new Logging Level
• Set the duration DNA Center should
keep this logging level change
• Intervals: 15 / 30 / 60 minutes or forever
39
Required information to report an issue
• RCA file
[Sun Feb 11 14:26:00 UTC] maglev@10.90.14.247 (maglev-master-1)
• SSH to server using maglev user $ rca
ssh –p 2222 maglev@<dnacenter_ip_address> ===============================================================

Verifying ssh/sudo access
• rca ===============================================================
[sudo] password for maglev: <passwd>
Done
• Generated file can be copied using scp/sftp from mkdir: created directory '/data/rca'
external server changed ownership of '/data/rca' from root:root to maglev:maglev
===============================================================
scp –P 2222 Verifying administration access
===============================================================
maglev@<dnacenter_ip_address>:<rca_filename> [administration] password for 'admin': <passwd>
User 'admin' logged into 'kong-frontend.maglev-
system.svc.cluster.local' successfully
• Error Screenshot from UI

===============================================================
RCA package created on Sun Feb 18 14:26:14 UTC 2018
===============================================================
• API Debug log using 2018-02-18 14:26:14 | INFO | Generating log for 'date'...
browser debugging mode tar: Removing leading `/' from member names
/etc/cron.d/
/etc/cron.d/.placeholder
/etc/cron.d/clean-elasticsearch-indexes
/etc/cron.d/clean-journal-files
Logs ISE
Bring-up Issues
Issues Issues
Cisco DNA Center – ISE Integration
Administration  pxGrid Services
• Pxgrid service should be enabled on ISE.
• SSH needs to be enabled on ISE.
• Superadmin credentials will be used for trust establishment for SSH/ERS
communication. By default ISE Super admin has ERS credentials
• ISE CLI and UI user accounts must use the same username and password
• ISE admin certificate must contain ISE IP or FQDN in either subject name or SAN.
• DNAC system certificate must contain DNAC IP or FQDN in either subject name
or SAN.
• Pxgrid node should be reachable on eth0 IP of ISE from DNAC.
• Bypass Proxy for DNAC on ISE server
Cisco DNA Center – ISE Integration Workflow
After Trust establishment

Check the subscriber
status in ISE pxGrid
Offline, Pending approval, Online
Trust Status on Cisco DNA •Center
Identity source status: (Under System360)
• AAA server Status (Settings – Auth/Policy Server) • INIT
• INPROGRESS • Available/Unavailable (PxGRID state)
• ACTIVE • TRUSTED/UNTRUSTED
• FAILED
• RBAC_FAILURE
#CLMEL BRKCRS-2813 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Troubleshooting
ISE - Cisco DNA Center Integration
Checking pxGrid service status

• Login to ISE server using SSH
• Run “show application status ise” to check for the services running.
Increasing log level to debug

• Go to Administration  Logging  Debug Log Config
• Select the ISE server and Edit
• Find pxGrid, ERS, Infrastructure Service from the list.
Click Log Level button and select Debug Level
Troubleshooting
ISE - Cisco DNA Center Integration
On Cisco DNA Center check On ISE check logs

• network-design-service • ERS
• identity-manager-pxGrid-service • pxGrid
• Cisco DNA Center common-service • Infrastructure Service logs
Example Error:
2017-08-01 05:24:36,794 | ERROR | pool-1-thread-1 | identity-manager-pxGrid-service |
c.c.e.i.u.pxGridConfigurationUtils | An error occurred while retrieving pxGrid endpoint
certificate. Request: PUT https://bldg24-ise1.cisco.com:9060/ers/config/endpointcert/
certRequest HTTP/1.1, Response: HttpResponseProxy{HTTP/1.1 500 Internal Server Error
[Cache-Control: no-cache, no-store, must-revalidate, Expires: Thu, 01 Jan 1970 00:00:00 GMT,
Set-Cookie: JSESSIONIDSSO=9698CC02E88780EC4415A6DE80C37355; Path=/; Secure; HttpOnly, Set-
Cookie: APPSESSIONID=03A609099AD604812984C6DF27CF7A19; Path=/ers; Secure; HttpOnly, Pragma:
no-cache, Date: Tue, 01 Aug 2017 05:24:36 GMT, Content-Type: application/json;charset=utf-8,
Content-Length: 421, Connection: close, Server: ] ResponseEntityProxy{[Content-Type:
application/json;charset=utf-8,Content-Length: 421,Chunked: false]}} |
Logs ISE
Bring-up Issues
Issues Issues
Device Discovery
Step 1
Verify all devices are green after Discovery
Step 2
Check if all devices in Managed state
New Configuration after Discovery
FE250#show archive config differences flash:underlay system:running-config
!Contextual Config Diffs:
+device-tracking tracking
+device-tracking policy IPDT_MAX_10
+limit address-count 10
+no protocol udp
+tracking enable
+crypto pki trustpoint TP-self-signed-1978819505

+enrollment selfsigned
+subject-name cn=IOS-Self-Signed-Certificate-1978819505 New RSA Keys are created
+revocation-check none
+rsakeypair TP-self-signed-1978819505
+crypto pki trustpoint 128.107.88.241

+enrollment mode ra Secure connection to DNA Center using the
+enrollment terminal interface 1 IP address as the certificate name
+usage ssl-client
See Notes for Complete Configurations
Troubleshooting – Discovery/Inventory
• Check for IP address reachability from DNAC
to the device
• Check username/password configuration in
Settings
• Check whether telnet/ssh option is properly
selected
• Check using manual telnet/ssh to the
device from DNAC or any other client
• Check SNMP community configuration
matches on switch and DNA-C
• Discovery View will provide additional
information.
Services Involved on DNA:

apic-em-inventory-manager-service
#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Logs ISE
Bring-up Issues
Issues Issues
Time to Provision Devices
Pre-deployment Summary
BLD2-FLR2-DST2
System Details Device Name: BLD2-FLR2-DST2
Platform Id: WS-C3650-

12X48UR-E
Device IP: 192.168.1.8
Device Location: SJ-22
Network Settings NTP Server:
AAA Primary Server: 172.25.0.170
DNS Domain Name: cisco.com
DNS Primary Server: 172.25.14.105

Verifying Config Push
• While DNA Center is evolving to use NETCONF and YANG APIs, at this time it
pushes most configuration by SSH.
• Exact configuration commands can be seen via show history all
FE2050#show history all
CMD: 'enable' 13:29:55 UTC Tue Jan 16 2018
CMD: 'terminal length 0' 13:29:55 UTC Tue Jan 16 2018
CMD: 'terminal width 0' 13:29:55 UTC Tue Jan 16 2018
CMD: 'show running-config' 13:29:55 UTC Tue Jan 16 2018
CMD: 'config t' 13:29:56 UTC Tue Jan 16 2018
CMD: 'no ip domain-lookup' 13:29:56 UTC Tue Jan 16 2018
CMD: 'no ip access-list extended DNA Center_ACL_WEBAUTH_REDIRECT' 13:29:57 UTC Tue Jan 16 2018
*Jan 16 13:29:57.023: %DMI-5-SYNC_NEEDED: Switch 1 R0/0: syncfd: Configuration change requiring
running configuration sync detected - 'no ip access-list extended DNA
Center_ACL_WEBAUTH_REDIRECT'. The running configuration will be synchronized to the NETCONF
running data store.
CMD: 'ip tacacs source-interface Loopback0' 13:29:57 UTC Tue Jan 16 2018
CMD: 'ip radius source-interface Loopback0' 13:29:57 UTC Tue Jan 16 2018
CMD: 'cts role-based enforcement vlan-list 1022' 13:29:57 UTC Tue Jan 16 2018
AAA Configuration
FE2050#show running-config | sec aaa
aaa new-model AAA server (ISE) is now
aaa group server radius dnac-group used to authenticate
server name dnac-radius_172.26.204.121 device logins
ip radius source-interface Loopback0
aaa authentication login default group dnac-group local
aaa authentication enable default enable
aaa authentication dot1x default group dnac-group
aaa authorization exec default group dnac-group local
aaa authorization network default group dnac-group
aaa authorization network dnac-cts-list group dnacs-group
aaa accounting dot1x default start-stop group dnac-group
aaa server radius dynamic-author

client 172.26.204.121 server-key cisco123
FE2050#show aaa servers

RADIUS: id 1, priority 1, host 172.26.204.121, auth-port 1812, acct-port 1813 AAA server up and
State: current UP, duration 546s, previous duration 0s running from IOSd
Dead: total time 0s, count 0
Platform State from SMD: current UNKNOWN, duration 546s, previous duration 0s
SMD Platform Dead: total time 0s, count 0
Global Cisco TrustSec (CTS) Configurations
Global AAA Configuration for all IOS Switches

aaa new-model
!
aaa authentication dot1x default group ise-group
TrustSec authorization should use cts-list AAA servers aaa authorization network default group ise-group
aaa authorization network cts-list group ise-group
cts authorization list cts-list aaa accounting dot1x default start-stop group ise-group
!
For SGT policy enforcement, if switch has to access control aaa server radius dynamic-author
client <Switch_IP> server-key cisco
cts role-based enforcement !
cts role-based enforcement vlan-list <VLANs> radius server ise
address ipv4 <ISE_IP> auth-port 1812 acct-port 1813
pac key <PAC_Password>
!
aaa group server radius ise-group
server name ise
!
ISE and ‘Network Device’ Transact Securely Using PAC keys
Switch authenticates with Cisco ISE for Secure EAP FAST Channel
Environmental Data Switch# cts credential id <device_id> password <cts_password>
TrustSec Egress Policy
RADIUS EAP FAST Channel RADIUS PAC* keys pushed by ISE. Switch uses this to talk to ISE securely
ISE IOS bldg24-edge-3650-1#show cts pacs
AID: 5079AA777CC3205E5D951003981CBF95
PAC-Info:
PAC-type = Cisco Trustsec
AID: 5079AA777CC3205E5D951003981CBF95
I-ID: FDO1947Q1F1
A-ID-Info: Identity Services Engine
Credential Lifetime: 15:30:58 PST Mon May 28 2018
PAC-Opaque:
000200B800010211000400105079AA777CC3205E5D951003981CBF950006009C0003
0100C25BAEC6DC8B90034431914E48C335DC000000135A95A90900093A8087E1E4
7B8EA12456005D6E38C41F69C19F86B884B370177982EB65469F1E5F6B2B6D96B7
1C99DA19B240FE080757F8F8BBD543AE830A5959EA4A999C310CE1FEC427213AA
552406796C8DDDA695DBCF08FB3473249DCC025598D27CD280E4D01E7877F14C6
F211CC3BAB5E3B836A6B42A9C5EE4E0E6F997549D10561
Refresh timer is set for 11w3d
Environmental Data
Switch# show cts environment-data
CTS Environment Data
====================
Current state = COMPLETE
Last status = Successful
Local Device SGT:
SGT tag = 2-00:TrustSec_Infra_SGT
Server List Info:
Installed list: CTSServerList1-0001, 1 server(s):
IOS *Server: 10.1.1.222, port 1812, A-ID 3E465B9E3F4E012E6AD3159B403B5004
ISE Status = DEAD
Security Group Name Table:
auto-test = TRUE, keywrap-enable = FALSE, idle-time = 60 mins, deadtime
= 20 secs 0-00:Unknown
Multicast Group SGT Table: 2-00:TrustSec_Infra_SGT
Security Group Name Table:
0-00:Unknown 10-00:Employee_FullAccess
2-00:TrustSec_Infra_SGT 20-00:Employee_BYOD
10-00:Employee_FullAccess 30-00:Contractors
20-00:Employee_BYOD
30-00:Contractors 100-00:PCI_Devices
100-00:PCI_Devices 110-00:Web_Servers
110-00:Web_Servers
120-00:Mail_Servers
120-00:Mail_Servers
255-00:Unregist_Dev_SGT 255-00:Unregist_Dev_SGT
Environment Data Lifetime = 86400 secs
Last update time = 21:57:24 UTC Thu Feb 4 2016
Env-data expires in 0:23:58:00 (dd:hr:mm:sec)
Env-data refreshes in 0:23:58:00 (dd:hr:mm:sec)
Cache data applied = NONE
State Machine is running
If CTS is not Configured, Verify the Device is a NAD
Configuration Configuration not pushed to the
Issues network device
Save
Check
state?
Device should be Reachable and Managed
Debug Inventory Issue
Different Types of Error
Validation Check Configuration Error Internal Error
When stale config is CLI errors out on the No config change is

present on the device device pushed to the device.
and DNAC config
validation throws an
error.
vrf Campus is already configure % 10.9.3.0 overlaps with Vlan12
Fix the configuration on the device
(config)#no vrf definition Campus
Navigate to Device inventory
Select the device and click “Resync”
Before You Add to Fabric
Configure Loopback 0 If you are using Automated Underlay

skip this setup
interface Loopback0
ip address <> This is only required for Manual
Underlay configuration
ip router isis
Don’t forget to select the device and click “Resync”
91
SD-Access Fabric
Provisioning
Fabric Edge Configuration
LISP configuration
VRF/VLAN configuration
SVI configuration
Interface configuration
SDA Provisioning – Workflow
Services Involved Start Provisioning from UI
Pre-Process-Cfs-Step Determine all the namespaces this config applies to
NB API
SPF Service Validate-Cfs-Step Validate whether this config is consistent and conflict free
Process-Cfs-Step
Persist the data and take snapshot for all namespaces
in a single transaction
Target-Resolver-Cfs-Step
Orchestration Determine the list of devices this config should go to
SPF Device
Engine Translate-Cfs-Step Per device convert the config to the config that needs to go to the device
Messaging
Deploy-Rfs-Task Convert the config to Bulk Provisioning Message to send it to NP
Network
Programmer Rfs-Status-Updater-
Task Update the Device config Status based on response from NP
Rfs-Merge-Step
Complete Update the task with an aggregate merged message
SDA Provisioning – Task Status Check
Click on View Target Device List
Click on Show task
Status Check the status
Click on See Details

VLAN and VRF Configuration
FE2050#show run | beg vrf FE2050#show run | sec vlan
vrf definition BruEsc ip dhcp snooping vlan 1021-1024
rd 1:4099 vlan 1021
! name 192_168_1_0-BruEsc
address-family ipv4 vlan 1022
route-target export 1:4099 name 192_168_100_0-BruEsc
route-target import 1:4099 vlan 1023
exit-address-family name 192_168_200_0-DEFAULT_VN
vrf definition DEFAULT_VN cts role-based enforcement vlan-list 1021-1023
rd 1:4099
!
address-family ipv4
route-target export 1:4099
route-target import 1:4099
exit-address-family
One VLAN per IP Address Pool

One VRF per VN DHCP Snooping and CTS are enabled
Closed Authentication Configuration
ISBN 2.0 Template Interface Configuration
template DefaultWiredDot1xClosedAuth FE2051#show run int gi 1/0/1
dot1x pae authenticator switchport mode access
switchport access vlan 2047 device-tracking attach-policy IPDT_MAX_10
switchport mode access authentication timer reauthenticate server
switchport voice vlan 4000 dot1x timeout tx-period 7
dot1x max-reauth-req 3
mab
source template DefaultWiredDot1xClosedAuth
access-session closed
spanning-tree portfast
access-session port-control auto
authentication periodic
authentication timer reauthenticate server
service-policy type control subscriber PMAP_ D
Troubleshooting – Device / Fabric Provision Issues
Services involved:
• orchestration-engine-service • spf-device-manager-service
• spf-service-manager-service • apic-em-network-programmer-service
Cisco SD-Access Fabric
Troubleshooting DHCP
DHCP Packet Flow in Campus Fabric
B DHCP
1 The DHCP client generates a
DHCP request and broadcasts it
on the network
FE1 BDR
1 2 FE uses DHCP Snooping to add
it’s RLOC as the remote ID in
Option 82 and sets giaddress the
2 Anycast SVI
Using DHCP Relay the request is
forwarded to the Border.
4 3 DHCP Server replies with offer
3
5 to Anycast SVI.
4 Border uses the remote ID in
option 82 to forward the packet.
5 FE installs the DHCP binding
and forwards the reply to client
DHCP Binding on Fabric Edge
FE#show ip dhcp snooping binding

MacAddress IpAddress Lease(sec) Type VLAN Interface
------------------ --------------- ---------- ------------- ---- --------------------
00:13:a9:1f:b2:b0 10.1.2.99 691197 dhcp-snooping 1021 TenGigabitEthernet1/0/23
FE#debug ip dhcp snooping ? Debug ip dhcp snooping

H.H.H DHCP packet MAC address Enables showing detail with regards to DHCP snooping
agent DHCP Snooping agent and the insertion of option 82 remote circuit
event DHCP Snooping event Debug ip dhcp server packet
packet DHCP Snooping packet Enables debug with regards to the relay function , insertion
redundancy DHCP Snooping redundancy giaddress and relay functionality to the Server
Debug dhcp detail
Adds additional detail with regards to LISP in DHCP debugs
Received DHCP Discover
015016: *Feb 26 00:07:35.296: DHCP_SNOOPING: received new DHCP packet from input interface
(GigabitEthernet4/0/3)
015017: *Feb 26 00:07:35.296: DHCP_SNOOPING: process new DHCP packet, message type: DHCPDISCOVER,
input interface: Gi4/0/3, MAC da: ffff.ffff.ffff, MAC sa: 00ea.bd9b.2db8, IP da: 255.255.255.255, IP
sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0,
DHCP chaddr: 00ea.bd9b.2db8, efp_id: 374734848, vlan_id: 1022
Adding Relay Information Option

015018: *Feb 26 00:07:35.296: DHCP_SNOOPING: add relay information option.
015019: *Feb 26 00:07:35.296: DHCP_SNOOPING: Encoding opt82 CID in vlan-mod-port format
015020: *Feb 26 00:07:35.296: :VLAN case : VLAN ID 1022
015021: *Feb 26 00:07:35.296: VRF id is valid
015022: *Feb 26 00:07:35.296: LISP ID is valid, encoding RID in srloc format
015023: *Feb 26 00:07:35.296: DHCP_SNOOPING: binary dump of relay info option, length: 22 data:
0x52 0x14 0x1 0x6 0x0 0x4 0x3 0xFE 0x4 0x3 0x2 0xA 0x3 0x8 0x0 0x10 0x3 0x1 0xC0 0xA8 0x3 0x62
015024: *Feb 26 00:07:35.296: DHCP_SNOOPING: bridge packet get invalid mat entry: FFFF.FFFF.FFFF,
packet is flooded to ingress VLAN: (1022)
015025: *Feb 26 00:07:35.296: DHCP_SNOOPING: bridge packet send packet to cpu port: Vlan1022.
Option 82 0x3 0xFE = 3FE = VLAN ID 1022 LISP Instance-id 4099 RLOC IP 192.168.3.98
0x4 = Module 4 , 0x3 = Port 3
Continue with Option 82
015026: *Feb 26 00:07:35.297: DHCPD: Reload workspace interface Vlan1022 tableid 2.
015027: *Feb 26 00:07:35.297: DHCPD: tableid for 1.1.2.1 on Vlan1022 is 2
015028: *Feb 26 00:07:35.297: DHCPD: client's VPN is Campus.
015029: *Feb 26 00:07:35.297: DHCPD: No option 125
015030: *Feb 26 00:07:35.297: DHCPD: Option 125 not present in the msg.
015032: *Feb 26 00:07:35.297: DHCPD: Sending notification of DISCOVER:
015033: *Feb 26 00:07:35.297: DHCPD: htype 1 chaddr 00ea.bd9b.2db8
015034: *Feb 26 00:07:35.297: DHCPD: circuit id 000403fe0403 Circuit ID
015035: *Feb 26 00:07:35.297: DHCPD: table id 2 = vrf Campus 0x3 0xFE = 3FE = VLAN ID 1022
015036: *Feb 26 00:07:35.297: DHCPD: interface = Vlan1022 0x4 = Module 4 , 0x3 = Port 3
015037: *Feb 26 00:07:35.297: DHCPD: class id 4d53465420352e30
Sending Discover to DHCP server Anycast Gateway IP address
015040: *Feb 26 00:07:35.297: DHCPD: Looking up binding using address 1.1.2.1

015041: *Feb 26 00:07:35.297: DHCPD: setting giaddr to 1.1.2.1.
015042: *Feb 26 00:07:35.297: DHCPD: BOOTREQUEST from 0100.eabd.9b2d.b8 forwarded to 192.168.12.240.
015043: *Feb 26 00:07:35.297: DHCPD: BOOTREQUEST from 0100.eabd.9b2d.b8 forwarded to 192.168.12.241.
Forwarding ACK
015089: *Feb 26 00:07:35.302: DHCPD: Reload workspace interface LISP0.4099 tableid 2.
015090: *Feb 26 00:07:35.302: DHCPD: tableid for 1.1.7.4 on LISP0.4099 is 2
015091: *Feb 26 00:07:35.302: DHCPD: client's VPN is .
015092: *Feb 26 00:07:35.302: DHCPD: No option 125
015093: *Feb 26 00:07:35.302: DHCPD: forwarding BOOTREPLY to client 00ea.bd9b.2db8.
015094: *Feb 26 00:07:35.302: DHCPD: Forwarding reply on numbered intf
015096: *Feb 26 00:07:35.302: DHCPD: Clearing unwanted ARP entries for multiple helpers
015097: *Feb 26 00:07:35.303: DHCPD: src nbma addr as zero
015098: *Feb 26 00:07:35.303: DHCPD: creating ARP entry (1.1.2.13, 00ea.bd9b.2db8, vrf Campus).
015099: *Feb 26 00:07:35.303: DHCPD: egress Interfce Vlan1022
015100: *Feb 26 00:07:35.303: DHCPD: unicasting BOOTREPLY to client 00ea.bd9b.2db8 (1.1.2.13).
015101: *Feb 26 00:07:35.303: DHCP_SNOOPING: received new DHCP packet from input interface (Vlan1022)
015102: *Feb 26 00:07:35.303: No rate limit check because pak is routed by this box
015103: *Feb 26 00:07:35.304: DHCP_SNOOPING: process new DHCP packet, message type: DHCPACK, input
interface: Vl1022, MAC da: 00ea.bd9b.2db8, MAC sa: 0000.0c9f.f45d, IP da: 1.1.2.13, IP sa: 1.1.2.1,
DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 1.1.2.13, DHCP siaddr: 0.0.0.0, DHCP giaddr: 1.1.2.1, DHCP chaddr:
00ea.bd9b.2db8, efp_id: 374734848, vlan_id: 1022
Client Adding to Device Tracking
015104: *Feb 26 00:07:35.304: DHCP_SNOOPING: binary dump of option 82, length: 22 data:
0x52 0x14 0x1 0x6 0x0 0x4 0x3 0xFE 0x4 0x3 0x2 0xA 0x3 0x8 0x0 0x10 0x3 0x1 0xC0 0xA8 0x3 0x62
015105: *Feb 26 00:07:35.304: DHCP_SNOOPING: binary dump of extracted circuit id, length: 8 data:
0x1 0x6 0x0 0x4 0x3 0xFE 0x4 0x3
015106: *Feb 26 00:07:35.304: DHCP_SNOOPING: binary dump of extracted remote id, length: 12 data:
0x2 0xA 0x3 0x8 0x0 0x10 0x3 0x1 0xC0 0xA8 0x3 0x62
015107: *Feb 26 00:07:35.304: actual_fmt_cid OPT82_FMT_CID_VLAN_MOD_PORT_INTF global_opt82_fmt_rid
OPT82_FMT_RID_DEFAULT_GLOBAL global_opt82_fmt_cid OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 6
015108: *Feb 26 00:07:35.304: DHCP_SNOOPING: opt82 data indicates local packet
015109: *Feb 26 00:07:35.304: DHCP_SNOOPING: opt82 data indicates local packet
015117: *Feb 26 00:07:35.405: DHCP_SNOOPING: add binding on port GigabitEthernet4/0/3 ckt_id 0
GigabitEthernet4/0/3
015118: *Feb 26 00:07:35.405: DHCP_SNOOPING: added entry to table (index 1125)
015119: *Feb 26 00:07:35.405: DHCP_SNOOPING: dump binding entry: Mac=00:EA:BD:9B:2D:B8 Ip=1.1.2.13 Lease=21600
Type=dhcp-snooping Vlan=1022 If=GigabitEthernet4/0/3
015120: *Feb 26 00:07:35.406: No entry found for mac(00ea.bd9b.2db8) vlan(1022) GigabitEthernet4/0/3
015121: *Feb 26 00:07:35.406: host tracking not found for update add dynamic
Client Added to Device Tracking
(1.1.2.13, 0.0.0.0, 00ea.bd9b.2db8) vlan(1022)
015122: *Feb 26 00:07:35.406: DHCP_SNOOPING: remove relay information option.
015123: *Feb 26 00:07:35.406: platform lookup dest vlan for input_if: Vlan1022, is NOT tunnel, if_output:
Vlan1022, if_output->vlan_id: 1022, pak->vlan_id: 1022
015124: *Feb 26 00:07:35.406: DHCP_SNOOPING: direct forward dhcp replyto output port: GigabitEthernet4/0/3.
Cisco SD-Access Fabric
Troubleshooting
Host Onboarding
Typical SD-Access Environment
Control Plane Node
 Underlay Network
(CP)
C  Routing ID (RLOC) – IP address of
the LISP router facing ISP
B B  Overlay Network
10.2.100.1
Border Node  Endpoint Identifier(EID) - IP address
(BDR) of a host
10.2.100.2 10.2.100.3
 VRF - Campus
 Instance Id - 4099
10.2.120.1 10.2.120.2 10.2.120.3
 Dynamic EID – 10_2_1_0-Campus
Fabric Edge 1 Fabric Edge 3
(FE1)  VLAN – 1021
(FE3)
10.2.1.99 Fabric Edge 2 10.2.1.89
(FE2)
Fabric Domain
(Overlay)
Here Is How You Begin
Host Registration Host Resolution
External Connectivity
East West Traffic Host Mobility
Different hosts
Wired Client Access Wireless

Point Client
Case 1: Host Registration – Wired Client
CP
C
10.2.120.1 10.2.100.1
IP Network
FE1
router lisp
10.2.1.99 site site_sjc
...
eid-prefix instance-id 10.2.1.0/24 accept-more-specifics
exit
router lisp
...
eid-table Campus instance-id
dynamic-eid 10_2_1_0-Campus
database-mapping 10.2.1.0/24 locator-set campus_fabric
exit
Registration
Message flow
!
C Client send ARP, DHCP or DATA

1
pkt
FE1 CP
1 2 FE saves the host info in local
database. Send the registration
message to CP (Map–server)
2
3 3 CP receives the registration

message saves the host tracking
database and send the reply
C
MAC
B B
Address ?
1 FE1#show mac address
1021 0013.a91f.b2b0 DYNAMIC Te1/0/23
If you don’t see the MAC address entry, then it’s a SILENT HOST.
ARP
Entry ?
2 FE1#show arp vrf Campus
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.2.1.99 0 0013.a91f.b2b0 ARPA Vlan1021
IP Device
Tracking ?
3 FE1#show device-tracking database
Network Layer Address Link Layer Address Interface vlan
ARP 10.2.1.99 0013.a91f.b2b0 Te1/0/23 1021
Fabric Edge Fabric Edge can learn the IP address from ARP, DHCP or DATA pack. If device tracking entry is
missing then check if client got an IP
C
B B
LISP local
database ?
FE1#show ip lisp instance-id 4099 database
4 LISP ETR IPv4 Mapping Database for EID-table vrf Campus (IID 4099)
LSBs: 0x1 Entries total 3, no-route 0, inactive 0
10.2.1.99/32, dynamic-eid 10_2_1_0-Campus, locator-set rloc_021

Fabric Edge Locator Pri/Wgt Source State
10.2.120.1 10/10 cfg-intf site-self, reachable
Instance
Enable debug if the database entry is missing ID
EID
FE1 RLOC
C
B B
If No Local Database Entry ?

debug lisp control-plane local-eid-database
*Jan 17 01:47:15.101: LISP-0: Local EID IID 4099 prefix 10.2.1.99/32, Setting state to
active (state: inactive, rlocs: 0/0, sources: NONE).
debug lisp control-plane dynamic-eid

*Jan 17 01:47:15.102: LISP-0: Local dynEID 10_2_1_0-Campus IID 4099 prefix 10.2.1.99/32
RLOC 10.2.120.1 pri/wei=10/10, Created (IPv4 intf RLOC Loopback0) (state: active, rlocs: 1/1, sources: dynamic).
debug lisp forwarding data-signal-discover-dyn-eid

*Jan 17 01:47:15.102: LISP-0: DynEID IID 4099 10.2.1.99 [10_2_1_0-Campus:Vlan1021] Created.
Dynamic EID
FE1 RLOC EID
Instance
ID
C
B B
LISP Control
Plane Entry ?
CP#show lisp site instance-id 4099
5 Site Name Last Up Who Last Inst EID Prefix
C Register Registered ID
site_sjc never no -- 4099 10.2.1.0/24
3d23h yes# 10.2.120.1 4099 10.2.1.99/32
Enable debug on FE and Control Plane if the database entry is missing
FE1 RLOC Instance EID

ID
C
Check if FE has sent the registration B B
message ?
debug lisp control map-request
*Jan 17 01:56:01.045: LISP: Send map request for EID prefix IID 4099 10.2.1.99/32
debug lisp forwarding data-signal-map-request

*Jan 17 01:56:02.204: LISP-0: EID-AF IPv4, Sending map-request from 10.2.1.99 to 10.2.1.99 for EID
10.2.1.99/32, ITR-RLOCs 1, nonce 0x0B5B0D11-0x5110DF55 (encap src 10.2.120.1, dst 10.2.100.1).
FE1 RLOC Control

Plane
C
B B
Verification for registration message

debug lisp control-plane map-server-registration
*Jan 17 01:57:27.716: LISP-0: MS EID IID 4099 prefix 10.2.1.99/32 site site_sjc, Forwarding map request to
ETR RLOC 10.2.120.1
FE1 RLOC
debug lisp forwarding eligibility-process-switching B B
*Jan 17 01:56:02.209: LISP: Processing received Map-Reply(2) message on

TenGigabitEthernet1/0/1 from 10.2.100.1:4342 to 10.2.120.1:4342
Control
Plane FE1 RLOC
Case 1b: Host Registration – Access Point
B
CP
C
10.2.120.1 10.2.100.1
IP Network
FE1
router lisp
...
eid-prefix instance-id 10.2.1.0/24 accept-more-specifics
exit
router lisp
...
exit
SD-Access Wireless Basic Workflows
AP Join 1
B Cisco
DNA Center
IP Network
FE1 C
Fabric WLC
1 Admin configures AP pool in Cisco DNA Center in INFRA_VN. Cisco DNA Center pre-
provision a configuration macro on all the FEs
AP Join 1
FE1
B Cisco
DNA Center
IP Network
2 CDP C
Fabric WLC
AP directly connected (*)
2 AP is plugged in and powers up. FE discovers it’s an AP via CDP and applies the macro
to assign the switch port the the right VLAN
(*) AP can be connected also through an “Extended node” switch
AP Join 1
FE1
B
IP Network DHCP
2 CDP C
DHCP 3
Fabric WLC
AP directly connected (*)
2 AP is plugged in and powers up. FE discovers it’s an AP via CDP and applies the macro
to assign the switch port the the right VLAN
3 AP gets an IP address via DHCP in the overlay
AP Join FE1 5
B
CAPWAP Join CAPWAP in VXLAN
exchange SDA Fabric
C
CAPWAP traffic 4 7 AP RLOC?
AP EID register 6 AP Check
Fabric WLC
4  Fabric Edge registers AP’s IP address and MAC (EID) and updates the Control Plane (CP)
5  AP learns WLC’s IP and joins using traditional methods. Fabric AP joins in Local mode
6  WLC checks if AP is fabric-capable (Wave 2 or Wave 1 APs)
7  If AP is supported, WLC queries the CP to know if AP is connected to Fabric
LISP Control
Plane Entry ?
1 CP#show lisp site instance-id 4099
C Site Name Last Up Who Last Inst EID Prefix
Register Registered ID
site_sjc never no -- 4099 10.2.1.0/24
3d23h yes# 10.2.120.1 4099 10.2.1.7/32
Is AP
discovered?
FE1 RLOC
2
(Cisco Controller) >show ap summary EID
Instance
Number of APs.................................... 1 ID
WLC AP Name Slots AP Model Ethernet MAC IP Address

---------------------- ----- -------------------- ----------------- ---------------
AP00A6.CA36.08D 2 AIR-AP3802P-T-K9 00:a6:ca:36:08:d8 10.2.1.7
Is AP fabric
enabled? (Cisco Controller) >show fabric summary
3
Fabric Support................................... enabled
Enterprise Control Plane MS config

--------------------------------------
Primary Active MAP Server

IP Address....................................... 10.2.100.1
WLC
VNID Mappings configured: 1
Name L2-Vnid L3-Vnid IP Address/Subnet

-------------------------------- ---------- ---------- ---------------------------------
ap_10_0_0_0 8041 4099 10.2.1.0 / 255.255.255.0
(Cisco Controller) >show ap config fabric AP00A6.CA36.08D8
Fabric Configuration Information For AP: AP00A6.CA36.08D8

Fabric status - Enabled
Fabric L3vnid - 4099
Fabric L2vnid - 8041
Fabric rlocIp - 10.2.120.1
FE1#show lisp instance-id 8041 ethernet database wlc
Is VXLAN WLC clients/access-points information for router lisp 0 IID 8041
tunnel UP?
4 Hardware Address Type Sources Tunnel Update
---------------- ------ ------- -------------

00d7.8fed.dba0 AP 1 Signaled
Fabric Edge FE1#show access-tunnel summary
Access Tunnels General Statistics:
Number of AccessTunnel Data Tunnels = 1
Name SrcIP SrcPort DestIP DstPort VrfId

------ --------------- ------- --------------- ------- ----
Ac0 10.2.120.1 N/A 10.2.1.7 4789 2
Name IfId Uptime

------ ------------------ --------------------
Ac0 0x0000000000000057 4 days, 07:28:25
Case 1c: Host Registration – Wireless Client
B
CP
C
10.2.120.1 10.2.100.1
IP Network
FE1
router lisp
...
eid-prefix instance-id10.2.1.0/24 accept-more-specifics
exit
router lisp
...
exit
Client Onboarding
FE1 B
CAPWAP in VXLAN ISE
SDA Fabric
C
Client Join
1
Fabric WLC
Client SGT/VNID and RLOC
• Client authenticates to a Fabric enabled WLAN. WLC gets SGT from ISE, updates AP
1
with client L2VNID and SGT
Client Onboarding
B
Client in FWD
3
table SDA Fabric
FE1
C
2
Client MAC register Fabric WLC
• Client authenticates to a Fabric enabled WLAN. WLC gets SGT from ISE, updates AP
1
with client L2VNID and SGT
2 • WLC knows RLOC of AP from internal DB . WLC proxy registers Client L2 info in CP;
this is LISP modified message to pass additional info, like the client SGT
3 • FE gets notified by CP and knows it’s a client; FE adds client MAC in L2 forwarding
table and go and fetch the client policy from ISE based on the client SGT
SD-Access Wireless Basic Workflows DHCP
Client Onboarding DHCP flow
B
6 SDA Fabric
5
FE1 C
4 DHCP packet + L2 vnid
Fabric WLC
4 • Client initiates DHCP Request
5 • AP encapsulates it in VXLAN with L2 VNI info (and SGT)
• Fabric Edge maps L2 VNID to the VLAN interface and forwards the DHCP
6
packet in the overlay (same as for a wired Fabric client)
SD-Access Wireless Basic Workflows DHCP
Client Onboarding 7
FE1 B
SDA Fabric
8 C
Client IP, L3 VNI, RLOC IP

Fabric WLC
7 • Client receives an IP address from DHCP
• DHCP snooping triggers the client EID registration by the Fabric Edge to the CP.
8
(If client has a static IP, then ARP or any other IP packet will trigger the
registration)
This completes Client onboarding process
1 (Cisco Controller)>show fabric summary
VNID Mappings configured: 1
Name L2-Vnid L3-Vnid IP Address/Subnet

-------------------------------- ---------- ---------- ---------------------------------
ap_10_0_0_0 8041 4099 10.2.1.0 / 255.255.255.0
WLC Fabric Enabled Wlan summary

WLAN ID WLAN Profile Name / SSID Vnid Tag Peer ip
------- ----------------------------------- ---------- ------- -----------
Is client 2 fabric_wlan51 / fabric_wlan51 8041 0 0.0.0.0
Associated?
2
(Cisco Controller)>show client summary
Number of Clients................................ 1
Number of PMIPV6 Clients......................... 0
Number of EoGRE Clients.......................... 0
WLC MAC Address AP Name Slot Status WLAN Auth Protocol Port Wired Tunnel Role
----------------- ---------------- ---- ------------- ----- ---- ---------------- ---- ----- ------- --------
b8:27:eb:ac:4c:d8 AP00A6.CA36.08D8 0 Associated 2 Yes 802.11n(2.4 GHz)

1 No No Local
Is WLAN
fabric enabled ?
3 (Cisco Controller)>show client detail b8:27:eb:ac:4c:d8
Client MAC Address............................... b8:27:eb:91:0b:80
Client Username ................................. N/A
. . .
Client State..................................... Associated
Client User Group................................
Client NAC OOB State............................. Access
WLC Wireless LAN Id.................................. 2
. . .
Authentication Algorithm......................... Open System
802.1P Priority Tag.............................. disabled
Security Group Tag............................... 1000
. . .
Fabric Configuration
--------------------
Fabric Status: .................................. Enabled
Vnid: ........................................... 8041
Is client
registered?
4
CP#show lisp instance-id 8041 ethernet server
C LISP Site Registration Information
Site Name Last Up Who Last Inst EID Prefix

site_sjc never no -- 8041 any-mac
00:11:34 yes# 10.2.120.1 8041 18F6.43E1.3FFB /48
Is client
entry on
access-tunnel ?
5
FE1#show mac address-table vlan 1021
Mac Address Table
-------------------------------------------
Fabric Edge
Vlan Mac Address Type Ports
---- ----------- -------- -----
1021 18F6.43E1.3FFB DYNAMIC Ac0
Is AP to FE
VXLAN tunnel up
? 6
AP00A6.CA36.08D8#show ip tunnel fabric
Fabric GWs Information:
Tunnel-Id GW-IP GW-MAC Adj-Status Encap-Type Packet-In Bytes-In
1 10.2.120.1 00:42:5A:91:89:46 Forward VXLAN 930 100370
AP
Is client
entry on
access-tunnel ?
7
AP00A6.CA36.08D8#show controllers dot11Radio 0 client 18:F6:43:E1:3F:FB
mac radio vap aid state encr Maxrate is_wgb_wired wgb_mac_addr
18:F6:43:E1:3F:FB 0 1 2 FWD OPEN M7 false 00:00:00:00:00:00
fabric client details:

client IP_ACL SGT VNID GW_IP
AP 18:F6:43:E1:3F:FB 0 8041 10.2.120.1
LISP Control
Plane Entry ?
8 CP#show lisp site instance-id 4099
C Site Name Last Up Who Last Inst EID Prefix
site_sjc never no -- 4099 10.2.1.0/24
3d23h yes# 10.2.120.1 4099 10.2.1.89/32
Wired and Wireless Host Resolution
CP
C
Wired Wireless
Clients Clients
B B
10.2.100.1
10.2.120.1 10.2.120.3
FE1 FE3
10.2.1.99 10.2.1.89
Map Cache FE1#show ip lisp map-cache instance-id 4099
Entry ? LISP IPv4 Mapping Cache for EID-table vrf Campus (IID 4099), 5 entries
1
10.2.1.89/32, uptime: 00:05:16, expires: 23:57:59, via map-reply, complete
Locator Uptime State Pri/Wgt
10.2.120.3 00:04:23 up 10/10
Fabric Edge
Control Plane
Entry ?
2
C CP#show lisp site instance-id 4099
site_sjc never no -- 4099 10.2.1.0/24
3d23h yes# 10.2.120.1 4099 10.2.1.99/32
3d23h yes# 10.2.120.3 4099 10.2.1.89/32
Host Resultion
Message flow
!
C 1 A client wants to establish

communication to a Host2
2 No local map-cache entry Host2
FE1 CP FE3
on FE1. Map-Request is sent to
1 the CP(Map-Resolver)
3 CP(Map Server) forwards the original
2 Map-Request to the FE3(ETR) that
last registered the EID subnet
3\ 3 4 FE3(ETR) sends to the FE1(ITR) a

Map-Reply containing the
requested mapping information
4
5 FE1(ITR) installs the mapping
information in its local map-cache
5
C
Verify map-request messages sent to the fabric B B
control-plane ?

*Jan 18 16:12:57.610: LISPdata-signal: sending signal for 10.2.1.99 ->10.2.1.89 on in
IPv4:Campus
Host1
debug lisp forwarding eligibility-process-switching EID
*Jan 18 16:12:57.741: LISP-0: EID-AF IPv4, Sending map-request from 10.2.1.89 to 10.2.1.89
for EID 10.2.1.89 /32, ITR-RLOCs 1, nonce 0x0579975B-0x0823B8E4 (encap src 10.2.120.1, dst
10.2.100.1).
Host2
EID
Verify map-request messages are sent
C
to the fabric control-plane

B B


*Jan 18 16:12:57.610: LISPdata-signal: sending signal for 10.2.1.99 ->10.2.1.89 on in IPv4:Campus
Host1
debug lisp forwarding eligibility-process-switching EID
*Jan 18 16:12:57.741: LISP-0: EID-AF IPv4, Sending map-request from 10.2.1.89 to 10.2.1.89 for EID
10.2.1.89 /32, ITR-RLOCs 1, nonce 0x0579975B-0x0823B8E4 (encap src 10.2.120.1, dst 10.2.100.1).
Host2
EID
C
B B
Verification on Control Plane ?

site_sjc never no -- 4099 10.2.1.0/24
3d23h yes# 10.2.120.1 4099 10.2.1.99/32
3d23h yes# 10.2.120.3 4099 10.2.1.89/32
debug lisp control map-server-map-request

*Jan 18 16:15:27.529: LISP: Received map request for IID 4099 10.2.1.89/32, source_eid IID
4099 10.2.1.99, ITR-RLOCs: 10.2.120.1, records 1, nonce 0x0579975B-0x0823B8E4
*Jan 18 16:15:27.529: LISP-0: MS EID IID 4099 prefix 10.2.1.89/32 site site_sjc,
Forwarding map request to ETR RLOC 10.2.120.3.
FE1 RLOC FE3 RLOC
C
B B
Verify map-request forwarded to the fabric edge?

Jan 18 16:12:58.531: LISP: Received map request for IID 4099 10.2.1.89/32, source_eid IID
4099 10.2.1.99, ITR-RLOCs: 10.2.120.1, records 1, nonce 0x0579975B-0x0823B8E4
Jan 18 16:12:58.531: LISP-0: Sending map-reply from 10.2.120.3 to 10.2.120.1.
FE3 RLOC FE1 RLOC
B B
Verify map-reply recevied from FE 3?

*Jan 18 16:12:57.748: LISP: Processing Map-Reply mapping record for IID 4099
10.2.1.89/32, ttl 1440, action none, authoritative, 1 locator 10.2.120.3 pri/wei=10/10 LpR
It is the Same Sequence if Border is Requesting
C
B B
Map Cache
10.2.1.99/32,
Locator 10.2.120.1
Local Database
10.2.1.99/32,
Locator 10.2.120.1
Case: 4 - External Connectivity
CP
C
router lisp
site site_sjc
40.1.1.40 ...
B B eid-prefix instance-id 4099 10.2.1.0/24 accept-more-specifics
exit
10.2.100.1
BDR
10.2.100.2 router lisp
encapsulation vxlan
!
eid-table Campus instance-id 4099
map-cache 10.2.1.0/24 map-request exit
10.2.120.3
router lisp
FE3 ...
10.2.1.89 database-mapping 10.2.1.0/24 locator-set campus_fabric
exit
C
Verification on Control Plane

site_sjc never no -- 4099 10.2.1.0/24
3d23h yes# 10.2.120.3 4099 10.2.1.89/32
B B
Verification at the FE
FE3#show ip lisp map-cache instance-id 4099
LISP IPv4 Mapping Cache for EID-table vrf Campus (IID 4099), 5 entries
32.0.0.0/4, uptime: 00:01:30, expires: 00:00:21, via map-reply, forward-native

Encapsulating to proxy ETR
B B
B
Verification at the Border

BDR#show ip lisp map-cache instance-id 4099
LISP IPv4 Mapping Cache for EID-table vrf Campus (IID 4099), 5 entries

10.2.120.3 00:04:23 up 10/10
B B
SD-Access Borders
Border Node is an entry & exit point for all data traffic
coming in or going out of the Fabric
! C ?
There are 2 Types of Border Nodes: Known Unknown
Networks Networks
B B
• Fabric Border (Internal)
• Used for “Known” Routes in your company
• Default Border (External)

• Used for “Unknown” Routes outside your company
Fabric Edge Nodes
In case of Internal Border
Verify the routes that are being imported
Internal-BDR#show ip lisp route-import map-cache instance 10

LISP IPv4 imported routes for EID-table vrf PACAF (IID 10)
Config: 1, Entries: 7 (limit 1000)
Prefix Uptime Source RLOC-set Cache/DB State
10.1.18.0/24 21:59:17 bgp 65002 installed
10.1.100.1/32 21:59:17 bgp 65002 installed
100.1.1.0/24 21:59:17 bgp 65002 installed
101.1.1.0/24 21:59:17 bgp 65002 installed
192.168.111.0/24 21:59:17 bgp 65002 installed
192.168.206.0/24 21:59:17 bgp 65002 installed
Case: 5 - East West Traffic
CP
C
router lisp
site site_sjc
B B ...
eid-prefix instance-id 4099 10.2.1.0/24 accept-more-specifics
exit
router lisp
...
exit
FE1 FE3
router lisp
...
Host1 Host2 database-mapping 10.2.1.0/24 locator-set campus_fabric
exit
C
Verification on Control Plane ?

site_sjc never no -- 4099 10.2.1.0/24
2d05h yes# 10.2.120.1 4099 10.2.1.99/32
2d02h yes# 10.2.120.2 4099 10.2.1.89/32
4d02h yes# 10.2.120.2 4099 10.2.1.88/32
If any of Host IP are missing.
Run Host Registration

flow (Case 2).
Verification at the FEs
FE1#show ip lisp instance-id 4099 database 10.2.120.1
10.2.1.99/32, locator-set rloc_021a8c01-5c45-4529-addd-b0d626971a5f
Locator Pri/Wgt Source State

10.2.120.3 00:00:06 up 10/10
10.2.120.3

10.2.120.1 00:00:06 up 10/10
Case: 6 - Host Mobility
CP
C
B B
FE1 FE2 FE3
Host1 Host2
Map Request Message flow 1 Host1 moves from FE1 to FE2
C 2 FE2 saves the host info in local
database. Send the registration
message to control plane
FE1 CP FE2 3 The Map-Server adds to the
1 database the entry for the
specific EID, associated to the
RLOCs
2
4 The Map-Server sends a Map-
Notify message to the last FE1
3 that registered the 10.2.1.99/32
prefix
4 5 FE1 receives the Map-Notify
message from the CP and adds
route associated to the 10.2.1.99
5 EID to away table
Verification at the FEs
FE1#show ip lisp away instance-id 4099
LISP Away Table for router lisp 0 (Campus) IID 4099
Entries: 1
Prefix Host EID
Producer
10.2.1.99/32 local EID

FE2

10.2.120.1 00:00:06 up 10/10
FE1
1 The LISP process on FE1
receiving the first data packet
Map Request Message flow creates a control plane message
SMR and sends it to the remote
FE3(ITR) that generated the packet
C
2 Send a new Map-Request for the
desired destination (10.17.1.99) to
the Map-Server
FE3 FE1 CP FE2
1 3 Map-Request is forwarded by
the Map-Server to the FE2 that
registered last the /32 EID
address
2 4 FE2 replies with updated
mapping information to the
3 remote FE3
5 FE3 updates the information in
its map-cache, adding the
4 specific /32 EID address
5 associated to the xTRs deployed
in the East site (10.2.120.1 and
5
10.2.120.2)
Q&A
Complete your online session evaluation
Give us your feedback to be entered

into a Daily Survey Drawing.
Complete your session surveys through
the Cisco Live mobile app or on
www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available for viewing
on demand after the event at www.CiscoLive.com/Online.
Continue Your Education
• Demos on the Cisco stand
• Walk-in Self-Paced Labs
• Meet the Expert 1:1 meetings
• Related sessions
179
Thank you
#CLMEL
DNA Center Services not coming up
How to Check Service Status from GUI
System Settings  System360: Services
https://<dnacenter_ip>/dna/systemSettings

BRKCRS 2813

Uploaded by

Copyright:

Available Formats

BRKCRS 2813

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKCRS 2813

Uploaded by

Copyright:

Available Formats

#CLMEL

• SD-Access Fabric Troubleshooting

Single Appliance for DNAC (Automation + Assurance)

DESIGN PROVISION POLICY ASSURANCE

Routers Switches Wireless Controllers Wireless APs

ISE Appliance Cisco DNA Center

Design | Policy | Provision | Assurance

Cisco Switches | Cisco Routers | Cisco Wireless

Devices REST pxGrid

ISE-PSN ISE-PAN ISE-PXG

Design Provision Policy Assurance

• Global Settings • Fabric Domains • Virtual Networks • Health Dashboard

Planning & Preparation

APIs, SDK & Packaging APIs, SDK & Packaging

• Check network connectivity

Avoid console login or don’t run

Verify sufficient disk and memory available

Verify number of CPUs to be minimum 88

Docker health check

Package Update Ordering

$ maglev catalog package delete network-visibility:2.1.1.60067 Delete the package

$ maglev package undeploy network-visibility. Undeploy failed package using the

$ maglev catalog package pull network-visibility:2.1.1.60067

Switch 1 Switch 2 Switch 3

Cisco Cisco Cisco

Cluster nodes MUST be on the same version

To Configure node-2 point to first node-1 as first step of software install

Repeat the same for node-3 after node-2 completes installation

Fusion Services NDP Services

GlusterFS MongoDB Kubernetes

RabbitMQ Cassandra Docker

Kong Kong Kong

Fusion Services NDP Services

GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes

RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker

Kong Kong Kong

CatalogServer MaglevServer WorkflowServer DockerRegistry WorkflowWorker

GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes

RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker

Enable Service Distribution Not

Enable Service Distribution shows

Enabling HA using CLI

Service Distribution happened

Cisco Cisco Cisco Cisco Cisco Cisco

Node fails, automation services are automatically distributed

Current re-distribution takes 25 minutes (unplanned)

Failure of two nodes will bring the cluster down

Persistent notification of failure:

Node down notification

 2nd and 3rd node will form a quorum

Nodes are up but one or more

Some services are pending and

Node down Some services

Node down Banner changed from Node Lost to Services

Node Up Fully restored so banner gone

Gracefully removing a node

For Chrome/Firefox Browsers

• For clarity clear existing log.

• Run the task from DNA Center GUI