Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                

SNMP Cisco

Download as pdf or txt
Download as pdf or txt
You are on page 1of 120

GETTING THE RIGHT EVENTS FROM

NETWORK ELEMENTS
NMS-A01

Benoit Claise

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 1


Housekeeping

• Please switch off your mobile phones!


• Don’t forget to complete your evaluations - you can access
them on-line via Schedule Builder!
• Visit the World of Solutions on Level -01!
• Your session printouts have been prepared by
Print Center on Level -01

• Please remember this is a ‘No Smoking’ venue!


• Please remember to wear your badge at all times including the
Party!

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 2


This Tutorial Is…

• NOT about
Fault Management Return On Investment
A level 1 type of presentation
Marketing slides
Polling the device to “discover” the fault
Fault Management Applications details

• About
Features, tricks, information, examples, etc.
on “How to generate the right events from your
network elements!”

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 3


Polling vs. Event Notification
Polling–Request
Polling–Response

SNMP Agent SNMP Manager

Event After the event…


Then potentially some
event-based polling
• Message:
Let the network elements monitor themselves
Let’s tune the right fault management events from the
network elements
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 4
Polling vs. Event Notification

Polling Event

Network Manager Network Engineer,


Load on Station, Links, Initially, to Configure
Network Devices the Event
Performance
Management Proactive
(Availability Fault Monitoring,
Application Monitoring, Utilization, Operational
and Forecasting) Monitoring
Fault Management

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 5


Agenda

• SNMP Notification: Traps and Informs


• Syslog Messages
• Embedded Syslog Manager
• RMON Event/Alarm
• EVENT-MIB
• EXPRESSION-MIB
• Specific MIBs and Scenarios
• Embedded Event Manager

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 6


SNMP NOTIFICATION

EVERYBODY KNOWS ABOUT TRAPS!

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 7


SNMP Notifications

• Notifications are the messages being generated


from the SNMP agent, regardless of the mechanism
to deliver them
• SNMP notification implemented in SNMPv2:
Traps
Unacknowledged UDP packet
Implemented since SNMPv1
Informs
Acknowledged UDP packet
Implemented since SNMPv2c

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 8


SNMP Trap Notification
Agent Generates Notification
2 3
linkDown
Remote ifIndex.4 4 1
Device ifDescr.4 Serial 1/2
ifType .4 propPointToPointSerial(22) Load MIB
Goes Down
ifLocReason.4 Keepalive failed

SNMP Notification Instance 4


Serial 1/2
SNMP SNMP
Agent Manager

OID for
linkDown
4
5
Trap PDU 1.3.6.1.2.1.11.0.2
linkDown Notification Delivered to the NMS
• SNMP trap notification contains:
Varbinds: ifIndex, ifDescr, ifType, ifLocReason
• OID: linkDown notification
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 9
How to Enable SNMP Traps Notification?

• On a Cisco router:

Router (config)# snmp-server enable traps


<trap_type>
Router (config)# snmp-server host <NMS host>
version <v1/v2c/v3 [auth | noauth | priv]>
<trap_community> <trap_type>

• On a Cisco switch:

Switch>(enable) set snmp trap enable <trap_type>


Switch>(enable) set snmp trap <NMS_host>

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 10


Traps-Show Commands
Router#show snmp

22689 SNMP packets output
0 Too big errors (Maximum packet size 1500)
229 No such name errors
0 Bad values errors
0 General errors
22450 Response PDUs
172 Trap PDUs Router(config)# snmp-server
queue-length <length>

SNMP logging: enabled


Logging to 10.48.71.130.162, 0/10, 86 sent, 0 dropped.
Logging to 144.254.7.167.162, 0/10, 85 sent, 1 dropped.

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 11


linkUp/linkDown Notification

linkDown
ifIndex.4 4
Cisco Redefinition
ifDescr.4 Serial 1/2
CISCO-GENERAL-TRAPS
ifType .4 propPointToPointSerial(22)
locIfReason.4 keepalive failed
Instance 4
Serial 1/2
linkDown
IETF Notification
ifIndex.4 4
ifAdminStatus.4 Down
IF-MIB
(RFC2233/RFC2863)
ifOperStatus.4 lowerLayerDown

router(config)# snmp-server trap link ietf

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 12


How to Enable SNMP Inform Notification?
Enable Trap and Inform Notifications; Ideally “Notification”!
Router(config)# snmp-server enable traps ...
Router(config)# snmp-server host <host-id> informs
version [2c | 3 [auth | noauth | priv]]
<community-string>...
Router(config)# snmp-server informs [retries
retries] [timeout seconds] [pending pending]
By default: 3 retries, 30 sec timeout, 25 informs pending for
acknowledgement

• “snmp-server enable informs…” no functionality!


• Switches:
So far, needed the SNMPv3 architecture
8.3(1): simplified v2c inform CLI
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 13
Informs: Show Commands
Router#show snmp

SNMP Manager-role output packets …
20 Inform-request PDUs
0 Timeouts
0 Drops

SNMP Manager-role input packets …
20 Response PDUs
0 Response with errors

SNMP informs: enabled


SNMP informs: enabled
Informs in flight 0/25 (current/max)
Logging to 10.48.71.163.162
2 sent, 0 in-flight, 0 retries, 0 failed, 0 dropped

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 14


SNMP Source Trap Notification

Eth0 (10.1.1.1)
SNMP SNMP
Agent Manager
S0

L0

Router(config)# snmp-server trap-source ethernet 0


(notification sent even if ethernet 0 is down)

Or even better

Router(config)# snmp-server trap-source loopback 0

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 15


SNMP Traps vs. Informs

Traps Informs

Reliability None Some

Retries Not Applicable 3 (Default)

Resources x X

Source Interface
Source Configuration
Not Implemented

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 16


Notification Deduplications
Notion of Time

SNMPv2-Notification-PDU sysUpTime
in the Device

PDU type Request- ErrorStatus ErrorIndex Variable-bindings:


id =0 =0 sysUpTime

• The notifications have no notion of UTC time


NTP is of NO USE between network elements and notification
receiver
for SNMP notifications
Big drawback of SNMP notifications
• No solution for network-wide alarm deduplications or analysis
Only track: the fault management applications look at the time the
notifications are received
• Similar problem with SNMPv1: trap also sends the sysUpTime

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 17


How to Find Out About Notifications?

Router(config)# snmp-server enable traps ?


atm Enable SNMP atm traps
bgp Enable BGP state change traps
config Enable SNMP config traps

• TAC Web document


http://www.cisco.com/warp/customer/477/SNMP/
SNMPTrapsInImages.html
• What device supports which MIB?
http://www.cisco.com/go/mibs

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 18


NOTIFICATION-LOG-MIB

• RFC-3014 “NOTIFICATION-LOG MIB”


• Notification buffer: allow a management station
to retrieve notifications that have been missed
• Notifications visualization without a receiver;
useful for troubleshooting!
• No persistence across reload

Router(config)#snmp mib notification-log ?


default create/configure default log
globalageout modify the global ageout
globalsize modify the global size

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 19


NOTIFICATION-LOG-MIB

Router#show snmp mib notification-log all


Notification ID cisco.0.1
sysUpTime when logged 4057361, Accessed by
1 log(s), contains 8 varbinds
Notification ID snmpTraps.4
sysUpTime when logged 4098180, Accessed by
1 log(s), contains 6 varbinds

The MIB returns the MIB values, not the CLI

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 20


SYSLOG MESSAGES

WHAT HAPPENS IF THE NOTIFICATION DOESN’T EXIST?


OR IF THERE IS NO SNMP NOTIFICATION RECEIVER?

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 21


Syslog Message

Fan Failure
Syslog
Agent
Receiver

2
1

Syslog Message Sent:


Cisco InfoCenter
*Apr 10 08:21:32 CET CW RME Syslog Analyzer
%CI-3-PARTFANFAIL : Single fan failure (UNIX Syslogd)

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 22


Syslog Message

• Syslog produces (mostly) structured logs of information;


allowing software subsystems to
report and save important error messages either locally or to
a remote logging server
• Very basic reporting mechanism: no variable bindings, plain
English text
• Text messages sent to a Syslog daemon,
on UDP port 514
• Very basic “standard”, informational RFC 3164
• RFC 3195: reliable delivery for Syslog
• Complementary to other events (SNMP notifications)

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 23


Syslog Message Format

Message Header: Message Body


(Next Slide)
< >, Timestamp, Tag string, …

<facility(X)>.<level(Y)> WHAT Messages Are Logged?


emergency 0, alert 1, critical 2,
error 3, warning 4, notification 5,
information 6, debug 7

WHERE Is the Message Logged in the Syslog Server?


local0…local7, cron, user, etc.

• <facility.level> is not retained in the Syslog message file


• Additional timestamp is added by logging host
• Header example
Message Header: local7.emergency

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 24


How to Enable Syslog Message
on Cisco IOS?

• On a Cisco router:
Router(config)# logging on
Router(config)# logging <server_ip_address>
Router(config)# logging facility local6
Router(config)# service sequence-numbers
Router(config)# service timestamps log [datetime
| uptime]
Router(config)# service timestamps log datetime
[msec] [localtime] [show-timezone] [year]

Optional: Default Is in UTC with No


Milliseconds and No Time Zone
Note: UTC, Universal Time, since 1970

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 25


Syslog Message “Body” Format
in the Cisco IOS

* Sep 20 01:12:31: %SYS-5-CONFIG_I: Configured from


CONSOLE console by vty1 (144.254.9.79)

Timestamp Cisco IOS® Severity Mnemonic Message-text


M
Component es
Timestamp sa
from the Server ge
te
SERVER Sep 20 01:07:00 router.cisco.com 571: * Sep 20 01:12:31: xt
%SYS-5-CONFIG_I: Configured from console by vty1
(144.254.9.79)
Router
Service Sequence- Timestamp from
• NTP is needed! Numbers the Router

• Header:level can be different than Body:severity


NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 26
Syslog: Show Commands on Cisco IOS
(Cont.)

Router# show logging


Syslog logging: enabled (0 messages dropped, 13 messages rate-limited, 0
flushes, 0 overruns)
Console logging: level debugging, 34 messages logged
Monitor logging: level debugging, 0 messages logged
Buffer logging: level debugging, 47 messages logged
Logging Exception size (8192 bytes)
Trap logging: level debugging, 51 message lines logged
Logging to 10.48.71.225, 51 message lines logged

Log Buffer (8192 bytes):


*Apr 10 08:21:32 CET: %SYS-5-RESTART: System restarted --
*Apr 10 08:21:32 CET: %SNMP-5-COLDSTART: SNMP agent on host popo is
undergoing a cold start
*Apr 10 08:21:32 CET: %LINK-5-CHANGED: Interface FastEthernet5/1,
changed state to administratively down

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 27


How to Enable Syslog Message on
Catalyst OS?

On a Cisco Catalyst Switch:


Switch(enable)> set logging session enable
Switch(enable)> set logging server
<Server_ip_address>
Switch(enable)> set logging server facility
local7
Switch(enable)> set logging server severity 3
Switch(enable)> set logging console enable
Switch(enable)> set logging timestamp enable

Local Time Configured


on the Switch (Optional)

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 28


Syslog Source Interface

Eth0 (10.1.1.1)
Syslog Syslog
Agent Manager
S0

Router(config)# logging source-interface loopback 0

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 29


Syslog Message Filtering: Example 1

• How to get the error messages which have severity


level equal or lower than error?

Router(config)# logging 10.10.10.10 Confusing!


Router(config)# logging facility local6 Should be
Level!!
Router(config)# logging trap errors (The One in
Router(config)# logging console debugging the Syslog
Header)

• On the Syslog server (UNIX), the corresponding


line in Syslog.conf file is:
local6.errors /var/log/mylog

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 30


Syslog Message Filtering: Example 2

• How to only log the error messages related to


spanning tree?
Switch> (enable) set logging session enable
Switch> (enable) set logging server 10.10.10.10
Switch> (enable) set logging server severity 0
Switch> (enable) set logging level spantree 0
Switch> (enable) set logging server facility local5
Switch> (enable) set logging console enable

• On the Syslog server (UNIX), the corresponding line


in Syslog.conf file is:
local5.emerg /var/log/spantree
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 31
Convert a Syslog Message
to a SNMP Notification?

• Why?
Not all error messages are supported via notifications
Syslog daemon not running in the NMS
Events correlation need
• Send a trap/inform from the CISCO-SYSLOG-MIB
when a new Syslog message is generated
• How to convert to a trap?
Router (config)# snmp-server enable traps syslog

• How to convert to an inform? Attention to the


<all> Keyword !!!

Router (config)# snmp-server host <x.x.x.x>


informs version 2c public syslog
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 32
Syslog Writing to Flash

• System error and debug messages saved on the


router’s CompactFlash disks (also known as ATA
Flash disks)
• Persistent across reboot
• Introduced in 12.0(26)S
Router(config)# logging buffered
Router(config)# logging persistent url
disk1/:syslog size 134217728 filesize 16384

Router# copy slot0:/syslog


ftp://myuser/mypass@192.21.1.129/syslog

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 33


Syslog Issue
Consistent Message Format

• Syslog isn’t consistently used across different


Cisco platforms and Cisco IOS versions
Example: environmental monitor initiated shutdown event
Cisco IOS 11.2 ENVM-1-SHUTDOWN
Cisco IOS 12.0 ENVM-0-SHUT

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 34


How to Find Out About Syslog Messages?

• ‘Cisco IOS Software System Error Messages’ per


Cisco IOS release
For Cisco IOS version 12.2:
http://www.cisco.com/en/US/partner/products/sw/iosswrel/ps1835/
products_system_message_guide_book09186a008009e73d.html
• ‘System message’ per Cisco switch, Cisco 6000 switch:
http://www.cisco.com/en/US/partner/products/hw/switches/
ps700/products_system_message_guide_chapter09186a00800f27
09.html
• Error Message Decoder
http://www.cisco.com/cgi-bin/Support/Errordecoder/home.pl
• Output Interpreter
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 35


XML Interface to Syslog Messages

• Enable Syslog messages to be sent in an Extensible Markup


Language (XML) format
• Logs in a standardized XML format can be more readily used
in external customized monitoring tools
• Tags are hard-coded
• Available in 12.2(15)T
• Configuration:

Router(config)#logging console xml


Router(config)#logging monitor xml 6
Router(config)#logging host 128.107.165.215 xml
Router(config)#logging host 171.69.1.129
Router(config)#logging buffered xml 10000

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 36


XML Interface to Syslog Messages
Events Comparison

000013: *Oct 11 14:52:10.039: %SYS-5-CONFIG_I:


Configured from console by vty0 (172.19.208.14)

<ios-log-msg>
<facility>SYS</facility>
<severity>5</severity>
<msg-id>CONFIG_I</msg-id>
<seq>000013</seq>
<time>*Oct 11 14:52:10.039</time>
<args>
<arg id="0">console</arg>
<arg id="1">vty0 (172.19.208.14)</arg>
</args>
</ios-log-msg>

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 37


Syslog Messages vs. SNMP Notifications

Syslog Notification
NMS Syslog Daemon Trap Receiver
Protocol/ UDP 514 UDP 162
Port
Filtering Yes Limited
Easy-to-Read Format, More Rigid Format,
Format No MIB Needed Parse Able
None (RFC 3195
None with Traps
Reliable Syslog)
Reliability Some with Informs
(Syslog Writing to
(NOTIFICATION-LOG MIB)
Flash)
Note: the Syslog message could be sent faster!
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 38
EMBEDDED SYSLOG MANAGER

SOME MORE FLEXIBILITY FOR THE SYSLOG MESSAGES

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 39


Embedded Syslog Manager (ESM)

• Post-process
ESM Syslog messages
Modules with selected ESM
filters (proactive
rules-based
analysis)
• User definable
scripting (TCL)
• New message
queue in parallel
with classic
Syslog
• Available in
images with TCL
Buffer Console tty Syslog Server 8.3.4, in 12.3(2)T,
12.2(25)S
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 40
Embedded Syslog Manager (ESM)
Configuration Example

Router(config)# logging filter <URL> [<position>]


[args <argstring>]

• “URL”, the location of the TCL script (Cisco IOS,


Flash, Web, TFTP server)
• “Position”, ordering of filters when multiple exist
• “Args”, arguments to the TCL script

Router(config)# logging console filtered


Router(config)# logging host <x.x.x.x> filtered [stream_id]

• The stream_ID is added by the script, for event


routing
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 41
Embedded Syslog Manager (ESM)
Example 1

• Severity escalation: messages that Cisco deemed


low priority may be very important to some
customers
• Example: escalate syslog messages that contain
the word ‘CONFIG_I’ to severity level of 4 (they are
by default level 5)

Router(config)# logging filter slot0:escalate.tcl args CONFIG_I 4

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 42


Embedded Syslog Manager (ESM)
Example 1
# Embedded Syslog Manager, Severity Escalation Module
# ===================================================================
# Usage: Set CLI Args to "mnemonic new_severity"
# Namespace: global
# Check for null message

if { [string length $::orig_msg] == 0} {


return ""
}

if { [info exists ::cli_args] } {


set args [split $::cli_args]
if { [ string compare -nocase [lindex $args 0] $::mnemonic ] == 0 } {
set ::severity [lindex $args 1]
set sev_index [ string first [lindex $args 0] $::orig_msg ]
if { $sev_index >= 2 } {
incr sev_index -2
return [string replace $::orig_msg $sev_index $sev_index \
[lindex $args 1]]
}
}
}
return $::orig_msg
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 43
Embedded Syslog Manager (ESM)
Example 2

• Message correlation: to help reduce the volume


of messages when certain well-known network
events occur, ESM can correlate local events,
and summarize them
• Example: link-flapping messages can be counted
over a period of time, and a single Syslog
message sent

00:22:11: %LINK-3-UPDOWN: serial1 flapping


(4 changes to up/4 changes to down between 00:21:09 and 00:22:11)

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 44


Embedded Syslog Manager (ESM)
Other Examples

• Message routing: categorize messages using


criteria other than facility or severity
Example: send all spanning tree messages to a
separate syslog server (setting a specific stream ID
in the TCL script)

• SMTP-based email alerts: capability for


notifications using TCP to external servers, such
as TCP-based Syslog collectors or Simple Mail
Transfer Protocol (SMTP) servers
Example: “configuration changes” sent to administrators
via an email message

• Your example…the possibilities are endless!


NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 45
RMON EVENT AND ALARM

SOMETIMES THE EXACT NOTIFICATION DOESN’T EXIST!


BUT THE SNMP OBJECTS TO TRIGGER THE NOTIFICATION
DO EXIST!

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 46


RMON Event and Alarm
1

Load RMON-MIB

High CPU!!

SNMP
NMS
Agent
2

Sent an Alarm when


3
CPU load > 80%
ALARM: “cpu too busy”

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 47


RMON Event and Alarm

• Allows proactive monitoring:


The device polls itself
• RMON-MIB used to configure SNMP notification:
Traps and informs NEW
Integer32, Counter32, Counter64, Gauge, or Timeticks
may be sampled
• Included in all Cisco IOS software images
Since Cisco IOS 11.1
CLI or SNMP configuration
• Included in all the switches images
Only SNMP configuration

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 48


How to Enable RMON Event and
Alarm via CLI?

• Configure RMON to generate a trap if CPU utilization reaches


80%, and rearm the trap if utilization drops below 40%,
sampling interval is 20 seconds
T (sec)
Router(config)#rmon alarm 1
cpmCPUTotalEntry.3.0 20 absolute
rising-threshold 80 1 falling-threshold
40 2 owner me Rising Condition Triggering Event #1
Triggering Event #2
Router(config)#rmon event 1 log Trap
public description "cpu busy" owner me
Router(config)#rmon event 2 log
description "cpu not too busy"
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 49
RMON Reaction Condition

Alert!! Alert!!
Threshold Threshold
Thresholds Violation
>80%
No Alert Violation
>80%

Rising 80%

Falling 40%

T Alert!!
Threshold
violation <40%

T: Internal Sampled Interval (20 sec)


NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 50
How to Enable RMON Event and
Alarm via SNMP?

Send a trap when the number of bytes going into interface with
ifIndex 12, during the last two minutes is above 140000000
snmpset -c private <router> eventStatus.123 integer 2
snmpset -c private <router> eventDescription.123 string “above 140000000“
snmpset -c private <router> eventType.123 integer 4
snmpset -c private <router> eventCommunity.123 string "public“
snmpset -c private <router> eventOwner.123 string "event_owner“
snmpset -c private <router> eventStatus.123 integer 1
snmpset -c private <router> alarmStatus.321 integer 4
snmpset -c private <router> alarmStatus.321 integer 2
snmpset -c private <router> alarmInterval.321 integer 120
snmpset -c private <router> alarmVariable.321 integer ifInOctets.12
Snmpset –c private <router> alarmSampleType.321 integer 1
snmpset -c private <router> alarmRisingThreshold.321 integer 140000000
snmpset -c private <router> alarmRisingEventIndex.321 integer 123
snmpset -c private <router> alarmOwner.321 string "alarm_owner"
snmpset -c private <router> alarmStatus.321 integer 1
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 51
Which MIB Variables to Monitor?

dot3StatsCarrierSenseErrors bufferFail
ciscoEnvMonTemperatureState
cpmCPUTotal5min

ifOutDiscards ciscoEnvMonFanState
bufferNoMem
locIfResets
locIfCollisions
locIfCollisions ifOperStatus
ciscoMemoryPoolFree
locIfInputQueueDrops
locIfCarTrans
locIfInCRC
bufferFail
locIfOutputQueueDrops

See the APPENDIX


NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 52
Fault Management
Which MIB Variables to Monitor?

Interface
Poll Thres-
Object Descr OID
Int hold
Number of Times the
locIfResets .1.3.6.1.4.1.9.2.2.1.1.17 15 Min
Interface Internally Reset
The Current Operational
State of the Interface; the
ifOperStatus Testing (3) State Indicates .1.3.6.1.2.1.2.2.1.8 5 Min != 1
That No Operational Packets
Can Be Passed
Number of Times
locIfCarTrans Interface Saw the Carrier .1.3.6.1.4.1.9.2.2.1.1.21 15 Min
Signal Transition

Number of Output Collisions


locIfCollisions 1.3.6.1.4.1.9.2.2.1.1.25 15 Min
Detected on This Interface

Number of Input Packets


locIfInCRC Which Had Cyclic Redundancy .1.3.6.1.4.1.9.2.2.1.1.12 15 Min
Checksum Errors

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 53


ifIndex and RMON Persistence

• ifIndex persistence
Router

router(conf) snmp-server ifindex persist


router(conf-if) snmp-server ifindex persist

Switch: ifIndex persistence by default

• RMON persistence
Router: event/alarm saved in the startup configuration
Switch: no event/alarm persistence

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 54


EVENT-MIB

SOMETIMES THE EXACT NOTIFICATION DOESN’T EXIST!


BUT THE SNMP OBJECTS TO TRIGGER THE
NOTIFICATION DO EXIST!

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 55


Event-MIB

• The EVENT MIB provides a superset of the


capabilities of the RMON alarm and event
• The EVENT MIB calls “triggers”
The RMON MIB calls “alarms,”
but the concepts are the same
• More flexible test types with the EVENT-MIB
Existence test: absent, present, changed
Boolean test: <>, =, <, <=, >, >=

• Event MIB proposed by Cisco to IETF DISMON


Working Group, accepted standard track RFC-2981

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 56


Event-MIB Advantages vs.
RMON Event and Alarm

• EVENT MIB can monitor


Any MIB object (existence)
Any integer/counter (Boolean, threshold)
• RMON MIB can only monitor
Integer/counter (threshold)
• EVENT-MIB allows wildcarding
• EVENT-MIB sends an SNMP notification in
response to a trigger (like RMON) but add the
concept of setting a MIB object (integers)
• EVENT-MIB can specify which variables to add to
the notification

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 57


Cisco IOS Support

• Event MIB Support in Cisco IOS release 12.1(3)T


and 12.0(12)S
• RFC 2981-compliant support is in Cisco IOS
release 12.2(4)T
http://www.cisco.com/en/US/products/sw/iosswrel/ps1839/
products_feature_guide09186a00800c391a.html

• Only configuration support via SNMP so far: no CLI


Scriptable Interface for adding command line support
in 12.3(7)T
However “show management event” exists
However “debug management event mib” exists

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 58


EVENT-MIB: Example 1
Wildcarding

• Wildcarding is a powerful functionality which


allows you to monitor multiple instances of
an object
• Can specify a single OID for monitoring, or use
wildcarding to specify a group of OIDs
• Example:
Monitor ifInOctets for all interfaces; the EVENT-MIB is
configured to monitor the delta values of ifInOctets for
all interfaces once per minute; if any of the samples
exceed the delta rising or falling triggers, a trap
notification will be sent

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 59


EVENT-MIB: Example 2
Add Variable to Notification

LinkDown!!
E0
SNMP S1 NMS
Agent

1 2

Look at the Status


Customized New
of ALL the Interface:
linkUP/Down Notification Sent
ifOperStatus.*
with Extra VARBINDS
(sysLocation, sysName… )

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 60


EVENT-MIB: Example 3
SNMP Set

1. The Trigger 2. The Event 3. The SNMPSet

TFTP
Server
S1
Change in
the config
mteTriggerValueID =>
ccmHistoryRunningLastChanged

mteTriggerBooleanComparison <> 0
=> change in the config

SNMP Set, using CONFIG-COPY-MIB


to copy the config to TFTP server

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 61


How to Enable the EVENT-MIB?

Step 1: Any MIB Object Type

• Each trigger is configured to watch a single object or a group


of objects specified by a wildcard
• The object-type can be any on of the types:
INTEGER_TYPE OCTET_PRIM_TYPE
NULL_TYPE OBJECT_ID_TYPE
SEQUENCE_TYPE INTEGER_32_TYPE
IP_ADDR_PRIM_TYPE COUNTER_TYPE
GAUGE_TYPE TIME_TICKS_TYPE
OPAQUE_PRIM_TYPE COUNTER_32_TYPE
GAUGE_32_TYPE UNSIGNED32_TYPE
COUNTER_64_TYPE
• However, the type of sampling dictates the types of objects
that can be monitored

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 63


How to Enable the EVENT-MIB?

Step 2: Possibility: Sampling Type


• The Event MIB process checks the state of this
watched object at predefined intervals
• The type of sampling that can be done on an
object is of two types:
Absolute
Delta

• Configurable observation interval

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 64


How to Enable the EVENT-MIB?

Step 3: Test Type and Parameters


• The test that can be done on the watched object
is one or a combination of the following:
Existence
Absent, Present, Changed
Boolean
Unequal, Equal, Less, LessOrEqual, Greater,
GreaterOrEqual
Threshold
Rising, Falling, Rising or Falling

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 65


How to Enable the EVENT-MIB?

Step 4: Actions
• This could be one or both of the following:
Notifications (Traps/Informs), with the possibility to
add extra Object IDs to the notification
SNMP set

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 66


How to Enable EVENT-MIB?
Define the Trigger Define Which Variable(s)
mteTriggerTable to Add to the Notification
INDEX: mteOwner, IMPLIED mteObjectsTable
mteTriggerName
INDEX: mteOwner,
mteTriggerObjects mteObjectsName,
mteTrigger*Event mteObjectsIndex
Define the Notification Define the Event
mteEventNotificationTable mteEventTable
INDEX: mteOwner, IMPLIED
mteEventName INDEX: mteOwner, IMPLIED
mteEventName
Define the SNMP Set mteEventAction
mteEventSetTable
INDEX: mteOwner, IMPLIED AND/OR
mteEventName
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 67
EVENT-MIB Feature
MIB Persistence

• Allows the MIB to be persistent across reloads, i.e.,


MIB information retains the same set object values
each time a networking device reboots

Router(config)# snmp mib persist [event]

• Write to NVRAM by using the “write mib-data“


• Any modified MIB data must be written to NVRAM
memory using the “write mib-data”

Router# write mib-data

• Added in 12.2(4)T3
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 68
Event MIB Summary

• If we want a trigger:
Threshold based,
On the local device (not remote),
Without wildcard,
With no extra objects in notification,
With no SNMP Set

• Then it is easier to use the RMON Event/Alarm


• Else the EVENT-MIB is your friend ☺

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 69


EXPRESSION-MIB

SOMETIMES THE DESIRED SNMP OBJECT DOESN’T EXIST


BUT CAN BE DERIVED FROM MULTIPLE OTHER OBJECTS

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 71


EXPRESSION-MIB

• Allows you to create new SNMP objects based upon existing


MIB variables and formulas
• Interesting when combined with the EVENT-MIB
• EXPRESSION MIB proposed by Cisco to IETF DISMON
Working Group, accepted standard track RFC-2982
Cisco implementation based on IETF draft, again in the DISMON
Working Group, and numbered in Cisco’s namespace
• Only configuration support via SNMP so far: no CLI
Scriptable interface for adding command line support
in 12.3(7)T
However “show management expression” exists
However “debug management expression mib” exists

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 72


EVENT-MIB and EXPRESSION-MIB
Example 1: Notification

• An access router would like to send a trap only for


the high-speed interface
• Router A sends a trap when Serial0 has:
BW>100Kbits & OperStatus=DOWN

• Steps:
Create an expression that will return “1” when
the condition is TRUE and “0” when FALSE
Expression
-MIB Exp1 = (ifSpeed > 100000) && (ifOperStatus == 2)
If Exp1 == “1” generates an event; this will be
checked every minute
Event-MIB

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 73


EVENT-MIB and EXPRESSION-MIB
Example 1: Notification
snmpset -v 2c -c private RouterA expNameStatus.101.49.101.120.112
integer 6
snmpset -v 2c -c private RouterA expNameStatus.101.49.101.120.112
integer 5 e1exp in ASCII
snmpset -v 2c -c private RouterA expExpressionIndex .101.49.101.120.112
gauge 1
snmpset -v 2c -c private RouterA expExpressionComment.1 octetstring "e1
expression“
snmpset -v 2c -c private RouterA expExpression.1 octetstring '$1 <
100000 && $2 == 2‘
snmpset -v 2c -c private RouterA expObjectID.1.1 objectidentifier
ifSpeed.16
snmpset -v 2c -c private RouterA expObjectID.1.2 objectidentifier
ifOperStatus.16
.16=ifIndex Serial0 Absolute(1)
snmpset -v 2c -c private RouterA expObjectSampleType.1.1 integer 1
snmpset -v 2c -c private RouterA expObjectSampleType.1.2 integer 1
snmpset -v 2c -c private RouterA expObjectStatus.1.1 integer 1
snmpset -v 2c -c private RouterA expObjectStatus.1.2 integer 1
snmpset -v 2c -c private RouterA expNameStatus.101.49.101.120.112
integer 1
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 74
EVENT-MIB and EXPRESSION-MIB
Example 1: Notification
#mteTriggername =
#N Characters for trigger1
the mteOwner mteOwner = tom

mteTriggerEntry Index=3.116.111.109.116.114.105.103.103.101.114.49 = Y
mteEventEntry Index= 3.116.111.109.101.118.101.110.116.49 = Z
snmpset -v 2c -c private RouterA mteTriggerEntryStatus.Y integer 6
#mteEventname
snmpset -v 2c -c private RouterA mteTriggerEntryStatus.Y integer 5 = event1
snmpset -v 2c -c private RouterA mteTriggerValueID.Y objectidentifier
1.3.6.1.4.1.9.10.22.1.4.1.1.2.1.0.0.0
snmpset -v 2c -c private RouterA mteTriggerValueIDWildcard.Y integer 2
snmpset -v 2c -c private RouterA mteTriggerTest.Y o "40“ Existance(0)
Boolean(1)
snmpset -v 2c -c private RouterA mteTriggerFrequency.Y gauge 60
snmpset -v 2c -c private RouterA mteTriggerSampleType.Y integer 1 Threshold(2)
snmpset -v 2c -c private RouterA mteTriggerEnabled.Y integer 1
snmpset -v 2c -c private RouterA mteEventEntryStatus.Z integer 6 Absolute (1)
snmpset -v 2c -c private RouterA mteEventEntryStatus.Z integer 5
snmpset -v 2c -c private RouterA mteEventActions.Z o “80” When Condition
Is met>send
Notification
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 75
EVENT-MIB and EXPRESSION-MIB
Example 1: Notification

snmpset -v 2c -c private RouterA mteTriggerBooleanValue.Y i 1


snmpset -v 2c -c private RouterA mteTriggerBooleanComparison.Y i 2
snmpset -v 2c -c private RouterA mteTriggerBooleanObjectsOwner.Y o “tom”
snmpset -v 2c -c private RouterA mteTriggerBooleanObjects.Y o “objects1”
snmpset -v 2c -c private RouterA mteTriggerBooleanEventOwner.Y o “tom”
snmpset -v 2c -c private RouterA mteTriggerBooleanEvent.Y o “event1”
Creating the ObjectTable Unequal(1)
snmpset -v 2c -c private RouterA mteObjectEntryStatus.Z.1 i 6 Equal(2)
snmpset -v 2c -c private RouterA mteObjectEntryStatus.Z.1 i 5 Less(3)…
snmpset -v 2c -c private RouterA mteObjectsID.Z o ifAdmin.13
snmpset -v 2c -c private RouterA mteObjectEntryStatus.Z.1 i 1
Attaching the object to the event:
snmpset -v 2c -c private RouterA mteEventNotificationObjectsOwner.Z o “tom”
snmpset -v 2c -c private RouterA mteEventNotificationObjects.Z o “objects1”
Activating the Trigger and the Event

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 76


EVENT-MIB and EXPRESSION-MIB
Example 2: Simple Capacity Planning

• If my link utilization is above 50% for an hour,


it’s time to upgrade the link
• Steps:
Create an expression
Expression- utilization = (ifInOctets + ifOutOctets) *800/hour/ifSpeed
MIB
If utilization is above 50% of the bandwidth after one hour,
generates an event
Event-MIB

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 77


EVENT-MIB and EXPRESSION-MIB
Example 3: Table Entry Count

• Sometimes there is no counter for the number of


table entries in the MIB definition
• Create an expression1 that will match all entries
• Create an expression2 that will sum expression1
• Other examples:
Number of Ethernet interfaces up
Number of entries in the CAM table
Number of static route in the routing table

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 78


ARP Table Entry Count
Show Command Example

Router# show management expression


Expression: e1exp is active
Expression to be evaluated is $1==3 where:
$1 = ipNetToMediaEntry.4
Object Condition is not set
Sample Type is absolute
ObjectID is wildcarded
Expression: e2exp is active
Expression to be evaluated is sum($1) where:
$1 = ciscoExperiment.22.1.4.1.1.4.1.0.0
Object Condition is not set
Sample Type is absolute
ObjectID is wildcarded

• This example specifies an expression e2exp that


sums up all the static ARP entries

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 79


Expression MIB Feature
MIB Persistence

• Allows the MIB to be persistent across reloads, i.e.,


MIB information retains the same set object values
each time a networking device reboots

Router(config)# snmp mib persist [expression]

• Any modified MIB data must be written to NVRAM


memory using the “write mib-data”

Router# write mib-data

• Added in 12.2(4)T3

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 80


EVENT-MIB and EXPRESSION-MIB
Summary

• Very flexible and useful MIBs


• Not that easy to set up
• You should work from existing examples
Drop me an email

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 81


SPECIFIC MIBS AND SCENARIOS

SPECIFIC INTERFACES,
MPLS/VPN SYSLOG & SNMP NOTIFICATION,
IP SLA & SNMP,
ENHANCED OBJECT TRACKING

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 82


Disabling the Logging of Some Interfaces

• Limit the amount of output that is logged from the


group-async interface and ISDN D channels
Router(config)# interface Group-Async 1
Router(config-if)# no logging event link-status
Router(config-if)# no snmp trap link-status

• Depending on the layering…


Router(config)# snmp ifmib trap throttle
Router(config-if)#no logging event subif-link-status

• Depending on the encapsulation…


Router(config-if)#no logging event dlci-status-change

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 83


VRF Aware Notifications

VPN_A VPN_A
CE CE
VPN_B VPN_A
CE P P
PE PE CE
VPN_A PE
CE PE
P P
VPN_B
VPN_B CE CE

Notifications sent to a receiver in a VRF


Router(config)#snmp-server host
<receiver-ip-addr> vrf yellow public

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 84


VRF Aware Syslog

VPN_A VPN_A
CE CE
VPN_B VPN_A
CE P P
PE PE CE
VPN_A PE
CE PE
P P
VPN_B
VPN_B CE CE

• Syslog messages sent to a server in a VRF


• New in 12.2(24)S
Router(config)#logging host vrf <yellow>
<syslog-ip-address>
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 85
Monitoring Service
IP SLA

Management
Application

Configure IP Host
Collect Data
SNMP Trap ure
as Trigger Other Operations
Reconfigure e
M Based on Thresholds/Timeouts
Source Target
Measure

Measure Performance

IP SLA, Previously Service Assurance Agent


NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 87
Monitoring Service with IP SLA
VoIP Example

ip sla 11
udp-jitter 198.198.198.1 3000 codec g711alaw

ip sla reaction-configuration 11 react connectionLoss


threshold-type immediate action-type trapOnly
ip sla reaction-configuration 11 react jitterDSAvg
threshold-value 10 5 threshold-type immediate action-
type trapOnly
ip sla reaction-configuration 11 react jitterSDAvg
threshold-value 10 5 threshold-type immediate action-
type trapOnly
ip sla reaction-configuration 11 react mos threshold-
value 390 220 threshold-type immediate action-type
trapOnly

ip sla schedule 11 start-time now

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 88


Enhanced Objects Tracking for IP SLA

• The Enhanced Object Tracking feature separates


the tracking mechanism from the protocol and
creates a separate standalone tracking process
that can be used by any other process
• Subset of the Enhanced Object Tracking Cisco IOS
feature:
Track the output from the IP SLA objects and use the
provided information to trigger an action

• Aspects of an IPSLA operations which can be


tracked: state and reachability
• Introduced in 12.3(4)T and 12.2(25)S

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 89


Example: HSRP and IPSLA Tracking

Server

Internet

IP SLA ISP 2
ISP 1

Router 1 Router 2
.1 HSRP: 10.10.10.10 .2

10.10.7.1
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 90
Example: HSRP and IP SLA Tracking
Router1(config)#
ip sla 18
icmp-echo <server>
ip sla schedule 18 start-time now life forever
track 100 rtr 18 state
interface FastEthernet0/0
ip address 10.10.10.1 255.255.255.224
standby 1 ip 10.10.10.10
standby 1 priority 105
standby 1 preempt
standby 1 track 100 decrement 10

Without Enhanced Object Tracking:


“Standby 1 track serial 0”
With Enhance Object Tracking:
The Object 100 Is Tracked

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 91


Example 2: Injecting Routes and IP SLA

User1 Don’t Advertise the


Route Anymore
Internet

ISP 1 BGP BGP ISP 2


IP SLA
Customer Customer Customer
Router 1 Network Router 2

X X
Server
Traffic “Black Holed”

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 92


Example 2:
Injecting Routes Into Routing Tables

Router1(config)#
ip sla 1
icmp-echo <server>
ip sla schedule 1 start-time now life forever

track 123 rtr 1 reachability


ip route <server_network> 255.255.255.0 Null0 track 123
(more specific routes will be used to forward packets)
router bgp 65505
redistribute static

The static route, advertised by BGP, will only


exit if the reachability to the server Is OK!

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 93


EMBEDDED EVENT MANAGER

THE NEWEST IN CISCO IOS

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 94


Embedded Event Manager (EEM)

• In-box monitoring of different components of


the system via a set of software agents (event detectors)
• Event detectors (ED) notify EEM when an event
of interest occurs; based on this, an action can
be taken
• Advantages:
Local programmable actions, triggered by specific events
• Version 1.0 introduced in 12.0(26)S, 12.3(4)T
• Version 2.0 introduced in 12.2(25)S
• Version 2.1 introduced in 12.3(14)T
• Version 2.2 introduced in 12.4(2)T

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 95


Embedded Event Manager (EEM) 1.0
The Framework

Syslog SNMP Other


Event Notifications Events

Syslog ED SNMP ED Future EDs…

Event Detectors

EEM Policies
Embedded Event Actions
Manager 1.0

Syslog

Switch Over (*)

Reload (*)
(*) Initially Developed for High Availability

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 96


Embedded Event Manager 1.0
Example 1: Syslog ED

• Applets are groupings of an ‘event specification’


and a policy action that is taken when the specified
event occurs

event manager applet fe0trans


event syslog pattern .*UPDOWN.*FastEthernet0/0.*
action 1.0 syslog priority emergencies msg “New
syslog $_syslog_msg“

• Example: causes an emergency-level Syslog


message when a log message indicates that the
FastEthernet0/0 port changed state to either up
or down
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 98
Embedded Event Manager 1.0
Example 2: Notification ED

ciscoMemoryPoolFree

event manager applet memory-demo


event snmp oid 1.3.6.1.4.1.9.9.48.1.1.1.6.1 get-
type exact entry-op lt entry-val 512000 poll-
interval 10
action 1.0 syslog priority critical msg "Memory exh
austed; current available memory is $_snmp_oid_val
bytes"
action 2.0 force-switchover

• Example: the applet will run when the available


memory on the primary RP falls below the specified
threshold of 512000 bytes

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 99


Embedded Event Manager 1.0
Environment Variables

• These environment variables can be used in ‘msg’ text


• Will be replaced with the relevant text
• Environment variable available for all events
$_event_type The event type that triggered the event
$_event_pub_time The time at which the event type was published
• Environment variable available for SNMP events
$_snmp_oid The SNMP object OID that caused the event to be
published
$_snmp_oid_val The SNMP object ID value when the event was published
• Environment variable available for Syslog events
$_syslog_msg The syslog message that caused the event to be
published
• A lot more environment variables in the version 2.0, 2.1, and 2.2
check the documentation

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 100


Embedded Event Manager 2.0
The Framework
Process Interface
Syslog SNMP Scheduler Descriptor
Event Notifications Database Blocks
Application
Interface Timer Counter Specific
Syslog ED SNMP ED Watchdog ED Event ED
Counter ED Event ED Event ED
Event Detectors

EEM Policies
Embedded Event
Actions
Manager 2.0

Syslog

Cisco TCL Scripts

Modify Counter

Application Specific Event


(*) Initially Developed for High Availability SNMP Notifications Switch Over
or Reload (*)
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 101
Embedded Event Manager 2.0
Example 3: Watchdog ED

event manager applet IOSWD_SNMPENGINE


event ioswdsysmon sub1 cpu-proc taskname "SNMP
ENGINE" op ge val 20 period 10
action 2.0 snmp-trap intdata1 20 strdata "SNMP
Engine above 20%"
snmp-server enable traps event-manager

• Watchdog event detector example:


Monitor the “SNMP engine” process from “show processes
cpu”; sent a trap if the CPU is above 20% for 10 seconds

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 102


Embedded Event Manager 2.1
The Framework
Process Interface
Syslog SNMP Scheduler Descriptor
Event Notification Database Blocks
Application
CLI OIR
Interface Timer Counter Specific
Syslog ED SNMP ED Watchdog ED ED ED
Counter ED Event ED Event ED Event ED
Event Detectors

EEM Policies
Embedded Event
Manager 2.0 Actions

Syslog

User TCL Scripts

Modify Counter
Short Email notification
Application Specific Event
CLI command
SNMP Notifications Switch Over
Get info or Reload (*)
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 103
Embedded Event Manager 2.1
Example 4: CLI ED

event manager applet cli-match


event cli pattern "router bgp 1" sync no skip no occurs 1
action 1.0 syslog priority critical msg "$_cli_msg /
configured at $_event_pub_time"

Router(config)#router bgp 1
Router(config-router)#
*Nov 22 12:05:59.047: %HA_EM-2-LOG: cli-match: router bgp 1
configured at Nov 22 12:05:59.047

• Command Line Interface (CLI) event detector


example:
When the “router bgp 1” CLI command is entered, a syslog
message is sent

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 104


Embedded Event Manager 2.2
The Framework
Interface Event manager run
Process OIR
Syslog Descriptor CLI command
SNMP Scheduler Event
Event Blocks
Notification Database
Appl.
Interface Timer Counter Spec. CLI None OIR
Syslog Watchdog Event Event EOT ERM RF
Counter Event ED ED ED
ED SNMP ED ED ED ED ED ED ED
ED ED
Event Detectors

applet
EEM Policies
Embedded Event script
Manager 2.2 Actions

Syslog

User TCL Scripts

Modify Counter Read/Set


Short Email notification a tracked object
Application Specific Event
CLI command
SNMP Notifications Switch Over
Get info or Reload (*)
NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 105
Embedded Event Manager 2.2

• Amongst others, Enhanced Object Tracking Event


Detector
• Advantages:
IP SLA state/reachability is supported by EOT
EOT now supported by EEM
EEM has got an implicit IP SLA Event Detector

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 106


Embedded Event Manager
Advantages

• More and more event detectors


• Automated local action triggered by events
TCL: scripting for Cisco written policies
Control is in the customer’s hands: full customization
• Customized notifications, syslog, and email
Environment variables, redefined syslog priority level, etc.
• Can define your own applications in the router
• Some examples in the documentation
• My personal view:
EOT used for object tracking
+ EEM for the action/fault management
+ TCL for the customized action/fault management
= the perfect framework for any policy-based management

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 107


SUMMARY

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 108


Principles of Fault Management
in Cisco Devices

• Quick fault detections is strategic to network


management
• Systematic network element polling doesn’t
always scale
• Let’s put some more NMS intelligence into the
network elements
• Let’s tune the right fault management events from
the network elements themselves
• We investigated a few ways
• Then potentially event-based polling…

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 109


Other Network Management Sessions

• NMS-A02 Advanced Network Performance measurement with Cisco


IOS IPSLA
• NMS-A03 Advanced NetFlow Usage
• NMS-A04 Advanced IOS Management Tools
• NMS-A05 Managing the MPLS core and MPLS VPNs
• NMS-A06 Operating your MPLS Core and IP VPNs
• NMS-D01 Zero Touch Provisioning and Configuration Management
• NMS-D02 Performance Measurement with Cisco Devices
• NMS-D03 Securely Managing Your Network and SNMPv3
• NMS-D04 Cisco Accounting Techniques
• NMS-D05 Management of Cisco IOS-XR platforms
• NMS-D06 Management of IP Telephony (for Enterprises)
• NMS-T01 CiscoWorks LMS 2.5 - A Practical Guide
• NMS-T02 MPLS Operations and Management: A Practical Guide

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 110


Recommended Reading

• Continue your
Networkers learning
experience with further
reading from Cisco
Press.
• Visit the on-site Cisco
company store, where
the full range of Cisco
Press books is available
for you to browse.

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 111


Management & Operations

• Benoit Claise – Distinguished Service Engineer

• Bruno Klaser – Consulting Systems Engineer

• David Melton – Systems Engineer

• Monique Morrow – Distinguished Consulting Engineer

• Emmanuel Tychon – Software Engineer

• Anders Viden – Product Marketing Manager

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 112


Q and A

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 113


NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 114
APPENDIX

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 115


Fault Management
Which MIB Variables to Monitor?

Interface
Threshol
Object Descr OID Poll Int
d

Number of Times the


locIfResets .1.3.6.1.4.1.9.2.2.1.1.17 15 Min
Interface Internally Reset

The Current Operational State


of the Interface; the Testing(3) >= 1
ifOperStatus .1.3.6.1.2.1.2.2.1.8 5 Min
State Indicates that No Operational
Packets Can Be Passed

Number of Times Interface Saw


locIfCarTrans .1.3.6.1.4.1.9.2.2.1.1.21 15 Min
the Carrier Signal Transition

Number of Output Collisions


locIfCollisions 1.3.6.1.4.1.9.2.2.1.1.25 15 Min
Detected on this Interface

Number of Input Packets Which 15 Min


locIfInCRC had Cyclic Redundancy .1.3.6.1.4.1.9.2.2.1.1.12
Checksum Errors

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 116


Fault Management
Which MIB Variables to Monitor?

Interface
Object Descr OID Poll Int Threshold

The Total Number of Octets


ifOutOctets Transmitted out of the Interface, .1.3.6.1.2.1.2.2.1.16 30 Min
Including Framing Characters
> 1% of
locIfInputQueue The Number of Packets Dropped
.1.3.6.1.4.1.9.2.2.1.1.26 30 Min Incoming
Drops Because the Input Queue Was Full
Traffic
The Number of Packets Dropped > 10% of
locIfOutputQueue
Because the Output Queue .1.3.6.1.4.1.9.2.2.1.1.27 30 Min Outgoing
Drops
Was Full Traffic
The Number of Inbound Packets
Which Were Chosen to Be
Discarded even Though No Errors
Had Been Detected to Prevent Their
ifInDiscards .1.3.6.1.2.1.2.2.1.13
Being Deliverable to a Higher-Layer
Protocol; One Possible Reason for
Discarding Such a Packet Could be
to Free up Buffer Space

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 117


Fault Management
Which MIB Variables to Monitor?

Ethernet
Object Descr OID Poll Int Threshold

dot3StatsCarrier Number of Times that the Carrier


SenseErrors Sense Condition Was Lost or Never
.1.3.6.1.2.1.10.7.2.1.11 15 Min >= 2
Asserted When Attempting to
Transmit a Frame
A Count of Frames for Which the
First Transmission Attempt on a
dot3StatsDeferred
Particular Interface Is Delayed .1.3.6.1.2.1.10.7.2.1.7 15 Min
Transmissions
Because the
Medium Is Busy
Count of Frames for Which 0.2% of
dot3StatsExcessive Traffic
Transmission Failed Because of .1.3.6.1.2.1.10.7.2.1.9 15 Min
Collisions
Excessive Collisions
1% of
Count of Frames for Which Incoming
dot3StatsInternalMac
Reception Fails Because of an ..1.3.6.1.2.1.10.7.2.1.16 15 Min Traffic
ReceiveErrors
Internal MAC Sublayer Receive Error

1% of
Count of Frames for Which 15 Min Outgoing
dot3StatsInternalMac
Transmission Fails Because of an .1.3.6.1.2.1.10.7.2.1.10 Traffic
TransmitErrors
Internal MAC Sublayer Transmit Error

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 118


Fault Management
Which MIB Variables to Monitor?

Memory
Object Descr OID Poll Int Threshold

Number of Buffer
bufferFail .1.3.6.1.4.1.9.2.1.46 15 Min
Allocation Failures

Number of Buffer Create >= 1


bufferNoMem .1.3.6.1.4.1.9.2.1.47 15 Min
Failures Due to No Free Memory

Number of Bytes from the


ciscoMemoryPool
Memory Pool that Are Currently 1.3.6.1.4.1.9.9.48.1.1.1.6 30 Min
Free
Unused on the Managed Device

ciscoMemoryPool Largest Number of


Contiguous Bytes from the .1.3.6.1.4.1.9.9.48.1.1.1.7 30 Min
LargestFree Memory Pool Currently Unused

ciscoMemoryPool Number of Bytes from 30 Min


the Memory Pool that .1.3.6.1.4.1.9.9.48.1.1.1.5
Used Are Currently in Use

ciscoMemoryPool Number of Bytes from the


Memory Pool that Are Currently .1.3.6.1.4.1.9.9.48.1.1.1.6 15 Min
Free Unused on the Managed Device

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 119


Fault Management
Which MIB Variables to Monitor?

Environment
Object Descr OID Poll Int Threshold

The Current State


ciscoEnvMon
of the Fan Being .1.3.6.1.4.1.9.9.13.1.4.1.3 15 Min >= 1
FanState
Instrumented
The Current State of
ciscoEnvMon >= 1
the Power Supply .1.3.6.1.4.1.9.9.13.1.5.1.3 15 Min
SupplyState
Being Instrumented
The Current State of
ciscoEnvMon
the Testpoint Being .1.3.6.1.4.1.9.9.13.1.3.1.6 15 Min != 1
TemperatureState
Instrumented
The Current State of
ciscoEnvMon
the Testpoint Being .1.3.6.1.4.1.9.9.13.1.2.1.7 15 Min != 1
VoltageState
Instrumented

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 120


Fault Management
Which MIB Variables to Monitor?

Miscellaneous
Object Descr OID Poll Int Threshold

Overall CPU Busy


Percentage in the Last
cpmCPUTotal 5 Min Period; this Object
.1.3.6.1.4.1.9.9.109.1.1.1.1.5X 5 Min < 30000
5min Deprecates the avgBusy5
Object from the OLD-
CISCO-SYSTEM-MIB
System Uptime in
sysUpTime .1.3.6.1.2.1.1.3 5 Min < 30000
1/100ths of Seconds

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 121


Recommended Reading

• Continue your
Networkers learning
experience with further
reading from Cisco
Press.
• Visit the on-site Cisco
company store, where
the full range of Cisco
Press books is available
for you to browse.

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 122


Management & Operations

• Benoit Claise – Distinguished Service Engineer

• Bruno Klauser – Consulting Systems Engineer

• David Melton – Systems Engineer

• Monique Morrow – Distinguished Consulting Engineer

• Emmanuel Tychon – Software Engineer

• Anders Viden – Product Marketing Manager

NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 123


NMS-A01 © 2005 Cisco Systems, Inc. All rights reserved. 124

You might also like