BRKNMS-1942 0101

Monitoring and Trending Application Performance in Data Center Services
BRKNMS-1942
BRKNMS-1942_c1
2009 Cisco Systems, Inc. All rights reserved.
Cisco Public
Challenges for Data Center Application Service Management

Overall health of network? How to measure performance degradation? Network deficiencies? Network performance? Distributed architecture Systematic problem identification is a challenge
BRKNMS-1942_c1
Cisco Public
Agenda
Data Center Application Network Management (DC), a Challenge DC Application Network Management, a Solution Service Flows and Key Performance Indicators (KPIs) Some Key DC Protocol Overview Network Management, Performance Counters Explained A Service Flow CASE Study and a Tool Implementation, Deep Dive Virtualization KPIs Service Flow, and Application Network Management Layering Concept Network Element Monitoring through EMS and NE Some Application Profiling Services Summary
BRKNMS-1942_c1 2009 Cisco Systems, Inc. All rights reserved. Cisco Public
What Is Distributed Data Center?

www.app.com
Client
GSS
Data Center 1
Data Center 2
DCCore
Si
Si
DCCore
Si
Si
Aggregation
Aggregation
Access
Access
BRKNMS-1942_c1
Cisco Public
Data Center Network Management Problem and Value Proposition

Situation Data Center (DC) growth has been phenomenal There are many desperate systems( with individual components) in the DC A system, can be a load balancing flow, SAN data flow, a security flow or a WAAS flow and so on
Complication Therearenoturnkeynetworkmanagementoftheentiresystem(wholeview)isabiggap Thereisnoeffectivewaytrackingtheentiresystem(orflow)currently
Implications Customercannoteffectivelydocapacityplanning,troubleshooting,developanykindof SLAs Theresult,customerhasdowntime
WhichResultsin LossofRevenue
5
BRKNMS-1942_c1
Cisco Public
Data Center Network Management Problem and Value Proposition (Cont.)

Action So what do we do? We tackle the problem by identifying the service flow through key performance indicators( KPIs ) in the system
Plan WhataretheseKPIs?Theseareperformancecounterscollected onaperiodicbasisfromeach flowsegmentinasystem PseudorealtimeRoundRobinDatabases(RRD)arecreated withthesecounters DashBoards arecreatedfromtheseRRDs,foreach datacenter

A DC Service Flow with Key Performance Indicators (KPIs)
BRKNMS-1942_c1
Cisco Public
Service Flow and Key Performance Indicators

What Is a Service Flow? It can be DNS service, a HTTP service, SMTP service, a SIP service, a ftp service and so on The flow of this service is basically the specific IP packets traversing the network elements that are carrying the service The best way to track this flow is through key performance indicators along the different network element hops in the service flow
BRKNMS-1942_c1
Cisco Public
Service Flow and Key Performance Indicators (Cont.)

The KPIs could very much vary for different data center environments As an example DNS service flow traversing the network would constitute of request/answer performance counters Followed by the traffic traversal performance counters related to the answer( the VIP IP) from the client and the respective server response
BRKNMS-1942_c1
Cisco Public
KPIs Together Represent the Application Service Flows in a Data CenterDNS Example
ServiceFlowSegments KeyPerformanceIndicators
DNS Answer Total Hits DNS Individual Answer Hits L2 Input/Output Unicast L3 Input/Output Unicast Load Balancer Global Input/Output VLAN Stats Load Balancer Individual Service/VIP Client Packet Stats DNS Domain Total Hits DNS Individual Answer Hits L2 Input/Output Multicast L3 Input/Output Multicast Load Balancer Global L7-L4 Performance Stats Load Balancer Individual Service/VIP Server Packet Stats
GSSStats
CoreSwitch MSFCVLANStats LoadBalancerStats LoadBalancer VIPStats
Firewall Stats SSLOffload
BRKNMS-1942_c1
Cisco Public
10
A LB Service/Performance Counter Flow for a HTTP Service

ServiceVlanInput/Output GSLB Stats Client MSFC
QueryforDOMAIN www.kalap.com #shstatisticsdnsdomain DomainTotalHits testkal.cisco.com5 kalap.test.com3 GlobalStats #shstatisticsdnsglobal DnsQueriesRcvd=56 DnsHostAddrQueriesRcvd=56 DnsResponsesSent=56 DnsResponsesNoError=6 DnsResponsesErrors=50 DnsServfailRCode=6 DnsNxdomainRCode=44 DnsNotimpRCode=0 DnsRefusedRCode=0 DnsQueriesUnmatched=44
L3Switch Stats
VlanId:332 L2UnicastPackets:304853 L2UnicastOctets:28373494 L3InputUnicastPackets:34934 L3InputUnicastOctets:3524869 L3OutputUnicastPackets:490221 L3OutputUnicastOctets:38803286 L3OutputMulticastPackets:0 L3OutputMulticastOctets:0 L3InputMulticastPackets:0 L3InputMulticastOctets:0 L2MulticastPackets:407863
GSS
ServiceVlanInput/Output LB Stats LB
Policymap:L4_POLICY Status:ACTIVE Interface:vlan332 servicepolicy:L4_POLICY class:L4_CLASS nat: natdynamic1vlan2332 currconns:0,hitcount:220 droppedconns:0 clientpktcount:1032,clientbytecount:1468317 serverpktcount:360,serverbytecount:498309 maxconnlimit:0,dropcount:0 connratelimit:0,dropcount:0 bandwidthratelimit:0,dropcount:0 loadbalance: L7loadbalancepolicy:SSLID_32_POLICY VIPRouteMetric:77 VIPRouteAdvertise:ENABLEDWHENACTIVE VIPICMPReply:ENABLEDWHENACTIVE VIPState:INSERVICE currconns:0,hitcount:331 droppedconns:21 clientpktcount:1295,clientbytecount:1673649 serverpktcount:360,serverbytecount:498309 maxconnlimit:0,dropcount:0 connratelimit:0,dropcount:0 bandwidthratelimit:0,dropcount:0
++ + ServiceConnectionstatistics + TotalConnectionsCreated:1696043 TotalConnectionsCurrent:10 TotalConnectionsDestroyed:542 TotalConnectionsTimedout:1614174 TotalConnectionsFailed:81317
Access Layer
++ + HTTPstatistics+ ++ LBparseresultmsgssent:347,TCPdatamsgssent :193 Inspectparseresultmsgs:0,SSLdatamsgssent :0 sent TCPfin/rstmsgssent:32,Bouncedfin/rstmsgssent:0 SSLfin/rstmsgssent:0,Unproxymsgssent :46 Drainmsgssent:0,Particlesread :338 Reusemsgssent:0,HTTPrequests :301 Reproxiedrequests:0,Headersremoved :0 Headersinserted:0,HTTPredirects :0 HTTPchunks:0,Pipelinedrequests :0 HTTPunproxyconns:31,Pipelineflushes :0
WebServer
11
Example KPIs Explained
BRKNMS-1942_c1
Cisco Public
12
DNS Rates and Hit Counts at CLI

GSS-Pod1.bxb.com>#
show statistics dns answer Answer Type Total Hits 1-Min 5-Min 30-Min 4-Hr ----------------------------------------------------------------192.168.200.1 VIP 0 0 0 0 0 192.168.150.1 VIP 0 0 0 0 0 192.168.50.41 VIP 0 0 0 0 0 10.86.209.232 VIP 0 0 0 0 0
Raw DNS Answer Stats
GSS-Pod1.bxb.com>#
show statistics dns domain Domain Total Hits 1-Min 5-Min 30-Min 4-Hr ---------------------------------------------------------------www.bxb.com 0 0 0 0 0 www.rtp.com 0 0 0 0 0
Raw DNS Domain Stats
BRKNMS-1942_c1
Cisco Public
13
MSFC, Over All Stats and VLAN Stats KPIs

cce02swdclb1015-da01#sh vlan counters
* Multicast counters include broadcast packets Vlan Id L2 Unicast Packets L2 Unicast Octets L3 Input Unicast Packets L3 Input Unicast Octets L3 Output Unicast Packets L3 Output Unicast Octets L3 Output Multicast Packets L3 Output Multicast Octets L3 Input Multicast Packets L3 Input Multicast Octets L2 Multicast Packets L2 Multicast Octets : 250 : 94262 : 6144108 :0 :0 :0 :0 : 40 : 2638 :0 :0 : 31949864227 : 2099855890392 Vlan Id L2 Unicast Packets L2 Unicast Octets L3 Input Unicast Packets L3 Input Unicast Octets L3 Output Unicast Packets L3 Output Unicast Octets L3 Output Multicast Packets L3 Output Multicast Octets L3 Input Multicast Packets L3 Input Multicast Octets L2 Multicast Packets L2 Multicast Octets : 2250 : 2253 : 146172 :0 :0 :0 L3 Specific :0 VLAN Stats : 124 : 8178 :0 :0 : 31928071021 : 2098462266576
L2 Specific Stats
BRKNMS-1942_c1
Cisco Public
14
ACE Load Balancer StatsKPIs

cce02swdclb1015-da01-ace1s7/SLB1# sh +------------------------------------------+ +------- Connection statistics ------------+ +------------------------------------------+ Total Connections Created : 37260 Total Connections Current : 0 Total Connections Destroyed: 0 Total Connections Timed-out: 14 Total Connections Failed : 37246 +------------------------------------------+ +-------------- HTTP statistics -----------+ +------------------------------------------+ LB parse result msgs sent : 0 Inspect parse result msgs : 0 sent TCP fin/rst msgs sent SSL fin/rst msgs sent Drain msgs sent Reuse msgs sent Reproxied requests Headers inserted HTTP chunks HTTP unproxy conns Whitespace appends Header insert errors Static parse errors Invalid path errors :0 :0 :0 :0 :0 :0 :0 :0 :0 :0 :0 :0 , Bounced fin/rst msgs sent: 0 , Unproxy msgs sent , Particles read , HTTP requests , Headers removed , HTTP redirects , Pipelined requests , Pipeline flushes , Second pass parsing , Analysis errors , Max parselen errors , Resource errors :0 :0 :0 :0 :0 :0 :0 :0 :0 :0 :0 , TCP data msgs sent , SSL data msgs sent :0 :0
stats
Load cce02swdclb1015-da01-ace1s7/SLB1# sh Balancer stats Global Stats

+------------------------------------------+ +------- Loadbalance statistics -----------+ +------------------------------------------+ Total version mismatch Total Layer4 decisions Total Layer4 rejections Total Layer7 decisions Total Layer7 rejections Total Layer4 LB policy misses Total Layer7 LB policy misses Total ACL denied :0 :0 :0 :0 :0 :0 :0 :0
Load Balancer App. Stats

sh service-policy
cce02swdclb1017-da03-ace1s7/SLB1#
Total times rserver was unavailable : 0 +------------------------------------------+ +----------- Sticky statistics ------------+ +------------------------------------------+ Total sticky entries reused prior to expiry Total active sticky entries Total active sticky conns Total static sticky entries :0 :0 :0 :0
Response entries recycled : 0
, Bad HTTP version errors : 0
Policy-map : G4W05423OLATPITG_MM_2382 Status : ACTIVE ----------------------------------------Context Global Policy: service-policy: G4W05423OLATPITG_MM_2382 class: G4W05423OLATPITG_2382 loadbalance: L7 loadbalance policy: G4W05423OLATPITG_2382 VIP Route Metric : 77 VIP Route Advertise : DISABLED VIP ICMP Reply : ENABLED VIP state: OUTOFSERVICE :0 , hit count : 26 curr conns dropped conns : 15 client pkt count : 73 , client byte count: 11413 , server byte count: 10604 server pkt count : 48
+------------------------------------------+ +--------- HTTP Inspect statistics --------+ +------------------------------------------+ Total request/response : 0 Total allow decisions Total drop decisions :0 :0
Total logging decisions : 0

15
Some Relevant Protocols Overview
BRKNMS-1942_c1
Cisco Public
16
OSI Layers
BRKNMS-1942_c1
Cisco Public
17
Relevant Protocols
IP Protocol Stack
BRKNMS-1942_c1
Cisco Public
18
Relevant Protocols
IP Header Fields
BRKNMS-1942_c1
Cisco Public
19
Relevant Protocols
UDP: User Datagram Protocol TCP: Transmission Control Protocol DNS: Domain Name System HTTP: Hypertext Transfer Protocol SSL: Secure Sockets Layer SIP
BRKNMS-1942_c1
Cisco Public
20
DNS
Applications, like browsers, connect to servers using server names
The operating system resolver contacts the configured DNS server to get the IP address Applications use the address provided by the resolver When multiple addresses are provided, applications can behave differently: use first IP, use random IP, use first IP, and move to the next one if unsuccessful
21
BRKNMS-1942_c1
Cisco Public
DNS
Client Local DNS Server
User Datagram Protocol, Src Port: 1302 (1302), Dst Port: domain (53) Domain Name System (query) Transaction ID: 0x002a Flags: 0x0100 (Standard query) 0... .... .... .... = Response: Message is a query .000 0... .... .... = Opcode: Standard query (0) .... ..0. .... .... = Truncated: Message is not truncated .... ...1 .... .... = Recursion desired: Do query recursively .... .... ...0 .... = Non-authenticated data is unacceptable Questions: 1 Answer RRs: 0 Authority RRs: 0 Additional RRs: 0 Queries www.cisco.com: type A, class inet Name: www.cisco.com Type: Host address Class: inet
22
DNS
Client Local DNS Server
User Datagram Protocol, Src Port: domain (53), Dst Port: 1302 (1302) Domain Name System (response) Transaction ID: 0x002a Flags: 0x8580 (Standard query response, No error) Questions: 1 Answer RRs: 1 Authority RRs: 2 Additional RRs: 2 Queries <--snipped--> Answers www.cisco.com: type A, class inet, addr 198.133.219.25 Name: www.cisco.com Type: Host address Class: inet Time to live: 1 day Data length: 4 Addr: 192.168.1.1 Authoritative nameservers <--snipped--> Additional records <--snipped-->
23
HTTP
"http:" "//" host [":"port]/[abs_path["?"query]] TCP Port TCP Port DNS Resolution DNS Resolution
Path and File Name / Path and File Name/
Scheme Scheme
Additional Information Additional Information
BRKNMS-1942_c1
Cisco Public
24
HTTP 1.0
Client Web Server
SYN SYN_ACK ACK GET/HTTP 1.0 ACK HTTP/1.0 200 OK Continuation ACK FIN_ACK FIN_ACK ACK
25
HTTP: Cookies
www.cookiecentral.com
Server sends cookie to client:

Set-Cookie:NAME=VALUE;expires=DATE;path=PATH; domain=DOMAIN_NAME; secure=YES
Client sends cookie back to the server on subsequent visits

GET / HTTP/1.1\r\n Host: DOMAIN_NAME\r\n Cookie: NAME=VALUE;
26
SIP Overview
SIP Client Server
Call Setup
INVITE 180 Ringing 200 OK ACK Call Dialog (not Part of SIP Exchange) BYE 200 OK
BRKNMS-1942_c1
Cisco Public
27
SIP Methods
Consists of requests and responses Requests (unless mentioned, each has a response)
REGISTER (UA registers with registration server) INVITE (request from a UA to initiate a call) ACK (confirms receipt of a final response to INVITE) BYE (sent by either side to end a call) CANCEL (sent to end a call not yet connected) OPTIONS (sent to query capabilities)
Messages contain SIP headers and body; body might be SDP or an attachment or some other application
28
BRKNMS-1942_c1
Cisco Public
Network Management Performance Counters
BRKNMS-1942_c1
Cisco Public
29
Network Management and Performance Counters

One of the key network management aspects is monitoring of performance counters or performance pegs Performance counter collection and reporting
Typically in NMS/EMS and NE the performance counters is collected as data and then reports of this information Performance counters are collected in various time buckets, these buckets keep historic and pseudo-real time data. The pseudo real-time buckets can be reset for immediate troubleshooting These reports are also periodically dumped to disk as flat files. These files are then pulled off to a data store to perform data mining
30
Network Management and Performance Counters (Cont.)

Common usage of performance counters by service providers
Preemptive trend analysis for capacity planning Service Level Agreements (SLA) Quality-of-service monitoring and network troubleshooting
BRKNMS-1942_c1
Cisco Public
31
Case Study and Tool Architecture
BRKNMS-1942_c1
Cisco Public
32
Data Center Application Network ManagementSolution and Value Proposition

The trends captured through the dashboards would help capture/encompass preemptive capacity planning, generating baseline behavior of the network, for identifying anomalies and pseudo real time trouble shooting. They can also serve to provide data for SLAs for internal/external customers The case study in this presentation focuses on global site selector, Catalyst 6500 with ACE blades and the MSFC for VLAN counter stats But this concept can be expanded to include, other LB blades, appliance boxes, core switches, firewalls, WASS and so on Currently CLI scraping is done to collect the stats, but enhancements can be made to pull the data through XML interface Although the data collections is only based on open source (gnu) tools, north bound interface for CISCO products like ANM can also be integrated
33
Tool Architecture Deep Dive
BRKNMS-1942_c1
Cisco Public
34
DC Service Management Strategy

Methodology Service KPI Identification Service KPI Collection Implementation
Collection through SNMP, scripts (expect), XML (agents) Storage RRD, flat files, SQL DB Correlation through scripts, correlation systems MRTG graphs, reporting systems
Cisco Public
Service KPI Storage
Service KPI Correlation
Service Management Dashboard

BRKNMS-1942_c1 2009 Cisco Systems, Inc. All rights reserved.
35
XML Used for KPI Collection

Extended markup language (XML) is being used extensively in the industry these days to address a variety of KPI collection needs Most of the NEs and management devices have a XML interface Provides maintainability and scalability
BRKNMS-1942_c1
Cisco Public
36
XML Applied in DC Network

Reporting Engine
XML Billing Reports Collected Periodically
Flow Through Provisioning

XML-Based Provisioning Communication
Network Monitoring
Data Store
XML Reports Offloaded Periodically
XML Agent
XML-Based Network Resource Information Collection Through XML API
Management System
XML Reports Offloaded Periodically XML-Based Provisioning and Stats Collection from NE XML Agent
37
NE
NE
XML Example
XML API and Collection Script Example Sample TCL Script

#!/usr/bin/tclsh # Global login credentials set user "username" set pass "password"
The running-configuration can be monitored via the XML agent
to Collect ACE LB Resource Stats
xml_cmd=<request_raw>show runningconfig</request_raw>
# Global mgmt IP address of Admin context set mgmt_ip "xx.xx.xx.xx"
<response_xml> <exec_command> <command> show running-config </command> <status code="100" text="XML_CMD_SUCCESS"/> <xml_show_result>
# this procedure will execute a curl command to send the XML # command to ACE. If the command fails to execute properly, # the script will exit with and error. # If the command executes with no problem, then the output # of the XML command is returned proc issue_command { cmd } {
global user pass mgmt_ip
if { [catch {set output [exec -- curl -s\
Output removed for brevity

{
http://${user}:${pass}@${mgmt_ip}/bin/xml_agent\ -d "xml_cmd=<request_raw>${cmd}</request_raw>"] } error] } puts "Problem with exec: $error" exit 0 }
</xml_show_result> </exec_command> </response_xml>
return $output
}
38
Tool Architecture
Service KPIs discussed at length earlier A set of scripts (Perl, Expect, Shell), database and WEB-based GUI collectively make up the data center service dashboard Data center performance data collection engine
Performance data was collected from an Enterprise customer The data were collected at 15 min intervals. 12 weeks worth of this data was collected. The collection was done through a tool, driven off a seed file containing the network devices and contexts to poll The data was collected through an EXPECT script running over a Linux server
Correlation and storage engine

Based on Perl scripts, which flattened the collected data and stored it as RRD Round Robin Database (RRD)
Visual dashboard
A Perl CGI(DRRAW) script was used to stack together the KPI dashboards to create the service dashboard
BRKNMS-1942_c1
Cisco Public
39
Tool Architecture
Input Seed File
Client
HTTP
Linux Server
Apache DRRAW CGI
Perl Script
Pull PM Data (Expect scripts) Parsing
MSFC
Client
Create RRD DB
ACE
RRD
Pull Data Parse Data Populate Data DRRAW CGI Client View
BRKNMS-1942_c1 2009 Cisco Systems, Inc. All rights reserved.
VLAN Counter GSS Counter ACE Counter
GSS
Cisco Public
40
Tool Architecture (Cont.)

RRD and DRRAW
The RRD databases are fed into an gnu CGI script call DRRAW. DRRAW is used to create the dashboards
This combination is used to create a pseudo real-time system wide view of the data center data flow, enabling us to do capacity planning, and keep on top of SLA
BRKNMS-1942_c1
Cisco Public
41
Round Robin Database (RRD) Overview

RRD tool is a database and set of tools that store, retrieve, and display a time series-based data It is stores data in a very efficient way and provides mechanism to generate very nice time series-based graphs RRD is effective for short-term trending, the data sources would get wrapped around once its set limit is reached, as RRD database size for a data source is determined at DB creation time
BRKNMS-1942_c1
Cisco Public
42
Performance Counter Raw Data GSS/MSFC/ACECollector Report

TIME_STAMP,DOMAIN_HTS TIME_STAMP,ANS_HTS 1196992645,179706 1196992645,39911 1196992645,179706 1196992645,39911 1196992843,179730 TIME_STAMP,L2U_PKTS, L3IU_PKTS, L3OU_PKTS L3OM_PKTS, L3IM_PKTS, L2M_PKTS MSFC 1196992645,1980541055,0,0,0,0,165550263 VLAN 2253 1196992843,1980579235,0,0,0,0,165554094 1196993806,1980823036,0,0,0,0,165568630 1196994722,1981053500,0,0,0,0,165586433 1196995638,1981314830,0,0,0,0,165604168
gss_domain_login.portal
GSS ANS 15.203.208.14
1196992645,1011585 1196992645,1011585 1196992843,1011686 1196993806,1012072 1196994722,1012441 1196995638,1012806
16.225.138.71-7-SLB1_stats
TIME_STAMP,TOTAL_CONN,CURRENT_CONN, DESTROYED_CONN,TIMEOUT_CONN, FAILED_CONN 1196992645,116398376,1424,112831941,3560072,4939 1196992843,116404942,1386,112838475,3560142,4939 1196993806,116431395,1390,112864622,3560444,4939 1196994722,116458865,1476,112891634,3560816,4939 1196995638,116483835,1532,112916244,3561120,4939
TIME_STAMP,CURRENT_CONN, DESTROYED_CONN, FAILED_CONN 1196992645,262704,218308604341,2979179043 1196992843,262704,218309140869,2979239008 1196993806,262704,218318299454,2979769928 1196994722,262704,218322705963,2980161653
16.225.138.71-7-SLB1SMTPRELAY_MM_25SMTPRELAY_25_policy
BRKNMS-1942_c1
Cisco Public
43
Performance Counter Raw Data SIP Trunk

TIMESTAMP, call_agent_id, CONDITION, TRKGRP_TYPE,tgn_id, TRKGRP_INCOM_ATTMP, TRKGRP_OUTG_ATTMP, TRKGRP_OUTBOUND_FAIL, TRKGRP_TOTAL_OVERFLOW, TRKGRP_TOTAL_TRK, TRKGRP_INCOM_BUSY_TRK, TRKGRP_OUTG_BUSY_TRK, TRKGRP_TOTAL_OOS_TRK, TRKGRP_INCOM_USAGE, TRKGRP_OUTG_USAGE, TRKGRP_TOTAL_USAGE, TRKGRP_AVERAGE_USAGE 2005-08-21 00:15:00,CA101,Normal,SS7,1002,0,0,0,0,24,0,0,4,0,0,0,0
2005-08-21 19:50:00,CA101,Normal,ANNC,9802,0,0,0,0,96,0,0,6,0,0,0,0
SIP Trunk 2005-08-21 19:40:00,CA101,Normal,ISDN,9950,0,0,0,0,23,0,0,0,0,0,0,0

2005-08-21 19:45:00,CA101,Normal,CAS,9401,0,0,0,0,4,0,0,0,0,0,0,0 19:45:00,CA101,Normal,SIP,9601,0,1,0,0,0,0,0,0,0,0,0,0
Info
BRKNMS-1942_c1
Cisco Public
44
Performance Counter Raw Data GSS

The raw data reflects a snippet from a trunk group performance counter file The switch collects and stores various trunk groups related PM counters: SIP PM counter information reflects the how many CICs are OOS along with other trunk information
BRKNMS-1942_c1
Cisco Public
45
RRD Creation Trunk Group Example

rrdtool create tg_ss7.rrd \ --start 1124600400 \ --step 900 \ values DS:TRKGRP_TOTAL_OFLOW:ABSOLUTE:900:0:100000 \ DS:TRKGRP_INCOM_BSYTRK:ABSOLUTE:900:0:100000 \ DS:TRKGRP_OUTG_BSYTRK:ABSOLUTE:900:0:100000 \ DS:TRKGRP_TUSAGE:ABSOLUTE:900:0:100000 \ RRA:AVERAGE:0.5:1:57600 600 days worth of data UTC time in ticks min, max 15 min. intervals
DS:TRKGRP_OFAIL:ABSOLUTE:900:0:100000 \
BRKNMS-1942_c1
Cisco Public
46
RRD Creation Trunk Group Example

RRD update command called by a Perl script with UTC time and all the data sources as arguments rrdtool update tg_sip.rrd \
$epoc_local:$TRKGRP_OUTBOUND_FAIL:\ $TRKGRP_TOTAL_OVERFLOW:\ $TRKGRP_INCOM_BUSY_TRK:\ $TRKGRP_OUTG_BUSY_TRK:\ $TRKGRP_TOTAL_USAGE
BRKNMS-1942_c1
Cisco Public
47
DRRAW CGI Overview

Drraw is a graphing system based on RRD tool It comprises of set of CGI scripts written in perl which allow creation of real time graphs very easily through a web front-end It also allows creation of consolidated dashboard views of these graphs
BRKNMS-1942_c1
Cisco Public
48
DRRAW Graph Front-End
BRKNMS-1942_c1
Cisco Public
49
Tool Dash Boards and Case Study
BRKNMS-1942_c1
Cisco Public
50
Dashboard in Depth to Capture the Application Service Flow

GSS domain stats dashboard captures the number of hits for gss_domain_login.portal GSS answer stats dashboard captures the number of answers with VIP 15.203.208.14 MSFC vlan stats dashboard captures packets for vlan 2253 ACE global stats dashboard captures the connection information for ACE context SLB1 16.225.138.71-7SLB1_stats ACE policy stats dashboard captures the connection information for 16.225.138.71-7-SLB1SMTPRELAY_MM_25-SMTPRELAY_25_policy
51
Dash Board Case Study Main Dashboard View Replace
A Collective View of All Dashboards
BRKNMS-1942_c1
Cisco Public
52
Dash Board Case StudyGSS Domain Stats Dashboard View Replace
BRKNMS-1942_c1
Cisco Public
53
GSS Domain Service Stats Explained

The small graph represents a baseline traffic behavior for domain gss_domain_login.portal
BRKNMS-1942_c1
Cisco Public
54
Dash Board Case StudyGSS Domain Stats Dashboard View (Cont.)

A Drop Anomaly Is Seen in the Login Domain
BRKNMS-1942_c1
Cisco Public
55
Dash Board Case StudyGSS Answer Stats Dashboard View Replace
BRKNMS-1942_c1
Cisco Public
56
Dash Board Case StudyGSS Answer stats Dashboard View (Cont.)
BRKNMS-1942_c1
Cisco Public
57
GSS Answer Stats Explained

A base line traffic is observed for answers for VIP 15.203.208.14
BRKNMS-1942_c1
Cisco Public
58
Dash Board Case Study MSFC VLAN Dashboard View Replace
BRKNMS-1942_c1
Cisco Public
59
Dash Board Case Study MSFC VLAN Dashboard View (Cont.)

L2 Pkt Baseline
L3 Pkt Burst
BRKNMS-1942_c1
Cisco Public
60
Explain the MSFC Counters

The dashboard clearly reflects a baseline pattern for L2 packet A spike is displayed for L3 packet, reflecting an anomaly
BRKNMS-1942_c1
Cisco Public
61
Dash Board Case Study ACE Context Dashboard View
BRKNMS-1942_c1
Cisco Public
62
ACE Slot/Context Stats Explained

A baseline traffic captured for current connections, total connections, destroyed connections
BRKNMS-1942_c1
Cisco Public
63
Dash Board Case Study ACE Policy Stat Dashboard View

Client and Server Connection Info
BRKNMS-1942_c1
Cisco Public
64
ACE Policy Stats Explained

A Service for SMTP traffic is displayed
BRKNMS-1942_c1
Cisco Public
65
SIP Performance Counters

BTS uses SIP to interact with voicemail equipment and SIP trunks Dashboard SIP counters
SIP outgoing messages, SIP outgoing success, 5xx errors
Problem trends
An increase in retransmits or increase in 5xx errors is a visible indication that voice mail server or SIP trunks is having issues
BRKNMS-1942_c1
Cisco Public
66
SIP Performance Counters (Cont.)

Increase in SIP invites TX with lower number of RX could also indicate resource issue on the voice mail system In a case where a new network equipment has been added, could cause SIP packets to be dropped which will be indicated in the trending datai.e., a new firewall addition in the path
BRKNMS-1942_c1
Cisco Public
67
SIP Counters: Example
TOT SUCC MSG. AVG. Overlayed TOT OUTG MSG. AVG.
BRKNMS-1942_c1
Cisco Public
68
SIP Counters: Example (Cont.)

Base-Line Trending Observed
Trending Observed over Several Months
BRKNMS-1942_c1
Cisco Public
69
SIP Graph Analysis

Graph is pointing out SIP success rate and shows a pattern to it. Any increase in the message could potentially help in voice mail capacity planning Any number of TX messages without RX would be an alarm for troubleshooting SIP issues It is also indicative of successful outbound SIP traffic
BRKNMS-1942_c1
Cisco Public
70
Trunk Group Performance

Three Key Measurements Could Be Used to Identify Problems with the Trunking Capability; These Include:
TRKGRP_OUTBOUND_FAIL is a count of the number of occurrences that a trunk member is available, however, some type of trouble prevented the call from being established on the trunk. These errors may indicate trouble with the related signalling for the particular trunk TRKGRP_TOTAL_OVERFLOW is a count of the number of calls that attempted to utilize a trunk from the group and encountered all trunks in use situation. This could stem from equipment failure or traffic congestion. This measurement should be monitored for trending information TRKGRP_TOTAL_OOS_TRK is a count of the number of trunks entering an Out of Service state for maintenance reasons. This could be triggered by operator action or by the system in response to trunk failures. A non-zero value should be investigated
71
Trunk Usage Counters

Dashboard trunk counters
Trunk total overflow, incoming trunk busy, outgoing trunk busy, total trunk usage
A pattern is seen
We can see that most of the trunk seizers are for outgoing trunks Incoming trunk seizures are low Overflow of trunks is very low
Problem indication
Total trunk usage goes high, overflow of trunks goes high are indications of capacity issue
BRKNMS-1942_c1
Cisco Public
72
Offnet Trunk Capacity Planning

For trunk capacity planning, following performance counters can be used:
TRKGRP_INCOM_BUSY_TRKterminating or ingress call completions TRKGRP_OUTG_BUSY_TRKoriginating or egress call setups TRKGRP_TOTAL_USAGEtotal of #1 and #2
BTS polls the above three performance counters every 100 seconds
BRKNMS-1942_c1
Cisco Public
73
Trunk Usage Counters: Example
TRUNK OUTG BUSY AVG.
TRUNK USGAE AVG.
TRUNK INCOM BUSY AVG.
BRKNMS-1942_c1
Cisco Public
74
Trunk Usage Counters: Example (Cont.)

Base-Line Trending Observed
Trending Observed over Several Months
BRKNMS-1942_c1
Cisco Public
75
A Virtualization KPI Overview
BRKNMS-1942_c1
Cisco Public
76
Virtualization KPIs
Continuing with our service KPI strategy we need to identify some key metrics for a virtualized environment The key metrics that we want to highlight are
CPU Memory Disk Load
We will focus on VMWare 3.5x and its powerful CLI command esxtop to collect the KPI information Other sources of collection are VIM API and vm-support
77
BRKNMS-1942_c1
Cisco Public
VMWARE KPI Collection CPU and Memory
A Load Average of 1.0 Represents CPUs Fully Utilized
High Ready Time, Indicates Contention
A MEMORY over Commitment of 1.0 Represents 100% over Commitment Increase in MEMCTL and SWAP Will Indicate Ballooning and Memory over Subscription
BRKNMS-1942_c1
Cisco Public
78
VMWARE KPI CollectionNetwork
Traffic Performance Egress and Ingress Traffic Drop Egress and Ingress
BRKNMS-1942_c1
Cisco Public
79
Service Flow, and Application Network ManagementLayering
BRKNMS-1942_c1
Cisco Public
80
Service Flow RecapNetwork Management Layering Concept

A key concept, LAYERING So far though we have tracked application service flows through the KPI dashboards then we go to specific element management layer If an anomaly occurs, affecting the service we can easily zoom in at where it occurred in the Network through the KPI dashboards We deep dive into a lower element level layer Specific element management systems which specialize in sets of devices in DC space Used to trouble shoot the localized anomaly Baseline of the service flow is developed so it can be used to drive capacity planning, ongoing trend analysis Develop in-house customer SLAs, for example you can gauge the server-farm traffic, basically egress traffic for particular services
81
Service Flow RecapNetwork Management Layering Concept (Cont.)

EMS in general can throw false alarms, which if not filtered properly can cause loss of countless tech trouble shooting hours Following this layering approach would make strides towards achieving five nines by reducing down time An example would be of a SMTP server farm Based on the baseline traffic, you identify the busy hour behavior Once busy hour traffic is identified you can potentially trend the capacity of your server farm on a pseudo-real basis
A DC Downtime 5 9s Perspective
Availability % 90% 95% 98% 99% 99.50% 99.80% 99.90% 99.95% 99.99% 100.00% 100.00% Downtime per Year 36.5 days 18.25 days 7.30 days 3.65 days 1.83 days 17.52 hrs 8.76 hrs 4.38 hrs 52.6 min 5.26 min 31.5 sec Downtime per Month* 72 hrs 36 hrs 14.4 hrs 7.20 hrs 3.60 hrs 86.23 min 43.2 min 21.56 min 4.32 min 25,9 sec 2.59 sec Downtime per Week 16.8 hrs 8.4 hrs 3.36 hrs 1.68 hrs 50.4 min 20.16 min 10.1 min 5.04 min 1.01 min 6.05 sec 0.605 sec
82
Reports and Potential Alerting Mechanism

Quarterly reports
Reports reflecting the base line traffic can be generated Reports trending the various deployed applications can generated Thus allowing for internal client to better do capacity planning for server farms Reports to make visible high and least resource intensive services
Alerting based on thresholds

Since the KPI information is already obtained on a periodic basis A high and low watermark level threshold based alerting mechanism introduced for each segment, thus further reducing the time to detect an anomaly
83
Data Center Element Management and Monitoring
BRKNMS-1942_c1
Cisco Public
84
A Typical Network Management Layout

Service Dashboard at this Layer
BRKNMS-1942_c1
Cisco Public
85
Utilization of ANM and NAM

Application network management can be effectively achieved by ANM for content devices as ACE, CSM and GSS NAM blades are used for traffic profiling and performing trouble shooting activities
BRKNMS-1942_c1
Cisco Public
86

ANM
BRKNMS-1942_c1
Cisco Public
87
Application Networking Manager (ANM)

Centralized Management and Administration
Centralized configuration, operations, and monitoring of Cisco data center networking equipment and services Simplified Management: GUI-driven ACE multiservice, virtualized provisioning, configuration, maintenance Operations Excellence: secure delegation of service and server tasks for ACE, CSS, CSM, GSS Application Performance: interactive monitoring of device and service health, performance and utilization IT Agility: highly granular role-based access control with user activity logging supports managing multi-tenant/use
ANM 2.0 Cisco App Delivery Infrastructure Management Solution
BRKNMS-1942_c1
Cisco Public
88
Product Overview
Cisco Application Networking Manager (ANM) helps enable centralized provisioning, operations, and basic monitoring of Cisco data center networking equipment and services Version 2.0 of Cisco ANM focuses on providing provisioning capability for Cisco Application Control Engine (ACE) devices, including ACE modules and ACE 4710 appliances It also supports operations management and monitoring for ACE devices as well as for Cisco Content Services Switch (CSS), Cisco Content Switching Module (CSM), Cisco Content Switching Module with SSL (CSM-S) and Cisco ACE Global Site Selector (GSS)
BRKNMS-1942_c1
Cisco Public
89
Product Overview
Enables device and virtualization provisioning for up to 50 ACE devices and operations support for up to 40 CSS and CSM devices per Cisco ANM server and up to three clusters of GSS Graphical interface for simplified and standardized service provisioning for basic, advanced and expert users Device and service monitoring Secure user access and delegation of responsibilities Up to 25 users can simultaneously manage multiple devices via web browser; runs from a centralized server running LINUX
BRKNMS-1942_c1
Cisco Public
90
Complete, Single-View
Provisioning/Operations/Monitoring
Configure, Monitor, Admin Task-Defined Screens Device and Virtual Context Grouping and Selection
Virtual Server (a.k.a. VIP) View and Configuration

BRKNMS-1942_c1
Intuitive Sequencing of Actions with Ability to Drill Down Levels, Sort and Filter
Cisco Public
91

NAM
BRKNMS-1942_c1
Cisco Public
92
The Cisco NAM

Cisco NAM Feature
Traffic analysis integrated in the network Critical points, Web-based GUI Real-time and historical monitoring Applications, hosts, conversations Application response time monitoring Users experience of the network Troubleshooting Packet capture and decode
Benefits
Eases Deployment, Management, and Support Detects How Applications and Users Use the Network and Receive Services Reveals How Applications Are Performing Isolates Problems Before They Impact Users
BRKNMS-1942_c1
Cisco Public
93
Embedded Traffic Analyzer Software

Configuration of the NAM
Network parameters Selection of traffic to monitor Types of statistics to gather
Real-time and historical reports

MIB-II monitoring Application, hosts, and conversation monitoring Packet capture and decode Application response time monitoring Voice over IP (VoIP) and video monitoring Differentiated services (DiffServ) monitoring
94
Interface Monitoring (ISR NAM)

View traffic statistics for all interfaces Drill-down to obtain more details including TopN applications, hosts, and conversations
Monitoring Router Interfaces Is a Good Starting Place to Learn How Network Traffic Is Being Used
95
Switch Port Monitoring (Cat6K NAM)

View traffic and error statistics for all interfaces Select a port and drill-down to obtain more details
Port-level statistics include:

Utilization, packets, errors, collisions
96
Application, Host, and Conversation Monitoring

Protocol Distribution Conversation Pair Statistics
NAM Detects the Applications, the Bandwidth They Consume, and the Hosts Using Costly Network Resources
Detailed Host and Conversation Statistics
97
Packet Capture and Decode
Support Troubleshooting Efforts with Trigger-Based Captures, Filters, Decodes, and a Capture Analysis Toolset
BRKNMS-1942_c1
Cisco Public
98
Application Profiling Service
BRKNMS-1942_c1
Cisco Public
99
Application ProfilingServer Discovery

Discover all the servers and software in fixed number of datacenters
Discovery appliance installed at a fixed location It is assumed that discovery appliance can access all the servers that need to be discovered in the datacenter Credentials will be available for all the servers that need to be discovered (user id, password) For Windows server discovery, discovery needs one m/c per domain to run a slave discovery service component
Identify all the Common Off The Shelf (COTS) software components such as Oracle, Apache, MS Exchange, etc. Scope
Number of servers < 1500 Operating systemsUNIX, Linux, and Windows ProtocolsIP only (no legacy protocols such as IPX, SNA, etc.)
100
Application ProfilingApplication Transport Mapping

Listen to selected LAN segments (VLANs) continuously for a period of fixed number of days and identify the inter-server communications From the data captured, identify the data exchanged by various application components From the data captured, provide TCP/IP characteristics of known application components From the data capture, provide enterprise services (DNS, NTP, etc.) utilization reports Scope
Capture device and location data Capture from switch only Capture VLAN traffic using span ports Capture data for a fixed interval of time
101
Application Visibility and Management (AVM)
BRKNMS-1942_c1
Cisco Public
102
Network-Based Application Visibility Is Critical

Every Transaction Passes Through the Network
11001 01110 11001
11001011101100101011011010
1011001
110010111011001
Data Center
001 011 11 101 100 1
A more efficient way to instrument everything on the network A holistic approach to application visibility is required Common services around support replace the current stovepipe model
103
BRKNMS-1942_c1
Cisco Public
What Cisco Application Visibility and Management Does

Leverage the ubiquitous nature of the network Discover and profile applications, components and infrastructure inside the Network Monitor distributed applications for high availability and performance Create a holistic view of applications health Pinpoints issues at an early stage
Discover
Monitor
Analyze
BRKNMS-1942_c1
Cisco Public
104
Service/Product Discussion
Open Forum
BRKNMS-1942_c1
Cisco Public
105
In Summary
The current study is based on the data obtained from a major customer It encompasses a very small set of devices to show proof of concept The data taken has been morphed into the dash board This has been done through a collection, correlation and reporting methodology We can clearly observe patterns
Base-line traffic can observed Also potential anomalies can be seen Current capacity can be gauged SLA requirements can be derived Audit reports (capacity, etc.) can also be generated, as traffic is already profiled Alerting mechanism can also included for certain threshold crossings as an enhancement
106
Please Visit the Cisco Booth in the World of Solutions

See the technology in action
Network Infrastructure and Systems
NS1 Cisco Catalyst Series: Optimize and Virtualize NS2 Cisco Catalyst Series: Fueling Collaboration NS3 Cisco ISR: Application Integration at Branch NS4 Enhance Collaboration with Cisco WebEx Node NS5 Optimize the WAN with Cisco ASR 1000 Series NS6 Pedal Power for the Cisco Catalyst 4500
BRKNMS-1942_c1
Cisco Public
107
Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Dont forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.
108
BRKNMS-1942_c1
Cisco Public
BRKNMS-1942_c1
Cisco Public
109

BRKNMS-1942 0101

Загружено:

Сведения о документе

Исходное описание:

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

BRKNMS-1942 0101

Загружено:

Авторское право:

Доступные форматы

Monitoring and Trending Application Performance in Data Center Services

2009 Cisco Systems, Inc. All rights reserved.

Challenges for Data Center Application Service Management

2009 Cisco Systems, Inc. All rights reserved.

What Is Distributed Data Center?

2009 Cisco Systems, Inc. All rights reserved.

Data Center Network Management Problem and Value Proposition

Complication Therearenoturnkeynetworkmanagementoftheentiresystem(wholeview)isabiggap Thereisnoeffectivewaytrackingtheentiresystem(orflow)currently

Implications Customercannoteffectivelydocapacityplanning,troubleshooting,developanykindof SLAs Theresult,customerhasdowntime

2009 Cisco Systems, Inc. All rights reserved.

Data Center Network Management Problem and Value Proposition (Cont.)

Plan WhataretheseKPIs?Theseareperformancecounterscollected onaperiodicbasisfromeach flowsegmentinasystem PseudorealtimeRoundRobinDatabases(RRD)arecreated withthesecounters DashBoards arecreatedfromtheseRRDs,foreach datacenter

A DC Service Flow with Key Performance Indicators (KPIs)

2009 Cisco Systems, Inc. All rights reserved.

Service Flow and Key Performance Indicators

2009 Cisco Systems, Inc. All rights reserved.

Service Flow and Key Performance Indicators (Cont.)

2009 Cisco Systems, Inc. All rights reserved.

CoreSwitch MSFCVLANStats LoadBalancerStats LoadBalancer VIPStats

Firewall Stats SSLOffload

2009 Cisco Systems, Inc. All rights reserved.

A LB Service/Performance Counter Flow for a HTTP Service

++ + ServiceConnectionstatistics + TotalConnectionsCreated:1696043 TotalConnectionsCurrent:10 TotalConnectionsDestroyed:542 TotalConnectionsTimedout:1614174 TotalConnectionsFailed:81317

Example KPIs Explained

2009 Cisco Systems, Inc. All rights reserved.

DNS Rates and Hit Counts at CLI

Raw DNS Answer Stats

Raw DNS Domain Stats

2009 Cisco Systems, Inc. All rights reserved.

MSFC, Over All Stats and VLAN Stats KPIs

2009 Cisco Systems, Inc. All rights reserved.

ACE Load Balancer StatsKPIs

Load cce02swdclb1015-da01-ace1s7/SLB1# sh Balancer stats Global Stats

Load Balancer App. Stats

Response entries recycled : 0

, Bad HTTP version errors : 0

Total logging decisions : 0

Some Relevant Protocols Overview

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

Additional Information Additional Information

2009 Cisco Systems, Inc. All rights reserved.

Server sends cookie to client:

Client sends cookie back to the server on subsequent visits

2009 Cisco Systems, Inc. All rights reserved.

2009 Cisco Systems, Inc. All rights reserved.

Network Management Performance Counters

2009 Cisco Systems, Inc. All rights reserved.

Network Management and Performance Counters

Network Management and Performance Counters (Cont.)

2009 Cisco Systems, Inc. All rights reserved.

Case Study and Tool Architecture

2009 Cisco Systems, Inc. All rights reserved.

Data Center Application Network ManagementSolution and Value Proposition

Tool Architecture Deep Dive

2009 Cisco Systems, Inc. All rights reserved.

DC Service Management Strategy

Service KPI Storage