Security Metrics

SAJENDRA KUMAR IIMT, IET, MEERUT

Overview
Why do we care? What is a metric? How do we decide which metrics to collect? How are they collected? Effective risk analysis through security metrics How do security metrics make a corporation money (operational risk)? Compliance competition and security ROI

Why do we care?
How do we know how secure an organization is?
Metrics help define secure Metrics let us benchmark our security investments against other organizations Compliance The metrics gathering process often leads to identification of security inconsistencies or holes

Why do we care: Example

Manager asks, Are we secure? Without metrics: Well that depends on how you look at it. With metrics: No doubt about it. Look at our risk score before we implemented that firewall project. Its down 10 points. We are definitely more secure today than we were before.

Why do we care: Example

Manager Asks: Have the changes that we implemented improved our security posture? Without metrics: Sure. They must have, right? With metrics: Absolutely. Look at our risk score before we made the recommended changes, and now its down 25 points. No question, the changes reduced our security risk.

Motorola CISO on Metrics

Security experts can't measure their success without security metrics, and what can't be measured can't be effectively managed.

What is a metric?
The National Institute of Standards and Technology (NIST) define metrics as tools designed to facilitate decision-making and improve performance and accountability through collection, analysis, and reporting of relevant performance-related data. Metrics are simply a standard or system of measurement. In this case, it is a standard for measuring security, specifically measuring an organizations security posture. Although there are some published standards for measuring security, ideally security metrics should be adjusted and tuned to fit a specific organization or situation.

Examples of metrics
Total number of remote connections over a one month period (VPN, ISDN, dial-up, remote desktop) Maximum number of concurrent remote by user The percentage of total applications that have a contingency plan by application criticality. Time to analyze and recommend action on a security event Number of Linux servers at least 90% compliant with the Linux platform security standard

Security Metric Categories

Platform
Number of Linux servers that are compliant with EFS policy

Network
DMZ port scans

Incident
Number of hosts infected with worm XYZ

Vendor
Average security rating for vendors that touch active customer files

People
Number of terminated employees with administrator access

Industry
Number of public security incidents in sector ABC with severity score Z

Political
Hacktivism scores, amount of sites listing sector/company ABC as potential target

Security Metric Types

Real Time
Number of concurrent connections to VPN Usually from incident response systems

Polled
Number of password reset requests (monthly), Usually from SAs or SMEs

Incident based
Number of machines infected with worm XYZ Number of vendors suffering from infections of worm XYZ Usually from industry intelligence/incident response/SAs and SMEs

How do we decide which ones to collect?

Policy Mining / Easy to Spot Anomalies Risk Scoring ROI / Vendor Evaluations Regulatory / Cover the industry standards Tips / Visionaries

Policy Mining Example

Policy Statement: All users who connect remotely must be uniquely authenticated. Enforcement Mechanism: Users are required to authenticate with a username, password and securID token to gain access to the internal network. Network Policy (VPN): A user Kerberos account must authenticate with both the Radius and securID privilege granting servers before VPN connectivity is established Question: How do we tell if a user is uniquely authenticated? Metric: Maximum number of remote connections by user in a month. Metric: Maximum number of concurrent connections for a single user Metric: Total time connected in a single month Metric: Number of users granted remote VPN privileges Metric: Number of securID reset requests in a given month Metric/Alert: user connecting to VPN from different countries simultaneously

Risk Scoring
Metric: Maximum number of remote connections by user in a month.
Impact: 6/10 (we care about this 6/10 relative to other metrics in this policys risk, this score may come from SMEs/upper management/industry direction) Risk: 20% + (10% * last months count) this is where the soft analysis takes place

ROI / Vendor Evaluations

We spent $XXXXX on 4 new application penetration testers, are our applications more secure now? Should we hire another one?
Metrics specific to applications not pen-tested, and those that are

We are spending $YYYY on product XYZ, is it worth it to renew the contract or should we start looking for a new solution?
Metrics specifically surrounding product XYZ and the problem it is solving
Number of successful social engineering attacks and their impact before and after the online training seminar

Regulatory
(doesnt yet exist for most industries)
Baseline metrics (from Spire Security)
Number of patched machines / total Number services running on external facing machines Port Scanning Incidents

Standards (from Spire Security)

Finances Market Cap Overall Revenue/Funding level Overall Expenses Workforce Number of Employees Number of contractors/temps Number of locations with dedicated IT IT Spending Budgets for Operations, Maintenance, Capital Employees Equipment Count of Servers, appliances, databases, client PCs, Laptops, PDAs Network Traffic Count of flows, possible flows, actual flows, blocked flows, sessions, commands, transactions Security Spending Operations, Maintenance, Capital Expenditures, Number of Security FTEs

Standards contd.
Identity Management Management budget Management FTEs Total User Repositories Total User Accounts Count of user accounts created, accounts modified, password resets, accounts disabled/deleted, accounts evaluated Authentication Events Number of failed authentications Vulnerability Management Spending Number of servers/applications/PCs scanned Number of Vulnerability Management FTEs Count of open ports, known vulnerabilities, patches, configuration changes Trust Management spending Count of Trust Management FTEs, policies written, certificates issued, signed documents, encrypted documents Threat Management Spending Count of Threat Management FTEs, alerts, compromised systems

Tips / Visionaries
Investigations/Government/Regulator may ask information security to monitor specific activities A visionary (author/upper management/consultant) will come up with a new/derived metric to collect in order to report on a new phenomenon

How are metrics collected?

Categorize and define the metric and its owner Determine and document metric source
Automated
database connection Script file output

Manual
Email polling Form entry Manual file updating Report analysis/research

How are metrics collected?

Define/document collection process for each metric
A pull replication query mirrors the critical IDS alerts from server ABC database BCA to the metrics collection server DEF database BCA. DEF then sums and categorizes the alerts. The final counts are archived in table QRS in database BCA on server DEF. Joe runs a stored procedure on server XYS database YZD which he manually correlates with Radius logs aging over the past 3 months. The report is then stored on share ABCD and Joe sends an email to Sally indicating the metric is updated. Sally then enters the metric information into the metric collection database using the form at URLQYZX

Effective risk analysis through security metrics

How do we make decisions based on the metrics now that we have them? Metrics which are collected should match high impact risk items. (only spend money collecting those with high risk scores)

Risk Breakdown Example

Risk Measurement: Federation information security risk score (akin to homeland security colors extremely vague, policy generally shouldnt be created based on such high indicators) Risk Components: Network, Incident, Vendor, People, Industry, Political Subcomponent: Federation-Global-Network-TradingFTSE Metric inventory for subcomponent: ID4786(A),ID2235(B),ID8674(C) Subcomponent risk score calculation: 50%(A*(last 4 months(B))) + (50% * Cs rolling average) Security risk analysts and SMEs create score weightings

This is complicated and expensive, why do we do it this way?

As people, we are generally bad at concentrating on more than 7 factors/metrics/indicators at a time Risk scoring lets us define and objectively monitor the big picture information security view Correlations Alerting / Smarter automated responses

How do security metrics make a corporation money (operational risk)?

Legislation (Basel II) says you have to withhold 15% of last years revenue unless you can prove that you have mitigated your risk Metrics are your proof, risk scores are your slice description of the state of the union In general, the less money you have to withhold/spend on insurance, the more money you make

Ask the question, do we have policy and guidelines to mitigate/monitor these combinations?
If not, create them In our case, assume we do Mine these policies for potential metrics

Asset Credit Card Database on CFAC Credit Card Database on CFAC Payment Processing Availability

Threat Insider Employee External hackers Competition/Hactivists Natural Disasters

Vuln Mal-intent System Administrators Network hopping/ Sniffing Denial of service Physical equipment destruction

Risk Measurement: Customer Credit Card Privacy Score

Risk Components: Network, Incident, Vendor, People, Industry Subcomponent: Firmwide-Platform-Database-Internal-Access Control Metric inventory for subcomponent:
(A)(Real-Time)Number of databases with more than 100 credit cards which do not store credit card numbers in an encrypted manner: (B)(Real-Time)Number of system administrators with view level access to non-encrypted databases with more than 100 credit cards: (C)(Real-Time)Number of system administrators who have criminal history with view level access to non-encrypted databases with more than 100 credit cards (D)(Real-Time)Number of system administrators who have criminal history with view level access to encrypted databases with more than 100 credit cards (E)(Monthly) Number of employees leaving the firm which have had access to nonencrypted databases with more than 100 credit cards: (F)(Real-Time)Number of databases with encrypted credit cards (G)(Real-Time)Number of administrators with ability to decrypt encrypted credit cards (H)(Real-Time)Age of keys used to encrypt credit cards (days) (I)(Manual)Industry time to break one credit card encryption (days) (J)(Manual)% of administrators who have attended social engineering seminars Subcomponent risk score calculation: (0.75 * A * ((B * .1) + (C * .3))) * (0.25 * F * ( ....) )

How do we do the risk score calculation?

Risk = Threat * Vulnerability * Expected Loss ALE (Average Loss Expectancy) = probability of loss * total loss potential Asset Valuing
Productivity Value Revenue Value Liquid Financial Assets Value Intellectual Property Value Confidentiality Integrity Availability Productivity Liability

Potential Loss

How do we do the risk score calculation?

Measuring Risk
Manifest risk = ratio of malicious events to total events
(sessions, commands transactions)

Inherent risk = likelihood that a configuration will contribute to a compromise

Open ports and services running compared to historical vulnerabilities on those ports

Contributory risk = measure of process errors during normal course of operations that contribute to a compromise
User Account Management procedures

How do we do the risk score calculation?

Correlation
Expert systems, obvious correlations

Historical testing
Look at data leading up to an incident, see what changed

SME predictions/insight
Sometimes they just know

Industry trends
We dont really care about 1024 bit key breaking in 2005, will we in 2012?

Firm specific
Explosion in certain incident types may necessitate a change in the equation

Optimization
After doing risk calculation, dont spend time/money to collect metrics that dont change the final score that much Analyze risk score equation against reality, are we really reporting the proper state of the union?
Add reliability fields to metrics and weigh according to them as well.

Correlation/expert system analysis Prioritize future metrics to be collected (find the value of a metric (risk of not collecting the metrics))

Problems Experienced
Systems unable to report highly critical metrics System integration SAs not willing/able to invest time setting up processes to collect metrics Ad-Hoc requests/reports and their implication on the overall view We didnt do a penetration test on most of our servers, how can call our network secure? Vague risk descriptions, Vague Metric Requests Impossible metrics (usually external) Number of credit card accounts compromised globally across any firm Missing/incomplete historical data Mistrust/inaction/devaluing because of qualitative components Complete trust

References
http://www.secmet.org http://csrc.nist.gov/publications/nistpubs/8 00-55/sp800-55.pdf http://www1.netsec.net/content/securitybri ef/archive/2004-09_Metrics.pdf http://www.cert.org/octave/

Homework
Pretend you are a security analyst at Polytechnic. You have just had the following (highly simplified and fictitious) conversation with a senior manager. Manager: Another engineering school was just sued because students transcripts could be accessed by anyone online. How secure is our new grade transaction server? You: We believe the new design is secure, however, havent allocated time, money and effort to post-implementation security evaluation. Manager: I need to show the board that we are not prone to this type of humiliation, what can you pull together for me in the next 3 months? Your assignment is to describe how you would structure your new metrics proposal which includes the following sections.
Description of which metrics you will be collecting (Based on risk analysis. Remember, this should be a minimal set, you only have 3 months to set this up) A metric collection process example for one of the metrics Suggested, simple weighting of metrics to calculate the overall risk score of the system.