Skip Navigation LinksFoundation / Foundation Research / Security Metrics / Executive Summary of Metrics Research

Executive Summary of Metrics Research

​Security metrics support the value proposition of an organization’s security operation. Without compelling metrics, security professionals and their budgets continue largely on the intuition of company leadership. With metrics, the security function grounds itself on measurable results that correlate with investment, and the security professional can speak to leadership in a familiar business language. Security metrics are vital, but in the field and in the literature one finds few tested metrics and little guidance on using metrics effectively to inform and persuade senior management.

To address the gap, the ASIS Foundation sponsored a major research project designed to add to the body of knowledge about security metrics and to empower security professionals to better assess and present metrics. The Foundation awarded a grant to Global Skills X-change (GSX), partnered with Ohlhausen Research, to carry out the project.

This report provides the project’s findings, including its three practical, actionable products:

  • The Security Metrics Evaluation Tool (Security MET), which security professionals can self-administer to develop, evaluate, and improve security metrics

  • A library of metric descriptions, each evaluated according to the Security MET criteria

  • Guidelines for effective use of security metrics to inform and persuade senior management, with an emphasis on organizational risk and return on investment

Methodology

With input from an advisory board and expert panel, the research team performed the following tasks:

  • Review and summarize literature on the use of security metrics to inform and persuade corporate management. The review cites approximately 100 sources.

  • Develop and refine a Security Metrics Evaluation Tool (Security MET). The Security MET is a written tool that security managers can use to assess the quality of specific security metrics. The tool was revised throughout the research process, based on feedback from the advisory board and expert panel.

  • Collect data to identify and evaluate current practices in the use of security metrics. This task included an online survey and detailed follow-up interviews by telephone.

  • Create a database of evaluated security metrics. The project report contains 16 metric summaries (Appendix B), each evaluated by three reviewers using the Security MET.

  • Develop guidelines for effective use of security metrics to persuade senior management. Chapter VII of this report presents guidelines gathered from a variety of sources: the literature review, the online survey, the follow-up telephone interviews, the advisory board, and the expert panel.

Literature Review

The literature review examined reasons to use metrics, characteristics of existing metrics, methods for communicating metrics, and means of evaluating metrics. Overall findings from the literature:

  • Descriptions of existing security metrics are often vague, making it difficult to adopt those metrics. The focus is more on counting events than creating meaningful, risk-based metrics.

  • Strategies for communicating metrics are general and may be hard to implement.

  • Typically, evaluation criteria are only presented at a conceptual level within the security literature, without explicit definitions.

  • Few examples of empirically sound metrics (with statistical justification and evidence) are present within the security literature. Physical security and information security appear to have more metrics in use than other security fields.

Security Metrics Evaluation Tool

The Security Metrics Evaluation Tool (Security MET) is a written tool that security managers can use to assess the quality of specific security metrics. Users will be able to determine whether an existing or proposed metric possesses scientific validity, organizational relevance (such as clear alignment with corporate risks or goals), return on investment, and practicality. Basically, the tool was designed to help a user identify a metric’s strengths and weaknesses so that the weaknesses can be corrected. The Security MET is presented in Appendix A.

The tool was developed through a lengthy, iterative process that involved synthesizing scientific literature, security industry standards, and input from metrics experts on the project’s advisory board and expert panel. (The advisory board and expert panel consisted primarily of senior security professionals with experience in the use of security metrics.) To develop the criteria (the characteristics that make an empirically sound security metric), the research team turned to measurement and testing literature, as well as industry benchmarks, and developed criteria in three categories: technical, operational, and strategic.

The tool includes the following criteria for evaluating a security metric. Definitions for and relevant research associated with the criteria are presented in Section IV.

Technical Criteria – Category 1
  1. Reliability
  2. Validity
  3. Generalizability
Operational (Security) Criteria – Category 2
  1. Cost
  2. Timeliness
  3. Manipulation
Strategic (Corporate) Criteria – Category 3
  1. Return on Investment
  2. Organizational Relevance
  3. Communication

For each criterion, the Security MET presents a definition, concept illustration, behavioral summary scale, and sample applications to help users understand how to evaluate the metric. A score sheet is presented at the end of the Security MET to tabulate the metric’s score across the nine criteria. Lower scores on particular criteria show where a metric has room for improvement.

The Security MET is designed to help the user review and understand all the behaviors associated with the criteria at varying levels. It establishes a common frame of reference for metrics users to employ when examining and rating their metrics. This frame of reference is further reinforced by the examples presented that highlight how the example metrics should be scored based on the criteria presented. Finally, this instrument is easy to score, imposes little to no time burden on staff, and could easily be placed on a wide variety of online systems.

Online Survey

On August 7, 2013, with the help of ASIS International and in concert with the ASIS Leadership & Management Practices Council, the research team invited more than 3,000 ASIS members to participate in an online survey. Invitations were e-mailed to all ASIS council members and the CSO Roundtable, plus an ASIS-created pool of top-level security professionals. A total of 297 people responded. Complete survey results, including detailed, open-ended responses, are presented in Appendix D.

Given the limitations of the sample (e.g., participation was optional, and those who chose to participate probably are not representative of all security managers), the survey did not attempt to ascertain the prevalence of particular metrics practices in the field. Instead, the survey helped the research team discover metrics practices and identify metrics users for follow-up interviews.

​Survey Questions

​Q1: Collection and Use of Security Metrics
Q2: Metric Comparison to External Benchmarks
Q3: Would You Use Metrics?
Q4: Measured Security Program Aspects
Q5: Who Records Metrics?
Q6: Metrics Provisions to Non-Security Persons
Q7: Metrics Provisions to Non-Security Persons – If No, Why Not?
Q8: Metrics Provisions to Non-Security Persons – Who?
Q9: Metrics Provisions to Non-Security Persons – How Often?
Q10: Metric Elements Shared with C-Suite Personnel 

​Q11: Most Important Metrics – Senior Management
Q12: Most Important Metrics – Why?
Q13: Metric Alignment With Risk/Objectives
Q14: Metric Alignment With Risk/Objectives – How?
Q15: Dashboard Tool Usage
Q16: Who Developed Dashboard Tool?
Q17: Third-Party Dashboard Tool Name
Q18: Metrics Interview Volunteers
Q19: Work Region
Q20: Desire Information Regarding Metrics


Respondents demonstrated a high degree of interest in the topic of metrics:

  • Seventy-seven percent of respondents said they are collecting and using security metrics.

  • Of respondents who said they are not using security metrics, 78 percent said they would use metrics if they knew more about how to create them and use them effectively.

  • Out of all respondents, 55 percent said they would like to receive more information from ASIS regarding metrics and supplied their names and e-mail addresses.

They also provided the research team with a detailed view of the many ways in which security professionals are using metrics today:

  • Metrics topics. Respondents were asked which aspects of the security program they measure. They were given a list of 13 categories (plus “other”) and asked to check all that apply. The top five categories of metric focus were security incidents, criminal incidents and investigations, cost against budget, security training and education, and guarding performance (turnover, inspections, etc.).

  • Sharing and reporting. Eighty percent provide their metric findings to persons outside the security department. Recipients of the information include senior management (listed by 79 percent of those who share metrics outside the security department), managers of other departments (59 percent), supervisors (51 percent), and people who report to the security department (47 percent). Those who share metrics provide the information quarterly (43 percent), monthly (40 percent), or annually (17 percent).

  • Topics shared with C-suite. Respondents who share metrics with C-suite personnel were given a list of 13 categories of topics (plus “other”) and asked which elements they share (selecting all that apply). The top choices were security incidents (80 percent), cost against budget (62 percent), criminal incidents and investigations (57 percent), regulatory compliance (44 percent), and risk analysis process (40 percent).

  • Alignment with organizational risk or objectives. Eighty percent of respondents who use metrics said their metrics are tied to, aligned with, or part of the larger organizational risk process or organizational objectives.

  • Dashboard tool. Only 44 percent of respondents using metrics perform their data collection, review, or sharing via a security management dashboard tool.

Metrics Summaries

The researchers developed 16 summaries of metrics that were in use in the security field as of 2013. The summaries were developed primarily through telephone interviews. Participants were identified through the project’s online survey, which asked respondents if they were currently using metrics and would be willing to describe their practices to a researcher. About half the interviewees also supplied examples of the graphics they use to convey their metrics to senior management. All 16 summaries are presented in Appendix B, along with evaluations. Each metric was scored against the Security Metrics Evaluation Tool (Security MET) by two members of the project’s expert panel and one member of the research team.

The summaries may serve as examples for security professionals considering ways to use metrics. Combining the summaries with scoring and expert reviews provides insights not only into the metrics, but also into the use of the Security MET.

These metrics measure a variety of issues and come from a variety of industries (as well as different countries).

​Metrics Collected and Evaluated

​1. Office Space Usage Metric
2. Security Activity Metric
3. Environmental Risk Metric
4. Averted External Loss Metric
5. Security Audit Metric
6. Officer Performance Metric Panel
7. Security-Safety Metric
8. Security Incident Metric

​ 9. Personnel Security Clearance Processing Metric
10. Loss Reduction-Security Cost Metric
11. Operations Downtime Reduction Metric
12. Due Diligence Metric
13. Shortage-Shrinkage Metric
14. Phone Theft Metric
15. Security Inspection Findings Metric
16. Infringing Website Compliance Metric

 

​Sources of Metrics (Industries)

​Defense/aerospace 
Energy/oil
Finance/banking
Government
Insurance
Manufacturing/industrial products

​Pharmaceutical
Real estate management
Retail
Security services
Shipping/logistics
Telecom

 

Some of the metrics are more sophisticated and detailed than others, providing a range of examples for potential users to consider. The metrics are not presented as models of perfection. Rather, they are authentic examples that security professionals can follow, refine, or otherwise adapt when developing their own metrics.

Next: Presenting Metrics to Senior Management