Before we tackle what a metric is, let's go back for a second and re-examine what a basic control is.

To control is an activity conducted to bring into check (to manage or to verify), or to constrain (to restrict or confine) something, the results of which bring forth a demonstrable outcome.

In this light, when we are employing the controls within our organization, we are creating a demonstrable outcome. Let's say that the organization must follow this control:

Establish assurance levels for information types The organization will ensure that its assurance classification guidelines take account of business needs for sharing of restricted information and the business impacts associated with such needs (e.g., unauthorized access or damage to the information). [CCI 00602]

This calls for the organization to apply its assurance strategy for information classification (and therefore information sharing) on all key IT assets. How would you go about reporting that you've accomplished this? One way would be to divide the number of key IT assets for which an assurance strategy has been implemented (you can determine this by checking their configuration policies) by the total number of IT assets as found within your configuration management database.

In formal terms, metrics present a system of measurement and analysis. What this system must define is:

1. what should be measured,

2. the unit of measurement,

3. the target for success, and

4. which sources must be drawn from for the measurement to be accurate.

Formally stated, the metric for the control in our description would read like this:

• • Title Key assets for which an assurance strategy has been implemented

  • Formula # of key assets for which an assurance strategy has been implemented / # of IT assets in total

  • Target 100%

  • Data sources The number of assets that have assigned configuration standards and other controlling policies and procedures / the total number of assets in the organization's configuration management database

In order to be useful, these metrics must indicate the degree to which goals are being met. In other words, they need to drive actions that will improve the organization's compliance processes. As one of our field editors stated, "metrics are only meaningful if they answer a question that someone wants answered." The same field editor pointed out that, "as with most if not all management decisions there are trade offs that need to be made. What these metrics need to speak to is risk -- again where are we relative to where we need to be? For example, if a given investment will reduce our exposure then what are the metrics that will improve as a result? Are we seeing such results?" Indeed, that is the type of input that metrics should be providing.

So now you're probably thinking, okay, I get it. Should I be making up a bunch of these metrics for each of my controls? The answer is, only if you are really, really bored and have nothing else to do. Because there are over 125 metrics that have already been defined from which you can choose.

As a matter of record, there are four well documented sources (and one revision) dealing specifically with information assurance measurement and metrics. The first was developed as early as 2003 and the latest in 2008.

• • July 2003 - Security Metrics Guide for Information Technology Systems, NIST SP 800-55

  • November 2004 - Corporate Information Security Working Group: Report of the best practices and metrics teams; subcommittee on technology, information policy, intergovernmental relations and the census; Government Reform Committee, United States House of Representatives (aka CISWG I)

  • March 2005 - IIA Global Technology Audit Guide (GTAG): Information Technology Controls

  • May 2006 - Guide for Developing Performance Metrics for Information Security, NIST SP 800-80

  • July 2008 - Performance Measurement Guide for Information Security, NIST SP 800-55 Revision 1

In addition to these resources, the Center for Internet Security, SecurityMetrics.org, and other such groups are continuously developing and documenting new metrics to meet the information assurance reporting challenges we all face.

The three audiences for metrics

More specifically, the four cornerstone metrics documents call out three groupings of metrics and their audiences:

• • Governance implementation metrics measure the implementation of assurance policies & procedures in connection with its information assurance responsibilities. These metrics should be reported at the Board and CXO level for use in connection with its information assurance responsibilities.

  • Effectiveness & efficiency metrics measure the results of individual policies and procedures. These metrics should be reported to the CIO, CISO, and IT Director level to determine the effectiveness of their processes and procedures.

  • Technical measurement metrics are reports such as those created by the backup and recovery tools, various logs, incident management tickets, etc. These IT operational metrics should be reviewed by IT managers, incident managers, HR managers, and others to ensure that their various control systems are in place and working effectively.

How to judge a metric

Each of the four documents above seems to greatly overlap the other in terms of both scope and content, with four points being reiterated in each of the documents:

1. Is the metric useful to (and desired by) the stakeholders of the organization?

Metrics should identify gaps in performance and expectations. The size and nature of the gap, as well as the importance of the activity being measured, will define the need to close the gap.

2. Is the metric based on defined and repeatable policies and procedures?

For the metric to be real, it has to be based upon repeatable measurements. A solitary report shows no trend. Also, in establishing a metrics program, the organization must establish the program around its current policies and procedures. Why? Because policies and procedures should be establishing what the organization does, and the metrics should be used to document how well those policies and procedures are working.

3. Can the data supporting the metric be readily and cost-effectively obtainable?

This is the big one - obtaining readily available and cost effective measurements. What you can gather, and the cost effectiveness of gathering those measurements, will be determined by your level of compliance maturity. Think for a moment in the terms of a Capability Maturity Model (CMM), that measures awareness, policies and procedures, skills and expertise, responsibility and accountability, and tools and automation, with level 1 being no formal compliance processes and level 5 being your organization is completely buttoned up and continuously improving. At level 1 and 2 of those areas, your metrics won't be trusted because it would take a Herculean effort to even collect the data, which is probably non-existent. It won't be until the organization is into level 3 and 4 that you are getting anything more substantial than simple percentages or binary metrics.

4. Does the metric yield quantifiable (percentage, average, etc.) information?

This is the million dollar question. Most "how-to" metrics guides that we have read come up with all sorts of wild numbers, costs, and items to measure. How accurate are they? Mostly, not. The good news, though, is that the Unified Compliance Framework has mapped, and will continue to map, all of the metrics guidelines published by the various authority documents. There are over 150 measurements that should be taken, and each of those measurements has been pre-defined along with calculations for how to systematize and report those measurements.

Presenting metrics

With the exception of Andrew Jacquith's book on security metrics (which is outstanding), if you look online at the various offerings of metrics books, metrics chart packs, and metrics how-tos, you'll find that each of them promises well over 500 different metrics from which to choose, and almost as many chart types to present the information. One book we found on Amazon promised over 900 different metrics. A chart series we found had around 60 different metrics reports, and over 50 different ways of presenting those metrics. Is more really better? In a word, no.

After examining all of the metrics called for within the hundreds and hundreds of IT regulatory guidelines being mapped by the UCF, we found that there are over 125 different metrics described by the various authority documents. That's it. Not 500, and especially not 900. And of those different metrics described by the various authority documents, we found that all of them can be presented in one of three different chart types:

1. A simple pie chart

2. A waterfall chart

3. A stacked column chart

So why complicate things if you don't have to? Each of the three chart types above fits the three basic types of comparisons found within the world of IT metrics: component comparison, component build, and time series categorized comparison. Each of the three comparisons presents a different message to the audience and each should be presented in a different graphic manner to give the audience clues for recognizing that message.

Component comparisons: a simple percentage of the total

Component comparisons show the size of each part as a percentage of the whole. Any time the word percentage, portion, or share is used, you are more than likely dealing with a component comparison. A component comparison should be presented using a simple pie chart.

Component comparison

Almost 99% of the predefined compliance metrics are component comparisons as they read something like this sample "Report on the current security plans that have been reviewed and adjusted to reflect the current conditions and risks."

Component build: an itemized breakdown of the total

When dissecting the parts of a whole in order to show the itemized breakdowns of the parts to the whole, a pie chart isn't as clear as one would hope. That's when you should turn to a waterfall chart. A waterfall is a simple plus and minus system. It adds and/or subtracts to either build or take away from a whole. Waterfalls take the place of columns of numbers that are summed. They visually display the values of the numbers being summed and, therefore, communicate a stronger message.

Component build

The base metric "Report on the estimated damage or loss in dollars resulting from all security incidents" is best displayed as a waterfall instead of a pie because of the number of elements being dissected.

Time series categorized comparison: changes over time

When comparing how categorized data has changed over time (whether the changes show a trend or not), the pie chart is not sufficient. However, a stacked column (or bar) can show the segments of a whole like a pie would. Use stacked bar/column charts that have the segments represented as percentages to take the place of multiple pie charts on a page. A single stacked bar/column has the same information as a single pie chart; however, many bars/columns can reside on a baseline and, therefore, visually display comparisons between the segments much better than multiple pies on a page.

Time series

While the base metric for "Report on the number of security incidents that took place that did not cause a loss of confidentiality, integrity, or availability beyond SLAs for thresholds" is a component comparison, comparing the last several quarters is a perfect example for the use of a time series comparison.

Always remember that you are presenting the message found in the data

Remember. you are presenting the message found within the data; that the servers are protected, or incidents are down this quarter, or unexpected changes have been brought down to nothing. You aren't presenting metrics to show off your PowerPoint prowess. You are there for a business reason, one that should be communicated clearly and precisely. You can do that with just three chart types. If you keep it simple and keep it clear, the message will carry the day.

When management wants glitz

So, what do you do when management wants glitz and glamour in your metrics presentations? What we in the Unified Compliance Framework do is follow Nancy Reagan's advice and "just say no." But we also know that we aren't the final word. So we are running an online poll to find out what you would do. As of this writing, here are the stats (you can take the poll yourself by clicking HERE):

• • 17% of you allow the glitz and glamour to override the message.

  • 38% of you tell management that you are not going for glamour and that you are focusing on content instead.

  • 13% of you sidestep the issue and ask management to redesign the presentations to fit (knowing that will never happen).

  • 4% of you simply file any and all requests for glitz and glamour in the trash.

  • 29% of you follow another route.

Here's what some of our field editors have had to say about this:

I have spent quite a bit of time in this area (4 years) and have studied the importance of presentation and the significance of no glitz because it is simply noise and prohibits the user from quickly understanding the data so that decisions can be made. How I've worked around this issue in the past is to work with the user to identify the real requirements, make suggestions for a better display of the data, all the while explaining the science and goal of displaying data. I've been 100% successful with this method. -- Signe Jackson

We would simply explain why management shouldn't want glitz and glamour in metrics presentations. But anyway, they do not want that and never will. -- Markku Povari

On the other hand, another distinguished field editor, Hugh Burley, had this comment on why every so often glitz does do the job:

"When management wants glitz, it is usually because they need to present content in a way that is compelling, in order to maximize the time they have in front of the executive or some other group. The challenge is to add the glitz while including the core message(s).

As an example, at a meeting of the executive level Information Security Committee, we needed to get action on the highest priority items in an orderly manner. This committee consists of Legal Council, the AVP Finance, the CIO and AVP of ITS, the Director of Facilities, a faculty member, and a student. They meet once a month for one hour. Several meetings had taken place with little progress, although large amounts of detailed information had been presented and discussed. The committee was in danger of losing momentum. The amount of information seemed overwhelming and the group appeared to be stuck in an endless cycle of discussion and reiteration of previous detail.

In order to get this process moving, a single, somewhat glitzy slide that crammed the information into a visual image capturing all the pertinent data was presented. There was some joking about the use of color and translucence to convey data, but the decision to move ahead with the highest priority item was taken, the committee was re-energized, and moved on to address the most critical issue. Score one for glitz."

And there you have it.