Creating your own metrics
If, for some reason, you've gone through all of the metrics that the Unified Compliance Framework has already mapped, and you can't find a metric that you want to employ, you are going to have to create one (or some) yourself.
And, as one of our distinguished field editors, Betsy Nichols, pointed out, "The truth is that 90% of all large public companies will want to create their own metrics, so don't assume this is an unusual circumstance."
If this is the case, before we get into the technical aspects of metrics, it bears repeating what we've said about good metrics before. In order for measurement reporting to be effective, measurements have to be presented to the right audience, and be actionable, timely, and relevant to the situation.
Following guidelines set by the National Institute of Standards and Technology, and a great whitepaper by Elizabeth Nichols and Andrew Sudbury, here are the main steps you'll need to follow to create your own metrics.
Identify the key stakeholders
You should have already done that in your first step. Remember that metrics are not simply measurements. They are analyses that help move the organization's compliance efforts forward, and are therefore subject to the needs and constraints of the stakeholders who are employing them and relying on them.
Each stakeholder might require his or her own set of customized measurements and analytics that provide a view of compliance performance for that stakeholder's area of responsibility.
Understanding and managing the business value is a key work function that must be assigned to someone on your metrics team if you are going to be defining your own metrics. In order to work with the stakeholders and be able to communicate about their business and compliance needs properly, whomever you assign the task of translating their needs into proper metrics should have at least the following knowledge or skills:
-
-
Knowledge of business processes and distributed computing
-
Knowledge of the organization's strategic plan, business conditions, and future goals
-
Ability to project impact of technology systems on business efficiencies
-
Knowledge of partner needs and business value and constraints
-
Knowledge of legal constraints and opportunities, and trends in legislation affecting business partnerships
-
Knowledge of business and cost issues in technology implementation
-
Knowledge of customer needs and business value and constraints
-
Set your goals
Is the goal of the metric clear to everyone involved? In other words, what decision are you trying to influence with this goal?
Remember that, in order to be useful, metrics must indicate the degree to which goals are being met. In other words, they need to drive actions that will improve the organization's compliance processes. Are you using these metrics for short term goals (to reduce vulnerabilities), or long term goals (to comply with certain aspects of ongoing contractual obligations)? The goal(s) for your metrics will determine which information you need to capture, and will, therefore, point you in the direction of whether or not you are gathering that information based upon defined and repeatable policies and procedures.
For our example, we'll say that our goal is to determine if we are protecting key information sources in a way that fulfills our compliance requirements. To begin with, we'll need to know how many systems have even been identified as falling under compliance requirements.
Determine the information sources are reliable
Is the metric based on defined and repeatable policies and procedures?
For the metric to be real, it has to be based upon repeatable measurements. A solitary report shows no trend. Also, in establishing a metrics program, the organization must establish the program around its current policies and procedures and compliance efforts. Why? Because policies and procedures should be establishing what the organization does, and the metrics should be used to document how well those policies and procedures are working. Policies and procedures provide the structure an organization needs in order to produce repeatable measurements.
This means that you are going to have to conduct a compliance framework review to ensure that the goals are tied to information sources as described in the compliance controls you are following. Continuing our example, let's say that you are developing metrics around the topic of risk management. One of the stakeholders wants to know (the goal of the metric) if you are hitting your mark of managing key systems (such as those systems that support the processing of payment card information) properly. One of the measures would be whether or not you have defined and identified (i.e., scoped) those systems appropriately. How would you know?
You'd know by checking against a reliable information source (your configuration management database) to see how many items in the database are tagged as being a part of a system managing payment card information. In this case, you could say that your information source matches the controls set forth by your compliance framework (for our example, "Identify information processes, applications, and systems significant to the organization [UCF Control ID 00688]").
Whose job is it to know if the information source for a metric matches that found within the compliance framework, and the policies and procedures that support the framework? The best answer we can give is whoever has the work function Manage IT and Compliance Policies and Standards assigned to them. That work function requires the following responsibilities:
Ensure the information is measurable
Does the metric yield quantifiable (percentage, average, etc.) information?
You can't easily measure happiness. You can measure monetary success. You can measure the number of systems that you have configured properly versus those that you have not. Make sure that, when you are deciding what to measure, you can explain the quantifiable yield of that measurement in a simple sentence that a cab driver or board member can understand. Formally stated, the metric for the control in our description would read like this:
-
-
Title Key assets for which an assurance strategy has been implemented
-
Formula # of key assets for which an assurance strategy has been implemented / # of IT assets in total
-
Target 100%
-
Data sources The number of assets that have assigned configuration standards and other controlling policies and procedures / the total number of assets in the organization's configuration management database
-
In order to ensure that this information is truly measurable, you are going to need to assign an information owner. The information owner's responsibilities will be to ensure that the data can be collected in a way that makes the data measurable. For instance, and to fit our example, the work function of Infrastructure Management is responsible for managing the current IT configuration [UCF CCI 00860], as well as maintaining configuration control and status accounting for each system [UCF CCI 00863]. Therefore, as a part of that responsibility, the person would need to ensure that the information can be gathered in a way that is measurable.
Ensure the information is repeatably obtainable
Can the data supporting the metric be readily and cost-effectively obtainable?
This is the big one - obtaining readily available and cost effective measurements. What you can gather, and the cost effectiveness of gathering those measurements, will be determined by your level of compliance maturity. Think, for a moment, in terms of a Capability Maturity Model (CMM), that measures awareness, policies and procedures, skills and expertise, responsibility and accountability, and tools and automation, with level 1 being no formal compliance processes and level 5 being your organization is completely buttoned up and continuously improving. At level 1 and 2 of those areas, your metrics won't be trusted because it would take a Herculean effort to even collect the data, which is probably non-existent. It won't be until the organization is into level 3 and 4 that you are getting anything more substantial than simple percentages or binary metrics.
With a nod to the Nichols and Sudbury whitepaper we mentioned earlier, here are the work function responsibilities for the function of performing monitoring and measurement that are assigned to this step:
Controlled responsibilities
-
-
Develop and maintain a metrics reporting standard and template [CCI 02157]
-
Establish and maintain an reporting methodology program [CCI 02072]
-
Non-controlled responsibilities
-
-
Analyze system performance to baseline.
-
Organize types of information required into categories for incorporation into metrics.
-
Connect the types of information produced by the data sources and to the categories of information required for incorporation into metrics.
-
Develop information models for each metric calculation, using the available data.
-
Identify any gaps remaining between information needs and availability when incorporating information into metrics. Some desired data may not be directly observable; as such, proxies or secondary models can be used.
-
Codify the metrics in a specific formula that can be recorded and communicated.
-
Express metrics by dimensions, obtained by joining source data with organizational or other information.
-
Complete the metric design by establishing and recording the metric specification.
-
Re-ensure that the metric is useful
After you've gone through all of that, you have to circle back and around and ask whether or not the metric is useful to (and desired by) the stakeholders of the organization. After you've analyzed whether the metric is readable, measurable, and obtainable, does it still meet the same goals as originally described? If yes, you are home free. If not, you'll need to revisit what changed.
Metrics should identify gaps in performance and expectations. The size and nature of the gap, as well as the importance of the activity being measured, will define the need to close the gap.
Mapping your controls to metrics
If you are creating your own metrics, chances are that you are doing so because you have reportable controls that don't have metrics already associated with them. In this case, you'll need to examine your controls and then define metrics for them.
Let's say that you have to follow the following control of "assign an appropriate number of IT personnel to the security awareness role." In order to create a metric for this, you'll need to ask yourself some questions and track your responses in a standard format.
Before you can even give your metric an ID and a title, you'll want to ask if you have standard policies and job descriptions that assign a certain number of people in IT positions to the work function of security awareness. If this policy is well documented, then you are on the right track. However, if no one knows that you are supposed to be doing this, then this isn't a metric you are going to want to tackle right now.
Once you know that you have a repeatable process to measure, the next question you'll want to ask is whether or not the process produces quantifiable information. And the answer in this case is yes, because you can count people in IT who are assigned the work function of security awareness versus the people in IT who are not. The next question for you will be to have someone in your organization make a definitive statement about the target number you should achieve. In this case, it would be goofy to have 100% of your IT personnel assigned the role of security awareness. So, we'll go with a target of 25% for discussion purposes.
The third question you'll want to answer involves where and how to get the data. In this case, it's pretty simple. You can get your base number (the total number of IT people within your organization) from human resources. You are then going to want to examine those job descriptions to check which ones have been assigned the work function of security awareness.
Finally, you'll want to formalize your calculation by dividing the number to be examined by the base number. If the resulting measurement is useful to your stakeholders, then you have defined your first metric. If it isn't, then either change the metric or toss the whole thing and move on to the next control.
Post a comment