All compliance programs can be broken down into six distinct categories that can be found within a CMMI, or capability maturity model; awareness, policies and procedures, tools and automation, skills and expertise, responsibility and accountability, and metrics. Each category is ranked from level 1 (ad hoc) through level 5 (mature and continuously improving).

A typical CMMI compliance matrix

The metrics within any compliance program will vary in their usefulness depending upon the level of maturity of the other five categories. If the organization is just beginning to recognize the need for a certain policy, that means that the approach to the policy and the supporting processes are probably ad hoc, ownership hasn't been firmly established, and thus, there are no metrics for that policy.

The effects of compliance maturity on metrics

There are three key initiatives an organization must undertake in any metrics program; ensuring the measurement data is available, collecting the measurement data, and then automating the metrics reporting process. We'll look at several levels of compliance maturity and discuss how each of these levels affects the compliance metrics program.

When metrics can't be trusted

What we've found is that there is no effect caused by an organization's level of maturity within the category of awareness of whether or not a control or suite of controls needs to be followed. Once the organization is aware that they need to follow a control, it doesn't matter if management is firmly aware of the need to act or if they are using sophisticated tools to share their belief in taking action. Awareness (other than not being aware of the control) simply doesn't affect a metrics program.

When metrics can't be trusted

It also seems to be the case that organizations have to be at least to level 3 of all other categories within compliance maturity before a real metrics program can take place. And that makes sense, because of policies and procedures aren't formalized, there are no real measurements that can even be taken. And how good would any standard of measure be if responsibility and accountability for a measured control hasn't even been formally assigned? Therefore, we can conclude that:

• • Data Availability is almost non-existent, as most processes are still informational and responsibility is informal as well.

  • Collection difficulty is a Herculean effort at best.

  • Collection automation does not exist.

When metrics are binary

We've also found that the organization can begin collecting basic "we are doing this or we aren't" type of metrics once the organization has reached maturity level 3 across the board, with tools and automation affecting data collection, but responsibility and accountability (once responsibility has been assigned) having little or no further effect.

When metrics are binary

Simply put, once policies and procedures are formalized, people are assigned responsibilities, people are trained in those responsibilities, a metrics program can begin in earnest. Once the program is running, collection difficulty depends both upon the tool set put into place and the collection methodologies the organization is using. Therefore, at this level we can conclude that:

• • Data Availability is dependent upon the policy and procedures including metrics capture.

  • Collection difficulty is dependent upon the "metric repositories" and collection methodologies that the organization puts into place.

  • Collection automation is dependent upon the tools, logs, and interpretation systems put into place.

When you can use metrics to note changes in status

Beyond stating "yes we do this because we can report it," metrics should be used as a methodology to report on changes in status, such as "the program is getting better/worse." That can only take place at about level 4, when policies and procedures are fully disseminated, leaders are leading, and the staff is trained.

When metrics can be used to note changes in status

Once the organization knows what it should be doing, and is training their staff to support the staff's responsibilities and accountability levels, the real use for metrics kicks into full swing because it can be used to point to trends of success or that a particular program or control did not make the grade. Therefore, at this level we can conclude that:

• • Data Availability is dependent upon the policy and procedures including metrics capture.

  • Collection difficulty should now be minimized because storage methodologies have been worked out and metrics reports are being treated like legal documents that are non-repudiable.

  • Collection automation is still going to be dependent upon how well the organization has defined and integrated their log and reporting tools into their metrics dashboards.

Metrics can be treated as statistically valid and fully analyzed

Not until an organization is very mature in all other areas of compliance capability can compliance metrics be treated as statistically valid measurement devices. The organization must be at a solid level 4 or greater in all areas.

Metrics as statistically valid measurement devices

Gathering metrics on your metrics

Signe Jackson, one of our distinguished field editors suggested that we needed to add a bit here at the end about metrics regarding the metrics you are using. Gong back to an earlier quote by one of our field editors who stated that metrics are only useful if they are used by someone, Signe states this:

"In order to determine if the metrics being reported on are effective and changing behavior, you need report on the % of reported metrics that have results that do not change over several reporting periods. Identifying the metrics that either have out lived their value or are just being reported to fulfill a requirement will help to either cull the dead metrics or modify the metrics awareness program."

As another field editor, Tonia Dudley, pointed out, It is important that once you've created your metrics, that you maintain them. This begins, then, wither measuring which metrics are being reported to the board and which are being discarded.

From there, you'll want to analyze which metrics both the audience, and the presenter, feel are best suited to the business demands the organization is currently facing.

And once you know which are in use, and which are useful, you can then tailor the more subtle points of each to determine if the target is correct (for you), if the information source is the best one for the type of reporting you need, or if there is a better source, and so on.

Basically, you'll want to go back through all of those steps we talked about earlier in creating metrics for yourself, and then apply them to maintaining and updating your metrics.

If you can do that, then you are on the road to metric optimization, which is a very good thing.