The problem sounded simple, harmonize the various audit questions like we did with control titles. It turned out to be anything but simple.

A seemingly simple question such as "Are all of the network access points accounted for in the network inventory?" has a lot of depth to it. Let's break this down to explain its subtle and underlying complexity. The key items in the question are network access points and network inventory.

Let's start with the inventory. In order for the organization to know what to inventory, they have to have a policy regarding inventory and change management, and within that policy they have to have a control that states what should be inventoried and how the inventory should be recorded. This tells us that we have to correlate the inventory records with the control and the policy to ensure that the evidence chain matches up.

And to answer the question, we have to then compare what we found in the inventory (by association of the control in the policy) to what we would find when we did "auditing by walking around" to see if they match.

In UCF parlance, the harmonized audit question format becomes "Examine a record sample of Records associated with the Control in the cDoc. Comparing this to a sample of Assets does this verify Default_Answer?"

We can then use that format to examine the Control mapped to the cDoc (policy, standard, procedure, etc.) as well as identify the Record Examples and Assets that are mapped to the Control. The UCF version can then be presented as:

Examine the sample of Network Inventory records associated with the control entitled Identify and control all network access points [UCF CE ID 00529] in the Network Change Management policy. Comparing this to a sample of Network Access Points does this verify the assets are listed in the records?

The UCF version of the audit question is more explicit and provides direct pointers to the auditable artifacts needed to support the claim (the records and policy).

Then there's the question of dealing with the assets; in this case, network access points, which are really routers and firewalls. Some audit question authors will write "access points" while others will write "routers and firewalls." It doesn't really matter, as they are the same. For simplicity in our audit questions, the UCF team have mapped the highest level asset in our hierarchy that applies. Because both Firewalls and Routers belong to Network Access Points in our hierarchy, we can ask a single question and allow the hierarchy to take care of the rest. If you care to take the audit documentation beyond the simple question format level, you can add additional content, such as the pointer to the hierarchical list of assets that meet the audit question criteria. Here's what the UCF team uses as an example:

The following is a list of applicable assets that meet the configuration criteria for Network:Network Access Points:::::: [UCF Asset ID 0001526]:

Network:Network Access Points:Routers::::: [UCF Asset ID 0001469]

Network:Network Access Points:Firewalls::::: [UCF Asset ID 0001524]

Audit Question Format harmonization

The biggest problem for harmonization came in when we decided that the best way to do this wasn't to attempt to harmonize the actual texts of the audit questions we found, but rather, to harmonize the methodologies that we found. Big difference. Harmonizing the texts would have presented us with hundreds of audit questions. Some unique, some duplicated. Harmonizing the methods have brought that number down under fifty. A very manageable number, and we think, a more robust and simpler way to present audit questions.

The answer to Audit Question Formats we are providing you is simple. We are providing you two methods to display the audit question, and two methods to handle answering the audit question.

To date, we have found 35 unique ways that the various audit guides have asked their audit questions in order to verify the answer they are looking for.

We arrived at this number of audit question formats using the following methodology.

First, when examining the audit questions, we ensured that we could reword the question so that the answer would be either Yes, No, or N/A. Second, we identified the various audit artifacts the audit questions were looking for in support of the answer. We found that the audit questions would rely upon examining the following either individually or collectively:

• • That the Compliance Document exists within the organization

  • Content, e.g., controls within the various compliance documents (policies, standards, procedures, et. al.)

  • Whether or not behaviors are being set or processes followed as documented within the content of the various compliance documents

  • Whether or not an Asset exists in the organization

  • Whether or not the Configurable Item exists for the asset in question

  • Configuration settings within the configurable items of assets

  • Whether or not Records exist in the organization

  • That Roles are defined within the organization

  • Whether roles are assigned to assets, records, or controls

  • Whether roles are staffed within the organization

Third, we looked at all of the ways the audit questions were being put together and came up with a list of de-duplicated methods. By de-duplicated methods, we mean that we examined each of the audit question formats we were reading to ensure that the same information wasn't being covered elsewhere. One audit question might state "examine a sample of records assigned to a control to ensure they coordinate with the compliance document" while a second question might reword the same question as "examine the compliance document to ensure they coordinate with the sample of records assigned to a control." By setting up a calculation to track each question's focus (Examine|cDoc|Control|Records|Role||, Examine|cDoc|Control|Records|Record_Category||) we could tell when a question format was unique or was a duplicate of an existing format.

Each time we found a new, format we entered the question method in the UCF's database. If the question calculated as unique, we added it to the list. If the question calculated to duplicate, we created a pointer to the unique version.

We also created a field we simply call default answer. This is a "free form" text field that our mappers use to add additional information to the questions so that they match the original intent as close as possible. By combining the various audit artifacts listed above (and bolded in the examples below) with harmonizing text, we've created a set of calculations to complete the UCF_Audit_Item_Question field. These are the question methods we have as of this writing.

• • Compare a cross sampling of records that contain the data element of DataField in the Record_Category to their related asset of Assets. Are they Default_Answer?

  • Examine the cDoc. Are a sample of Assets defined in the Control Default_Answer?

  • Examine the sample of Records as compared to their related asset of Assets. Is the Control in the cDoc being adhered to?

  • Examine the sample of Records associated with the Control in the cDoc. Comparing this to a sample of Assets does this verify Default_Answer?

  • Examine the sample of Records associated with a sample of Assets assigned to the Control in the cDoc. Does this verify the Configurable_Item Default_Answer?

  • Examine the setting Configuration_Setting of the configurable item Configurable_Item for a sample of Assets. Does this correlate with the Control in the cDoc?

  • Examine the Control in the cDoc. For a sample of Assets matching the setting Configuration_Setting of the configurable item Configurable_Item, does this verify Default_Answer?

  • Examine the setting Configuration_Setting of the configurable item Configurable_Item for a sample of Assets. Does this correlate with the role of Role and the Control in the cDoc?

  • Examine the setting Configuration_Setting of the configurable item Configurable_Item for a sample of Assets. Does this verify Default_Answer?

  • Examine the setting Configuration_Setting of the configurable item Configurable_Item for a sample of Assets. Does this correlate with the sample of Records?

  • Examine the setting Configuration_Setting of the configurable item Configurable_Item for a sample of Assets. Does this verify Default_Answer as defined in the sample of Records?

  • Examine the sample of Records. Can you verify they leverage automated tools such as a sample of Assets?

  • Examine the sample of Records. Do they correlate with training the role of Role to use a sample of Assets?

  • Examine the sample of Records. Do they correlate with how the role of Role uses a sample of Assets regarding the Default_Answer?

  • Examine the cDoc. If it exists, has it been reviewed in the last Review Interval N*?

  • Examine the cDoc. Is the Control included in it?

  • Examine the Control in the cDoc. Does this verify the DocType Default_Answer?

  • Examine the sample of Records. Do they correlate with the Control in the cDoc?

  • Examine the sample of Records. Do they correlate with the Control in the cDoc regarding Default_Answer?

  • Examine the Control in the cDoc as compared to records in the Record_Category. Do the two correlate?

  • Examine the sample of Records assigned to the role of Role. Does this coordinate with the Control in the cDoc?

  • Examine the sample of Records. Do they correlate with training the role of Role for the Control in the cDoc?

  • Examine the cDoc. Is the role of Role assigned to the Control?

  • Examine the cDoc. Is the role of Role Default_Answer?

  • Examine a cross sampling of records that contain the data element of DataField in the Record_Category. Are they Default_Answer?

  • Examine the Metric. Does this verify the Default_Answer?

  • Examine the Record_Category. Does the sample of Records exist?

  • Examine records in the Record_Category. Does the sample of Records Default_Answer?

  • Examine the sample of Records assigned to the role of Role. Does this verify Default_Answer?

  • Interview a sample of people in the role of Role about the Control in the cDoc. Does this verify the DocType Default_Answer?

  • Interview a sample of people in the role of Role. Are they using the cDoc to create and manage Records?

  • Interview a sample of people in the role of Role to ensure the Metric is being reported. If it is, is it timely and actionable?

  • Interview a sample of people in the role of Role about the Metric. Does this verify the metric Default_Answer?

  • Interview a sample of people in the role of Role about the sample of Records. Does this verify the Default_Answer?

  • Observe a sample of people in the role of Role performing the Control in the cDoc while working with records across the Record_Category. Does this confirm they are instructed to Default_Answer?

  • Observe a sample of people in the role of Role performing the Control in the cDoc while working on Records. Does this confirm they are instructed to Default_Answer?

  • Observe a sample of people in the role of Role performing the Control in the cDoc. Does this confirm they are instructed to Default_Answer?

  • Test a sample of Assets to ensure the configuration item of Configurable_Item is Configuration_Setting. Is this configured correctly?

  • Test the setting Configuration_Setting of the configurable item Configurable_Item for a sample of Assets. Does this verify Default_Answer?

Displaying and modifying the Audit Question

There are two ways you can handle the display of the audit question. You can either simply display it as is, or you can modify it to suit your needs.

If you don't want to use our format to create the question, you can always create your own calculation based off of the UCF_Audit_Item_Question_Method element.

Let's look at how to break down the UCF_Audit_Item_Question element and make a calculation out of it. Our sample will be:

Examine the cDoc. Are a sample of assets entitled Assets defined in the Control Default_Answer?

There are four UCF compound elements in this question method that are found within the Audit Basic Info element of the Audit list:

1. cDoc should be linked to UCF_cDoc_ID.

2. Assets should be linked to UCF_Asset_ID.

3. Control should be linked to UCF_CE_ID.

4. Default Answer should be linked to UCF_Audit_Item_Default_Answer.

In order to create the text for the element UCF_Audit_Item_Question, we created a Case statement calculation:

Audit_Item_Question_Method ="Examine the cDoc. Are the Assets defined in the Control default an-swer?";"

Examine the " & Lower(cDocs to Audit::UCF_cDoc_Type) & " entitled " & cDocs to Audit::Title with ID Reference & ". Are a sample of assets entitled " & Assets to Audit::TitlewithID_c & " defined in the control entitled " & Substitute(Controls to Audit::UCF_CE_Control_Title;".";"") & " [UCF CE ID " & Controls to Audit::UCF_CE_ID & "]"& " " & UCF_Audit_Item_Default_Answer & "?"

If, for any reason you don't like the way we've calculated our answer field, you can either create your own calculation and keep the information proprietary, or, you can suggest a better way for the UCF team to create the calculation. Trust us when we say we think we have a good answer to the problem, but we are more than happy to listen to any other method that works better.

How we are handling specific instances of variables in questions

During the initial testing of our audit question methodology (by running through about 500 different audit questions), we found that around ten percent of the questions had what we call "instance variables" associated with them. Many times audit questions will be very specific about aspects they are looking for in an audit answer. Aspects that each organization can legally determine on their own. For instance, many audit questions will demand that a certain length (such as password length) or complexity (such as using both upper and lower case characters in the password) be a part of the answer. When in truth, the organization doesn't have to follow the exact aspect as defined by the audit questions for many reasons. Therefore, when we found audit questions that were asked in a way that we could interchange a variable, we swapped out the specific as stated in the original audit question with the variable "N*".

Instead of reading "whether the inspections were carried out in the last quarter" for one audit guideline or "year" in another audit guideline or "annually/as necessary" in a third audit guideline, we wrote it as "whether the inspections have been carried out in the last N*".

So far, we have found variables of the following types in the audit questions:

• • Timeframe

  • Length

  • Complexity

  • Capacity

  • Retention Period

  • Frequency

  • Interval

  • Range

  • Compound

Each time the string N* appears in the UCF_Audit_Question element, we check the element for the type of variable that was presented and then enter the variable type in the element UCF_Audit_Item_Agreed_Upon_Variable. As an example, an audit question that would have the text "whether the inspections have been carried out in the last N*" in it would produce "Frequency" as the content of the UCF_Audit_Item_Agreed_Upon_Variable element.

We have one variable type we call compound because there are some instances where the audit question calls for a compound variable (such as the screen lock out variable, which can be based upon minutes of idle activity or some manual method).

What the heck do we do with the variables?

In strict auditing terms (and the reason we named the XML element this way), the answer to these variables has to be determined during the pre-questioning phase of the audit. Each time a question comes up that contains a variable regarding frequency, complexity, intervals, etc., it is the job of the auditor and the organization being audited to agree upon the contents of the variable.

Therefore, you'll want to ensure that your audit tool has the capability of asking about these variables before the audit begins, recording the agreed upon content of the variable, and incorporating that agreed upon content of the variable into the actual question and answer.

A simple substitute command can examine any question with the content of N* and replace it with the contents of a holder field. An example is:

Substitute(UCF_Audit_Question;"N*";Agreed_Upon_Content) where Agreed_Upon_Content represents the holder field, would result in N* being replaced by "every six months".

Why we use the term "Samples" in the questions

No one audits a router unless it is the only router the organization has. All of the routers that the organization and the auditor have determined are in scope are audited. This process of determining which organizational assets, records, processes, policies, etc. are "in play" for an audit is called scoping. The end result of the scoping process is a sample list of auditable artifacts, or even people, who are going to be brought into the audit process. Therefore our audit question language is defined to match this process by stating "sample" each time a question involves the investigation and auditing of more than one instance of the artifact or person being audited.

What can be sampled?

There are both simple samples and more complex samples that an audit can call for. We'll go through each of them here.

Records

When notifying the organization about record sampling, you'll want to call out the type of records that need to be examined, such as the specific log type, or client files in use. We suggest that you include the UCF records classification ID number when specifying the record type. You can always substitute your own classification index, but ours has been pretty well vetted already and contains specific items that are subject to IT audits, including client interaction files, and specific types for specific logs. Here is one of the examples we use when notifying the organization about a record sample they will need in order to answer the audit question:

Please list the specific instances of Asset-based Business Usage Documentation records [UCF Record Example ID 0000025] your organization will provide as samples in order to answer this audit question.

How your tool or the organization actually links those samples to the specific audit question is out of our hands. This could be anything from assigning legal BATES numbers to printed or PDF documents, attaching PDF files to a database record for the audit, to assigning a URI to find the files in question. Again, that's all up to you.

Roles

Remember that roles have to be assigned to people. And most of the audit questions that involve roles involve either the auditor interviewing the person in the role, or observing the person in the role do something. So when the audit question is calling for a "sample of persons in the role of...", that means someone is going to have to be in contact with one or more human beings assigned to the role. Below is a notification example that we use for the audit question. How your organization or tool tracks the contact information, and what the person's response to the audit question was, is out of the hands of the UCF team.

Please list the contact information for the person or persons assigned to Analyze and Manage Risks [UCF_Role_ID 0000065] we may contact in order to answer this audit question.

Assets

Now the sampling becomes a bit more complicated. Asset sampling doesn't just involve gathering a selected list of items for the audit. Asset sampling involves a hierarchical list of assets that are configured a very specific way because each UCF audit question links an asset to the question through a specific configuration state.

As documented in our write-up about how the UCF team maps configurable items, the UCF Asset List is a hierarchical list that begins with the asset category of "All Assets." Below that, we have several other categories of assets, including (but not limited to):

• • Operating Systems

  • Applications

  • Network

  • Hardware

When we map an asset record to a control, the UCF team always maps the highest-level hierarchical record possible. For our example, we'll use one of PCI's audit questions which asks the organization to check both routers and firewalls for a certain type of configuration. Instead of mapping two audit questions (one for routers and another for firewalls), the UCF team mapped the audit question to the asset record Network Access Points. The hierarchy in our asset list for Network Access Points puts the record one level below Network and one level above both Routers in general and Firewalls in general as depicted in the abbreviated hierarchy below:

Network
Network Access Points
Routers
DLink blah blah
Cisco blah blah
Firewalls
CheckPoint blah blah
Comodo blah blah

Back to our example, if we were presenting an audit question that was mapped at the highest level to a Network Access Point, the text we would present would be:

The following is a list of applicable assets that meet the configuration criteria for Network:Network Access Points:::::: [UCF Asset ID 0001526]:

Network:Network Access Points:Routers::::: [UCF Asset ID 0001469]
Network:Network Access Points:Routers:DIR:655::: [UCF Asset ID 0001487]
Network:Network Access Points:Firewalls::::: [UCF Asset ID 0001524]
Network:Network Access Points:Firewalls:PIX:500::: [UCF Asset ID 0001962]

Using the additional information to create an audit scoping document

To bring this home, just like variables, the sample set for anything being sampled needs to be agreed upon prior to the audit commencing. There is a concept in auditing called "scoping" in which the auditor and the group being audited meet to determine the values to be set for the variables and the scope of the sampled items for each audit question. The great thing about the UCF's audit questions is that you can either simply present the question as is, or you can modify it and provide different or additional information. In this case, you can use the information we just showed you to create a either a large scoping document to take into account all audit questions, or you can create a scoping document per audit question.

A sample audit question scope would look something like what we've presented below:

Sample Audit Question Scoping Worksheet