To determine the reliability of data, the auditor may consider all but which of the following?

                                                                 
Assessing the Reliability of Computer-Processed Data (01-OCT-02, 
GAO-03-273G).							 
                                                                 
GAO published a guide to assist its auditing staff in ensuring	 
the reliability of computer-based data. The guidance provides a  
flexible, risk-based framework for data reliability assessments  
that can be geared to the specific circumstances of each	 
engagement. The framework is built on (1) making use of all	 
existing information about the data; (2) performing at least a	 
minimal level of data testing; (3) doing only the amount of work 
necessary to determine whether the data are reliable enough for  
GAO's purposes; (4) maximizing professional judgment; and (5)	 
bringing the appropriate people, including management, to the	 
table at key decision points.					 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-03-273G					        
    ACCNO:   A05523						        
  TITLE:     Assessing the Reliability of Computer-Processed Data     
     DATE:   10/01/2002 
  SUBJECT:   Accounting procedures				 
	     Auditing standards 				 
	     Computer security					 
	     Data integrity					 
	     Information systems				 
	     Internal controls					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-03-273G

a

GAO United States General Accounting Office

Applied Research and Methods

October 2002 External Version 1 Assessing the

Reliability of Computer- Processed Data

GAO- 03- 273G

Page i GAO- 03- 273G Assessing Reliabillity

Contents Preface iii Section 1: Introduction 1 Section 2: Understanding
Data Reliability

3 Section 3: Deciding If a Data Reliability Assessment Is Necessary

6 Conditions Requiring a Data Reliability Assessment 8 Conditions Not
Requiring a Data Reliability Assessment 8

Section 4: Performing a Data Reliability Assessment

10 Timing the Assessment 10 Documenting the Assessment 10

Section 5: Viewing the Entire Assessment Process

12 Section 6: Taking the First Steps

14 Reviewing Existing Information 14 Performing Initial Testing 15 Dealing
with Short Time Frames 16

Section 7: Making the Preliminary Assessment

18 Factors to Consider in the Assessment 18 Outcomes to Consider in the
Assessment 21

Contents Page ii GAO- 03- 273G Assessing Reliabillity

Section 8: Conducting Additional Work

23 Tracing to and from Source Documents 24 Using Advanced Electronic
Testing 25 Reviewing Selected System Controls 26 Using Data of
Undetermined Reliability 27

Section 9: Making the Final Assessment

28 Sufficiently Reliable Data 29 Not Sufficiently Reliable Data 29 Data of
Undetermined Reliability 30

Section 10: Including Appropriate Language in the Report

31 Sufficiently Reliable Data 31 Not Sufficiently Reliable Data 31 Data of
Undetermined Reliability 32

Glossary of Technical Terms

33 Figures Figure 1: Factors to Consider in Making the Decision on Using
the

Data 1 Figure 2: Decision Process for Determining If a Data Reliability

Assessment Is Required 7 Figure 3: Data Reliability Assessment Process 13
Figure 4: The First Steps of the Assessment 14 Figure 5: The Preliminary
Assessment 19 Figure 6: Choosing and Conducting Additional Work 23 Figure
7: Making the Final Assessment 28

Page iii GAO- 03- 273G Assessing Reliability

Preface Computer- processed data, often from external sources,
increasingly underpin audit reports, including evaluations (performance
audits) and financial audits. Therefore, the reliability of such data has
become more and more important. Historically, computer- processed data
have been treated as unique evidence. However, these data are simply one
form of evidence relied on, although they may require more technical
assessment

than other forms of evidence. In addition, the very nature of the
information system creating the data allows opportunities for errors to be
introduced by many people.

This guidance is intended to demystify the assessment of computerprocessed
data. It supplements GAO*s *Yellow Book* (Government

Auditing Standards, 1994 Revision), which defines the generally accepted
government auditing standards (GAGAS), and replaces the earlier GAO
guidance, Assessing the Reliability of Computer- Processed Data (GAO/ OP-
8. 1. 3, Sept. 1990). For all types of evidence, various tests are used*
sufficiency, competence,

and relevance* to assess whether the evidence standard is met. You
probably have been using these tests for years and have become quite
proficient at them. But because assessing computer- processed data
requires more technical tests, it may appear that such data are subject to
a higher standard of testing than other evidence. That is not the case.
For

example, many of the same tests of sufficiency and relevance are applied
to other types of evidence. But in assessing computer- processed data, the
focus is on one test in the evidence standard* competence* which includes
validity and reliability. Reliability, in turn, includes the

completeness and accuracy of the data. This guidance, therefore, provides
a flexible, risk- based framework for data reliability assessments that
can be geared to the specific circumstances of each engagement. The
framework also provides a structure for planning and reporting,
facilitates bringing the right mix of skills to each engagement, and
ensures timely management buy- in on assessment strategies. The framework
is built on

 making use of all existing information about the data,  performing at
least a minimal level of data testing,

Preface Page iv GAO- 03- 273G Assessing Reliability

 doing only the amount of work necessary to determine whether the data
are reliable enough for our purposes,  maximizing professional judgment,
and  bringing the appropriate people, including management, to the table
at

key decision points. The ultimate goal of the data reliability assessment
is to determine whether you can use the data for your intended purposes.
This guidance is designed to help you make an appropriate, defensible
assessment in the most efficient manner. With any related questions, call
Barbara Johnson, focal point for data reliability issues, at (202) 512-
3663, or Barry Seltser, the Acting Director of GAO*s Center for Design,
Methods, and Analysis, at (202) 512- 3234.

Nancy Kingsbury Managing Director, Applied Research and Methods

Page 1 GAO- 03- 273G Assessing Reliability

Section 1: Introduction This guidance explains what data reliability means
and provides a framework for assessing the reliability of computer-
processed data. It begins with the steps in a preliminary assessment,
which, in many cases, may be all you need to do to assess reliability.
This guidance also helps you decide whether you should follow up the
preliminary assessment with additional work. If so, it explains the steps
in a final assessment and the

actions to take, depending on the results of your additional work. The
ultimate goal in determining data reliability is to make the following
decision: For our engagement, can we use the data to answer the research
question? See figure 1 for an overview of the factors that help to inform

that decision. Not all of these factors may be necessary for all
engagements.

Figure 1: Factors to Consider in Making the Decision on Using the Data Use
the data or not? Degree of risk Results of advanced electronic testing
Results of review of selected system controls Strength of corroborating
evidence Results of preliminary assessment Significance of data in
answering research question Results of tracing to or from source documents

Source: GAO.

Section 1: Introduction Page 2 GAO- 03- 273G Assessing Reliability

In addition, this guidance discusses suggested language* appropriate under
different circumstances* for reporting the results of your assessment.
Finally, it provides detailed descriptions of all the stages of the

assessment, as well as a glossary of technical terms used (see p. 33). An
on- line version of this guidance, which will include tools that may help
you in assessing reliability, is currently being developed. The overall
process is illustrated in figures 2 (p. 7) and 3 (p. 13).

Page 3 GAO- 03- 273G Assessing Reliability

Section 2: Understanding Data Reliability Data reliability refers to the
accuracy and completeness of computerprocessed data, given the intended
purposes for use. Computer- processed data include data (1) entered into a
computer system and (2) resulting from computer processing. Computer-
processed data can vary in form* from electronic files to tables in
published reports. The definition of computerprocessed data is therefore
broad. In this guidance, the term data always refers to computer-
processed data.

The *Yellow Book* requires that a data reliability assessment be performed
for all data used as support for engagement findings, conclusions, or
recommendations. 1 This guidance will help you to design a data
reliability assessment appropriate for the purposes of the engagement and
then to evaluate the results of the assessment.

Data are reliable when they are (1) complete (they contain all of the data
elements and records needed for the engagement) 2 and (2) accurate (they
reflect the data entered at the source or, if available, in the source
documents). A subcategory of accuracy is consistency. Consistency refers
to the need to obtain and use data that are clear and well- defined enough
to yield similar results in similar analyses. For example, if data are
entered at multiple sites, inconsistent interpretation of data rules can
lead to data that, taken as a whole, are unreliable. Reliability also
means that for any computer processing of the data elements used, the
results are reasonably

complete and accurate, meet your intended purposes, and are not subject to
inappropriate alteration.

Assessments of reliability should be made in the broader context of the
particular characteristics of the engagement and the risk associated with
the possibility of using data of insufficient reliability. Reliability
does not mean that computer- processed data are error- free. Errors are
considered acceptable under these circumstances: You have assessed the
associated risk and found the errors are not significant enough to cause a
reasonable

person, aware of the errors, to doubt a finding, conclusion, or
recommendation based on the data.

1 U. S. General Accounting Office, Government Auditing Standards, GAO/
OGC- 94- 4 (Washington, D. C.: June 1994), pp. 62- 87. 2 A data element is
a unit of information with definable parameters (for example, a Social
Security number), sometimes referred to as a data variable or data field.

Section 2: Understanding Data Reliability Page 4 GAO- 03- 273G Assessing
Reliability

While this guidance focuses only on the reliability of data in terms of
accuracy and completeness, other data quality considerations are just as
important. In particular, you should also consider the validity of data.
Validity (as used here) refers to whether the data actually represent what
you think is being measured. For example, if a data field is named *annual
evaluation score,* is this an appropriate measure of a person*s job
performance? Considerations of data validity and reliability issues should
be addressed early in the engagement, and appropriate technical
specialists* such as data analysts, statisticians, or information
technology specialists* should be consulted.

Page 5 GAO- 03- 273G Assessing Reliability

Page 6 GAO- 03- 273G Assessing Reliability

Section 3: Deciding If a Data Reliability Assessment Is Necessary

To decide if a data reliability assessment is necessary, you should
consider certain conditions. The engagement type and planned use of the
data help to determine when you should assess data reliability. See figure
2 for an illustration of the decision process that you should use.

Section 3: Deciding If a Data Reliability Assessment Is Necessary

Page 7 GAO- 03- 273G Assessing Reliability

Figure 2: Decision Process for Determining If a Data Reliability
Assessment Is Required

What is the type of engagement?

All other engagements Note: Primarily background information * Determine
if best available source  Disclose the source and that no

reliability assessment was performed Do you anticipate that the data will
be significant to findings,

conclusions, or recommendations? Conduct a computer system review and
disclose in OSM the work done, results, and any limitations found Will the
data be used on multiple future

engagements? Should you do a computer system review? Financial or
financial- related audit

Use guidance in FAM and FISCAM Continue with a data reliability assessment

No Yes Yes Not at this time No No Yes

Does the research question require a determination of the reliability of
an information

system? Yes

Source: GAO.

Section 3: Deciding If a Data Reliability Assessment Is Necessary

Page 8 GAO- 03- 273G Assessing Reliability

Conditions Requiring a Data Reliability Assessment

You should assess reliability if the data to be analyzed are intended to
support the engagement findings, conclusions, or recommendations. Keep in
mind that a finding may include only a description of the condition, as in
a purely descriptive report. In the audit plan for the engagement, you
should include a brief discussion of how you plan to assess data
reliability, as well as any limitations that may exist due to shortcomings
in the data.

Conditions Not Requiring a Data Reliability Assessment

You do not need to assess reliability if the data are used (1) only as
background information or (2) in documents without findings, conclusions,
or recommendations. Background information generally sets the stage for
reporting the results of an engagement or provides information that puts
the results in proper context. Such information could be the size of the
program or activity you are reviewing, for example. When you gather
background or other data, ensure that they are from the best available
source( s). When you present the data, cite the source( s) and state that
the data were not assessed.

Sometimes, as a best practice, however, you may want to do some assessment
of background data. Your judgment of the data*s importance and the
reliability of the source, as well as other engagement factors, can help
you determine the extent of such an assessment.

Section 3: Deciding If a Data Reliability Assessment Is Necessary

Page 9 GAO- 03- 273G Assessing Reliability

Finally, for financial audits and information system reviews, you should
not follow this guidance in assessing data reliability. For financial
audits, which include financial statement and financial- related audits,
you should follow the GAO/ PCIE Financial Audit Manual (FAM) and the
Federal Information System Controls Audit Manual (FISCAM). In an
information system review, all controls in a computer system, for the full
range of application functions and products, are assessed and tested. Such
a review includes (1) examining the general and application controls of a
computer system, 3 (2) testing whether those controls are being complied
with, and (3) testing data produced by the system. 4 To design such a
review, appropriate to the research question, seek assistance from
information technology specialists.

3 General controls refers to the structure, policies, and procedures*
which apply to all or a large segment of an organization*s information
systems* that help to ensure proper operation, data integrity, and
security. Application controls refers to the structure, policies, and
procedures that apply to individual application systems, such as inventory
or payroll.

4 Guidance for carrying out reviews of general and application controls is
provided in the U. S. General Accounting Office, Federal Information
System Controls Audit Manual,

GAO/ AIMD- 12.19.6 (Washington, D. C.: Jan. 1999).

Page 10 GAO- 03- 273G Assessing Reliability

Section 4: Performing a Data Reliability Assessment

To perform a data reliability assessment, you need to decide on the
timing* when to perform the assessment* and how to document it.

Timing the Assessment A data reliability assessment should be performed as
early as possible in the engagement process, preferably during the design
phase. The audit

plan should reflect data reliability issues and any additional steps that
still need to be performed to assess the reliability of critical data. The
engagement team generally should not finalize the audit plan or issue a
commitment letter until it has done initial testing and reviewed existing
information about the data and the system that produces the data. In
addition, the team should not commit to making conclusions or
recommendations based on the data unless the team expects to be satisfied
with the data reliability.

Documenting the Assessment

All work performed as part of the data reliability assessment should be
documented and included in the engagement workpapers. This includes all
testing, information review, and interviews related to data reliability.
In addition, decisions made during the assessment, including the final
assessment of whether the data are sufficiently reliable for the purposes
of the engagement, should be summarized and included with the workpapers.
These workpapers should be (1) clear about what steps the team took and
what conclusions they reached and (2) reviewed by staff with appropriate
skills or, if needed, technical specialists.

Page 11 GAO- 03- 273G Assessing Reliability

Page 12 GAO- 03- 273G Assessing Reliability

Section 5: Viewing the Entire Assessment Process

The ultimate goal of the data reliability assessment is to determine
whether you can use the data to answer the research question. The
assessment should be performed only for those portions of the data that
are relevant to the engagement. The extensiveness of the assessment is
driven by

 the expected significance of the data to the final report,  the
anticipated risk level of using the data, and  the strength or weakness
of any corroborating evidence. Therefore, the specific assessment process
should take into account these factors along with what is learned during
the initial stage of the assessment. The process is likely to be different
for each engagement.

The overall framework of the process for data reliability assessment is
shown in figure 3. The framework identifies several key stages in the
assessment, as well as actions and decisions expected as you move through
the process. The framework allows you to identify the appropriate mix of
assessment steps to fit the particular needs of your engagement. In most

cases, all of the elements in figure 3 would not be necessary in
completing the assessment. Specific actions for each stage are discussed
in sections 6- 10.

Section 5: Viewing the Entire Assessment Process

Page 13 GAO- 03- 273G Assessing Reliability

Figure 3: Data Reliability Assessment Process

Perform initial testing Obtain electronic or hard copy data What is known
about the data and the system? Making the Preliminary Assessment What is
the preliminary

assessment of reliability? Use data and disclose any limitations
Conducting Additional Work What is the final assessment of reliability?
Making the Final Assessment Take optional actions Take optional actions
Use data and disclose any limitations Anticipated significance of the data
in answering the research question Strength of corroborating evidence
Degree of risk involved Trace to or from source documents Use advanced
electronic testing Review selected system controls Sufficiently reliable
to answer research question Not sufficiently reliable to answer research
question Sufficientlyreliable to answer research question

Not sufficiently reliable to answer research question Action or
combination of actions Review existing information about the data and the
system Undetermined What is most appropriate

mix of additional

work?

Taking the First Steps

Consider these factors: Some options for additonal work:

Source: GAO.

Page 14 GAO- 03- 273G Assessing Reliability

Section 6: Taking the First Steps The data reliability process begins with
two relatively simple steps. These steps provide the basis for making a
preliminary assessment of data reliability: (1) a review of related
information and (2) initial testing (see figure 4). In some situations,
you may have an extremely short time frame for the engagement; this
section also provides some advice for this situation.

The time required to review related information and perform initial
testing will vary, depending on the engagement and the amount of risk
involved. As discussed in section 4, these steps should take place early
in the

engagement and include the team members, as well as appropriate technical
staff.

Figure 4: The First Steps of the Assessment

Reviewing Existing Information

The first step* a review of existing information* helps you to determine
what is already known about the data and the computer processing. The
related information you collect can indicate both the accuracy and
completeness of the entry and processing of the data, as well as how data
integrity is maintained. This information can be in the form of reports,
studies, or interviews with individuals who are knowledgeable about the
data and the system. Sources for related information include GAO, the
agency under review, and others.

Perform initial testing What is known about the data and the system?
Obtain electronic or hard copy

data Review existing

information about the data and the

system Action or combination of actions Source: GAO.

Section 6: Taking the First Steps Page 15 GAO- 03- 273G Assessing
Reliability

GAO GAO may already have related information in reports. Those from fiscal
year 1995 to the present are available via GAO*s Internet site. This site
also

provides other useful information: for example, as part of the annual
governmentwide consolidated financial audit, GAO*s Information Technology
Team is involved with reporting on the effectiveness of controls for
financial information systems at 24 major federal agencies.

Agency under Review Officials of the agency or entity under review are
aware of evaluations of their computer data or systems and usually can
direct you to both.

However, keep in mind that information from agency officials may be
biased. Consider asking appropriate technical specialists to help in
evaluating this information. Agency information includes Inspector General
reports, Federal Managers* Financial Integrity Act reports, Government
Performance and Results Act (GPRA) plans and reports, Clinger- Cohen Act
reports, and Chief Information Officer reports. (Some of this information
can be found in agency homepages on the Web.)

Others Other organizations and users of the data may be sources of
relevant information. To help you identify these sources, you can use a
variety of databases and other research tools, which include the
Congressional Research Service Public Policy Literature Abstracts and
organizations' Web

sites. Performing Initial Test i ng

The second step* initial testing* can be done by applying logical tests to
electronic data files or hard copy reports. For electronic data, you use
computer programs to test all entries of key data elements in the entire
data file. 5 Keep in mind that you only test those data elements you plan
to use for the engagement. You will find that testing with computer
programs

often takes less than a day, depending on the complexity of the file. For
5 Though an in- depth discussion of quality- assurance practices to be
used in electronic testing and analyses is beyond the scope of this
guidance, it is important to perform appropriate checks to ensure that you
have obtained the correct file. All too often, analysts receive an
incorrect file (an early version or an incomplete file). Appropriate steps
would include counting records and comparing totals with the responsible
agency or entity.

Section 6: Taking the First Steps Page 16 GAO- 03- 273G Assessing
Reliability

hard copy or summarized data* provided by the audited entity or retrieved
from the Internet* you can ask for the electronic data file used to create
the hard copy or summarized data. If you are unable to obtain electronic
data, use the hard copy or summarized data and, to the extent possible,
manually apply the tests to all instances of key data elements or, if the
report or summary is voluminous, to a sample of them.

Whether you have an electronic data file or a hard copy report or summary,
you apply the same types of tests to the data. These can include testing
for  missing data, either entire records or values of key data elements;
 the relationship of one data element to another;  values outside of a
designated range; and  dates outside valid time frames or in an illogical
progression. Be sure to keep a log of your testing for inclusion in the
engagement workpapers.

Dealing with Short Time Frames

In some instances, the engagement may have a time frame that is too short
for a complete preliminary assessment, for example, a request for
testimony in 2 weeks. However, given that all engagements are a function
of time, as well as scope and resources, limitations in one require
balancing the others.

Despite a short time frame, you may have time to review existing
information and carry out testing of data that are critical for answering
a research question, for example: You can question knowledgeable agency
staff about data reliability or review existing GAO or Inspector General
reports to quickly gather information about data reliability issues. In
addition, electronic testing of critical data elements for obvious errors
of

completeness and accuracy can generally be done in a short period of time
on all but the most complicated or immense files. From that review and
testing, you will be able to make a more informed determination about

whether the data are sufficiently reliable to use for the purposes of the
engagement. (See sections 7 and 8 for the actions to take, depending on
your determination.)

Page 17 GAO- 03- 273G Assessing Reliability

Page 18 GAO- 03- 273G Assessing Reliability

Section 7: Making the Preliminary Assessment The preliminary assessment is
the first decision point in the assessment process, including the
consideration of multiple factors, a determination of the sufficiency of
the data reliability with what is known at this point, and a decision
about whether further work is required. You will decide whether the data
are sufficiently reliable for the purposes of the engagement, not
sufficiently reliable, or as yet undetermined. Keep in mind that you are
not attesting to the overall reliability of the data or database. You are
only determining the reliability of the data as needed to support the
findings, conclusions, or recommendations of the engagement. As you gather

information and make your judgments, consult appropriate technical
specialists for assistance.

Factors to Consider in the Assessment

To make the preliminary assessment of the sufficiency of the data
reliability for the engagement, you should consider all factors related to
aspects of the engagement, as well as assessment work performed to this
point. As shown in figure 5, these factors include

 the expected significance of the data in the final report, 
corroborating evidence,  level of risk, and  the results of initial
assessment work.

Section 7: Making the Preliminary Assessment

Page 19 GAO- 03- 273G Assessing Reliability

Figure 5: The Preliminary Assessment

Expected Significance of the Data in the Final Report

In making the preliminary assessment, consider the data in the context of
the final report: Will the engagement team depend on the data alone to
answer a research question? Will the data be summarized or will detailed
information be required? Is it important to have precise data, making
magnitude of errors an issue?

Corroborating Evidence You should consider the extent to which
corroborating evidence is likely to exist and will independently support
your findings, conclusions, or

recommendations. Corroborating evidence is independent evidence that
supports information in the database. Such evidence, if available, can be
found in the form of alternative databases or expert views. It is unique
to each engagement, and its strength* persuasiveness* varies.

For help in deciding the strength or weakness of corroborating evidence,
consider the extent to which the corroborating evidence  is consistent
with the "Yellow Book" standards of evidence* sufficiency,

competence, and relevance;  provides crucial support;

What is the preliminary assessment of reliability? Sufficiently reliable
Not sufficiently reliable Undetermined Results of initial testing Results
of review of

existing information Strength of corroborating evidence Anticipated
significance of the data in answering the research question Degree of risk
involved Take optional actions Use data and disclose limitations, if any

Source: GAO.

Section 7: Making the Preliminary Assessment

Page 20 GAO- 03- 273G Assessing Reliability

 is drawn from different types of sources* testimonial, documentary,
physical, or analytical; and

 is independent of other sources. Level of Risk Risk is the likelihood
that using data of questionable reliability could have

significant negative consequences on the decisions of policymakers and
others. To do a risk assessment, consider the following risk conditions: 
The data could be used to influence legislation, policy, or a program that

could have significant impact.  The data could be used for significant
decisions by individuals or

organizations with an interest in the subject.  The data will be the
basis for numbers that are likely to be widely

quoted, for example, "In 1999, the United States owed the United Nations
about $1.3 billion for the regular and peacekeeping budgets."  The
engagement is concerned with a sensitive or controversial subject.  The
engagement has external stakeholders who have taken positions on the
subject.

 The overall engagement risk is medium or high.  The engagement has
unique factors that strongly increase risk. Bear in mind that any one of
the conditions may have more importance than another, depending on the
engagement.

Results of Initial Assessment Work

At this point, as shown in figure 5 (p. 19), the team will already have
performed the initial stage of the data reliability assessment. They
should have the results from the (1) review of all available existing
information about the data and the system that produced them and (2)
initial testing of the critical data elements. These results should be
appropriately documented and reviewed before the team enters into the
decision- making phase of the preliminary assessment. Because the results
will, in whole or in part, provide the evidence that the data are
sufficiently reliable* and therefore competent enough* or not sufficiently
reliable for the purposes

Section 7: Making the Preliminary Assessment

Page 21 GAO- 03- 273G Assessing Reliability

of the engagement, the workpapers should include documentation of the
process and results.

Outcomes to Consider in the Assessment

The results of your combined judgments of the strength of corroborating
evidence and degree of risk suggest different assessments. If the
corroborating evidence is strong and the risk is low, the data are more
likely to be considered sufficiently reliable for your purposes. If the
corroborating evidence is weak and the risk is high, the data are more
likely to be considered not sufficiently reliable for your purposes. The
overall assessment is a judgment call, which should be made in the context
of discussion with team management and technical specialists.

The preliminary assessment categorizes the data as sufficiently reliable,
not sufficiently reliable, or of undetermined reliability. Each category
has implications for the next steps of the data reliability assessment.

When to Assess Data as Sufficiently Reliable for Engagement Purposes

You can assess the data as sufficiently reliable for engagement purposes
when you conclude the following: Both the review of related information
and the initial testing provide assurance that (1) the likelihood of
significant errors or incompleteness is minimal and (2) the use of the
data

would not lead to an incorrect or unintentional message. You could have
some problems or uncertainties about the data, but they would be minor,
given the research question and intended use of the data. When the

preliminary assessment indicates that the data are sufficiently reliable,
use the data. When to Assess Data as Not Sufficiently Reliable for
Engagement Purposes

You can assess the data as not sufficiently reliable for engagement
purposes when you conclude the following: The review of related
information or initial testing indicates that (1) significant errors or
incompleteness exist in some or all of the key data elements and (2) using
the data would probably lead to an incorrect or unintentional message.

When the preliminary assessment indicates that the data are not
sufficiently reliable, you should seek evidence from other sources,
including (1) alternative computerized data* the reliability of which you
should also assess* or (2) original data in the form of surveys, case
studies, or expert interviews.

Section 7: Making the Preliminary Assessment

Page 22 GAO- 03- 273G Assessing Reliability

You should coordinate with the requester if seeking evidence from other
sources does not result in a source of sufficiently reliable data. Inform
the requester that such data, needed to respond to the request, are
unavailable. Reach an agreement with the requester to

 redefine the research questions to eliminate the need to use the data, 
end the engagement, or  use the data with appropriate disclaimers.
Remember that you* not the requester* are responsible for deciding what
data to use. If you decide you must use data that you have determined are

not sufficiently reliable for the purposes of the engagement, make the
limitations of the data clear, so that incorrect or unintentional
conclusions will not be drawn. Finally, given that the data you assessed
have serious reliability weaknesses, you should include this finding in
the report and recommend that the agency take corrective action. When to
Assess Data as of

Undetermined Reliability and Consider Additional Work

You can assess the data as of undetermined reliability when you conclude
one of the following:

 The review of some of the related information or initial testing raises
questions about the data*s reliability.

 The related information or initial testing provides too little
information to judge reliability.

 The time or resource constraints limit the extent of the examination of
related information or initial testing. When the preliminary assessment
indicates that the reliability of the data is undetermined, consider doing
additional work to determine reliability. Section 8 provides guidance on
the types of additional work to consider, as well as suggestions if no
additional work is feasible.

Page 23 GAO- 03- 273G Assessing Reliability

Section 8: Conducting Additional Work When you have determined (through
the preliminary assessment) that the data are of undetermined reliability,
consider conducting additional work (see figure 6). A range of additional
steps to further determine data reliability includes tracing to and from
source documents, using advanced electronic testing, and reviewing
selected system controls. The mix depends on what weaknesses you
identified in the preliminary assessment

and the circumstances specific to your engagement, such as risk level and
corroborating evidence, as well as other factors. Focus particularly on
those aspects of the data that pose the greatest potential risk for your
engagement. You should get help from appropriate technical specialists to

discuss whether additional work is required and to carry out any part of
the additional reliability assessment.

Figure 6: Choosing and Conducting Additional Work

What is most appropriate mix of additional work? Action or combination of
actions Consider these factors: Anticipated significance of the data in
answering the research question Strength of corroborating evidence Degree
of risk involved Trace to or from source documents Use advanced electronic
testing Review selected system controls Some options for additional work:
Results of initial testing Results of review of existing information
Source: GAO.

Section 8: Conducting Additional Work Page 24 GAO- 03- 273G Assessing
Reliability

Tracing to and from Source Documents

Tracing a sample of data records to source documents helps you to
determine whether the computer data accurately and completely reflect
these documents. In deciding what and how to trace, consider the relative
risks to the engagement of overstating or understating the conclusions
drawn from the data, for example: On the one hand, if you are particularly
concerned that questionable cases might not have been entered into the
computer system and that as a result, the degree of compliance may be
overstated, you should consider tracing from source documents to the
database. On the other hand, if you are more concerned that ineligible
cases have been included in the database and that as a result, the
potential

problems may be understated, you should consider tracing from the database
back to source documents.

The reason to trace only a sample is because sampling saves time and cost.
To be useful, however, the sample should be random and large enough to
estimate the error rate within reasonable levels of precision. Tracing a
random sample will provide the error rate and the magnitude of errors for
the entire data file. It is this error rate that helps you to determine
the data reliability. Generally, every data file will have some degree of
error (see example 1 for error rate and example 2 for magnitude of
errors). Consult statisticians to assist you in selecting the sampling
method most suited to the engagement.

Example 1: According to a random sample, 10 percent of the data records
have incorrect dates. However, the dates may be off by an average of only
3 days. Depending on what the data are used for, 3 days may not

compromise reliability.

Example 2: The value of a data element was incorrectly entered as $100,
000, rather than $1, 000, 000. The documentation of the database shows
that the acceptable range for this data element is between $100 and $5,
000, 000. Therefore, the electronic testing done in the initial testing
phase would have confirmed that the value of $100,000 fell within that
range. In this case, the error could be caught, not by electronic testing,
but only by tracing the data to source documents.

Section 8: Conducting Additional Work Page 25 GAO- 03- 273G Assessing
Reliability

Tracing to Source Documents

Consider tracing to source documents when (1) the source documents are
available relatively easily or (2) the possible magnitude of errors is
especially critical.

To trace a sample to source documents, match the entered data with the
corresponding data in the source documents. But in attempting to trace
entered data back to source documents, several problems can arise: Source
documents may not be available because they were destroyed, were never
created, or are not centrally located.

Several options exist if source documents are not available. For those
documents never created* for example, when data may be based on electronic
submissions* use interviews to obtain related information, any
corroborating evidence obtained earlier, or a review of the adequacy of
system controls.

Tracing from Source Documents

Consider tracing from source documents, instead of or in addition to
tracing a sample to source documents, when you have concerns that the data
are not complete. To trace a sample from source documents, match the
source documents with the entered data. Such tracing may be appropriate to
determine whether all data are completely entered. However, if source
documents were never created or are now missing, you cannot identify the
missing data.

Using Advanced Electronic Testing

Advanced electronic testing goes beyond the basic electronic testing that
you did in initial testing (see section 5). It generally requires
specialized computer programs to test for specific conditions in the data.
Such testing can be particularly helpful in determining the accuracy and
completeness of processing by the application system that produced the
data. Consider using advanced electronic testing for

 following up on troubling aspects of the data* such as extremely high
values associated with a certain geographic location* found in initial
testing or while analyzing the data;

Section 8: Conducting Additional Work Page 26 GAO- 03- 273G Assessing
Reliability

 testing relationships* cross- tabulation* between data elements, such as
whether data elements follow a skip pattern from a questionnaire; and

 verifying that computer processing is accurate and complete, such as
testing a formula used in generating specific data elements.

Depending on what will be tested, this testing can require a range of
programming skills* from creating cross- tabulations on related data
elements to duplicating an intricate automated process with more advanced
programming techniques. Consult appropriate technical specialists, as
needed.

Reviewing Selected System Controls

Your review of selected system controls* the underlying structures and
processes of the computer in which the data are maintained* can provide
some assurance that the data are sufficiently reliable. Examples of system
controls are limits on access to the system and edit checks on data
entered

into the system. Controls can reduce, to an acceptable level, the risk
that a significant mistake could occur and remain undetected and
uncorrected. Limit the review to evaluating the specific controls that can
most directly affect the reliability of the data in question. Choose areas
for review on the basis of what is known about the system. Sometimes, you
identify potential system control problems in the initial steps of the
assessment. Other times, you learn during the preliminary assessment that
source documents are not readily available. Therefore, a review of
selected

system controls is the best method to determine if data were entered
reliably. If needed, consult information system auditors for help in
evaluating general and application controls.

Using what you know about the system, concentrate on evaluating the
controls that most directly affect the data. These controls will usually
include (1) certain general controls, such as logical access and control
of changes to the data, and (2) the application controls that help to
ensure that the data are accurate and complete, as well as authorized.

The steps for reviewing selected system controls are  gain a detailed
understanding of the system as it relates to the data and  identify and
assess the application and general controls that are critical to ensuring
the reliability of the data required for the engagement.

Section 8: Conducting Additional Work Page 27 GAO- 03- 273G Assessing
Reliability

Using Data of Undetermined Reliability

In some situations, it may not be feasible to perform any additional work,
for example, when (1) given a short time frame (too short for a complete
assessment), (2) original computer files have been deleted, or (3) access
to needed documents is unavailable. See section 9 for how to proceed.

Page 28 GAO- 03- 273G Assessing Reliability

Section 9: Making the Final Assessment During the final assessment, you
should consider the results of all your previous work to determine
whether, for your intended use, the data are sufficiently reliable, not
sufficiently reliable, or still undetermined. Again, remember that you are
not attesting to the reliability of the data or database. You are only
determining the sufficiency of the reliability of the data for your
intended use. The final assessment will help you decide what actions to
take (see figure 7).

Figure 7: Making the Final Assessment

The following are some considerations to help you decide whether you can
use the data:

 The corroborating evidence is strong.  The degree of risk is low.  The
results of additional assessment (1) answered issues raised in the

preliminary assessment and (2) did not raise any new questions.

What is the final assessment of reliability? Not sufficiently reliable
Sufficiently reliable Use data and disclose limitations, if any Results of
any additional work Results of review of

existing information Results of Initial testing Degree of risk involved
Strength of corroborating evidence Significance of the data in answering
the research question optional actions Take Source: GAO.

Section 9: Making the Final Assessment Page 29 GAO- 03- 273G Assessing
Reliability

 The error rate, in tracing to or from source documents, did not
compromise reliability.

In making this assessment, you should consult with appropriate technical
specialists.

Sufficiently Reliable Data

You can consider the data sufficiently reliable when you conclude the
following: On the basis of the additional work, as well as the initial
assessment work, using the data would not weaken the analysis nor lead to
an incorrect or unintentional message. You could have some problems or

uncertainties about the data, but they would be minor, given the research
question and intended use of the data. When your final assessment
indicates that the data are reliable, use the data.

Not Sufficiently Reliable Data

You can consider the data to be not sufficiently reliable when you
conclude the following: On the basis of information drawn from the
additional assessment, as well as the preliminary assessment, (1) using
the data would most likely lead to an incorrect or unintentional message
and (2) the data have significant or potentially significant limitations,
given the research question and intended use of the data.

When you determine that the data are not sufficiently reliable, you should
inform the requester that sufficiently reliable data, needed to respond to
the request, are unavailable. Remember that you* not the requester* are
responsible for deciding what data to use. Although the requester may want
information based on insufficiently reliable data, you are responsible for
ensuring that data are used appropriately to respond to the requester. If
you decide to use the data for the report, make the limitations of the
data clear, so that incorrect or unintentional conclusions will not be
arrived at. Appropriate team management should be consulted before you
agree to

use data that are not sufficiently reliable. Finally, given that the data
you assessed have serious reliability weaknesses, you should include this
finding in the report and recommend that the agency take corrective
action.

Section 9: Making the Final Assessment Page 30 GAO- 03- 273G Assessing
Reliability

Data of Undetermined Reliability

You can consider the data to be of undetermined reliability when you
conclude the following: On the basis of the information drawn from any
additional work, as well as the preliminary assessment, (1) use of the
data could lead to a incorrect or unintentional message and (2) the data
have significant or potentially significant limitations, given the
research question and the intended use. You can consider the data to be of
undetermined reliability if specific factors* such as short time frames,
the deletion of original computer files, and the lack of access to needed
documents* are present. If you decide to use the data, make the
limitations of the data clear, so that incorrect or unintentional
conclusions will not be arrived at.

As noted above in the case of not sufficiently reliable data, when you
determine that the data are of undetermined reliability, you should inform
the requester* if appropriate* that sufficiently reliable data, needed to
respond to the request, are unavailable. Remember that you* not the

requester* are responsible for deciding what data to use. Although the
requester may want information based on data of undetermined reliability,
you are responsible for ensuring that appropriate data are used to respond
to the requester. If you decide to use the data in your report, make the

limitations clear, so that incorrect or unintentional conclusions will not
be arrived at. Appropriate team management should be consulted before you
agree to use data of undetermined reliability.

Page 31 GAO- 03- 273G Assessing Reliability

Section 10: Including Appropriate Language in the Report

In the report, you should include a statement in the methodology section
about conformance to generally accepted government auditing standards
(GAGAS). These standards refer to how you did your work, not how reliable
the data are. Therefore, you are conforming to GAGAS as long as, in
reporting, you discuss what you did to assess the data; disclose any data
concerns; and reach a judgment about the reliability of the data for use
in the report.

Furthermore, in the methodology section, include a discussion of your
assessment of data reliability and the basis for this assessment. The
language in this discussion will vary, depending on whether the data are
sufficiently reliable, not sufficiently reliable, or of undetermined
reliability. In addition, you may need to discuss the reliability of the
data in other

sections of the report. Whether you do so depends on the importance of the
data to the message.

Sufficiently Reliable Data

Present your basis for assessing the data as sufficiently reliable, given
the research questions and intended use of the data. This presentation
includes (1) noting what kind of assessment you relied on, (2) explaining
the steps in the assessment, and (3) disclosing any data limitations. Such
disclosure includes

 telling why using the data would not lead to an incorrect or
unintentional message,

 explaining how limitations could affect any expansion of the message,
and

 pointing out that any data limitations are minor in the context of the
engagement.

Not Sufficiently Reliable Data

Present your basis for assessing the data as not sufficiently reliable,
given the research questions and intended use of the data. This
presentation should include what kind of assessment you relied on, with an
explanation of the steps in the assessment.

Section 10: Including Appropriate Language in the Report

Page 32 GAO- 03- 273G Assessing Reliability

In this explanation, (1) describe the problems with the data, as well as
why using the data would probably lead to an incorrect or unintentional
message, and (2) state that the data problems are significant or
potentially significant. In addition, if the report contains a conclusion
or recommendation supported by evidence other than these data, state that
fact. Finally, if the data you assessed are not sufficiently reliable, you
should include this finding in the report and recommend that the audited
entity take corrective action.

Data of Undetermined Reliability

Present your basis for assessing the reliability of the data as
undetermined. Include such factors as short time frames, the deletion of
original computer files, and the lack of access to needed documents.
Explain the reasonableness of using the data, for example: These are the
only available data on the subject; the data are widely used by outside
experts or

policymakers; or the data are supported by credible corroborating
evidence. In addition, make the limitations of the data clear, so that
incorrect or unintentional conclusions will not be drawn from the data.
For example, indicate how the use of these data could lead to an incorrect
or unintentional message. Finally, if the report contains a conclusion or
recommendation supported by evidence other than these data, state that
fact.

Page 33 GAO- 03- 273G Assessing Reliability

Glossary of Technical Terms

accuracy. Freedom from error in the data.

completeness. The inclusion of all necessary parts or elements.

database. A collection of related data files (for example, questionnaire
responses from several different groups of people, with each group*s
identity maintained.)

data element. An individual piece of information that has definable
parameters, sometimes referred to as variables or fields (for example, the
response to any question in a questionnaire).

data file. A collection of related data records, also referred to as a
data set (for example, the collected questionnaire responses from a group
of people).

data record. A collection of related data elements that relate to a
specific event, transaction, or occurrence (for example, questionnaire
responses about one individual* such as age, sex, and marital status).

source document. Information that is the basis for entry of data into a
computer.

(460522)

GAO*s Mission The General Accounting Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve

the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO*s commitment to good government is reflected in its core
values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail this

list to you every afternoon, go to www. gao. gov and select *Subscribe to
daily E- mail alert for newly released products* under the GAO Reports
heading.

Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO

also accepts VISA and Mastercard. Orders for 100 or more copies mailed to
a single address are discounted 25 percent. Orders should be sent to:

U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548

To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax: (202)
512- 6061

To Report Fraud, Waste, and Abuse in Federal Programs

Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm E- mail:
[email protected] gao. gov Automated answering system: (800) 424- 5454 or (202)
512- 7470

Public Affairs Jeff Nelligan, managing director, [email protected] gao. gov (202)
512- 4800 U. S. General Accounting Office, 441 G Street NW, Room 7149

Washington, D. C. 20548

United States General Accounting Office Washington, D. C. 20548- 0001

Official Business Penalty for Private Use $300

Address Service Requested Presorted Standard

Postage & Fees Paid GAO Permit No. GI00
*** End of document. ***

Which of the following types of audit evidence is the least reliable?

Which of the following is correct regarding including client data in audit documentation for an issuer?

Which of the following is the best factor for determining the required data during audit planning phase?

Which of the following documents would be the most reliable evidence for an audit?