10-08 - Audit of SBA’s Efforts to Improve the Quality of Acquisition Data in the Federal Procurement Data System

U.S Small Business Administration

Memorandum

Office of Inspector General

To: Darryl K. Hairston

Associate Administrator Office of Management and Administration

/s/ Original Signed

From: Debra S. Ritt

Assistant Inspector General for Auditing

Subject: Audit of the SBA's Efforts to Improve the Quality of Acquisition Data in the Federal Procurement Data System, Report No. 10-08

This report presents the results of our audit of the quality of SBA' s acquisition data in the Federal Procurement Data System (FPDS). Based on data quality concerns raised by the Government Accountability Office (GAO), in 2007 the Office of Management and Budget (OMB) issued a directive requiring Federal agencies to annually certify to the accuracy of procurement data reported in FPDS. Data in FPDS must be accurate because it is used to report on Government contracting actions, procurement trends, achievement of small business goals, and to report contract activity under the American Reinvestment and Recovery Act (Recovery Act). In 2008, OMB also required that agencies annually implement data quality plans. Beginning in fiscal year (FY) 2005, OMB required Federal agencies to convert to a contract writing system that directly interfaces with FPDS.(1) SBA met this requirement in 2009 when it converted from Comprizon to PRIZM.

The objectives of the audit were to determine whether SBA had (1) taken adequate steps to ensure the accuracy of its FY 2008 acquisition data, and (2) developed a data quality plan to effectively address any data weaknesses. Additionally, at management's request, we determined whether the quality ofFPDS data improved with the conversion to a new contract writing system in FY 2009. To achieve the audit objectives, we examined SBA's process for ensuring the accuracy of its acquisition data; reviewed relevant policies, procedures, and regulations; reviewed a copy of SBA' s FY 2008 FPDS Data Quality Plan and SBA' s FY 2008 certification to OMB; and interviewed personnel in SBA's Office of Business Operations (OBO). We also analyzed a random sample of SBA's contract actions -36 in FY 2008 and 35 in FY 2009(2) to determine whether the data in FPDS matched information contained in SBA's contract files. In assessing data accuracy, we focused our review on 18 of the 46 key data elements identified by OMB, that were either legislatively mandated(3) or which described the nature of the contract action. The data elements selected for review and our sampling methodology are contained in Appendix I. We conducted our audit between May 2009 and December 2009 in accordance with Government Auditing Standards prescribed by the Comptroller General of the United States.

We found that SBA certified to the accuracy of its FY 2008 contracting data, although 92 percent of the contract actions in our sample contained one or more inaccurate or incomplete data elements in FPDS. While SBA had developed a data quality plan for FY 2008, it did not fully implement the plan, which contributed to the errors identified. Further, due to the volume of errors identified in FPDS, it appears that contracting personnel did not review FPDS data inputs to ensure they reflected accurate information, as required by the Federal Acquisition Regulations (FAR). Finally, while the accuracy of some data elements improved in FY 2009, overall there was a higher rate of error in the FY 2009 data. Approximately 97 percent of the contract actions in our sample contained one or more inaccurate or incomplete data elements, indicating again that SBA contracting personnel were not properly reviewing data entries.

We recommended that the Associate Administrator for Management and Administration identify steps that will be taken to ensure that contracting data is reviewed for accuracy and completeness, conduct an independent review to determine whether the Data Quality Plan has been fully implemented, and hold OBO personnel accountable for the accuracy of FPDS data. Management concurred with the report findings and recommendations and agreed to take appropriate action.

BACKGROUND

FAR requires Federal agencies to enter information on all of their reportable contract actions (4) into FPDS (5) and to certify annually to its accuracy.(6) Congress, Federal agencies, and the public rely on FPDS for a broad range of data on contracting actions and spending. This information is also used to assess the extent to which Federal agencies are meeting their small business goals and to report to Recovery.gov on contracting activity financed by the Recovery Act.

In September 2005, the GAO reported a lack of confidence in the ability of FPDS to provide timely and accurate data and recommended that agencies verify the accuracy and completeness of their data in FPDS. (7) Based on concerns raised by GAO and the need to ensure the public has accurate data on Federal procurements, beginning in 2007, OMB directed procuring agencies to take steps to improve the quality of data reported to FPDS.(8) Additionally in FY 2008, OMB required Federal agencies to submit Data Quality Plans, including actions that would be taken to validate contract actions entered into FPDS. In complying with this requirement, agencies were to determine the accuracy rates of 46 key data elements prescribed by OMB based on either a review of a statistical sample of contract actions or an alternative methodology. The FY 2008 Data Quality Plan was due to OMB by July 15,2008. Finally, by January 5, 2009, all agencies were required to certify that all reportable contract actions awarded during FY 2008 were entered into FPDS as fully and accurately as possible.

OBO is responsible for ensuring and certifying the accuracy and completeness of all SBA procurement information contained in FPDS. During FY 2008, SBA used Comprizon as its primary contract writing system, which did not have a direct interface with FPDS. Therefore, SBA manually entered the FY 2008 contract actions into FPDS. In FY 2009, the Agency converted to PRISM, which allowed contracting personnel to electronically transmit to FPDS procurement data for select fields. OBO contracting personnel manually enters information for all other fields.

SBA certified that all 855 contract actions for FY 2008, valued at $72 million, were entered into FPDS as fully and accurately as possible. At the time of our audit fieldwork, the Agency had not certified to the accuracy of its FY 2009 data. Since issuance of the draft report, SBA certified on January 5, 2010, that 99 percent of all contract actions for FY 2009, valued at $99 million, was entered into FPDS as fully and accurately as possible.

RESULTS

SBA Did Not Take Sufficient Action to Ensure the Accuracy of its FY 2008 FPDS Data

SBA's process for ensuring the accuracy of FY 2008 FPDS data was not adequate. According to OBO contracting personnel, after contract data was entered into FPDS, they printed a copy of the FPDS information and compared it to information in the contract file. If any discrepancies were identified between the two sets of information, corrections should have been made before finalizing the data in FPDS. While contract files contained a copy of the FPDS printout, our review of 36 sampled contract actions reported in FPDS disclosed that 33, or 92 percent, contained one or more inaccurate entries (9) and/or were missing data elements. Of the 33 contract actions that had problems, 32 involved inaccurate data, and 11 had incomplete data elements. Some of the more prevalent data errors included:

13 contract actions that had incorrect Data Universal Numbering System (DUNS) numbers-the contractor's unique identifier used for data retrieval. This type of error could result in an over-or understatement of contractor activity.

11 that incorrectly reported the size of the business, the majority of which consisted of businesses being reported in FPDS as small, but the contract file reflected that they were other than small. The accuracy of this data element is important for SBA to properly receive credit for meeting their small business goals.

6 that did not have North American Industrial Classification System (NAICS) codes that matched those in the contract files. The NAICS code is important because it provides the standard for determining whether a business is small, and eligible to receive small business contracts in a given industry.

11 contract actions did not contain an accurate award type, which restricts the data fields for data entry. For example, because a Delivery Order was incorrectly entered as a Purchase Order, pertinent data fields for the Delivery Order were not accessible in the drop-down menu. This type of error also skews the procurement statistics for the Agency and the Federal government that is reported in USAspending.gov.

18 did not contain correct zip codes for the place of contract performance, causing some contracts to be reported as performed in the wrong Congressional district.

Additionally, 11 contract actions (10 of which also had accuracy issues), had at least 1 data element that was blank in FPDS, even though it was a required element for the contract action based on the award type. For example, incomplete fields included the type of contract, description of requirements, and principal NAICS code, among others.

Based on our sample results, we estimate that at least 681 of the 850 contract actions reported in FY 2008 had inaccurate or incomplete information. Our statistical projections are detailed in Appendix I, and a further description of the data errors by element is provided in Appendix II.

SBA Did Not Fully Implement Its Data Quality Plan

In preparation for its FY 2008 certification, the Agency established and submitted a Data Quality Plan to OMB that detailed steps to ensure the accuracy and completeness of information being reported in FPDS. Specifically, the Plan called for:

Verifying and validating FPDS information to ensure the data entered was accurate and complete before awarding the contract.

Verifying that contract obligations matched the information entered into FPDS.

Verifying and validating FPDS data through a monthly statistical sampling of data entries.

Reviewing obligations to ensure that all contract actions were entered into FPDS.

Conducting an independent review, consisting of a statistically valid comparison of FPDS data and contract files.

Although the plan included steps to review and verify the completeness and accuracy of FPDS data, the Agency only compared contract actions in Comprizon to those in FPDS to identify and complete those that had not been entered into FPDS. While these reviews ensured that all contract actions were entered into FPDS, the Agency did not conduct additional tests as specified in the Plan, to ensure the data entered was accurate. The large volume of errors noted in our review would seem to indicate that contracting personnel were not reviewing FPDS inputs to ensure they accurately reflected contract actions. This occurred because contracting personnel were not held accountable for ensuring the accuracy of FPDS data. More importantly, the Agency did not conduct an independent review of a statistical sample of contract actions to validate the accuracy of FPDS data. According to OBO management, workload demands and a lack of resources hindered its ability to fully implement the Data Quality Plan. By not fully implementing the plan, the Agency inappropriately certified that its 855 contract actions were accurate.

The Percentage of Contract Actions with Data Quality Issues Increased in FY 2009 with the Conversion to a New Contract Writing System

Our review of 35 statistically sampled contract actions for FY 2009 determined that 34, or 97 percent, of the actions contained inaccurate (10) and/or incomplete data. Of the 34 actions, 30 contained 1 or more data elements that did not match data in contract files, and 12 were incomplete. Although more contract actions had data quality issues in FY 2009 than in FY 2008, the Agency's data entry improved for four of the five data elements discussed previously. For example:

8 had incorrect DUNS numbers, compared to 13 observed for FY 2008.

8 incorrectly reported the size of the business, compared to 11 that were identified for FY 2008.

8 did not have NAICS codes that corresponded to that reported in the contract files, compared to the 6 observed for FY 2008.

10 did not have matching award types, compared to 11 that were observed for FY 2008.

15 did not contain correct zip codes for the place of contract performance, compared to 18 that were observed for FY 2008.

Further, similar to what we observed with the FY 2008 data, 12 of the FY 2009 contract actions contained one or more data elements that were blank in FPDS. The NAICS code was the data element that was most frequently left blank. Based on our analysis of the sampled contract actions, we estimate that at least 266 of the 303 contract actions entered into FPDS by the SBA contained at least 1 error for the 18 data elements we reviewed. Complete results of our contract action analysis are detailed in Appendix I.

These results indicate that SBA' s contracting personnel continue to insufficiently review the data entered into FPDS to ensure the data is correct and that all required fields are completed. Since SBA contract actions in FPDS contain information not matching the data in the contract files, inaccurate information is being made available to Congress and the public on SBA contracting activities, including potential Recovery Act actions. Also, SBA's Goaling Report may contain inaccurate information concerning the extent to which the Agency met its small business goals.

In October 2009, OMB issued new guidance (11) on improving FPDS data quality for FY s 2009 and 2010 and eliminated the previous requirement to submit separate data quality plans and certification reports. The new guidance requires agencies to submit a single, annual Data Quality Report that includes the agency's certification of the completeness and accuracy of its FPDS data for the previous fiscal years, a description of activities to assure data input accuracy, and a summary of its policies and procedures for measuring and reporting data accuracy. Shortly after OMB issued its guidance, SBA issued an information notice (12) regarding the certification of FPDS data and provided guidance to SBA contracting personnel on ensuring data quality. However, this notice does not contain sufficient detail regarding the steps that will be taken to ensure data is reviewed for accuracy and completeness. This level of detail will be needed to meet OMB's new guidance for improving and validating the accuracy of FPDS data.

RECOMMENDATIONS

We recommend that the Associate Administrator for Management and Administration to:

Either update the FY 2008 Data Quality Plan or revise the information notice to include explicit steps that will be taken to ensure data is reviewed for accuracy and completeness.

Conduct an independent review to ensure that the Data Quality Plan or information notice requirements have been fully implemented.

Ensure that OBO contracting personnel are held accountable for the accuracy of FPDS data.

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL RESPONSE

On January 7,2010, we provided a draft of the report to SBA's Office of Management and Administration for comment. On February 18, 2010, the Associate Administrator for Management and Administration provided written comments, which are contained in their entirety in Appendix III. Management agreed with our findings and concurred with all three recommendations. The Agency's comments and our evaluation of them are summarized below.

Management Comments

Recommendation 1

The Associate Administrator, Office of Management and Administration agreed with the recommendation, and stated that he will revise and reissue the Agency Data Quality Plan to further delineate specific steps that must be taken to ensure data is properly reviewed for accuracy and completeness. The Associate Administrator also stated that he will provide training to the contracting specialists/ officers to reinforce this effort.

OIG Response

We consider management's comments to be responsive to the recommendation.

Recommendation 2

The Associate Administrator, Office of Management and Administration agreed with the recommendation, and stated that in September 2009 the OBO contracted with LS3 to conduct an independent review of the SBA's FPDS data. He noted that LS3 's findings confirmed our audit report conclusion that the FY 2009 data is more accurate than prior year data. The Associate Administrator also stated that the Office of Management and Administration will make necessary adjustments to the Agency's Data Quality Plan to ensure continued improvement of FPDS data quality. He also stated that LS3 will conduct semi-annual data quality reviews to ensure on-going compliance with applicable guidance and to ensure the accuracy and completeness of the data.

OIG Response

We consider management's comments to be responsive to the recommendation.

Recommendation 3

The Associate Administrator, Office of Management and Administration agreed with the recommendation, and stated that the annual Performance Business Commitment Plans for contracting specialists/officers will include a critical performance element holding them accountable for the accuracy and completeness of FPDS data.

OIG Response

We consider management's comments to be responsive to the recommendation.

ACTIONS REQUIRED

Please provide your management decision for each recommendation on the attached SBA Forms 1824, Recommendation Action Sheet, within 30 days from the date of this report. Your decision should identify the specific action( s) taken or planned for each recommendation and the target date(s) for completion.

We appreciate the courtesies and cooperation of the Small Business Administration during this review. If you have any questions concerning this report, please call me at (202) 205-[FOIA ex.2] or Riccardo R. Buglisi, Director, Business Development Programs Group, at (202) 205-[FOIA ex. 2].

APPENDIX I. ATTRIBUTE SAMPLING METHODOLOGY AND PROJECTIONS

For FY 2008, we reviewed a random sample of 36 of 850(13) contract actions made during the fiscal year to estimate a FY 2008 FPDS error rate. To estimate a FY 2009 FPDS error rate, we reviewed a random sample of 35 of 303 contract actions made. For each contract action, we reviewed 18 of 46 key data elements identified by OMB for assessing the accuracy ofFPDS data. We compared the information in FPDS to the information in SBA's contract file to determine the error rate for each data element. If the information for anyone of the 18 data elements did not match, we considered the overall contract action to be an error. FPDS data elements that required information, but were left blank, were also considered to be errors. In addition, if the Agency was unable to provide the file for a contract action in our sample, we counted the contract action as an error.

In statistical sampling, the estimate of attributes in the population universe has a measurable precision or sampling error. The precision is a measure of the expected difference between the value found in the sample and the value of the same characteristics that would have been found if a 100-percent review had been completed using the same techniques.

We calculated the population point estimates and the related lower and upper limits for the selected attributes, using the Defense Contract Audit Agency's "EZ Quant" software program, at a 90-percent confidence level. Projecting our sample results to the universe of 850 SBA contract actions, we estimated SBA's FY 2008 FPDS error rate to be at least 80 percent. Our calculations for the FY 2008 error rates are shown in Table 1 and 2. Projecting our sample results to the universe of 303 SBA contract actions, we estimated SBA's FY 2009 FPDS error rate to be approximately 88 percent. Our calculations for the FY 2009 rates are shown in Tables 3 and 4.

See the attached PDF for Tables 1, 2, 3, and 4.

APPENDIX II. RESULTS OF DATA ANALYSIS REVIEW

See attached PDF for Table 1 and 2.

APPENDIX III. AGENCY COMMENTS

See attached PDf for Appendix III.

FOOTNOTES

1 0MB Memorandum, Timely and Accurate Procurement Data, August 25, 2004.

2 The sample included contract actions that occurred between October 2008 and March 2009.

3 Public Law 109-282, Federal Funding and Transparency Act 0/2006, September 26, 2006.

4 Reportable Contract Actions are defIned by FAR Part 4.601 as any action that results in the purchase, rent, or lease of supplies or equipment, services or construction using appropriated dollars over the micro purchase threshold; or modifications to these actions regardless of value.

5 FAR Part 4.602(a).

6 FAR Part 4.604(c).

7 GAO-05-960R, Improvements Needed to the FPDS-Next Generation, September 27, 2005.

8 OMB Memorandam, Federal Procurement Data Verification and Validation, March 9, 2007, and Improving Acquisition Data Quality-FY 2008 FPDS Data, May 9, 2008.

9 For the purposes of our report, inaccurate means the FPDS report and the contract file for the contract action did not contain matching information or the information was missing from the contract file for 156 of the 648 data elements reviewed.

10 The FPDS report and the contract file for the contract action did not contain matching information or the information was missing from the contract file for 141 of the 630 data elements reviewed.

11 OMB Memorandum, Improving Acquisition Data Quality for Fiscal Years 2009 and 2010, October 7, 2009.

12 SBA Information Notice, Ensuring Accurate Procurement Data Reporting to the FPDS -Next Generation and Federal Assistance Award Database, October 19, 2009.

13 The sample number is based on the FPDS data file as of May 3, 2009, which was the date the OIG queried FPDS to identify the universe of contract actions for FY 2008. In January 2009, SBA's certification included 855 contract actions for FY 2008. The difference between the universe from which the OIG sampled and the number of contract actions that SBA certified to is attributed to the timing difference.