You are here
How We Measure Success - Part II
Comments welcome on this page. See Rules of Conduct.
How We Measure Success - Part II
Last week we kicked off a series of articles on measuring performance. This article, the second in the series, explains how we use the American Customer Satisfaction Index (ACSI) to gauge how satisfied our customers are with the information services provided on business.gov and how much time and money they believe they saved by visiting the site.
What is the ACSI and how is it used for Federal websites?
The ACSI methodology has been around since 1994 and is used in both the commercial and government sectors. The ACSI measures customer satisfaction with goods or services based on customer responses to a standardized survey. For E-Government sites, each measured website is given a customer satisfaction index score (ACSI score) based on a weighted average of three model questions:
- What is your overall satisfaction with [website]?
- How well does [website] meet your expectations?
- How does [website] compare to your idea of an ideal website?
Each index score is on a 0-100 scale and scores have tended to range from the low 50's to the high 80's.
To ensure that federal websites can be compared to each others, scores in the ACSI E-Government Index are organized by functional category and organizational structure. Business.gov is included in the News & Information function as an Agency site, under the Small Business Administration. The latest E-Government Satisfaction Index Report includes results for the fourth quarter of 2008. Business.gov is listed 32nd of a total of 51 web sites in its category with an ACSI score of 71.
How do we use the ACSI for Business.gov?
If you were to visit Business.gov today, you might be prompted to complete an ACSI survey after a few clicks. The survey is presented to a 'sampling' of customers based on a predetermined percentage after they have visited several pages in the website. The ACSI score is based on the responses of all customers who voluntarily completed the survey when prompted. All 35 survey questions are either multiple choice or type-in comments and none of the questions ask for any personal information.
The Business.gov ACSI survey currently includes 25 model questions and 10 custom questions. The model questions focus on gauging our customers' satisfaction with the website in the following categories:
- Look and Feel
- Site Performance
- Overall Satisfaction
The final three model questions gauge the impact of the customer's current visit to the website on their future behavior:
- How likely are you to use Business.gov as your primary resource for business compliance information?
- How likely are you to recommend Business.gov to someone else?
- How likely are you to return to Business.gov?
The Business.gov survey includes ten additional custom questions. Some of these questions are geared toward understanding more about our customers, including what stage of the business process they are in and what type of business they own.
Two of the custom questions aim to quantify the perceived value of the information services provided on Business.gov. In the last article we stated that the first and foremost goal, and the cornerstone of the Business Gateway program, is to save the small business community time and money. These two questions attempt to measure this outcome:
- By visiting Business.gov today, about how many total hours do you think you saved?
- By visiting Business.gov today, about how much money do you think you saved?
How satisfied are customers with Business.gov?
We mentioned above that the ACSI score for Business.gov for Q4 2008 was 71. Looking at specific categories, on average, our customers continuously rate Site Performance and Look and Feel the highest, while historically they have rated Functionality and Navigation the lowest.
This is based on an average of all customer responses. However, when we compare the responses of customers who classified themselves as one type of business owner vs.. another, there are some significant differences. For instance, customers who described themselves as Home-Based Business Owners or Self Employed were more satisfied in Q4 2008 than those who described themselves as Woman Business Owners.
Coming back to the goal of saving the small business community time and money, the results are harder to measure. For October - December 2008, 57% of ACSI respondents estimated saving between 1 and 10 hours by visiting Business.gov. For January - March 2009, this measure increased to 64%.
For the same time frames, 26% of ACSI respondents estimated saving between $1 and $100 by visiting Business.gov in October - December 2008. For January - March 2009, this increased to 31%.
How do we use the survey data?
We use this feedback, along with other feedback points, to help determine how and where to improve the services provided on Business.gov. This February we launched a new navigation scheme to help customers find information more quickly and easily; since then, we have seen an increase in the Navigation score from 72 to 74.
We continue to be encouraged by the small improvements noted and keep reaching out to our customers in new ways to find out how we can improve our services. We welcome your feedback and comments on this article.
Stay tuned for future articles in this series on measuring performance where we will discuss our second goal - engaging citizens to participate.