Standards: Standards for Usability Testing

Harry E. Blanchard

On March 5-6, 1998, a workshop was held at the offices of the U.S. National Institute of Standards (NIST) in Gaithersburg, Maryland. This is of direct interest to standards watchers in the United States, and perhaps bears watching for those elsewhere. The workshop was entitled "The Application of Usability Testing Results As Procurement Criteria for Software," and was co-chaired by Jean Scholtz of NIST and Keith Butler of Boeing. It was jointly sponsored by NIST and SIGCHI. The workshop is an attempt to define a standardized usability testing process and report format for the benefit of software consumer companies and software vendors.

Before describing more about this workshop, a short word about the role of NIST. NIST is a U.S. government agency, part of the Department of Commerce. It is not, however, a standards creating body for industry in the U.S., although it does have a minor role in creating and administering standards for the benefit of agencies in the U.S. government itself. NIST is neither a regulatory nor a standards enforcement agency. In the U.S., domestic industry standards are almost entirely created and administered by private agencies not associated with the government, mainly the American National Standards Institute (ANSI), or by professional associations and ad hoc industry interest groups. NIST does play a role in facilitating the creation of standards for the use of industry. Historically, NIST and been known for its role in keeping measurement standards (e.g. the most well known perhaps is the atomic clock based time standards broadcasts). The idea of defining a standard testing process for usability fits easily into historical role of NIST in providing measurement standards.

The workshop began with the following observation. Large corporations are significant consumers of software, such as desktop and word processing software, databases, and engineering and design packages. These corporations can suffer an uncontrollable amount of overhead due to usability problems with software: overhead from internal and external training and support costs, from the consequences of user errors, from lost productivity and reduced efficiency. User satisfaction with software could also affect morale.

Just as a company can exchange standard assessments of its raw materials with its vendors, why can't a company exchange a standard assessment of software on its ease of use? This would require a standard testing and report process, and that is the goal of this workshop. Such standardized test reports would then form part of the product selection process (where usability can be weighed in relation to other purchasing considerations).

The workshop goal is a general industry-wide process for providing visibility of software usability before product selection. The objectives of the workshop, to be fulfilled within the year, are to

define a minimal, standard testing process for usability
define a standard specification for reporting usability tests, and
define a pilot study to estimate the costs and benefits of the application of usability test results to software product selection.

Companies and agencies represented at the meeting included a mix of software vendor and consumers:

ACM SIGCHI
AT&T
Boeing
Cognetics
Digital Equipment Corp.
Eastman Kodak
Fidelity Investments
George Mason University
Hewlett Packard
IBM
Lucent Technologies
NIST
Northwestern Mutual
Oracle
PC Computing Magazine
Redish & Assoc.
Sun Microsystems
University of Maryland

(and several independent consultants)

Since the meeting, several other organizations have joined the group, including Bellcore, Microsoft, and US West.

The first half of the meeting was consumed with the exchange of comments on the goals and objectives of the workshop. The participants in the meeting shared some of their own usability testing and reporting processes done at their own organizations. ISO 13407 (Human-centered design process for interactive systems) was examined for its usability report format, it could provide a possible template for standardized reports in this standard.

The second half began with enumerating all the issues, problems, and tasks associated with the goal of providing a standard usability testing process and reporting format. A selection of these problems and issues included:

If a standard usability test were to be defined, what model would be followed for using and promoting the standard? For example, in an "FDA model" vendors report the results of tests) whereas in a Good Housekeeping Seal of Approval model, a third party might provide a minimal level of product certification. Who would do the testing?
What specific metrics would be chosen? Productivity, user satisfaction, learnability, support requirements, etc. How many metrics would be defined?
Who would be the user of the information?
The committee largely reject the idea of a standard design process, along the lines of process standards such as ISO 9000 or, in the usability domain, ISO 13407. The objective was seen to be a standardized test process.
At what stage of the product development process would a standard usability test be properly applied?
What role, if any, would there be for other usability evaluation techniques, notably, heuristic evaluation?
Would a standard usability testing process, with responsibility essentially directed toward the vendor, unfairly penalize small software companies, e.g. "garage" shops that distribute software over the Internet. Such shops are often a source of innovation in the industry yet would be unable to financially comply with a usability testing system.
How would the reliability and validity of a standardized test be insured?

This is only a sampling, a large number of such issues were examined and taken under study.

Following this, the workshop divided into three subteams. One team will examine all the issues associated with testing, vendors, and general management of such a standard usability test, and provide a management plan. The second group will define the testing and reporting methodology. The last group will define a plan for the pilot study, to test and validate the proposed standard.

This workshop on usability testing process standards will meet again in September of this year. The current plan is to have a draft created by the end of the year, and a pilot study executed by next year.

You can expect continual updates on the progress of the NIST workshop in this column. Meanwhile, you may direct reports of standards activities and question to me at heb@acm.org, by phone at +1 732 949-9745 or by fax at +1 732 949-8569.

Issue
Article
Vol.30 No.3, July 1998
Article
Issue