Issue
Article
Vol.30 No.1, January 1998
Article
Issue
Issue |
Article |
Vol.30 No.1, January 1998 |
Article |
Issue |
The increasing usage of interactive computer systems in major public and private sector corporations has been accompanied by a growing recognition of the benefits of usability to organisations (Bias & Mayhew, 1994). Twenty case studies summarised by Harrison, Henneman, & Blatt (1994) provide support for this view. Further, usability professionals have found that to successfully build usable systems, usability needs to be closely integrated with the information technology systems development process (Lindgaard, 1994). However, the increasing use of outsourcing arrangements for IT systems development presents substantial challenges to such integration. Overall IT outsourcing and it's consequences have generated a massive research literature that will not be reviewed in any detail here (for reviews, see for, example Lacity, Willcocks & Feeny, 1994). The majority of this work however does not address the issue of usability requirements and process in outsourced systems development, although concerns regarding quality have been identified (see, for example, Willcocks, Lacity & Fitzgerald, 1995). As part of the rationalisation in the Australian telecommunications industry, Telstra, Australia's major telecommunications company, is outsourcing its Information Technology Group. The purpose of this paper is to explore the challenges presented by IT outsourcing to usability practitioners and practise.
Overall, compared with in-house development, outsourcing of IT systems development results in three major changes to the development process. Specifically IT, outsourcing:
One major implication of these changes is likely to be a significant reduction in the customer organisation's ability to ensure the integration of usability into the development process and hence the successful development of a usable IT system.
The objectives of the present paper are thus:
The first objective is to assemble a previously scattered body of work and thus make it more readily accessible. The second and third objectives focus more specifically on incorporating usability requirements into the IT procurement process in telecommunications.
In what seems to be one of the first publications on this issue, Carey (1991) described a model for stating usability requirements which was being developed for the Central Computer & Telecommunications Agency of the UK government. Carey noted that the key problem in a contractual setting is that the relationships between those who commission the work and those who produce the deliverable do not match the basic assumptions that underpin usability engineering.
First, the formality of the contractual relationship can create difficulties for the types of interactions required by usability methods. For example, vendors may be reluctant to allow their staff to take part in usability activities that they do not perceive to be directly related to the production of the system. Also there are complex issues regarding proprietary information for both the vendor and the customer e.g. vendors may have proprietary rights over development tools and methodologies while customers have concerns about confidentiality of their work-practise and other internal information.
Second, in an outsourcing situation the high level "goals" of those who commission the work and those who produce the system are not always aligned in the way that usability methods typically assume. Typically, the customer "wants" a user interface which meets the needs of the organisation and the end-users. By contrast the vendor "wants" to produce a system that meets the customers stated requirements, on time and within budget. However, staff in the customer organisation involved in the IT procurement process are often unfamiliar with usability issues and do not include usability in their initial requirements definition. There is a substantial risk in this situation that the delivered system will have poor usability.
Following this argument, the main thrust of Carey's approach was the development and inclusion of testable usability goals in requirements documents to be issued to potential vendors. Carey argued that both usability features, that is specific design standards, and measurable usability objectives should be included in requirements specifications. He proposed that the key issues were how to state usability requirements in a manner which:
One of the key issues raised by Carey's analysis is that requirements specifications, including those relating to usability issues, need to address the needs and objectives of both parties.
A study by Allwood, & Kalen (1992) looked at usability issues in a hospital administration system and how these were related to the contracting process. They noted problems with the contracting process including:
They also identified as a key difficulty problems in the relationship between the various parties to the development process. They concluded that the specification of usability requirements in IT procurement appears to be a relatively neglected issue in usability engineering.
Hix, Hartson, Siochi & Ruppert (1994) discussed the specification of usability requirements in the Request for Proposal (RFP) for a geographical information system for the Bureau of Land Management. The approach they describe was to incorporate development process requirements into the RFP. By specifying process requirements they aimed to avoid the risk of being too specific in the RFP and having the developers take the UI specifications directly from the RFP. Rather, they argued that the aim is for the customer to specify their requirements from a usability perspective while leaving the developers free to design the interface. An important point made is that to effectively write and enforce usability requirements, the customer needs to be well informed about usability or be sufficiently aware of the issues to seek assistance. Thus, usability specialists either within the customer organisation or outside consultants may need to take on the role of adviser in the preparation of RFPs.
Building on this early work, Laura Downey and others organised a series of symposia under the auspices of the National Institute of Standards and Technology, NIST (Buie & Winkler, 94; Winkler & Buie 95; Downey, Laskowski, Buie & Hartson, 96) which addressed a range of challenges for usability in Government IT systems contracting. Some of the key challenges they identified were:
The solutions that emerged from the symposia were for usability specialists to exert influence on policy makers and hence on IT procurement processes and to use standards as leverage to integrate usability into standardized development methodologies. Given that, until recently in many countries including Australia, telecommunications have been Government owned, it is perhaps not surprising that IT systems in telecommunications share many of the challenges identified by the NIST group.
In the next section of this paper we present two case studies of usability inputs to the procurement of major new IT systems and discuss some of the lessons learned. The final section of this paper sets out an initial proposal for guidelines which will enable the integration of usability within the overall IT procurement process.
The Desktop Communicator project originated from within a development oriented business unit of Telstra. Telstra's 8500 front-of-house staff provide inquiry services to approximately 50 million customer calls annually.
In 1996, in line with trends in other large customer service organisations, Interactive Voice Response (IVR) applications were perceived to be a means for providing more timely customer service for simple inquiries such as notification of bill payment or inquiries on bill balance. Interactive Voice Response applications had been implemented in other parts of Telstra with partial success. An example was in residential faults departments, where customers could activate a simple line test through an IVR application, thus reducing the level of customer queuing and amount of handling by customer service representatives
However, while many customers welcomed the opportunity to undertake "do-it-yourself " servicing, many others appeared to feel uncomfortable with the new technology and preferred to talk to a human operator. The transfer of a customer from an IVR to a customer service representative was problematic.
Some of the problems included:
The solution to these problems was thought to involve technology that would facilitate rapid customer transfer and simultaneous transfer of data captured by the IVR application to service representatives with the customer. Some attempt had been made to better integrate voice and data transfer, but this technology was thought to be inadequate due to limited technological scalability and flexibility in the long term. A more strategic approach to a technological solution was required. The Call Centre Development Department delegated responsibility for the acquisition of this technology to an Internal Communications department. This department was also given responsibility for the implementation of this project. Other internal stakeholders involved included:
Table 1 outlines the tendering process as defined by the Internal communications group in consultation with the business project manager, information technology manager, technical architecture and usability consultants.
The usability group was interested in obtaining two types of information from tendering companies. First, they wanted to know whether tendering companies were willing to comply with Telstra's Corporate Usability Standards and Guidelines for interface design. Second they wanted to get a picture of the extent of each companies past experience with usability. This included whether tendering companies had processes for including usability within their IT development framework.
Overall, the outcome of the tender process resulted in a preferred supplier for technology being selected. However, none of the tenders satisfactorily met the usability selection criteria. All of the tendering companies stated their willingness to comply with Telstra's Corporate Usability Standards and Guidelines but none demonstrated any significant past experience with usability practice. On this basis it was decided that an in-house usability team would be responsible for the design and development of user interfaces. Other criteria within the initial Request for Tender were also downgraded following submissions from the tendering companies. For example, responsibility for technical architecture design was allocated to in-house groups as were project management responsibilities.
Our analysis of the events and documentation suggested a number of key process problems. First, until after the date documented in the table there was no formal business case for the project. This resulted in an ongoing lack of clarity about the business objectives for the system between the development group and the business owners. This can best be understood as a failure of process rather than a failure on the part of any stakeholders.
From a usability perspective, the usability group's involvement was requested too late, there being no time to conduct a user needs analysis prior to providing usability inputs to the RFP. Also the usability group was not able to gain a clear understanding of the business objectives, as these were not clearly identified when the usability group's involvement was requested. At the time this paper was written a final decision was pending but it was expected that the usability group would withdraw from further involvement with this project due to the process problems discussed.
Table 1: Summary of Major Usability Involvement in the Desktop Communicator Project Procurement Process
| Activity | Expression of Interest August 1996 | Request for Proposal September-October 1996 | Request for Tender Requirement Specification December 1996 |
|---|---|---|---|
| People |
|
|
|
| Purpose |
| To select a short list of potential vendors for distribution of the Request for Tender documentation based on ability of vendors to supply a highly scalable solution | To select from successful respondents to the RFP a preferred technology supplier, based on specific selection criteria |
| Process |
Broadly distribute the Expression of Interest document to likely vendor companies
Following receipt of documentation from interested vendors, a short list of possible suppliers were identified for distribution of the Request for Proposal | Distribute the Request for Proposal document to successful respondents to the EOI. |
Distribution of the documentation to a short list of potential vendors.
Provision of opportunities for RFT briefing sessions including invitations to demonstrations of Telstra's Corporate Graphical User Interface (Miller, 1996) |
| Criteria |
|
|
|
| Usability Involvement |
No usability involvement in the Express of Interest.
`Ease of use' was identified as an important objective but no evaluation criteria were documented |
Usability consultants were asked to review the software components of the RFP and to include specific usability selection criteria.
Usability selection criteria included requests for:
|
Usability criteria included:
|
The E-Commerce project was undertaken by Telstra in strategic partnership with the Australian Commonwealth Government. The objective of the project is to develop and implement an internet based electronic commerce system to support the full range of procurement transactions between supplier organisations and Government agencies involved in purchasing. The major project stakeholders and their roles are as follows:
The end-users for E-commerce system comprise two main groups purchasing officers in Australian government agencies involved in purchasing and staff in supplier organisation involved in selling goods and services to the Government. Both these user groups are extremely diverse and the two groups potentially have widely differing needs. Further, and critically in the current context neither of end-user groups are within the direct influence of either Telstra or Purchasing Australia.
The involvement of the TRL Human Factors group was sought by project management at an early stage in the project. At this time the vendor contract and system requirements specifications had not been finalised. The first usability inputs provided by the Human Factors group were usability specifications to the vendor contract and system requirements documents. Usability inputs to the vendor contract focussed on the development of an overall usability plan for the project and the development of user interface design standards. The usability specifications in the system requirements document addressed in more detail the development of user interface design standards including the specification of design grids for each page type in the site.
Currently the project is in its first commercial release with a basic set of functionality available. Development of the system is continuing with ongoing involvement of usability in the design process. Overall the involvement of the Human factors group with E-commerce project has resulted in two key successes thus far:
The major underlying factor supporting the successful involvement of the Human factors group was its early involvement in the development process. This, in turn, made possible:
Areas that have presented continuing challenges in the integration of usability in the development process have related to access to end-user for testing and evaluation and integration of some activities such as heuristic evaluations into the development process. The difficulties experienced in accessing end-users reflect the fact that for this project both of the major user groups are outside the direct control of the major stakeholders. Integration of usability activities in the development life-cycle could potentially be enhanced for future projects through specifying these as standard processes in the vendor contract.
The previous research reviewed along with the findings of the case studies suggests two key issues for including usability requirements in the IT procurement process:
It also seems likely that use of usability metrics will be important to the success of this approach.
We propose that usability issues need to be considered at several of the key points in the process. First, in the business context usability considerations relate to the basic objectives for procurement of any interactive system and this relationship should be made explicit in the business case. Second usability goals and the associated processes for developing usable systems need to be seen as a standard part of the IT procurement process addressed in the RFP/RFT, contract and system requirements specifications. Further, we believe that it will be important to provide methods such as cost benefit analysis to prioritize usability issues in development and usability plans, to track the progress of usability issues during the development process although these are not addressed in detail in the current paper.
Montaniz & Kissel (1996) noted that "the challenge is the same for each development project we undertake -- the project goals that are most readily interpretable from a business perspective receive higher priority than those whose ties to profitability are harder to assess." Work on cost benefits analysis of usability suggests that usability has an important relationship to profitability for IT systems (Bias and Mayhew 1994). The challenge for corporate usability specialists is to make this clear to business decision makers.
In general, the business case aims to assess the potential return to the business based on implementing specific functionality for a new IT system. What is often not considered is that for any interactive computer system, realisation of the business benefits depend on users being able to easily and efficiently access the functionality that this system will provide. Efficiency and ease of access are explicit and quantifiable goals for user interface design. Most often, however, these design goals are subsumed within vague unquantifiable statements such as "easy to use". When design goals are not stated in explicit, quantifiable terms, this will very often result in the loss of any clear link between user interface design and realisation of the business benefits of the system in the pressure of IT systems development work. For systems intended for "in-house" use this is likely to mean that projected savings from implementing the new system will not be realised due to higher training costs, high error rates, and end-user dissatisfaction. For systems intended to be sold to external customers this is likely to mean that the system will not be accepted by customers and the product will not achieve expected sales (Ehrlich & Rohn, 1994).
We propose that in the business case, the key usability issues relevant to realisation of the business benefits of the new system need to be identified (see Bloniarz & Larsen, (1997) for a discussion of these issues in a Government sector context). These measures need to be specified in quantifiable terms although at this early stage it is not likely to be possible to specify actual measures and goals. For "in-house" systems this is likely to involve identification of the direct contribution of usability to efficiency, ease of access and learnability of the system. For systems to be sold to external customers this is likely to include identification of the contribution of usability to initial sales and/or customer retention.
The most effective way to identify these issues would be for the customer organisation to conduct a user needs analysis as part of the preparation for producing an IT system business case. The overall direction for the user needs analysis would be set by the business objectives. The purpose of the UNA would be to enable the specification of the set of usability measures that directly related to the realisation of the business benefits of the system. This process would also be likely to result in the identification of "new" or unforseen usability issues.
The outcomes of the user needs analysis carried out for development of the initial business case should also inform the development of the RFP/ RFT.
Thus key usability issues identified from the user needs analysis should form the first element of usability specifications for the RFP/RFT. The other key usability requirement for RFPs is to specify usability process requirements for development of the user interface (Carey 1991; and Hix et al. 1994). The major process requirement is the use of an iterative development model for user interface development. More specifically:
There are in practise a number of ways by which iterative user interface development can be achieved. We propose that the RFP needs to obtain from potential vendors how they propose to enable iteration during the development process.
As discussed earlier, one of the key issues is to specify standard processes for usability assessment during development. The contract would specify standard measurement processes to be used to collect usability data during development. This is likely to include processes such as usability testing and also standards assessments of the user interface design. This strategy is consistent with established practice in the quality domain of specifying standard reviews during development (Carey, 1991).
In some areas it may be appropriate for the contract itself to directly reference usability standards. For example in domains where there are accepted measurable standards, compliance with these standards may be an appropriate contract specification. Alternatively it may be appropriate to specify that application specific UI design standards are to be formulated during development.
The use of an iterative development process for the user interface raises potentially difficult problems in a contractual context where a key requirement is to eliminate uncertainty. Developers need to be assured that iteration is a managed process and will not continue indefinitely. To achieve this, one key factor will be to develop and ratify measurable usability goals that can be included in the requirements specifications. We support the recommendation made by Carey (1991) that quantified usability goals should be an integral part of the system requirements specifications. In practice we believe that following Hix et al.(1994) these should be agreed between the vendor and customer and by discussion with other project stakeholders once the contract has been awarded.
The integration of usability in the procurement process could potentially provide a directions for research (Downey et al., 1996). First the approach outlined will require a range of reliable valid measures of usability. These are likely to include subjective usability rating scales, objective performance measures (e.g. time for task completion and numbers of errors), and measures of compliance with interface design standards. Further, there is a need for collaborative work to develop and/or refine cost benefit analysis methods that can set priorities for usability issues during development. We propose that as well as cost benefit analysis for usability per se, a key issue here is to identify the implicit usability assumptions made in cost benefit analyses for new IT systems in general.
In a business context, the adoption of this strategy will require persuading those who make decisions regarding IT systems procurement about the value of including usability requirements in the procurement process. We suggest several strategies to facilitate the process of informing and persuading management:
Rob Pedlow is a Human Factors Scientist at Telstra Research Laboratories a division of Telstra corporation, Australia's major telecommunications company. He is currently investigating the incorporation of Human Factors principles in the IT procurement process as part of a major project to enhance the development of customer access systems for Telstra.
Anne Miller is a Usability specialist now working with IBM Global Services, a joint venture between IBM, Telstra and Lend Lease. Having moved from in-house consultancy with in-house project development teams to working as an outsourced contractor, Anne is interested in the implications of outsourcing for the integration of usability processes and practice in the outsourced context.
Rob Pedlow
Telstra Research Laboratories
770 Blackburn Road Clayton
Victoria 3168, Australia
Phone: +61 3 9253-6373
Fax: +61 3 9253-6665
Email: r.pedlow@trl.telstra.com.au
The permission of the Director of Research -- Telstra Research Laboratories to publish this paper is gratefully acknowledged.
Allwood, C.,M. & Kalen, T. (1992) Lack of usability in a patient administrative system as a function of events in the system implementation process. In Proceedings of the sixth European Conference on Cognitive Ergonomics (ECCE 6) Balatonfured, Hungary, September 6-11.
Bias, R. G. and Mayhew, D. J. (eds) (1994) Cost Justifying usability. Academic Press: Boston.
Bloniarz, P. A., & Larsen, K. R. (1997) A cost/performance model for assessing WWW service investments. CTG.ISG-4. Centre for technology in Government: University at Albany, Albany, New York.
Buie, E., & Winkler, I. (1994) HCI challenges in Government contracting SIGCHI Bulletin, Vol 26, No 4, Oct 1994, pp 49-50
Carey, T. (1991) A usability requirements model for procurement life cycles. In Human Factors in Information Systems (ed. Jan M. Carey) Ablex, Norwood, New Jersey.
Downey, L. L., Laskowski, S. J., Buie, E. A., & Hartson, H. R. (1996) Usability engineering: Industry-Government collaboration for system effectiveness and efficiency (Symposium report). SIGCHI Bulletin, Vol 28, No. 4, Oct 1996, pp 66-7.
Ehrlich, K., & Rohn, J.A. (1994) Cost justification of usability engineering: A vendors perspective. (pp. 71-108) In Cost justifying usability. (Eds.) Randolph Bias and Deborah J. Mayhew. Academic Press: Boston.
Harrison, M. C., Henneman, R. L. and Blatt L. A. (1994). Bias, In: R. G. and Mayhew, D. J. (eds) (1994) Cost Justifying usability. Academic Press: Boston
Hix, D., Hartson, H. R., Siochi, A. C., Ruppert, D. (1994) Customer responsibility for ensuring usability: requirements on the user interface development process. Journal of Systems Software 24, 241-255.
Lacity, M. C., Willcocks, L. P., & Feeny, D. (1995) Sourcing information technology capability: a framework for decision making. OXIIM PAPER RDP 95/ 3
Lindgaard, G. (1994) Usability testing and system evaluation -- A guide for designing usable computer systems. London: Chapman and Hall.
Miller, A. (1996) Integrating Human Factors in Customer Support Systems development using a multi-level organisational approach. CHI 96 Conference proceedings. pp 368-375.
Montaniz, F., & Kissel, G. V. (1996) Reversing the charges. interactions, Vol 2 No 3, April 1996. http://www.acm.org/interactions/vol2no3/columns/business/business.htm
Willcocks, Lacity & Fitzgerald (95) IT outsourcing in Europe and the USA: Assessment Issues. OXIIM paper RDP 95/2
Winkler, I., & Buie, E., (1995) HCI challenges in government contracting. SIGCHI Bulletin Vol 27 No 4, pp 35-37.
Issue |
Article |
Vol.30 No.1, January 1998 |
Article |
Issue |