Issue
Article
Vol.29 No.4, October 1997
Article
Issue
Issue |
Article |
Vol.29 No.4, October 1997 |
Article |
Issue |
The reason for organizing this Special Interest Group originated from the work which currently goes on in the MEGATAQ project. MEGATAQ is an acronym for `MEthods and Guidelines for the Assessment of Telematics Applications Quality' (TE 2007). This two years project is funded by the European Commission in the Telematics Applications Programme (Telematics Engineering Sector). The aims of the MEGATAQ project is to make an evaluation reference model which indicates the evaluation areas involved, an inventory of existing evaluation methods, tools and techniques covering the areas of the reference model, and to provide these in the form of guidelines to telematics applications projects for usage and feedback.
The objective of the Special Interest Group was to share knowledge and experiences between practitioners and academics, and to create when possible a network of people who are interested in the subject of evaluation of telematics applications.
An international group of eleven people joined this special interest group.
Since the phenomenon of telematics applications is rather new. The SIG discussion focused around two basic questions: 1) what is telematics? and 2) what makes the evaluation of telematics applications different from traditional stand-alone `individual human-computer interaction' applications?
The first topic in the discussion was aimed at getting a description of the nature of telematics applications. It resulted in a first outline of a classification scheme for classifying the various type of telematics applications. The second topic showed that there are much more aspects to consider during the evaluation of telematics applications, which, in fact, raises the need for more theoretical notions at various levels of interaction, e.g. individual work level, communication level, group level, social level, and organizational level. Also, during the Special Interest Group the way of evaluating telematics applications was discussed. This resulted in a number of generic steps which should be taken in an evaluation process. The report ends with a number of conclusions.
The first dimension that was discussed can be formulated as `the kind of technology' used for telematics applications. The following technologies were recognized:
Tele-information applications are characterized by the retrieval of an individual of information in a database which is geographically dispersed, which is called by Bekkers & Smiths (1995) `consultation'. Tele-interaction applications allow their users to communicate, this may be in a synchronous (video conferencing or telephoning) or asynchronous (email) way. The tele-transaction applications are characterized by the transfer of structured information (e.g. Electronic Data Interchange; EDI), for example these kind of applications are used by transference of money between bank institutions. The tele-operation applications enable the users to operate equipment at a distance. The registration of processes or objects (e.g. for security and tele-metrics) at a distance can be conducted via tele-registration applications. The tele-action applications provide (geographically dispersed) users access to documents which they share and on which they might work together. These databases may be located centrally or locally.
All these technologies may be labeled as `complex applications' in terms of diversity of users separated in place and perhaps time, and even users which are not bound to one particular place (mobile). Also, these applications may feature various communication media.
Another way of looking at telematics applications is from a user perspective, in particular, from the kind of activities the users have to perform. The following activities have been identified by the participants of the Special Interest Group:
Of course, the above listed activities are not complete tasks in themselves. Almost every task requires communication, coordination or collaboration to some degrees. What is important, is the focus on one or more activities within the whole set of activities which constitutes a task.
The two dimensions `kind of technology' and `user perspective' can be brought together in one classification scheme (see figure 1). The dots in the figure indicate the kind of technologies necessary for the fulfillment of that particular activity.
According to the participants the value of these kind of classification schemes can be found in:
| User perspective | |||||
|---|---|---|---|---|---|
| Kind of Technology | Communication | Co-ordination | Collaboration | Group decisions | Control/steering |
| Tele-information | X | X | |||
| Tele-interaction | X | X | X | X | |
| Tele-operation | X | ||||
| Tele-registration | X | ||||
| Tele-action | X | X | |||
Figure 1: Telematics applications classification scheme
It has been recognized that the classification of applications should be based on the actual usage by the users. This might be problematic, because the actual usage of the application is quite often not yet known at the start of the development process. In cases, where the future work situation does not resemble the current work situation, extended analysis and extrapolation of this situation does not make much sense. However, there are techniques available which could spread some light on the future usage of the application, e.g. usage scenarios, story boards, and animation. One of the participants had rather good experience by drafting scenarios and play those scenarios in a simulated setting with the stakeholders of the application.
More classification schemes can be found in literature, e.g. Andriessen et al. (1993), Bekkers and Smits (1995), and Steinmetz and Nahrstedt (1995). In order to develop a usable classification scheme more research and discussions are necessary. Of course, the above scheme (figure 1) is only a first attempt.
Figure 2: Reference model
In this section of the report, the second question "what makes the evaluation of telematics applications different from traditional stand-alone `individual human-computer interaction' applications?" is addressed. Input for the discussion was provided by the MEGATAQ reference model (see figure 2).
The reference model includes several analysis levels, i.e. individual worker's level, the group level at which several individual workers communicate or collaborate, and the organizational level. Furthermore, it distinguishes between the context of use (input), the interaction (throughput), and the outcomes of the interaction (output).
At the individual level there is a user interacting with an application (tool) in order to fulfill a certain task in a specific environment (organizational, physical, and technical). During the task fulfillment the user may consult remote databases (tele-information). At this level one could say that the traditional HCI aspects play a role. Many evaluation tools and techniques are now available to assess the issues at this level. For example problems associated with human database interaction might occur with respect to the question whether it is the right version or whether it is the right information, i.e. integrity problems. Users should be informed about these kind of issues
At the group level two or more individuals have contact together in order to achieve a common goal (group task). This contact may be at the level of sharing information or active collaboration. Besides the earlier mentioned HCI aspect, other aspects becoming important. These aspects include social relations, communications, group dynamics. Besides the individual outcomes, other more group related outcomes become apparent, e.g. group effectiveness and efficiency but also such things as group cohesion. For example at this level network performance is becoming important, because this has a direct impact on the effectiveness and efficiency of the communication between users.
At the organizational level the contribution of the group to the organizational outcomes such as organizational effectiveness, processes, and structure play a role.
From the reference model it also becomes clear that the implementation of telematics applications involves different groups of people within and between organizations, which means that there are many more people, groups or organizations which have either a positive or negative interest in the new setting apart from the end-users. These groups are called stakeholders groups. It may be expected that these various stakeholders groups have different interests. At the beginning of a development project it is important to start with the identification of the stakeholders groups and the analysis of their expectations. The central question in the analysis is "what makes this application a success for you?" In other words, the stakeholders groups' critical success factors should be examined up front. Only when there is a proper overview of the various critical success factors involved (in for example stakeholders critical success factors profiles) and that there is a certain degree of consensus, a successful development can take place. Another problem is that the boundaries of organization are not always that clear-cut. Members of a stakeholders group might belong to various organizations, while other stakeholders groups might be found in only one organizational setting.
The above shows that the development and evaluation of telematics applications is quite a complex matter. Therefore, a multi-facet approach is recommended which ideally covers all three interaction levels presented in the reference model. In this view, the telematics interaction situation is seen as a system in which the system components have relationships with each other. An alteration in one component results in a total system change. Thus, aspects of all three levels should be included in evaluation procedures as well as an analysis of their relations.
As said before, most of the time totally new collaborative work situations are created by implementing telematics applications. This implies that not much can be learned from existing situations. In other words, there is not much empirical data available. In this case, probably much can be learned by applying valid theoretical notions. The following theoretical domains have been identified as being possibly relevant:
It has been recognized that not all these domains can be handled by one person. A multi disciplinary evaluation team might be indicated.
The evaluation process was also briefly discussed in this special interest group. The group identified a number of important steps in the process, but was not yet able to draft a complete evaluation approach. The first step in the process is probably the identification of the stakeholders groups and the analyses of their interests in the telematics application. In this phase the critical success factors are formulated. Next the telematics application under development is characterized on the basis of a classification scheme (such as presented in figure 2). This is done to determine the most relevant aspects of the future application which should be taken into account during development and evaluation. This might be an important step, because one has to be selective in the choice of aspects to evaluate. It is not possible to assess all aspects involved (too complex, time-consuming and costly).
On the basis of the critical success factors profiles and the relevant aspects assessment criteria must be formulated against which can be evaluated. Furthermore, an evaluation plan must be written which explains who are responsible, what aspects are evaluated, what are the evaluation procedures, analyses and interpretation, the cost involved, and the time schedule.
It should be noted that early evaluation is extremely important when developing telematics applications. This can be conducted by using low budget techniques such as story-boards, scenarios and playing the scenarios by the stakeholders.
It is not yet clear what telematics applications are. However, ideas were expressed which led to the formulation of a classification scheme which can be used for determination of the degree of importance of specific user activities and provision of direction of the application development and evaluation. Also it was recognized during the discussion that the evaluation of telematics applications is different from the traditional stand-alone human-computer interaction. Besides HCI issues, other issues such as group dynamics, communication, social, and organizational aspects should be included. Furthermore, the number of people who have interests in such systems is probably larger. Analyses of the expectations of the stakeholders groups in terms of critical success factors seems to be an appropriate starting point of the development and evaluation processes. Furthermore, some thoughts have been expressed about the steps of an evaluation approach.
As a separate issue the danger of privacy violation has been mentioned. Attention has to be paid to the possibility that centralized databases with privacy sensitive data are going to be misused or that people will continually be monitored.
Andersen, P.B. (1991) A Semiotic Approach to Construction and Assessment of Computer Systems. In: Nissen, H.-E., H.K. Klein & R. Hirschheim (Eds). Information Systems Research: Contemporary Approaches & Emergent Traditions. Amsterdam: North-Holland, pp. 465-514.
Andriessen, J.H.T.H. & Van der Velden, J.M. (1993) Teamwork Supported by Interaction Technology: the Beginning of an Integrated theory. European Work and Organizational Psychologist, 3(2), 129-143.
Bekkers, R. & J.Smits (1995) Mobiele telecommunicatie in Nederland en België. (In Dutch) Deventer: Kluwer.
Hacker, W. (1986) Allgemeine Arbeits- und Ingenieurpsychologie, Psychische Struktur und Regulation von Arbeitstätigkeiten. (In German) Bern: Verlag Hans Huber.
McGrath, J.E. & A.B. Hollingshead (1994) Groups Interacting with Technology. London: Sage Publications.
Nardi, B.A. (Ed.) (1996) Context and Consciousness. Cambridge, Mass., The MIT Press.
Rasmussen, J. (1986) Information Processing and Human-Computer Interaction. Amsterdam, North-Holland.
Steinmetz, R. & K. Nahrstedt (1995) Multimedia Computing, Communications & Applications. New Jersey: Prentice Hall.
Delft University of Technology
De Vries van Heystplantsoen 2
2628 RZ Delft, The Netherlands
+31 15 278 3720
arnold@wtm.tudelft.nl
vandenanker@wtm.tudelft.nl
Issue |
Article |
Vol.29 No.4, October 1997 |
Article |
Issue |