Issue
Article
Vol.28 No.2, April 1996
Article
Issue
Issue |
Article |
Vol.28 No.2, April 1996 |
Article |
Issue |
There has been little research into the study of long term adaptation by users or the extent to which usage is enhanced over a substantial period [1]. However, there is general agreement that some interfaces, such as Unix shells and certain editors, take years to master. In this paper we present evidence that users do indeed change their actions in the long term. Some implications of our findings are discussed.
The Experiment
The data for this paper was collected over a period of three years from 1991. The users were 2273 computer science students at the University of Sydney, who used the sam editor on a discretionary basis during their studies. The editor has a window interface with a mouse, and also a command window for issuing ed-like commands if desired [2]. The source of sam was modified to record all commands issued by the user. Thus this data represents just the commands across the interface between the user and the system. No information is directly available about tasks. The overall methodology is reported in Kay and Thomas [3].
For this work, we denote a z-curve to be the relationship between the command rank and its relative frequency. The most common command has rank 1, the second most frequent rank 2, and so on. Our frequencies approximate to an exponential distribution: the log of the relative frequency is linearly related to the command rank. Figure 1 shows this relationship for a single user during the periods corresponding to the first and eighth deciles of experience. A command of modest rank, e.g. twenty, has a relative frequency of about 0.1%.
Figure 1: Z-curves for an Individual User (Command rank vs. log relative frequency)
There is no doubt that users go on adding to their repertoire of commands over a considerable period. We define an exploration to occur whenever a command is invoked for the first time, regardless of intentionality or outcome. Figure 2 shows the cumulative explorations for a single user over a three year period. There is definite growth over the period, albeit at a slow rate. Figure 3 shows the explorations by a group of 63 users over three years. Apart from the massive activity at the left hand side, corresponding to initial system use, there is a fairly random pattern for the remainder of the commands. It cannot be assumed that a given command is explored at the same time by everyone, even when the tasks are approximately the same for all users.
Figure 2: Exploration Curve for an Individual User (Weeks of Use vs. Cumulative Number of Explorations)
Figure 3: Exploration Events
Users change their ranking of commands over time. The smoothed ranking is graphed in Figure 4 for the cut, paste and double click (select word) commands issued by an individual. The vertical axis shows command ranks -- those towards the top of the figure represent less popular commands, while the ranks at the bottom depict commonly used items. The horizontal axis is plotted in time periods, where a period corresponds to 4000 contiguous items of monitor data. Since there was more editor use in CS3 than CS1, the periods do not map linearly onto months.
Figure 4: Changes in Command Rank Over Time
It can be seen that the popularity of the cut and paste commands increases to a plateau over about half the total period, while the use of double click fell away and then recovered slightly in the second half. In Figure 5 on the other hand, the relative use of the two write file methods switches profoundly in period 10. In this case the user moved from a walk-through menu to a click-and-point method for writing a single file. Note that the overall ranking of the dominant write command did not change even though the method did.
Figure 5: Profound Switch in File Write Method
Work on the z-curves is still in progress. However it is clear that the z-curve holds well for an entire population over different snapshots of activity. It seems likely that there is predictable variation in the slope for individual users. Most seem to gradually specialise their command use and then back away towards a slightly more general distribution. High slope corresponds to a more hierarchical use of commands than lower slope.
Zipf frequency distributions have been observed in Unix commands [4, 5], although the constancy or otherwise of this over time was not shown. It may well turn out that the frequency distribution is a function of the interface, task and user. Much work could be done in this area.
Figures 4 and 5 show considerable variation in command ranking over time. It is suspected that these changes have to satisfy the overall constraint that the frequencies fit on the z-curve. It is easy to swap from one file write method to another -- the ranks are interchanged. Where there is no direct substitution available, several ranks may be adjusted. The z-curve constraint may in fact inhibit such change. The implication for interface design is considerable. It is suggested that adaptive systems may have failed in the past because this constraint has been violated.
Surprisingly, the exploration curves seem to be fairly constant. The initial sharp rise is necessary to start using an interface -- by definition nothing has been explored yet. During this early period the user may not perform any useful tasks at all -- this rise loosely corresponds to the guessability of the interface [6].
It seems appropriate in this paper to make some quite speculative remarks about possible implications of these observations. They may be helpful in formulating future research goals.
The first point is that changes in task may take time to be reflected in changes in behaviour. The length of the lag between task and behaviour will be influenced by (i) whether the task is mandatory, e.g. "I must use this unfamiliar font" or (ii) whether the user perceives a benefit from changing, e.g. more comfortable actions or a clear time saving. The lag in case (i) will be much smaller than for case (ii). Rieman [7] found that users were almost always prepared to undertake some task-oriented exploratory activity in cases where the real-world task took them beyond their current knowledge -- examples of case (i). Our data suggests that students tried out "relevant" commands at widely varying times, see Figure 3, even though the new task would have been introduced to everyone at the same time. Furthermore the students switched between file-write methods at different times, mainly during the last semester. Our students were usually influenced by (ii), but we have examples of (i) such as the first week in CS1 [8] or finding a given line number in a file during program development.
The second point is that the z-curve may well represent some sort of processing capacity constraint in the user. Capacity constraints have been observed elsewhere, in the work on M-spaces for example [9]. The implication is that some commands will always be relatively rarely invoked by some users, even if popular with others.
The third point is that rare commands may have a latency. People spend some seconds or minutes working out how to perform some exotic global substitution because they know the potential benefit. No doubt with tremendous practice, such commands could be performed far quicker. But the z-curve constraint militates against this. As another example of latency, consider how users are prepared to consult the Unix manual when trying to use infrequent command switches.
In the context of safety critical system interfaces, it may be that rare events need to be presented to the user for a sufficiently long time to counteract the latency. Usability of such systems will also be enhanced by use of more display resources for rare events [10]. It may be possible to include latency factors in user models of individual operators.
There is a need to understand the frequency constraints better. It is possible they will have significant implications for training, design and upgrading software. The interaction between task, user, interface and time on frequency distributions needs to be understood.
A body of empirical evidence about exploratory behaviour over time is beginning to emerge. There is a need to understand the response to different situations especially in regard to user motivation, task and style.
There is also a need to understand the modelling of time itself. In this paper the actual date of an exploration was used for Figures 2 and 3. In Figures 4 and 5 equally sized chunks of monitor data were employed as time periods in the calculations of rank. Some normalization method which enables comparison across individuals is desirable; it should embody notions of the intensity of work and elapsed time.
The concept of latency needs to be developed. Latency is believed to exist because it fits with the ways we appear to use software. Effort should be expended to determine the extent of latency by users -- does it appear for all commands which are infrequently used? Can latency be reduced when a command becomes more popular? How can it be counteracted? There is a research paradox though -- methods will have to be devised which overcome the rather infrequent nature of latency. Field studies would be long-term and the design of laboratory experiments would be challenging.
Judy Kay and Kathryn Crawford deserve special thanks. M- spaces were suggested by KC. Ronny Cook and Greg Ryan provided much valuable support. David Benyon helped in the early stages of the project. The University of Sydney, Basser Department of Computer Science, is also thanked.
Issue |
Article |
Vol.28 No.2, April 1996 |
Article |
Issue |