Issue
Article
Vol.28 No.2, April 1996
Article
Issue
Issue |
Article |
Vol.28 No.2, April 1996 |
Article |
Issue |
The advent of user interfaces using several media in order to provide users with information and to interact with them has raised the necessity of a clear understanding of the role of time in user interactions. In fact, information needs to be composed and distributed among media which have different times for performing processing and presentation of information.
Generally, being able to describe temporal parameters is essential in order to have a real description of temporal dependent systems, where particular states can be reached if some temporal actions have been executed. It is thus important to have structures and systematic approaches to specify Interactive Systems so that time-related aspects can be addressed as well. Having these types of methods is an important pre-condition to be able to address usability issues related with time.
The concept of an interactor is an abstraction which captures specific requirements of software objects which interact with users. The software architecture of an Interactive System can be built by composing instances of the interactor model which communicate with each other. The model allows designers and developers to identify easily the main elements to take into account in the selection of concrete instances of interaction objects and to develop software architectures in a systematic way. In [P94] a theory of user interaction objects was presented. This theory was developed mainly for graphical interaction objects and needs some extensions in order to capture the specific time requirements of multimedia user interfaces more efficiently. In this paper we give indications on how to perform such an extension.
The concept of an interactor has been expressed at a mathematical level, by couples of functions which express their double, bidirectional behaviour between the user and functional core, and at a formal level, by using the syntax and semantics of a formal notation, in this case we use LOTOS [PF92]. Thus we performed the time-oriented extension of the interactor concept by introducing some constraints that force the functions that describe interactors to perform their processing under specific temporal constraints and by identifying a temporal LOTOS version that is suitable for specifying these constraints.
In order to obtain a formal notation to correctly describe temporal dependent systems, several studies have been made. Many concern the time extension of process algebra, CSP[Hoa85], CCS[Mil89], LOTOS[ISO8807]. In this context, we considered the studies on the extension of the temporal concepts to the LOTOS specification language. The proposal in [LeL94], TLOTOS, was particularly interesting with regard to the temporal operators introduced in the specification language and for the semantic associated with these.
First we use the interactor model to specify temporal constraints which are needed to describe multimedia interactions. This is performed by identifying a design space for temporal constraints expressed by the interactor model at a mathematical level. We show how specifications structured within the interactor model can be expressed using TLOTOS. This notation is considered within the ISO environment in order to produce a standard of a formal notation which can address time-related problems as well. Furthermore, we introduce a simple user model so that we can reason better about human-machine interaction and identify the possible errors which an interface may cause the user to make.
In [P94] an abstract interactor was defined as a couple of functions I=(FI, FO), where (see Figure 1):
(a) FI: (Im × Bool) -> (Inp U phi) × If
(b) FO: (Ic × If × Bool ) -> (Out U phi).
FI describes the processing of the information received from the user side and sent to the application. Im indicates the domain of the data from the user side, and Inp the data generated by the FI function to send to the application. FO describes the information flow, from the application to the user side, where Ic is the data type that indicates what it can receive from the application side, If indicates what it can receive from the input part in order to produce feedback, and Out indicates what it can send to the user side. Booleans indicate trigger events which indicate when new information has to be generated; when they are false, no information is produced (which is indicated by phi).
Also, we have: Ing: (Inp U phi) × If ->(Inp U phi). The Ing function selects in the codomain of FI (the input component of an interactor), the part to deliver the result to the application.
To clarify what these function-based definitions mean we can consider a simple, common interaction such as a pop-up menu. In this concrete case the FI function indicates one function which takes a cursor position (the set of the possible cursor positions is described by the Im domain) and generates one of the possible selectable elements (this set of elements is described by the Inp domain) when the Boolean associated with the condition button released is true. The If codomain indicates, at any time, the current selected menu element.
In this case the FO function defines the appearance of the menu: it takes the graphical description of the menu from the Ic domain, the current selected element from the If domain (this is useful in order to highlight it, so that the user has feedback on the state of the interaction) and generate the appearance of the menu, which is described in the Out codomain, when the Boolean associated with the condition button pressed is true.
Figure 1: The Possible Communication of an Interactor
In order to reason about time-dependent aspects we have to take into account that the functions associated with interactors in some cases are instantaneous and in other cases take some time to perform their processing. In our approach, an Interactive System is composed of a set of interactors which receive and generate data. The data of interactors can be generated instantaneously. In this case we have a function T: D -> N which indicates when the data considered belonging to the D domain were generated. There are also cases where the data are not generated instantaneously and where a datum is composed of a series of data (for example when voice input is provided). In this case, we can define two functions Ts and Te which indicate the starting and final times associated with the receiving or producing of the data considered. Moreover, we define TI: FI -> N which, given an Input Function of one interactor provides its processing time and TO: FO -> N which provides the same result for Output Functions of Interactors.
These temporal values can change depending on the object modelled by the specific interactor.
If this object is a mouse, the TI is instantaneous; however, this is not true for media such as printers or speech recognition systems in which the internal delay has a considerable value.
Now we can classify interactors in terms of the temporal constraints which they apply on their data and results. Thus each temporal constraint defined indicates a class of functions which apply it on different data, using various processing functions. The following sections outline some examples of these classes of functions.
When we define the temporal constraints we will highlight when they have to be applied on input functions or on output functions or on both as these couples of functions characterise the interactor's behaviour. Hereafter FIX and FOX indicate the input and the output functions of the X interactor, respectively.
This class of interactors (C1FI) can describe inputs where voice inputs and graphical inputs have to be composed. We consider an interactor Comp which belongs to the C1FI class and which receives information from two interactors: one associated with input voice, Voice, and the other with graphical selection, GraphSel. The composition of the results can only be performed when the user has generated the two values within a specific interval of time.
For example, structured input information (Inp in expression (a)) is created by one interactor in the user interface of Matis [N94], by composing the information received from the vocal request "flights from Boston to here" uttered by the user, and a mouse selection of an arrival city. This mouse selection has to be executed within a time tk before or after the vocal request, or during it.
To describe this type of interaction we can define the class C1FI whose domain and codomain are structured as the general FI function: (Im × Bool) -> (InpU phi) × If on which we apply the constraint (1).
We have: Im= Im1 U Im2, and we consider two inputs im1, im2 arriving at interactor Comp from the other two interactors (Voice and GraphSel), where im1 elem Im1, im2 elem Im2, inp1, inp2 elem Inp and im1 = FIVoice(uim1, true) and im2 = FIGraphSel(uim2, true), uim1 and uim2 are the user-generated events associated with voice input and mouse selection (see Figure 2). (elem is the set operator 'is an element of', whose character is not available in html.)
Figure 2: The Composition of Interactors Considered
In the case of the FIComp function exemplifying the C1FI class of functions we add one constraint:
(1) Ing(FIComp(im1, im2, true)) =
inp1 if Ts(uim2)-tk < T(uim1) < Te(uim2)+tk
inp2 otherwise
This expression indicates that (Figure 3) the interactor produces a result inp1 to the application when uim1 is received within a time tk before or after the receiving of uim2 or during the receiving of the uim2 value, otherwise the inp2 value is produced.
Figure 3: Graphical Representation of the FIComp Results
Here we describe a class C2FO of couples of interactors which have to synchronise with each other for their output. Suppose we have two interactors I1, I2; we want to introduce a temporal constraint on the output results of these interactors.
We have FOT1 and FOT2 belonging to the C2FO class where
FOT1, FOT2: (Ic x If x Bool) -> Out with
out1 elem Out1 the output data of I1 and out2 elem Out2 the output data of I2
We define:
FOT1(ic1, phi, true) = out1
FOT2(ic2, phi, true) = out2 if Ts(out2) - Ts(out1) = T2
This function indicates that an output out2, which is sent from an interactor I2, must be produced in a specific instant, depending on the starting time of an output result out1, which is sent from an interactor I1. A typical example of this situation is two interactors associated with sound and video which have to synchronise their output.
This class of interactors (C3FI) models the situation in which, after receiving an application message, the user has a specific time to produce a particular input value; it is not possible to produce this value when the established time has passed.
If we have a generic interactor X belonging to the C3FI class, whose output function is defined in such a way that given FIX: (Im x Bool) -> (Inp U phi) x If,
if FOX(ic1, if1, true) = out1, and im1 elem Im, out1 elem Out, inp1 elem Inp, if1 elem If, ic1 elem Ic
we want the following constraint to be satisfied:
Ing(FIX(im1, true)) = inp1 if T(im1) - T(out1) < tk
= phi otherwise
Figure 4: Graphical Representation of the FIX results
The T(im1) - T(out1) < tk expression indicates that the difference between the time the input value belonging to the Im1 data type arrives and the output value to the user is produced has to be less than tk. For example, this function is useful for modelling a specific situation of the user interface for a machine that allows users to withdraw money by credit card. In this case, after receiving the system message to provide the personal code, the user has a specific time interval in order to carry out the required operation.
Below we give an example which shows the importance of introducing the time concept and what situations this concept allows us to describe. We use TLOTOS to describe the new interactor model. We want to highlight what information the user model can give.
We can consider, as an example, a graphical editor that supports speech and mouse interactions. Here, the user can say "create that there" and at the same time, use the mouse to select the object to create and to indicate the location. We thus have two levels of composition of information; the former combines the "create that there" expression with the selection of the object, the latter combines this with the selection of a point on the screen.
Figure 5: Example of a Multimedia Interaction with a Temporal Window
The first composition (i.e. "create that there" with the mouse-selected object) is possible if the interactor command1 verifies that the select_object data is generated by the user in a tk time interval before or after the voc_req data or during this; in this case identify_object is sent to the command2 interactor. Moreover, the interactor command2 has to perform a similar control.
The command1 and command2 interactors have input functions that belong to the C1FI class which was defined in section 2.1.
The following TLOTOS expression considers the two possible orders in receiving the input from the interactors associated with voice and graphical inputs. We assume that TI(FIObject) is instantaneous. TI(FIVoice) is the time spent by the vocal recognition to process the vocal request.
The sequence sent_voc_req and selected_object is performed only when the condition [tk > TI(FIVoice)] is verified (the two possibilities are represented in Figure 6) because this would allow the voice to arrive at the command1 interactor before mouse selection even if it takes some time (TI(FIVoice)) to be processed.
In the other order (first mouse selection and next voice arrive at the command1 interactor) we have to check that t2 (which is the time when the mouse selection occurs) happens within the temporal window [(t2 > Ts(voc_req) - tk) and (t2 < (Te(voc_req) +tk ))].
Figure 6: Graphical Representation of the Considered Interval Times
The command1 interactor can be described using TLOTOS as indicated below:
Process command1 [sent_voc_req, selected_object, identify_object]: noexit:=Endprocsent_voc_req; (selected_object {tk -TI(FIVoice)}; identify_object; command1[...]
[] delta tk -TI(FIVoice) command1[...])
[] selected_object@t2; sent_voc_req;
[(t2 > (Ts(voc_req) - tk)) and (t2 < (Te(voc_req) +tk ))] -> identify_object;
command1[...]
[] [not((t2 > Ts(voc_req) - tk) and (t2 < (Te(voc_req) +tk )))]-> command1[...]
In the specification, the expression g{time} indicates that the action g is expected in the time interval (this operator is called life reducer), the expression delta T g indicates that the action g is expected after a time value T (called delay) and g@ T records in the variable T the time when it was generated (this operator is called wait).
In this example the user model helps to illustrate that the change in the media interaction, from a vocal recognition to a mouse, looking for the graphical area and selecting the part of interest, may be so time consuming that the object request can no longer be created.
The internal and external actions related to a vocal request and to selecting an object from a list of elements are indicated in the table below. In this case the user is synchronized with the user interface on the gates voc_req and select_object.
In this context, we want to model the user as a process, which can perform both external and internal actions. The former include the perception of both graphical and sound aspects of what the system produces, the latter are associated with users' interpretations of their own perceptions. Decisions are thus made on what tasks to perform and what actions are needed. With a user model, we are better equipped to reason about human-computer interactions and to identify the errors which a user interface may cause the user to make. This distinction between internal and external actions can be directly mapped in the LOTOS model, where the actions performed by processes have the same type of distinction.
Figure 7: The User Model
Figure 8: User Actions in the Example Considered
In this paper we have shown first results of a proposal for an extension of the interactor concept to include the time notion in order to correctly describe multimedia user interfaces. We began the work by expressing classes of temporal constraints on the main aspects which characterise the interactor's behaviour. Moreover, we have outlined an example of an application of the new interactor using the TLOTOS notation.
Future work will be dedicated to building an automatic tool which will provide a graphical environment for specifying constraints, which should allow us to reason about time-dependent behaviours in user-system interactions and task performance.
Issue |
Article |
Vol.28 No.2, April 1996 |
Article |
Issue |