Authoring Multimedia Documents for the Web

Muriel Jourdan

INRIA Rhone-Alpes

Opera Project (www.inrialpes.fr/opera)

Muriel.Jourdan@inrialpes.fr

The WWW consortium defined the first version of Smil [W3C SMIL 1998]: a "tag-based" standard to describe synchronized multimedia documents on the web. Smil 1.0 allows to describe multimedia documents which are mainly predictive : the temporal organization of the document is known before its presentation (only some durations of continuous object may be unknown) and readers' interactions can only trigger hyperlinks either inside or outside the document. The second version of Smil [W3C SMIL Boston 2000] (whose recommendation is expected in june 2000) will merge predictive and unpredictive behaviors, to be able to describe synchronization such as starting a video when a user click occurs.

These two standards will make the delivery of synchronized multimedia documents on the web possible, but this delivery will be really effective if and only if friendly authoring tools are designed to help authors to handle the intrinsic complexity of the temporal synchronization specification task. Indeed, two features make this one very complex: the dynamic nature of a multimedia document (its presentation is closer to a program execution than to a static document visualization) and the incremental process of the design task (it is a cyclic action/visualization process). Thus the challenging issues are :

• finding an equivalent to the WYSIWYG paradigm used in word-processor (not valuable in a dynamic context) but avoiding to design something too close to a programming environment interface which will be only used by programmers.
• helping the author to propagate local changes (change of an object duration for instance) on the whole document without loosing some existing synchronizations.
• finding technics to analyze the document to detect well-known temporal errors in every temporal synchronization tasks like deadlocks (waiting for an event that will never occurs) and integrating these technics (used mainly in real-time programming environments) in a document authoring tool.

Solutions proposed by the Opera project to overcome these difficulties are based on the following principles:

• providing the author with multiple views interface and direct manipulation capabilities on these views [JRT 1998c]. Multiple views are necessary to show the different dimensions of the document (logical, spatial, temporal and hyperlink dimensions) and its execution. Direct manipulation is a friendly way to interact with objects : for instance, the spatial organization can be directly achieved in the presentation view by moving objects. To experiment this idea the Opera project built the Kaomi toolkit [JRT 1999b] which eases the design of such multiple views authoring tools whatever the paradigm use at the interface levels is (conditions/actions, hierarchical operators, constraints). We already test the Kaomi toolkit in different contexts : editing SMIL1.0 document [JRTV 1999], MHEG like documents, or multimedia document expressed by using temporal and spatial constraints. Multimedia documents designed with such authoring tools are mainly predictive. Our next challenge will be to extend the toolkit for the design of unpredictive documents.
• designing "intelligent" temporal view. The aim of this view is twofold :
  • firstly to visualize the temporal positions of the document objects deduced from the authoring actions the author has already performed. The difficulty here is that there is often several possible objects positions (due to unpredictive behaviors or solutions space introduced with constraints formalism).
  • secondly to help the author to make the link between such positions and its authoring actions, since some temporal positions are the result of joint effects of several authoring actions.

  An "intelligent" temporal view is a view which provide authors with good supports to handle these two difficulties. In [JRT 1998c], we present such a view for a constraint-based authoring tool (namely Madeus) based on the following principle : one solution of the constraints set is shown and the author can get through the solutions space by selecting and moving objects. Moreover some graphical representations of each authoring actions are displayed in the view. The major difficulty we still have to face here is to find the best way to draw this kind of graph : roughly speaking lines connect objects which are temporally related by an authoring action, knowing that this drawing is not a static one : the author will move objects. Simultaneously, we have to think about the design of such an intelligent view in a more unpredictive context.

• using as much as possible constraint technologies in the authoring tool we proposed. A constraint defines some relations between two variables which must be always maintained. In a multimedia document context, constraints can be used to express the relations between either objects begin and end points or objects width and height. Their are at least two strong advantages of handling such relations by using constraint-based formalisms. Firstly constraints algorithms can be used both to detect some errors in the document and to automatically propagate some local changes in the whole document while maintaining valid the other constraints. Secondly, since constraints can be used to describe not only one solution but a set of solutions, we can take benefit of this flexibility to adapt a document to some presentation conditions just before its presentation. For instance, it is possible to compute a solution such that the maximum duration for reading the document is less than a given value. With Madeus [JLRST 1998], the Opera project experiments the idea of a constraint-based authoring tool for 5 years. One biggest difficulties we still have to face is to find a better compromise beween a powerful constraints solver (in term of expressivity and capabilities to control which solution is computed) and a fast constraints solver [Tardif 00]. The other challenge consists in findind some ways to extend classical contraints technics (consistency, solution search) when variables values cannot be controlled in order to be able to specify unpredictive behaviors [LS 1996a]. Finally, we are currently experimenting how constraints can be used to internally handle temporal synchronizations between objects, even if the formalism used by the author is not based on constraints. This experiment takes place in our Smil 1.0 authoring tool [JRTV 1999].
• considering the twofold issues of error analysis and diagnosis making, as crucial ones even if a document is not a "critical" process like some real-time programs. Nevertheless, multimedia documents will be really used as a new medium to convey information if and only if authors (and people which pay authors) have some strong confidence in the document execution they designed. Little work has been already done in this direction. The intensive use of constraints to handle temporal synchronizations in a multimedia documents as promoted by the Opera project, provides the author with a way to detect some temporal inconsistencies but the diagnosis issue (how helping the author to correct this error) still stays an open and difficult problem. Moreover, in more unpredictive contexts real-time formal checking methods based on states diagram and events would be certainly more suitable to detect errors. Our plan in this direction is to begin by defining something like an "error typology" both for predictive and unpredictive documents.

In conclusion, future directions of Opera work on multimedia authoring can be summed up by :

• use constraint technics "at the right place" inside authoring tools,
• extend our work to unpredictive behaviors,
• pay more attention to the error analysis and diagnosis making issues.

References

[JRT 1998c] M. Jourdan, C. Roisin, L. Tardif, ``Multiviews Interfaces for Multimedia Authoring Environments'', Proceedings of the 5th Conference on Multimedia Modeling, pp. 72-79, IEEE Computer Society, Lausanne, octobre 1998.

[JLRST 1998] M. Jourdan, N. Layaïda, C. Roisin, L. Sabry-Ismail, L. Tardif, ``Madeus, an Authoring Environment for Interactive Multimedia Documents'', Proc. of the ACM Multimedia'98 Conference, pp. 267-272, Bristol (UK), septembre 1998.

[JRT 1999b]

M. Jourdan, C. Roisin, L. Tardif, ``A Scalable Toolkit for Designing Multimedia Authoring Environments'', numéro spécial `Multimedia Authoring and Presentation: Strategies, Tools, and Experiences' de Multimedia Tools and Applications Journal, à paraître, 1999.

[JRTV 1999]

M. Jourdan, C. Roisin, L. Tardif, L. Villard, ``Authoring SMIL documents by direct manipulations during presentation'', World Wide Web, Balzer Science Publishers, vol 2. num 4. 1999.

[LS 1996a] N. Layaïda, L. Sabry-Ismail, ``Maintaining Temporal Consistency of Multimedia Documents using Constraint Networks'', Multimedia Computing and Networking 1996, M. Freeman, P. Jardetzky, H. M. Vin, ed., pp. 124-135, SPIE 2667, janvier 1996.

[Tardif 00]

Laurent Tardif, Frédéric Bes and Cécile Roisin,"Constraints for multimedia documents", in proceedings of PACLP2000, Manchester (UK), april 2000

[W3C SMIL 1998]

W3C Recommendation, Synchronized Multimedia Integration Language (SMIL) 1.0 Specification, http://www.w3.org/TR/REC-smil, 15-June 1998.

[W3C SMIL Boston 2000]

W3C Working Draft,, Synchronized Multimedia Integration Language (SMIL) Boston Specification,http://www.w3.org/TR/2000/WD-smil-boston-20000225, 25-february 2000.