A Unified Model for Representing Timing in XML Documents

Patrick L. Schmitz
Microsoft Research

Introduction

Growing interest in SMIL underscores the importance of a common language to describe timing and synchronization in documents. SMIL 1 defined syntax and semantics for a basic multimedia language. This was an important first step, but does not allow authors to use the timing semantics with other languages, like HTML and CSS. Authors need to be able to describe timing for wide range of documents, and to combine the semantics of timing and synchronization with other existing tools and semantics.  In addition, they need the flexibility to accommodate different editing/authoring scenarios and constraints. 

We need to provide a common model for timing so that authors are not forced to learn and remember different models for different document types or different authoring scenarios. In addition, where different syntactic approaches are available (e.g. to address different authoring scenarios), it is important that the model allow authors to combine the different approaches in a straightforward manner.

Background

SMIL Boston has extended the SMIL timing support, and added support for integration of timing and synchronization markup with other languages. Working Drafts describe three approaches to this integration:

  1. Inline syntax, in which attributes are added to existing language elements.
  2. Styled timing, in which CSS or XSL stylesheets are used to apply timing to a language.
  3. Timesheets, in which timing is separated from both content and presentation (style) control.

SMIL Boston only specifies the inline syntax variant, mostly due to time constraints of the Working Group, but also due to the range of open issues with styled timing and timesheets. Looking to the future, each of these three options for integration is needed for a number of applications, and with this comes the need to combine the approaches. 

A unified model must be developed that allows authors and language designers to choose the most appropriate syntax, and to combine the different methods for a given document. This timing integration model must preserve common timing semantics, and should work well with other new document tools such as XSLT.

Approaches and Issues

Inline Timing integration

Inline syntax provides a straightforward and easy-to-understand means of adding timing to a language. This approach is fairly well understood, and is documented in some detail in the SMIL Boston working drafts. Even in documents that use styled timing and/or timesheets, the inline syntax can be useful to augment or override more general rules defined with the other methods. This presumes that there is a clear hierarchy among inline timing, styled timing and timesheets

CSS provides a good precedent for such a hierarchy, defining ordering and priorities in the stylesheet cascade. While the specifics of the CSS model may not apply, the general approach is a good one. Given the many authors familiar with CSS, a unified timing model should align to the general model and principles of the cascade model. In particular, inline syntax should override styled timing and timesheets.

Styled Timing integration

Styled timing is well suited to documents (or document sections) in which the document structure and the timing structure are fairly closely aligned. An example of this is list structures that are presented as a sequence. For example the HTML <ol> construct may be presented with a highlight style applied to each list item for a given duration, in sequence.

In order to layer styled timing with inline syntax, the semantics must be consistent between the two approaches. CSS again provides a good precedent for a unified model, supporting both stylesheets and inline style specification in the cascade. Based upon the experience with CSS, a unified model for timing integration should base styled timing upon the manipulation of the same set of properties that the inline syntax describes.

Styled timing has some inherent issues, especially related to distinguishing presentation style and timing. Two of the useful actions to perform over time are application of an inline style rule (i.e. as described by a "style" attribute), and application of a class to an element (effectively adding a token to the "class" attribute for an element). Both of these time actions can result in the modification of style properties on the element. In a styled timing model, this includes the possibility of modifying the timing properties. If timing invokes styling that in turn modifies the timing, the complexity of the time model increases considerably. It is then too easy for authors to create documents that have very non-obvious or undefined behavior.

A unified model should preclude this feedback complexity. One possible approach is to "lock" the timing properties while applying the time actions for an element so that the styling side-effects of the time action cannot feed back into the timing. Other approaches may also be explored.

Timesheets

Timesheets allows authors to specify timing and synchronization in a separate document, or in a separate section of a document. The name borrows from CSS stylesheets, and the idea applies some of the principles of stylesheets to the description of timing and synchronization. The timesheet looks much like the timing section of a SMIL language document, except that referenced content can be an individual element in an XML file (e.g. a <span> in XHTML). In addition, integration semantics such as "timeAction" are supported. 

Timesheets address those authoring scenarios in which it is easier or simpler to separate the timing and synchronization description from the content and presentation controls. The specific requirements that timesheets address are that it must be possible to:

  1. describe a timing structure that is orthogonal to the document structure. In some cases, the document containment does not at all align to the timing and synchronization relationships of elements in the document.
  2. apply timing and synchronization to a document without modifying the original document. In some cases the original document has copyright restrictions, and in others the logistics of the document creation and editing process require that the timing not be inline to the content.
  3. coordinate multiple documents. SMIL can coordinate multiple media files, and HTML+SMIL shows how timing can be integrated with other XML languages. These approaches must be merged and extended to allow references to multiple documents (including media files as well as fragments of documents like HTML).
  4. allow multiple, orthogonal time actions on an element. For example the visibility of the element may be controlled for a portion of time, and a highlight style may (independently) be applied in sync with some other part of the presentation.

The first three requirements are addressed simply by virtue of separating the timing and synchronization markup into the timesheet. The fourth requirement is related to the overriding need for a unified model for timing integration. Multiple time action support and a unified model are easily achieved if we model timesheets as animating properties. That is, all time actions are simply manipulations of some property over time. The property can be an abstract like the intrinsic action of an HTML <b> (bold) element, or a specific property like CSS visibility.  Multiple references to the same property on a given element are resolved as multiple animations of the property, according to the rules already detailed in SMIL Animation. 

Mixing timesheets, styled timing and inline timing can be thought of as a variant on the mix of XSL FO and CSS stylesheets that can be used to present XML documents. In effect, each stylesheet is a filter on the document. A key issue to work out is whether there is a fixed priority between timesheets and styled timing (i.e. a defined ordering of the filters), or whether they can be arbitrarily ordered. A fixed ordering may be easier to explain to authors, but may be too constraining.

Conclusions

A unified model for timing and synchronization can built upon a property based model for timing manipulation. This extends the SMIL Boston (inline) timing and synchronization syntax, integration controls and animation model to support styled timing and timesheet support. The model aligns with the widely understood model for cascading syntax, and should work well with a range of XML tools.