Publications Lynda Hardman

Many websites allow their users to personalize their profiles. As users subscribe to many personalization websites, such as social net- works or search systems, each user owns different profiles, which are seldom compatible. Yet, there is a strong need for comparing the profiles of different users to discover shared interests, e.g., by integrating all user profiles into a global one. In this paper, we propose a novel method for in- tegrating and ranking user interests from various profiles. Our approach relies on the identification of high-level concepts around which similar user interests are clustered. We compute the weight of each cluster with respect to the other ones, thus enabling the ranking of the most shared user interests between user profiles.

Traditionally the relations between concepts from a controlled vocabulary, such as the hierarchical and associative relations in a thesaurus, have been used to support users in their search process. In the context of the Semantic Web, multiple interlinked vocabularies are becoming available, providing a large number of different relations between concepts. However, for a specific search task, only a small fraction of these will be meaningful to the user, and currently we have little understanding of which methods can be used to determine this. In this paper, we describe a case study in the cultural heritage domain that investigates support for the specific task of finding artworks in a data set of multiple linked art collections and vocabularies. In a first experiment a number of use cases from domain experts are collected and the paths in the data graph by which artworks can be found are analysed. A number of different types of paths are identified and their usefulness is qualitatively evaluated. In a second experiment we explore how the different path types can be used in a semantic search algorithm to support the intended search behavior indicated by the experts. We conclude that effective end-user support requires a highly interactive application in which the user can explore multiple search strategies. Based on our findings we discuss the implications on the design of such an interactive search application.

Comparison search is an information seeking task where a user examines individual items or sets of items for similarities and differences. While this is a known information need among experts and knowledge workers, appropriate tools are not available. In this paper, we discuss comparison search in the cultural heritage domain, a domain characterized by large, rich and heterogeneous data sets, where different organizations deploy different schemata and terminologies to describe their artifacts. This diversity makes meaningful comparison difficult. We developed a thesaurus-based comparison search application called LISA, a tool that allows a user to search, select and compare sets of artifacts. Different visualizations allow users to use different comparison strategies to cope with the underlying heterogeneous data and the complexity of the search tasks. We conducted two user studies. A preliminary study identifies the problems experts face while performing comparison search tasks. A second user study examines the effectiveness of LISA in helping to solve comparison search tasks. The main contribution of this paper is to establish design guidelines for the data and interface of a comparison search application. Moreover, we offer insights into when thesauri and metadata are appropriate for use in such applications.

People conventionally refer to an action or occurrence taking place at a certain time at a specific location as an event. This notion is potentially useful for connecting individual facts recorded in the rapidly growing collection of linked data sets and for discovering more complex relationships between data. In this paper, we provide an overview and comparison of existing event models, looking at the different choices they make of how to represent events. We describe a model for publishing records of events as Linked Data. We present tools for populating this model and a prototype "event directory" web service, which can be used to locate stable URIs for events that have occurred, provide RDFS+OWL descriptions and link to related resources.

We performed a user experiment in which museum professionals used vocabularies from the Web for annotating the subject matter of museum objects. We study the requirements on the underlying RDF dataset, search algorithms and user interface design in a real world setting. We identify the advantages of reusing vocabularies from the Web and discuss how and to what extent the disadvantages can be overcome. The study is performed at the Print Room of the Rijksmuseum Amsterdam, where currently a large collection of prints, photographs and drawings is being catalogued. We report on the analysis of the current practice of professional cataloguers, the iterative design of an annotation tool and a qualitative evaluation of this tool with a user experiment in a realistic annotation environment. We discuss our findings in terms of their impact on the RDF data, the semantic search functionality and the user interface.

People conventionally refer to an action or occurrence taking place at a certain time at a specific location as an event. This notion is potentially useful for connecting individual facts recorded in the rapidly growing collection of linked data sets and for discovering more complex relationships between data. In this paper, we provide an overview and comparison of existing RDFS+OWL event models, looking at the different choices they make of how to represent events. We describe a recommended model for publishing records of events as Linked Data. We present tools for populating this model and a prototype of an "event directory" web service, which can be used to locate stable URIs for events that have occurred and to provide RDFS+OWL descriptions of them and links to related resources.

The Web of data enables fragments of information to be identified, described and connected together in a rich information environment. Users requiring information are faced with the problem of finding out what information is available, and obtaining sufficient fragments to successfully carry out their task. Systems supporting these tasks can use the fragments, descriptions of them and relationships among them, to improve both the selection and presentation of the information. Questions to be answered are: which information needs can be better supported, and how can the Web of data help. While the construction of the linked data cloud'' is necessary to even start thinking about providing this type of support for users, our claim is that we first need to establish the user's information needs before establishing the potential roles the linked data can play in information selection and presentation. In this paper, we discuss potential uses of linked data to support users' information needs, give examples of using linked data to support user information seeking tasks and highlight future research directions.

We describe two user studies that investigate organization strategies of autocompletion in a known-item search task: searching for terms taken from a thesaurus. In Study 1, we explored ways of grouping term suggestions from two different thesauri (TGN and WordNet) and found that different thesauri may require different organization strategies. Users found Group organization more appropriate to organize location names from TGN, while Alphabetical works better for WordNet. In Study 2, we compared three different organization strategies (Alphabetical, Group and Composite) for location name search tasks. The results indicate that for TGN autocompletion interfaces help improve the quality of keywords, Group and Composite organization help users search faster, and is perceived easier to understand and to use than Alphabetical.

Cultural heritage involves rich and highly heterogeneous collections of different people, organizations and collections. Preserved mainly by professionals it is challenging to convey this diversity of perspectives and information to the general public. Professionals also experience a great deal of obstacles archiving digital collections. This special issue presents current trends in employing AI and Web technologies to overcome such problems.

While many multimedia systems allow the association of semantic annotations with media assets, there is no agreed-upon way of sharing these among systems. As an initial step within the multimedia community, we identify a small number of fundamental processes of media production, which we term canonical processes. We specify their inputs and outputs, but deliberately do not specify their inner workings, concentrating rather on the information flow between them. We thus identify a small set of building blocks that can be supported in semantically aware media production tools. The processes are identified in conjunction with a number of different research groups within the community who supply, in the companion papers, descriptions of existing systems and a mapping to them. We give a basic formalisation of the processes and discuss how this fits with other formalisation endeavours. We present a number of frequently asked questions during the development of the model and this special issue.

In this article we describe a Semantic Web application for semantic annotation and search in large virtual collections of cultural-heritage objects, indexed with multiple vocabularies. During the annotation phase we harvest, enrich and align collection metadata and vocabularies. The semantic-search facilities support keyword-based queries of the graph (currently 20 M triples), resulting in semantically grouped result clusters, all representing potential semantic matches of the original query. We show two sample search scenario’s. The annotation and search software is open source and is already being used by third parties. All software is based on established Web standards, in particular HTML/XML, CSS, RDF/OWL, SPARQL and JavaScript.

Multimedia constitutes an interesting field of application for Semantic Web and Semantic Web reasoning, as the access and management of multimedia content and context depends strongly on the semantic descriptions of both. At the same time, multimedia resources constitute complex objects, the descriptions of which are involved and require the foundation on sound modeling practice in order to represent findings of low- and high level multimedia analysis and to make them accessible via Semantic Web querying of resources. This tutorial aims to provide a red thread through these different issues and to give an outline of where Semantic Web modeling and reasoning needs to further contribute to the area of semantic multimedia for the fruitful interaction between these two fields of computer science.

We report on our user study on the information seeking behavior of cultural heritage experts and the sources they use to carry out search tasks. Seventeen experts from nine cultural heritage institutes in the Netherlands were interviewed and asked to answer questionnaires about their daily search activities. The interviews helped us to better understand their search motivations, types, sources and tools. A key finding of our study is that the majority of search tasks involve relatively complex information gathering. This is in contrast to the relatively simple fact-finding oriented support provided by current tools. We describe a number of strategies that experts have developed to overcome the inadequacies of their tools. Finally, based on the analysis, we derive general trends of cultural heritage experts’ information seeking needs and discuss our preliminary experiences with potential solutions.

State of the art web search applications allow the user to aggregate information from many sources. Because of this, users are confronted with having to assess the reliability of information from different sources. This paper reports on an empirical user study on the effect of displaying credibility ratings of multiple cultural heritage sources (e.g. museum websites, art blogs) on users' search performance and selection. The study investigated whether source credibility has an influence on users' search performance when they are confronted with only a few information sources or where there are many. The results of our online interactive study (n=122) show that by presenting the source credibility information explicitly, people's confidence in their selection of information significantly increases, even though it does not necessarily make search more time efficient. Additionally, we highlight credibility issues that are applicable beyond the cultural heritage domain, such as issues related to credibility measures and choice of visualization.

In this paper we describe a model for automatically generating video documentaries. This allows viewers to specify the subject and the point of view of the documentary to be generated. The domain is matter-of-opinion documentaries based on interviews. The model combines rhetorical presentation patterns used by documentary makers with a data-driven approach. Rhetorical presentation patterns provide the viewer with an engaging viewing experience, while a data-driven approach can be applied to growing media repositories. To date, the modeling of rhetoric has been achieved in a top-down manner using closed repositories, while data-driven generation approaches were unable to implement non-trivial rhetorical presentation patterns. We describe an implementation of our model in a system, Vox Populi, and apply it to an online documentary shot by a group of independent amateur documentarists.

The Sapir-Whorf Hypothesis postulates a link between thought and language: if you haven't got a word for a concept, you can't think about it; if you don't think about it, you won't invent a word for it. The term "Web 2.0" is a case in point. It was invented by a book publisher as a term to build a series of conferences around, and conceptualises the idea of Web sites that gain value by their users adding data to them. But the concept existed before the term: Ebay was already Web 2.0 in the era of Web 1.0. But now we have the term we can talk about it, and it becomes a structure in our minds, and in this case a movement has built up around it.

In previous work, we proposed a unified approach for describing multimodal human-computer interaction and interaction constraints in terms of sensual, motor, perceptual and cognitive functions of users. In this paper, we extend this work by providing formalised vocabularies that express human functionalities and anatomical structures required by specific modalities. The central theme of our approach is to connect these modality representations with descriptions of user, device and environmental constraints that influence the interaction. These descriptions can then be used in a reasoning framework that will exploit formal connections among interaction modalities and constraints. The focus of this paper is on specifying a comprehensive vocabulary of necessary concepts. Within the context of an interaction framework, we describe a number of examples that use this formalised knowledge.

We report the results of a user study that captures knowledge on how cultural heritage experts search for information. We use a qualitative study technique with participants from four cultural heritage institutions in the Netherlands who were interviewed and asked to answer questionnaires about their daily work. Our goal is to acquire knowledge of their information seeking needs and the information sources they use. The paper provides an analysis and discussion of the issues that experts frequently face when searching for information.

Autocompletion is a commonly used interface feature in diverse applications. Semantic Web data has, on the one hand, the potential to provide new functionality by exploiting the semantics in the data used for generating autocompletion suggestions. Semantic Web applications, on the other hand, typically pose extra requirements on the semantic properties of the suggestions given. When the number of syntactic matches becomes too large, some means of selecting a semantically meaningful subset of suggestions to be presented to the user is needed. In this paper we identify a number of key design dimensions of autocompletion interface components. Our hypothesis is that a one-size-fits-all solution to autocompletion interface components does not exist, because different tasks and different data sets require interfaces corresponding to different points in our design space. We present a fully configurable architecture, which can be used to configure autocompletion components to the desired point in this design space. The architecture has been implemented as an open source software component that can be plugged into a variety of applications. We report on the results of a user evaluation that confirms this hypothesis, and describe the need to evaluate semantic autocompletion in a task and application-specific context.

Many Semantic Web applications provide access to their resources through text-based search queries, using explicit semantics to improve the search results. This paper provides an analysis of the current state of the art in semantic search, based on 35 existing systems. We identify different types of semantic search features that are used during query construction, the core search process, the presentation of the search results and user feedback on query and results. For each of these, we consider the functionality that the system provides and how this is made available through the user interface.

This paper describes ongoing work of a project aimed at exploiting Semantic Web techniques to support searching and annotating large cross-institutional digital-heritage collections. The project demonstrator contains multiple collections and multiple vocabularies. The architecture is fully based on Web standards. We show novel search and presentation techniques which make use of interoperability between the collections and between the vocabularies.

Semantic descriptions of non-textual media available on the web can be used to facilitate retrieval and presentation of media assets and documents containing them. While technologies for multimedia semantic descriptions already exist, there is as yet no formal description of a high quality multimedia ontology that is compatible with existing (semantic) web technologies. We explain the complexity of the problem using an annotation scenario. We then derive a number of requirements for specifying a formal multimedia ontology before we present the developed ontology, COMM, and evaluate it with respect to our requirements. We provide an API for generating multimedia annotations that conform to COMM.

The main objective of the MultimediaN E-Culture project is to demonstrate how novel semantic-web and presentation technologies can be deployed to provide better indexing and search support within large virtual collections of cultural-heritage resources. The architecture is fully based on open web standards, in particular XML, SVG, RDF/OWL and SPARQL. One basic hypothesis underlying this work is that the use of explicit background knowledge in the form of ontologies/vocabularies/thesauri is in particular useful in information retrieval in knowledge-rich domains.

Facet browsing has become popular as a user friendly interface to data repositories. The Semantic Web raises new challenges due to the heterogeneous character of the data. First, users should be able to select and navigate through facets of resources of any type and to make selections based on properties of other, semantically related, types. Second, where traditional facet browsers require manual configuration of the software, a semantic web browser should be able to handle any RDFS dataset without any additional configuration. Third, hierarchical data on the semantic web is not designed for browsing: complementary techniques, such as search, should be available to overcome this problem. We address these requirements in our browser, /facet. Additionally, the interface allows the inclusion of facet-specific display options that go beyond the hierarchical navigation that characterizes current facet browsing. /facet is a tool for Semantic Web developers as an instant interface to their complete dataset. The automatic facet configuration generated by the system can then be further refined to configure it as a tool for end users. The implementation is based on current Web standards and open source software. The new functionality is motivated using a scenario from the cultural heritage domain.

Finding relevant information is one step in the chain of understanding information. Presenting material to a user in a suitable way is a further step. Our research focuses on using semantic annotations of multimedia elements to increase the

Facet browsing has become popular as a user friendly interface to data repositories. We extend facet browsing of Semantic Web data in four ways. First, users are able to select and navigate through facets of resources of any type and to make selections based on properties of other, semantically related, types. Second, we address a disadvantage of hierarchy-based navigation by adding a keyword search interface that dynamically makes semantically relevant suggestions. Third, the interface of our browser, /facet, allows the inclusion of facet-specific display options that go beyond the hierarchical navigation that characterizes current facet browsing. Fourth, the browser works on any RDFS dataset without any additional configuration. These properties make /facet an ideal tool for Semantic Web developers that need a instant interface to their complete dataset. The automatic facet configuration generated by the system can then be further refined to configure it as a tool for end users. The implementation is based on current Web standards and open source software. The new functionality we provide is motivated using a scenario from the cultural heritage domain

Browsing is a useful way of exploring annotated media repositories. Sets of links can be automatically created from the annotations associated with the media items in the repository. When there are also relationships among the annotations themselves, such as when the annotation terms are part of a thesaurus, these relations can also be used in the link generation process. Using structured annotations and a thesaurus for generating link sets has two advantages. The first is to evaluate the effectiveness of the terms in the thesaurus for classifying the media items in the repository. The second is to be able to control the links being generated by changing relationships within the thesaurus. The work is illustrated using video segments annotated with argument structures, but we show that the method used is independent of the media types and applicable to systems that use similar annotation structures and typed relations among them.

This paper discusses generating document structure from annotated media repositories in a domain-independent manner. This approaches the vision of a universal RDF browser. We start by applying the search-and-browse paradigm established for the WWW to RDF presentation. Furthermore, this paper adds to this paradigm the clustering-based derivation of document structure from search returns, providing simple but domain-independent hypermedia generation from RDF stores. While such generated presentations hardly meet the standards of those written by humans, they provide quick access to media repositories when the required document has not yet been written. The resulting system allows a user to specify a topic for which it generates a hypermedia document providing guided navigation through virtually any RDF repository. The impact for content providers is that as soon as one adds new media items and their annotations to a repository, they become immediately available for automatic integration into subsequently requested presentations

This paper discusses generating document structure from annotated media repositories in a domain-independent manner. This approaches the vision of a universal RDF browser. We start by applying the search-and-browse paradigm established for the WWW to RDF presentation. Furthermore, this paper adds to this paradigm the clustering-based derivation of document structure from search returns, providing simple but domain-independent hypermedia generation from RDF stores. While such generated presentations hardly meet the standards of those written by humans, they provide quick access to media repositories when the needed document has not yet been written. The resulting system allows a user to specify a topic for which it generates a hypermedia document providing guided navigation through virtually any RDF repository. The impact for content providers is that as soon as new media items and their annotations are added to a repository, they become immediately available for automatic integration into subsequently requested presentations.

We use rhetorical annotations to specify a generation process that can assemble meaningful video sequences with a communicative goal and an argumentative progression. Our annotation schema encodes the verbal information contained in the audio channel, identifying the claims the interviewees make and the argumentation structures they use to make those claims. Based on this schema, we construct a semantic graph which is traversed by rhetoric-based strategies selecting video segments. The selected video segments are edited to form a meaningful video sequence.

Creating compelling multimedia presentations is a complex task. It involves the capture of media assets, then editing and authoring these into one or more final presentations. Tools tend to concentrate on a single aspect of media production to reduce the complexity of the interface and to tailor support to a specific task. While these tools are suited to the task they are designed for, very often there is no consideration for input requirements for the next tool down the line. Each tool in the multimedia production chain has the potential for adding semantic annotations to the media asset at hand, describing relevant aspects of the asset and why it is being used for a particular purpose. These annotations need to be included in the information handed over to the next tool in the production chain. We specify inputs and outputs to a number of canonical processes we identify in multimedia production. We do not specify the intricate workings of the processes, but concentrate on the information flow between them. To validate our model we describe a number of workflows in terms of the processes and specify the inputs and outputs. Our claim is that by specifying explicitly the input and output required for processes that occur in widely differing uses of media we can provide the tool-building community with a small set of building blocks that can be supported for semantically aware media production tools.

We use rhetorical annotations to specify a generation process that can assemble meaningful video sequences with a communicative goal and an argumentative progression. Our annotation schema encodes the verbal information contained in the audio channel, identifying the claims the interviewees make and the argumentation structures they use to make those claims. Based on this schema, we construct a semantic graph which is traversed by rhetoric-based strategies selecting video segments. The selected video segments are edited to form a meaningful video sequence.

Vox Populi is a system that automatically generates video documentaries. Our application domain is video interviews about controversial topics. Via a Web interface the user selects one of the possible topics and a point of view she would like the generated sequence to present, and the engine selects and assembles video material from the repository to satisfy the user request.

This article discusses the state of the art in metadata for audio-visual media in large semantic networks, such as the Semantic Web. Our discussion is predominantly motivated by the two most widely known approaches towards machine-processable and semantic-based content description, namely the Semantic Web activity of the W3C and ISO's efforts in the direction of complex media content modeling, in particular the Multimedia Content Description Interface (MPEG-7). We explain that the conceptual ideas and technologies discussed in both approaches are essential for the next step in multimedia development. Unfortunately, there are still many practical obstacles that block their widespread use for providing multimedia metadata on the Web. Based on a scenario to explain our vision of a media-aware Semantic Web, we derive in Part I a number of problems regarding the semantic content description of media units. We then discuss the multimedia production chain, in particular emphasizing the role of progressive metadata production. As a result we distill a set of media-based metadata production requirements and show how current media production environments fail to address these. We then introduce those parts of the W3C and ISO standardization works that are relevant to our discussion. In Part II of this article, we analyze their abilities to define structures for describing media semantics, discuss syntactic and semantic problems, ontological problems for media semantics, and the problems of applying the theoretical concepts to real world problems. Part II concludes with implications of the findings for future action with respect to the actions the community should take.

Authoring context sensitive, interactive multimedia presentations is much more complex than authoring either purely audiovisual applications or text. Interactions among media objects need to be described as a set of spatio-temporal relationships that account for synchronous and asynchronous interactions, as well as on-demand linking behavior. This article considers the issues that need to be addressed by an authoring environment. We begin with a partitioning of concerns based on seven classes of authoring problems. We then describe a selection of multimedia authoring environments within four different authoring paradigms: structured, timeline, graph and scripting. We next provide observations and insights into the authoring process and argue that the structured paradigm provides the most useful framework for presentation authoring. We close with an example application of the structured multimedia authoring paradigm in the context of our own structure-based system GRiNS.

Applications that use annotated multimedia assets need to be able to process all the annotations about a specific media asset. At first sight, this seems almost trivial, but annotations are needed for different levels of description, these need to be related to each other in the appropriate way and, in particular on the Semantic Web, annotations may not all be stored in the same place. We distinguish between technical descriptions of a media asset from content-level descriptions. At both levels, the annotations needed in a single application may come from different vocabularies. In addition, the instantiated values for a term used from an ontology also need to be specified. We present a number of existing vocabularies related to multimedia, discuss the above problems then discuss requirements for and the desirability of a lightweight multimedia ontology.

Most document structures define layout structures which implicitly define semantic relationships between content elements. While document structures for text are well established (books, reports, papers etc.), models for time based documents such as multimedia and hypermedia are relatively new and lack established document structures. Traditional document description languages convey domain-dependent semantic relationships implicitly, using domain-independent mark-up for expressing layout. This works well for textual documents a,s for example, CSS and HTML demonstrate. True device independence, however, sometimes requires a change of document model to maintain the content semantics. To achieve this we need explicit information about the discourse role of the content element. We propose a model in which content is marked-up with the discourse role it plays in the document. This way the formatter has knowledge about the function of a content element so it can make appropriate lay out choices.

The Semantic Web envisions a Web that is both human readable and machine processible. In practice, however, there is still a large conceptual gap between annotated content repositories on the one hand, and coherent, human readable Web pages on the other. To bridge this conceptual gap, one needs to select the appropriate content from the repository, structure and order the material, and design a Web page that effectively conveys the selected content and the chosen structure. In addition to this conceptual gap, there is also a technological gap. On one side of this gap, we find the semantic-oriented technology deployed to build annotated content repositories. This includes RDF, RDF Schema and OWL. On the other side of the gap is the syntax-oriented technology deployed to build Websites. This includes XML, XSLT, CSS, XHTML and SMIL. In this paper, we discuss the conceptual relationships between the world of explicit metadata semantics and the world of Web presentations and their underlying syntactic formats. We also explore to what extent this gap can be bridged automatically, and how current Web technologies can be used to support this process.

Traditionally, research in applying Semantic Web technology to multimedia information systems has focused on using annotations and ontologies to improve the retrieval process. This paper concentrates on improving the presentation of the retrieval results. First, our approach uses ontological domain knowledge to select and organize the content relevant to the topic the user is interested in. Domain ontologies are valuable in the presentation generation process, because effective presentations are those that succeed in conveying the relevant domain semantics to the user. Explicit discourse and narrative knowledge allows selection of appropriate presentation genres and creation of narrative structures, which are used for conveying these domain relations. In addition, knowledge of graphic design and media characteristics is essential to transform abstract presentation structures into real multimedia presentations. Design knowledge determines how the semantics and presentation structure are expressed in the multimedia presentation. In traditional Web environments, this type of design knowledge remains implicit, hidden in style sheets and other document transformation code. Our second use of Semantic Web technology is to model design knowledge explicitly, and to enable it to drive the transformations needed to turn annotated media items into structured presentations

Time-based, media-centric Web presentations can be described declaratively in the XML world through the development of languages such as SMIL. It is difficult, however, to fully integrate them in a complete document transformation processing chain. In order to achieve the desired processing of data-driven, time-based, media-centric presentations, the text-flow based formatting vocabularies used by style languages such as XSL, CSS and DSSSL need to be extended. The paper presents a selection of use cases which are used to derive a list of requirements for a multimedia style and transformation formatting vocabulary. The boundaries of applicability of existing text-based formatting models for media-centric transformations are analyzed. The paper then discusses the advantages and disadvantages of a fully-fledged time-based multimedia formatting model. Finally, the discussion is illustrated by describing the key properties of the example multimedia formatting vocabulary currently implemented in the back-end of our Cuypers multimedia transformation engine.

Multimedia presentation generation has to be able to balance the functional aspects of a presentation that address the information needs of the user and its aesthetic form. We demonstrate our approach using automatic colour design for which we integrate relevant aspects of colour theory. We do not provide a definition of the relative importance of form versus function, but seek to explore the roles of subjective elements in the generation process.

Multimedia presentation generation has to be able to balance the functional aspects of a presentation that address the information needs of the user, and its aesthetic form. We demonstrate our approach using automatic colour design for which we integrate relevant aspects of colour theory. Colour selection takes the relative importance of form and function into account through the use of weights in the generation process. We do not provide a definition of the relative importance of form versus function, but seek to explore the roles of subjective elements in the generation process.

With the rapid growth of interest in the Internet as a means for accessing multimedia presentations for education, entertainment and commerce, comes a corresponding need for systems to supply automatically generated, personalised presentations. Multimedia is a rich and complex genre of resources and the interrelated effects of content, style and structure ensure that automatic presentation generation is in itself a complicated and challenging task. Integrating a model for user personalisation adds a further layer of complication and ensuring that the requirements of user, supplier and platform are all met is a demanding undertaking. This project investigates the influence of information about a user in the process of generating a multimedia presentation. As a result an architecture taking into account these trade-offs is proposed. To evaluate this combined architecture, a framework has been implemented which adjusts colour choices based on the different influences involved. This paper describes the integration of a user modelling approach within an existing system architecture, discusses some of the issues involved in applying user modelling to multimedia presentation generation and describes the prototype implementation and how it addresses some of these issues.

Time-based, media-centric Web presentations can be described declaratively in the XML world through the development of languages such as SMIL. It is difficult, however, to fully integrate them in a complete document transformation processing chain. In order to achieve the desired processing of data-driven, time-based, media-centric presentations, the text-flow based formatting vocabularies used by style languages such as XSL, CSS and DSSSL need to be extended. The paper presents a selection of use cases which are used to derive a list of requirements for a multimedia style and transformation formatting vocabulary. The boundaries of applicability of existing text-based formatting models for media-centric transformations are analyzed. The paper then discusses the advantages and disadvantages of a fully-fledged time-based multimedia formatting model. Finally, the discussion is illustrated by describing the key properties of the example multimedia formatting vocabulary currently implemented in the back-end of our Cuypers multimedia transformation engine.

The 'Document Web' has established itself through the creation of an impressive family of XML and related languages. In addition to this, the 'Semantic Web' is developing its own family of languages based primarily on RDF. Although these families were both developed specifically for 'the Web', each language family has been developed from different premises with specific goals in mind. The result is that combining both families in a single application is surprisingly difficult. This is unfortunate, since the combination of semantic processing with document processing provides advantages in both directions --- namely using semantic inferencing for more intelligent document processing and using document processing tools for presenting semantic representations to an end-user. In this paper, we investigate this integration problem, focusing on the role of (RDF) semantics in selecting, structuring and styling (XML) content. We analyze the approaches taken by two example architectures and use our analysis to derive a more integrated alternative.

Traditionally, research in applying Semantic Web technology to multimedia information systems has focused on using annotations and ontologies to improve the retrieval process. This paper concentrates on improving the presentation of the retrieval results. First, our approach uses ontological domain knowledge to select and organize the content relevant to the topic the user is interested in. Domain ontologies are valuable in the presentation generation process, because effective presentations are those that succeed in conveying the relevant domain semantics to the user. Explicit discourse and narrative knowledge allows selection of appropriate presentation genres and creation of narrative structures, which are used for conveying these domain relations. In addition, knowledge of graphic design and media characteristics is essential to transform abstract presentation structures in real multimedia presentations. Design knowledge determines how the semantics and presentation structure are expressed in the multimedia presentation. In traditional Web environments, this type of design knowledge remains implicit, hidden in style sheets and other document transformation code. Our second use of Semantic Web technology is to model design knowledge explicitly, and to let it drive the transformations needed to turn annotated media items into structured presentations.

The research goal of the Multimedia and Human-Computer Interaction group at CWI is to investigate the automated generation of Web-based hypermedia presentations tailored to the abilities, preferences and platform of the user. This requires the description and processing of different types of information in order to assemble semantically annotated media items into a coherent presentation, i.e. a presentation that communicates the intended semantic relations to the user.

Web publishing systems have to take into account a plethora of Web-enabled devices, user preferences and abilities. Technologies generating these presentations will need to be explicitly aware of the context in which the information is being presented. Semantic Web technology can be a fundamental part of the solution to this problem by explicitly modeling the knowledge needed to adapt presentations to a specific delivery context. We propose the development of a emphSmart Stylelayer which is able to process metadata that describes content and use this metadata to improve the presentation of the content to human users. In the paper, we derive the requirements of such a Smart Style layer by considering Web design from both the document engineering and graphic design perspectives. In addition, design trade-offs made by human designers have to be taken into account for the automated process. After stating the requirements for a Smart Style layer, we discuss to what extent the currently available Web technology can be used and what its limitations are. The limitations are illustrated with examples of potential future extensions.

Until recently, the Semantic Web was little more than a name for the next generation Web infrastructure as envisioned by its inventor, Tim Berners-Lee. Now, with the introduction of XML and RDF, and new developments such as RDF Schema and DAML+OIL, the Semantic Web is rapidly taking shape. In this paper, we first give an overview of the state-of-the-art in Semantic Web technology, the key relationships with traditional hypermedia research, and a comprehensive reference list to various sets of literature (Hypertext, Web and Semantic Web). We then present a research agenda by describing the open research issues in the development of the Semantic Web from the perspective of hypermedia research.

First generation Web-content encodes information in handwritten (HTML) Web pages. Second generation Web content generates HTML pages on demand, e.g. by filling in templates with content retrieved dynamically from a database or transformation of structured documents using style sheets (e.g. XSLT). Third generation Web pages will make use of rich markup (e.g. XML) along with metadata (e.g. RDF) schemes to make the content not only machine readable but also machine processable - a necessary pre-requisite to the emphSemantic Web. While text-based content on the Web is already rapidly approaching the third generation, multimedia content is still trying to catch up with second generation techniques. Multimedia document processing has a number of fundamentally different requirements from text which make it more difficult to incorporate within the document processing chain. In particular, multimedia transformation uses different document and presentation abstractions, its formatting rules cannot be based on text-flow, it requires feedback from the formatting back-end and is hard to describe in the functional style of current style languages. We state the requirements for second generation processing of multimedia and describe how these have been incorporated in our prototype multimedia document transformation environment, emphCuypers. The system overcomes a number of the restrictions of the text-flow based tool sets by integrating a number of conceptually distinct processing steps in a single runtime execution environment. We describe the need for these different processing steps and describe them in turn (semantic structure, communicative device, qualitative constraints, quantitative constraints, final form presentation), and illustrate our approach by means of an example. We conclude by discussing the models and techniques required for the creation of third generation multimedia content.

In this article we claim that the linguistic-centered view within hypermedia systems needs refinement through a semiotic-based approach before real interoperation between media can be achieved. We discuss the problems of visual signification for images and video in dynamic systems, in which users can access visual material in a non-linear fashion. We describe how semiotics can help overcome such problems, by allowing descriptions of the material on both denotative and connotative levels. Finally we propose an architecture for a dynamic semiotic-aware hypermedia system.

For much the past decade, our research group has focused on developing multimedia authoring tools. We developed a research multimedia format and were involved in developing a multimedia standard. Our lab developed a research prototype of a multimedia editor and then spun off a company that turned it into a commercial editor for this standard. With several research concepts now having reached full maturity in the form of this standard and product, our lab is back where it started: at the beginning of the research idea life cycles, with new visions and hopes for what's next.

The paper describes the advantages of the use of constraint logic programming to articulate transformation rules for multimedia presentation in combination with efficient constraint solving techniques. It demonstrates the need for two different types of constraints. Quantitative constraints are needed to verify whether the final form presentation meets all the numeric constraints that are required by the environment. Qualitative constraints are needed to facilitate high-level reasoning and presentation encoding. While the quantitative constraints can be handled by off-the-shelf constraint solvers, the qualitative constraints needed are specific to the multimedia domain and need to be defined explicitly.

In this article we argue that the automatic generation of dynamic multimedia presentation requires both low-level collections of objective measurements for media units representing prototypical style elements, and high-level conceptual descriptions supporting contextual and presentational requirements. Only the combination of both facilitates the retrieval of adequate material and its user-centered presentation. We discuss the problems of visual signification for images in dynamic systems and explain how a combined approach can help overcome such problems. We then propose an architecture for such a system and present its applicability for a museum-oriented multimedia system with a working example.

In this article we argue that the automatic generation of dynamic multimedia presentation requires both low-level collections of objective measurements for media units representing prototypical style elements, and high-level conceptual descriptions supporting contextual and presentational requirements. Only the combination of both facilitates the retrieval of adequate material and its user-centred presentation. We discuss the problems of visual signification for images in dynamic systems and explain how a combined approach can help overcome such problems. We then propose an architecture for such a system and present its applicability for a museum-oriented multimedia system with a working example.

Also for multimedia, there is a need to create presentations in presentation-independent forms and have the final form(s) generated from the base document. We can input our expertise on the specific requirements for multimedia in automatic document generation. In addition, we have experience in processing multimedia documents in a Web context in general and related issues within SMIL 2.0 in particular. We can explain the use of alternative media for devices where text is inappropriate. In addition, different sorts of meta-data can be incorporated into a presentation-independent base document, and device/user-specific presentations can be generated from this base document, using the information captured in the meta-data. In addition, we can explain the advantages of incorporating timing as one of the dimensions of variation in a presentation. For example, adding temporal variations within HTML, using e.g. HTML+SMIL, can allow different types of trade-offs in a presentation.

Hypermedia structured in terms of the higher-level intent of its author can be adapted to a wider variety of final presentations. Many multimedia systems encode such high-level intent as constraints on either time, spatial layout or navigation. Once specified, these constraints are translated into specific presentations whose timelines, screen displays and navigational structure satisfy these constraints. This ensures that the desired spatial, temporal and navigation properties are maintained no matter how the presentation is adapted to varying circumstances. Rhetorical structure defines author intent at a still higher level. Authoring at this level requires that rhetorics can be translated to final presentations that properly reflect them. This paper explores how rhetorical structure can be translated into constraints, which are then translated into final presentations. This enables authoring in terms of rhetorics and provides the assurance that the rhetorics will remain properly conveyed in all presentation adaptation.

Hypermedia communicative devices are patterns of hypermedia presentation structure with specific affects on the user. Some devices have been determined for affecting the user with abstract authoring constructs, such as rhetorical structure. However, these devices each typically focus on particular aspects of the structure of hypermedia presentation. This paper presents a framework for hypermedia communicative devices that simultaneously affects multiple dimensions of presentation structure. In discussing this, we focus on the document's rhetorical structure as the input to be communicated through these devices.

Activation and deactivation of media items plays a fundamental role in the playing of multimedia and time-based hypermedia presentations. Activation and deactivation information thus has to be captured in an underlying document format. In this paper we show that a number of aspects of activation and deactivation information can be captured using both link structures and events in time-based hypermedia. In particular, we discuss how deactivation and activation can be specified, how the activations and deactivations can be initiated and potential (synchronization) relationships between the elements involved. We first introduce the notions of time-based scheduling and event-based scheduling and then present a short summary of linking. We discuss the similarities between event-based scheduling and linking. We describe a number of aspects of activation and deactivation that can be specified within a document. We then discuss how activation and deactivation information can be recorded in link structures and events.

This essay presents a brief discussion of combining temporal aspects of multimedia presentations with hypertext links. Three ways of combining linking with temporally synchronized components of a presentation are described. We describe work that has been done to incorporate both temporal and linking information within the W3C language SMIL (Synchronized Multimedia Integration Language). We conclude with a discussion of future directions, namely providing support for linking within and among non-linear presentations and the ability to add temporal information to existing XML document languages.

Having the content of a presentation adapt to the needs, resources and prior activities of a user can be an important benefit of electronic documents. While part of this adaptation is related to the encodings of individual data streams, much of the adaptation can/should be guided by the semantics in and among the objects of the presentation. The semantics involved in having hypermedia presentations adapt can be divided between adaptive hypermedia, which adapts autonomously, and adaptable hypermedia, which requires presentationexternal intervention to be adapted. Understanding adaptive and adaptable hypermedia and the differences between them helps in determining the best manner with which to have a particular hypermedia implementation adapt to the varying circumstances of its presentation. The choice of which type of semantics to represent can affect speed of the database management system processing them. This paper reflects on research and implementation approaches toward both adaptive and adaptable hypermedia and how they apply to specifying the semantics involved in hypermedia authoring and processing. We look at adaptive approaches by considering CMIF and SMIL. The adaptable approaches are represented by the SGML-related collection of formats and the Standard Reference Model (SRM) for IPMS are also reviewed. Based on our experience with both adaptive and adaptable hypermedia, we offer recommendations on how each approach can be supported at the data storage level.

This paper presents three user case studies of the multimedia standard SMIL. Each conveys a different kind of typical multimedia. The studies illustrate how SMIL can be used for these forms of multimedia. Analysis of these studies also show potential areas for extension of the language to better suit the needs of Web-based multimedia.

Most hypermedia documents don't incorporate time explicitly. This prevents authors from having direct control over the temporal aspects of a presentation. In this paper we discuss the concept of presentation time - the timing of the individual parts of a presentation and the temporal relations among them. We argue why time is necessary from a presentation perspective, and discuss its relationship with other temporal views of a presentation. We derive the requirements and present our solution for incorporating temporal and linking information in a model of time-based hypermedia.

Making hypermedia adaptable for multiple forms of presentation involves enabling multiple distinct specifications for how a given collection of hypermedia can have its presentation generated. The Standard Reference Model for Intelligent Multimedia Presentation Systems describes how the generation of hypermedia presentation can be divided into distinct but cooperating layers. Earlier work has described how specifications for generating presentations can be divided into distinct modules of code corresponding to these layers. This paper explores how the modules for each layer of a presentation specification can be exchanged for another module encoded for that layer and result in the whole specification remaining well functioning. This capability would facilitate specifying presentation generation by allowing for the use of pre-programmed modules, enabling the author to focus on particular aspects of the presentation generation process. An example implementation of these concepts that uses current and developing Web standards is presented to illustrate how wide-spread modularized presentation generation might be realized in the near future.

SMIL is the W3C recommendation for bringing synchronized multimedia to the Web. Version 1.0 of SMIL was accepted as a recommendation in June. Work is expected to be soon underway for preparing the next version of SMIL, version 2.0. Issues that will need to be addressed in developing version 2.0 include not just adding new features but also establishing SMIL's relationship with various related existing and developing W3C efforts. In this paper we offer some suggestions for how to address these issues. Potential new constructs with additional features for SMIL 2.0 are presented. Other W3C efforts and their potential relationship with SMIL 2.0 are discussed. To provide a context for discussing these issues, this paper explores various approaches for integrating multimedia information with the World Wide Web. It focuses on the modeling issues on the document level and the consequences of the basic differences between text-oriented Web-pages and networked multimedia presentations.

This paper describes the current use of the multimedia standard SMIL. SMIL features that relate to active areas of multimedia research are discussed. SMIL current implementation in existing browsers is described. Examples from the Web of SMIL applications representing different types of multimedia are presented. These discussions together provide an overview of how SMIL currently addresses the needs of multimedia distributed on the Web.

In order to facilitate adaptability of hypermedia documents a distinction is often made between the underlying conceptual structure of a document and the structure of its presentation. This distinction enables greater variety in how a presentation can be adapted to best convey these underlying concepts in a given situation. What is often confusing for those applying this distinction is that although both levels of structure often share similar components: transformation form the storage of a document to its presentation sometimes occurs directly between these similar components and sometimes does not. These similarities typically fall in the categories of space, time and relationships between document portions. This paper identifies some primary similarities between the structure of hypermedia storage and presentation. It also explores how the transformation from storage to presentation often does not follow these similarities. This discussion is illustrated with the Fiets hypermedia application, which addresses the issues of storage, presentation and transformation using public domain formats and tools. The intention is to help authors who separate storage from presentation to better understand this distinction.

This paper discusses the implementation of adaptability in environments that are based on the Standard Reference Model for Intelligent Multimedia Presentation Systems. This adaptability is explored in the context of style sheets, which are represented in such formats as DSSSL. The use of existing public standards and tools for this implementation of style sheet-based adaptability is described. The Berlage environment is presented, which integrates these standards and tools into a complete storage-to-presentation hypermedia environment. The integration of the SRM into the Berlage environment is introduced in this work. This integration illustrates the issues involved in implementing adaptability in the model.

The content and structure of an electronically published document can be authored and processed in ways that allow for flexibility in presentation on different environments for different users. This enables authors to craft documents that are more widely presentable. Electronic publishing issues that arise from this separation of document storage from presentation include (1) respecting the intent and restrictions of the author and publisher in the document's presentation, and (2) applying costs to individual document components and allowing the user to choose among alternatives to control the price of the document's presentation. These costs apply not only to the individual media components displayed but also to the structure created by document authors to bring these media components together as multimedia. A collection of ISO standards, primarily SGML, HyTime and DSSSL, enable the representation of presentation-independent documents and the creation of environments that process them for presentation. SMIL is a W3C format under development for hypermedia documents distributed on the World Wide Web. Since SMIL is SGML-compliant, it can easily be incorporated into SGML/HyTime and DSSSL environments. This paper discusses how to address these issues in the context of a presentation-independent hypermedia storage. It introduces the Berlage environment, which uses SGML, HyTime, DSSSL and SMIL to store, process, and present hypermedia data. This paper also describes how the Berlage environment can be used to enforce publisher restrictions on media content and to allow users to control the pricing of document presentations. Also explored is the ability of both SMIL and HyTime to address these issues in general, enabling SMIL and HyTime systems to consistently process documents of different document models authored in different environments.

Hypermedia documents are most often created with a particular presentation environment in mind. This requires the authoring of one document per presentation platform. As pointed out in [], much implementation effort can be avoided by specifying how the same underlying document can be presented in different environments. A style sheet defines a mapping from a source document to a presentation for it. We discuss the existing use of style sheets as applied to text and discuss their application to the case of hypermedia, and in particular how they need to be extended.

Being able to author a hypermedia document once for presentation under a wide variety of potential circumstances requires that it be stored in a manner that is adaptable to these circumstances. Since the nature of these circumstances is not always known at authoring time, specifying how a document adapts to them must be a process that can be performed separately from its original authoring. These distinctions include the porting of the document to different platforms and formats and the adapting of the document s presentation to suit the needs of the user and of the current state of the presentation environment. In this paper we discuss extensions to our CMIF hypermedia authoring and presentation environment that provide adaptability through this distinction between authoring and presentation specification. This extension includes the use of HyTime for document representation and of DSSSL for presentation specification. We also discuss the Berlage architecture, our extension to HyTime that specifies the encoding of certain hypermedia concepts useful for presentation specification.

We consider the generic hypermedia structure of a document to be a means of representing the document that allows it to be processed into a wide variety of presentations. Representing a document in this manner requires additional specification and resources to render it into any presentation. In this paper we discuss the relationship between the generic hypermedia structure of documents and the processing of this structure into presentation. Our discussion is expressed in terms of existing models for hypertext and hypermedia systems and also in terms of ISO standards for text and hypermedia document formatting and processing. The discussion and the resulting formalisms and the resulting formalisms are illustrated with extension designs for the hypermedia authoring and presentation environment developed at our laboratory.

The standards MHEG-5 and HyTime are interchange formats for hypermedia information. While they may seem to compete, they actually play separate and complementary roles in a complete and open hypermedia environment. MHEG-5 is used for portable final-form hypermedia presentations. HyTime is used for the long-term, presentation-independent storage of hypermedia documents. Given these tasks, MHEG-5 can be used to encode presentations of HyTime documents. This paper explores these two standards, the cooperative roles they play and their application to the CMIF hypermedia environment architecture. The issues discussed include the semantic overlap between the hypermedia models each standard represents and how the use of each standard affects the cooperative use of the other.

In this contribution we consider the design of a hypermedia information system which not only includes standard functionality of storage and presentation, but also the automatic generation of hypermedia presentations on the basis of a domain-dependent knowledge base. We identify and describe the data sources required and the processes involved.

The playing of computer-based business games has changed little with recent technological improvements. Teams make decisions which are written down on paper and handed to a tutor who then has to enter these into the computer. To make it a richer and more real-life experience we have brought a business simulation into a multimedia environment. In order to create a sense of the passing of time the simulation is continuous, and players can enter input values throughout the game. Important parameters, such as product sales, are displayed as continuously updated graphs. Background information such as news items is provided in multimedia form. Players also have direct access to online tools, such as spreadsheets, from the game environment. We describe the prototype built to illustrate our approach.

The integration of audio, video, graphics and text on the desktop promises to fundamentally challenge the centuries-old model of the printed document as the basis for information exchange. Before this potential can be realized, however, systems must be devised that enable the production and presentation of complex, inter-related media objects. These systems are generically called multimedia authoring tools. In this article, we consider the development of multimedia authoring tools, examine the current state of the art, and then discuss a set of research challenges that need to be addressed before the full potential of multimedia output technology can be effectively utilized to share information.

We give a short description of the Amsterdam Hypermedia Model followed by examples of its use in a number of existing and planned applications. The main application to date has been as a basis of the multimedia authoring system, CMIFed, along with its ability to specify trade-offs for resource use. We discuss the models potential for generating differing document formats, followed by future work on using it as a goal format for generating multimedia documents.

From Computing Reviews, by Jeanine Meyer The purpose of this paper is to convince the reader of the need for a general model for hypermedia and present the Amsterdam hypermedia model (AHM) as fulfilling that need. Hypertext, which is described by the authors as a 'relatively mature discipline,' has the Dexter model, but the authors show that enhancing that model for hypermedia is not a straightforward task. In particular, it requires attention to issues of synchronization. The AHM model includes attention to timing and composite objects as well as implementation issues such as channels and having the sources of components residing over a distributed system. The paper features one example and also describes an authoring and presentation environment called CMIFed. It is generally well written. The paper can be understood even if one has not studied the Dexter hypertext reference model or the CMIF multimedia document model and, in fact, this paper could serve as an introduction to the issues involved. Too much of the focus, however, is on other systems and not on what AHM actually is. The authors do not demonstrate the model by using it to express the featured example. Moreover, to really merit the term 'model,' AHM should be shown as serving a substantial role in describing and implementing applications in terms of two or more distinct authoring or runtime environments. This is not done, though it appears well within the experience and understanding of the authors.

Although there are many hypertext systems currently on the market there is little advice available on how authors can create easy-to-use hypertexts. This paper addresses the usability of Glasgow Online - a hypertext which contains tourist information on the city of Glasgow. The issues raised in the paper are based on an observational study in which a number of readers performed tasks representative of those undertaken by tourists visiting Glasgow. The usability issues raised by Glasgow Online are discussed in a wider context and their applicability to other styles of hypertext is considered.