INS2 Publications

ImageCLEF's Wikipedia Retrieval task provides a testbed for the system-oriented evaluation of multimedia information retrieval from a collection of Wikipedia images. The aim is to investigate retrieval approaches in the context of a large and heterogeneous collection of images (similar to those encountered on the Web) that are searched for by users with diverse information needs. This paper presents an overview of the resources, topics, and assessments of the Wikipedia Retrieval task at ImageCLEF 2010, summarizes the retrieval approaches employed by the participating groups, and provides an analysis of the main evaluation results.

The wikipedia image retrieval task at ImageCLEF provides a testbed for the system-oriented evaluation of visual information retrieval from a collection of Wikipedia images. The aim is to investigate the effectiveness of retrieval approaches that exploit textual and visual evidence in the context of a large and heterogeneous collection of images that are searched for by users with diverse information needs. This chapter presents an overview of the available test collections, summarises the retrieval approaches employed by the groups that participated in the task during the 2008 and 2009 ImageCLEF campaigns, provides an analysis of the main evaluation results, identifies best practices for effective retrieval, and discusses open issues.

Automatic image annotation using supervised learning is performed by concept classifiers trained on labelled example images. This work proposes the use of clickthrough data collected from search logs as a source for the automatic generation of concept training data, thus avoiding the expensive manual annotation effort. We investigate and evaluate this approach using a collection of 97,628 photographic images. The results indicate that the contribution of search log based training data is positive despite their inherent noise; in particular, the combination of manual and automatically generated training data outperforms the use of manual data alone. It is therefore possible to use clickthrough data to perform large-scale image annotation with little manual annotation effort or, depending on performance, using only the automatically generated training data. An extensive presentation of the experimental results and the accompanying data can be accessed at http://olympus.ee.auth.gr/~diou/civr2009/.

Improving the user's interaction with a video retrieval system requires to examine the search behavior of real users. We present in this article a study comparing the video search behavior between professional archivists and novice users. The comparison focuses on the use and effectiveness of different state-of-the-art video search methods offered by our retrieval system, and the result investigation behavior of the two user groups. We conducted our experiments in the context of TRECVID's 2009 interactive search task, using the provided collection and topics for our evaluation. The findings are based on a qualitative questionnaire analysis and a quantitative examination of the logged user actions on the search interface. The experimental results indicate that today's visual search techniques have improved in effectiveness, confirming a trend found in previous user studies. To our surprise, professional archivists used visual concept search in many of their searches. Queries containing visual concepts were more effective, resulting in more relevant shots found than the alternative methods. Overall, we conclude that professional archivists are more focused on recall in carrying out their search tasks, and are better at reflecting on their own search performance.

ImageCLEF's wikipediaMM task provides a testbed for the system-oriented evaluation of multimedia information retrieval from a collection of Wikipedia images. The aim is to investigate retrieval approaches in the context of a large and heterogeneous collection of images (similar to those encountered on the Web) that are searched for by users with diverse information needs. This paper presents an overview of the resources, topics, and assessments of the wikipediaMM task at ImageCLEF 2009, summarises the retrieval approaches employed by the participating groups, and provides an analysis of the main evaluation results.

We present a method that exploits `linked data' to determine semantic relations between consecutive user queries. Our method maps queries onto concepts in linked data and searches the linked data graph for direct or indirect relations between the concepts. By comparing relations between large numbers of user queries, we identify semantic modification patterns. The application of this method to the logs of an image search engine revealed interesting usage patterns, such as that users often search for two entities sharing a property (e.g., two players from the same team). These patterns can be used to generate query suggestions. Results of preliminary experiments show that the patterns enable us to generate suggestions for more queries than a method purely based on search-log statistics.

Previous research has studied query modifications on a syntactic level by focusing on the addition, elimination and substitution of terms between consecutive queries that have at least one term in common. In this paper, we determine semantic relations between queries by first mapping them onto concepts in linked data sources and then identifying the relations between the concepts. This enables us to find relations between queries that do not share any terms. Moreover, with this approach we can find more detailed and more meaningful query modification patterns than with a term-based analysis. Application of our method to search logs of two search engines shows the importance of studying query modifications on a semantic level. Our results indicate that users often search for entities that are related semantically, but not syntactically. Specifically, users often successively search for two entities sharing a common property, such as two actors starring in the same movie, or two entities with a specific relation, such as spouses. We discuss the implications of these findings for the design of search engines.

Comparison search is an information seeking task where a user examines individual items or sets of items for similarities and differences. While this is a known information need among experts and knowledge workers, appropriate tools are not available. In this paper, we discuss comparison search in the cultural heritage domain, a domain characterized by large, rich and heterogeneous data sets, where different organizations deploy different schemata and terminologies to describe their artifacts. This diversity makes meaningful comparison difficult. We developed a thesaurus-based comparison search application called LISA, a tool that allows a user to search, select and compare sets of artifacts. Different visualizations allow users to use different comparison strategies to cope with the underlying heterogeneous data and the complexity of the search tasks. We conducted two user studies. A preliminary study identifies the problems experts face while performing comparison search tasks. A second user study examines the effectiveness of LISA in helping to solve comparison search tasks. The main contribution of this paper is to establish design guidelines for the data and interface of a comparison search application. Moreover, we offer insights into when thesauri and metadata are appropriate for use in such applications.

Traditionally the relations between concepts from a controlled vocabulary, such as the hierarchical and associative relations in a thesaurus, have been used to support users in their search process. In the context of the Semantic Web, multiple interlinked vocabularies are becoming available, providing a large number of different relations between concepts. However, for a specific search task, only a small fraction of these will be meaningful to the user, and currently we have little understanding of which methods can be used to determine this. In this paper, we describe a case study in the cultural heritage domain that investigates support for the specific task of finding artworks in a data set of multiple linked art collections and vocabularies. In a first experiment a number of use cases from domain experts are collected and the paths in the data graph by which artworks can be found are analysed. A number of different types of paths are identified and their usefulness is qualitatively evaluated. In a second experiment we explore how the different path types can be used in a semantic search algorithm to support the intended search behavior indicated by the experts. We conclude that effective end-user support requires a highly interactive application in which the user can explore multiple search strategies. Based on our findings we discuss the implications on the design of such an interactive search application.

This text describes the ImageCLEF benchmark for multilingual, multimodal image annotation and retrieval. First, the general field of multimedia retrieval evaluation and the situation of ImageCLEF in this field are explained. Then, the ImageCLEF 2009 tasks, their objectives and the participation in these tasks are described. As of 2010, the format of CLEF (Cross Language Evaluation Forum) and ImageCLEF is changing; these changes are presented in detail to motivate readers of the IAPR Newsletter to participate in CLEF and ImageCLEF 2010.

Social media systems have encouraged end user participation in the Internet, for the purpose of storing and distributing Internet content, sharing opinions and maintaining relationships. Collaborative tagging allows users to annotate the resulting user-generated content, and enables effective retrieval of otherwise uncategorised data. However, compared to professional web content production, collaborative tagging systems face the challenge that end-users assign tags in an uncontrolled manner, resulting in unsystematic and inconsistent metadata. This paper introduces a framework for the personalization of social media systems. We pinpoint three tasks that would benefit from personalization: collaborative tagging, collaborative browsing and collaborative search. We propose a ranking model for each task that integrates the individual user’s tagging history in the recommendation of tags and content, to align its suggestions to the individual user preferences. We demonstrate on two real data sets that for all three tasks, the personalized ranking should take into account both the user’s own preference and the opinion of others.

People conventionally refer to an action or occurrence taking place at a certain time at a specific location as an event. This notion is potentially useful for connecting individual facts recorded in the rapidly growing collection of linked data sets and for discovering more complex relationships between data. In this paper, we provide an overview and comparison of existing event models, looking at the different choices they make of how to represent events. We describe a model for publishing records of events as Linked Data. We present tools for populating this model and a prototype "event directory" web service, which can be used to locate stable URIs for events that have occurred, provide RDFS+OWL descriptions and link to related resources.

We performed a user experiment in which museum professionals used vocabularies from the Web for annotating the subject matter of museum objects. We study the requirements on the underlying RDF dataset, search algorithms and user interface design in a real world setting. We identify the advantages of reusing vocabularies from the Web and discuss how and to what extent the disadvantages can be overcome. The study is performed at the Print Room of the Rijksmuseum Amsterdam, where currently a large collection of prints, photographs and drawings is being catalogued. We report on the analysis of the current practice of professional cataloguers, the iterative design of an annotation tool and a qualitative evaluation of this tool with a user experiment in a realistic annotation environment. We discuss our findings in terms of their impact on the RDF data, the semantic search functionality and the user interface.

This PhD project aims at understanding and supporting the complex activities of information gathering. To date, most search applications support one aspect of search namely low-level keyword-based search to find documents. However, in reality, users search tasks are often high-level search tasks, such as comparing differences between art objects. Thus, there is a mismatch between the users search needs and the available search tools. This research investigates information gathering characteristics and how alternative search interfaces could support them. Design recommendations and guidelines for interfaces to support information gathering will be designed and verified through lab experiments and evaluation in different domains.

Comparison search is an information seeking task where a user examines individual items or sets of items for similarities and differences. While this is a known information need among experts and knowledge workers, appropriate tools are not available. In this paper, we discuss comparison search in the cultural heritage domain, a domain characterized by large, rich and heterogeneous data sets, where different organizations deploy different schemata and terminologies to describe their artifacts. This diversity makes meaningful comparison difficult. We developed a thesaurus-based comparison search application called LISA, a tool that allows a user to search, select and compare sets of artifacts. Different visualizations allow users to use different comparison strategies to cope with the underlying heterogeneous data and the complexity of the search tasks. We conducted two user studies. A preliminary study identifies the problems experts face while performing comparison search tasks. A second user study examines the effectiveness of LISA in helping to solve comparison search tasks. The main contribution of this paper is to establish design guidelines for the data and interface of a comparison search application. Moreover, we offer insights into when thesauri and metadata are appropriate for use in such applications.

Identifying alignments between vocabularies has become a central knowledge engineering activity. A plethora of alignment techniques has been developed over the past years. In this paper we present a case study in which we examine and evaluate the practical use of three typical alignment techniques. The study involves the alignment of two vocabularies used in a semantic-search engine for cultural-heritage objects. We show that a sequence can be beneficial. The case study give s insight into evaluation issues, such as techniques for identification of false positives. We see this work as a step to a badly-needed methodology for alignment.

ImageCLEF's wikipediaMM task provides a testbed for the system-oriented evaluation of multimedia information retrieval from a collection of Wikipedia images. The aim is to investigate retrieval approaches in the context of a large and heterogeneous collection of images (similar to those encountered on the Web) that are searched for by users with diverse information needs. This paper presents an overview of the resources, topics, and assessments of the wikipediaMM task at ImageCLEF 2009, summarises the retrieval approaches employed by the participating groups, and provides a first analysis of the main evaluation results.

CWI's experiments investigate the usefulness of clickthrough data for improving the diversity of image retrieval results. We use the search logs provided to us by Belga to find relevant images; we consider that these correspond to images clicked for queries exactly matching or best matching a topic's title and cluster titles. To reduce the noise, we also filter these results and only consider those clicked images that are also retrieved by a text-based approach that uses the image captions. To promote diversity, we interleave the images retrieved in the previous step for each of the cluster titles (and also the title). However, given that the clickthrough data available to us cover only a small part of the collection used in the photo retrieval task, our experimental results are inconclusive, although they do provide indications on the reliability of using image search clickthrough data to identify relevant images.

We examine the reliability of clickthrough data as concept-based image annotations, by comparing them against manual annotations, for different concept categories. Our analysis shows that, for many concepts, the image annotations generated by using clickthrough data are reliable, with up to 90% of true positives in the automatically annotated images compared to the manual ground truth. Concept categories, though, do not provide additional evidence about the types of concepts for which clickthrough-based image annotation performs well.

A mismatch between different event spaces has been used to argue against rank equivalence of classic probabilistic models of information retrieval and language models. We question the effectiveness of this strategy and we argue that a convincing solution should be sought in a correct procedure to design adequate priors for probabilistic reasoning. Acknowledging our solution of the event space issue invites to rethink the relation between probabilistic models, statistics and logic in the context of IR.

Clustering methods cluster objects on the basis of a similarity measure between the objects. In clustering tasks where the objects come from more than one collection often part of the similarity results from features that are related to the collections rather than features that are relevant for the clustering task. For example, when clustering pages from various web sites by topic, pages from the same web site often contain similar terms. The collection-related part of the similarity hinders clustering as it causes the creation of clusters that correspond to collections instead of topics. In this paper we present two methods to restrict clustering to the part of the similarity that is not associated with membership of a collection. Both methods can be used on top of standard clustering methods. Experiments on data sets with objects from multiple collections show that our methods result in better clusters than methods that do not take collection information into account.

People conventionally refer to an action or occurrence taking place at a certain time at a specific location as an event. This notion is potentially useful for connecting individual facts recorded in the rapidly growing collection of linked data sets and for discovering more complex relationships between data. In this paper, we provide an overview and comparison of existing RDFS+OWL event models, looking at the different choices they make of how to represent events. We describe a recommended model for publishing records of events as Linked Data. We present tools for populating this model and a prototype of an "event directory" web service, which can be used to locate stable URIs for events that have occurred and to provide RDFS+OWL descriptions of them and links to related resources.

Automatic image annotation using supervised learning is performed by concept classifiers trained on labelled example images. This work proposes the use of clickthrough data collected from search logs as a source for the automatic generation of concept training data, thus avoiding the expensive manual annotation effort. We investigate and evaluate this approach using a collection of 97,628 photographic images. The results indicate that the contribution of search log based training data is positive; in particular, the combination of manual and automatically generated training data outperforms the use of manual data alone. It is therefore possible to use clickthrough data to perform large-scale image annotation with little manual annotation effort or, depending on performance, using only the automatically generated training data. The datasets used as well as an extensive presentation of the experimental results can be accessed at http://olympus.ee.auth.gr/~diou/civr2009/.

Embodied social agents are programmed to display human-like social behaviour to increase intuitiveness of interacting with these agents. It is not yet clear to what extent people respond to agents’ social behaviours. One example is touch. Despite robots’ embodiment and increasing autonomy, the effect of communicative touch has been a mostly overlooked aspect of human-robot interaction. This video-based, 2x2 betweensubject survey experiment (N=119) found that the combination of touch and proactivity influenced whether people saw the robot as machine-like and dependable. Participants’ attitude towards robots in general also influenced perceived closeness between humans and robots. Results show that communicative touch is considered a more appropriate behaviour for proactive agents rather than reactive agents. Also, people that are generally more positive towards robots find robots that interact by touch less machine-like. These effects illustrate that careful consideration is necessary when incorporating social behaviours in agents’ physical interaction design.

Even though robots' physical embodiment makes it likely humans will come into physical contact with robots, the effects of touch on attitudes in human-robot interaction are still relatively unknown. This survey and video-based, experiment (N=199) investigates the effects of touch and robots interactions. Results show that physical contact and autonomous behavior interact in their effects on perceived machine-likeness and dependability. Attitudes towards robots in general also affected the influence of touch on perceptions of a robot.

StreetTiVo is a project that aims at bringing research results into the living room; in particular, a mix of current results in the areas of Peer-to-Peer XML Database Management System (P2P XDBMS), advanced multimedia analysis techniques, and advanced information retrieval techniques. The project develops a plug-in application for the so-called Home Theatre PCs, such as set-top boxes with MythTV or Windows Media Center Edition installed, that can be considered as programmable digital video recorders. StreetTiVo distributes compute-intensive multimedia analysis tasks over multiple peers (i.e., StreetTiVo users) that have recorded the same TV program, such that a user can search in the content of a recorded TV program shortly after its broadcasting; i.e., it enables near real-time availability of the meta-data (e.g., speech recognition) required for searching the recorded content. StreetTiVo relies on our P2P XDBMS technology, which in turn is based on a DHT overlay network, for distributed collaborator discovery, work coordination and meta-data exchange in a volatile WAN environment. The technologies of video analysis and information retrieval are seamlessly integrated into the system as XQuery functions.

The Web of data enables fragments of information to be identified, described and connected together in a rich information environment. Users requiring information are faced with the problem of finding out what information is available, and obtaining sufficient fragments to successfully carry out their task. Systems supporting these tasks can use the fragments, descriptions of them and relationships among them, to improve both the selection and presentation of the information. Questions to be answered are: which information needs can be better supported, and how can the Web of data help. While the construction of the linked data cloud'' is necessary to even start thinking about providing this type of support for users, our claim is that we first need to establish the user's information needs before establishing the potential roles the linked data can play in information selection and presentation. In this paper, we discuss potential uses of linked data to support users' information needs, give examples of using linked data to support user information seeking tasks and highlight future research directions.

We demonstrate how to develop Web-based user interfaces for Semantic Web applications using commonly available, off-the-shelf Web widget libraries. By formally defining the underlying data model that is assumed by these widgets, Semantic Web application developers can use familiar RDF constructs to map their own data to the model implemented by the Widgets. As an example, we briefly describe the interface model underlying our own framework, and provide concrete examples showing how it has been used to create Semantic Web applications in two different domains. We conclude by discussing the advantages and limitations of our approach.

We describe two user studies that investigate organization strategies of autocompletion in a known-item search task: searching for terms taken from a thesaurus. In Study 1, we explored ways of grouping term suggestions from two different thesauri (TGN and WordNet) and found that different thesauri may require different organization strategies. Users found Group organization more appropriate to organize location names from TGN, while Alphabetical works better for WordNet. In Study 2, we compared three different organization strategies (Alphabetical, Group and Composite) for location name search tasks. The results indicate that for TGN autocompletion interfaces help improve the quality of keywords, Group and Composite organization help users search faster, and is perceived easier to understand and to use than Alphabetical.

The wikipediaMM task provides a testbed for the system-oriented evaluation of ad-hoc retrieval from a large collection of Wikipedia images. It became a part of the ImageCLEF evaluation campaign in 2008 with the aim of investigating the use of visual and textual sources in combination for improving the retrieval performance. This paper presents an overview of the task¿s resources, topics, assessments, participants' approaches, and main results.

While many multimedia systems allow the association of semantic annotations with media assets, there is no agreed-upon way of sharing these among systems. As an initial step within the multimedia community, we identify a small number of fundamental processes of media production, which we term canonical processes. We specify their inputs and outputs, but deliberately do not specify their inner workings, concentrating rather on the information flow between them. We thus identify a small set of building blocks that can be supported in semantically aware media production tools. The processes are identified in conjunction with a number of different research groups within the community who supply, in the companion papers, descriptions of existing systems and a mapping to them. We give a basic formalisation of the processes and discuss how this fits with other formalisation endeavours. We present a number of frequently asked questions during the development of the model and this special issue.

In this article we describe a Semantic Web application for semantic annotation and search in large virtual collections of cultural-heritage objects, indexed with multiple vocabularies. During the annotation phase we harvest, enrich and align collection metadata and vocabularies. The semantic-search facilities support keyword-based queries of the graph (currently 20 M triples), resulting in semantically grouped result clusters, all representing potential semantic matches of the original query. We show two sample search scenario’s. The annotation and search software is open source and is already being used by third parties. All software is based on established Web standards, in particular HTML/XML, CSS, RDF/OWL, SPARQL and JavaScript.

This article presents the CHIP demonstrator1 for providing personalized access to digital museum collections. It consists of three main components: Art Recommender, Tour Wizard, and Mobile Tour Guide. Based on the semantically enriched Rijksmuseum Amsterdam2 collection, we show how Semantic Web technologies can be deployed to (partially) solve three important challenges for recommender systems applied in an open Web context: (1) to deal with the complexity of various types of relationships for recommendation inferencing, where we take a content-based approach to recommend both artworks and art-history topics; (2) to cope with the typical user modeling problems, such as cold-start for first-time users, sparsity in terms of user ratings, and the efficiency of user feedback collection; and (3) to support the presentation of recommendations by combining different views like a historical timeline, museum map and faceted browser. Following a user-centered design cycle, we have performed two evaluations with users to test the effectiveness of the recommendation strategy and to compare the different ways for building an optimal user profile for efficient recommendations. The CHIP demonstrator received the Semantic Web Challenge Award (third prize) in 2007, Busan, Korea.

Using opportunistic software development principles in computer engineering education encourages students to be creative and to develop solutions that cross the boundaries of diverse technologies. A framework for opportunistic software development education helps to create a space in which students can combine systems that were never meant to work together or even to be reused, and thus produce innovative ideas and solutions. A case study involving students in a course on intelligent human-computer interaction design demonstrates the approach, and the authors discuss some lessons learned.

In cultural heritage, large virtual collections are coming into existence. Such collections contain heterogeneous sets of metadata and vocabulary concepts, originating from multiple sources. In the context of the E-Culture demonstrator we have shown earlier that such virtual collections can be effectively explored with keyword search and semantic clustering. In this paper we describe the design rationale of ClioPatria, an open-source system which provides APIs for scalable semantic graph search. The use of ClioPatria’s search strategies is illustrated with a realistic use case: searching for ”Picasso”. We discuss details of scalable graph search, the required OWL reasoning functionalities and show why SPARQL queries are insufficient for solving the search problem.

In this paper, we show how spreadsheets, an end-user development paradigm proven to be highly productive and simple to learn and use, can be used for complex service compositions. We identify the requirements for spreadsheet-based service composition, and present our framework that implements these requirements. Our framework enables spreadsheets to send requests and retrieve results from various local and remote services. We show how our tools support different composition patterns, and how the style of declarative dependencies of spreadsheets can facilitate service composition. We also discuss novel issues identified by using the framework in several projects and education.

Multimedia constitutes an interesting field of application for Semantic Web and Semantic Web reasoning, as the access and management of multimedia content and context depends strongly on the semantic descriptions of both. At the same time, multimedia resources constitute complex objects, the descriptions of which are involved and require the foundation on sound modeling practice in order to represent findings of low- and high level multimedia analysis and to make them accessible via Semantic Web querying of resources. This tutorial aims to provide a red thread through these different issues and to give an outline of where Semantic Web modeling and reasoning needs to further contribute to the area of semantic multimedia for the fruitful interaction between these two fields of computer science.

n this paper we present the CHIP demonstrator aimed at helping users to explore the Rijksmuseum Amsterdam collection both online and inside the museum. Cultural heritage data from various external sources is integrated to provide an enriched semantic knowledge structure. The resulting RDF/OWL graph is the basis for CHIP main functionality for recommendations, search and personalized interaction.

This paper discusses accuracy in processing ratings of and recommendations for item features. Such processing facilitates featurebased user navigation in recommender system interfaces. Item features, often in the form of tags, categories or meta-data, are becoming important hypertext components of recommender interfaces. Recommending features would help unfamiliar users navigate in such environments. This work explores techniques for improving feature recommendation accuracy. Conversely, it also examines possibilities for processing user ratings of features to improve recommendation of both features and items. This work’s illustrative implementation is a web portal for a museum collection that lets users browse, rate and receive recommendations for both artworks and interrelated topics about them. Accuracy measurements compare proposed techniques for processing feature ratings and recommending features. Resulting techniques recommend features with relative accuracy. Analysis indicates that processing ratings of either features or items does not improve accuracy of recommending the other.

We report on our user study on the information seeking behavior of cultural heritage experts and the sources they use to carry out search tasks. Seventeen experts from nine cultural heritage institutes in the Netherlands were interviewed and asked to answer questionnaires about their daily search activities. The interviews helped us to better understand their search motivations, types, sources and tools. A key finding of our study is that the majority of search tasks involve relatively complex information gathering. This is in contrast to the relatively simple fact-finding oriented support provided by current tools. We describe a number of strategies that experts have developed to overcome the inadequacies of their tools. Finally, based on the analysis, we derive general trends of cultural heritage experts’ information seeking needs and discuss our preliminary experiences with potential solutions.

Web 2.0 enables increased access to the museum digital collection. More and more, users will spend time preparing their visits to the museums and reflecting on them after the visits. In this context, the CHIP (Cultural Heritage Information Personalization) project offers tools to the users to be their own curator, e.g. planning a personalized museum tour, discovering interesting artworks they want to see in a 'virtual' or a 'real' tour and quickly finding their ways in the museum. In this paper we present the new additions to the CHIP tools, which target the above functionality - a Web-based Tour Preparation Wizard and an export of a personalized tour to an interactive Mobile Guide used in the physical museum space. In addition, the user interactions during a real museum visit are stored and synchronized with the user model, which is maintained at the museum Web site.

This position paper discusses our experience in evaluating our cultural search and annotation engine. We identify three aspects that determine the quality of a semantic web application as a whole, namely: the quality of data set, the quality of underlying search and inference software and the quality of the user interface. We argue that evaluation of semantic web applications is particularly difficult because of the strong interdependency between the three aspects.

State of the art web search applications allow the user to aggregate information from many sources. Because of this, users are confronted with having to assess the reliability of information from different sources. This paper reports on an empirical user study on the effect of displaying credibility ratings of multiple cultural heritage sources (e.g. museum websites, art blogs) on users' search performance and selection. The study investigated whether source credibility has an influence on users' search performance when they are confronted with only a few information sources or where there are many. The results of our online interactive study (n=122) show that by presenting the source credibility information explicitly, people's confidence in their selection of information significantly increases, even though it does not necessarily make search more time efficient. Additionally, we highlight credibility issues that are applicable beyond the cultural heritage domain, such as issues related to credibility measures and choice of visualization.

Perceptions of a system’s competence influence acceptance of that system [31]. Ideally, users’ perception of competence matches the actual competence of a system. This paper investigates the relation between actual and perceived competence of transparent Semantic Web recommender systems that explain recommendations in terms of shared item concepts. We report an experiment comparing non-transparent and transparent versions of a content-based recommender. Results indicate that in the transparent condition, perceived competence and actual competence (in specific recall) were related, while in the non-transparent condition they were not. Providing insight in what aspects of items triggered their recommendation, by showing the concepts that were the basis for a recommendation, gave users a better assessment of how well the system worked.

The Sapir-Whorf Hypothesis postulates a link between thought and language: if you haven't got a word for a concept, you can't think about it; if you don't think about it, you won't invent a word for it. The term "Web 2.0" is a case in point. It was invented by a book publisher as a term to build a series of conferences around, and conceptualises the idea of Web sites that gain value by their users adding data to them. But the concept existed before the term: Ebay was already Web 2.0 in the era of Web 1.0. But now we have the term we can talk about it, and it becomes a structure in our minds, and in this case a movement has built up around it.

In this paper we describe a model for automatically generating video documentaries. This allows viewers to specify the subject and the point of view of the documentary to be generated. The domain is matter-of-opinion documentaries based on interviews. The model combines rhetorical presentation patterns used by documentary makers with a data-driven approach. Rhetorical presentation patterns provide the viewer with an engaging viewing experience, while a data-driven approach can be applied to growing media repositories. To date, the modeling of rhetoric has been achieved in a top-down manner using closed repositories, while data-driven generation approaches were unable to implement non-trivial rhetorical presentation patterns. We describe an implementation of our model in a system, Vox Populi, and apply it to an online documentary shot by a group of independent amateur documentarists.

This paper presents an architecture for non-intrusive user interfaces in the interactive digital TV domain. The architecture is based on two concepts. First, the deployment of non-monolithic rendering for content consumption, which allows micro-level personalization of content delivery by utilizing different rendering components (e.g., sending video to the TV screen and extra information to a handheld device). Second, the definition of actions descriptions for user interaction, so that high-level user interaction intentions can be partitioned across a personalized collection of control components (e.g., handheld device). This paper introduces an over-all architecture to support micro-personalization and describes an implementation scenario developed to validate the architecture.

In this paper we present an approach for personalized access to museum collections. We use a RDF/OWL specification of the Rijksmuseum Amsterdam collections as a driver for an interactive dialog. The user gives his/her judgment on the artefacts, indicating likes or dislikes. The elicited user model is further used for generating recommendations of artefacts and topics. In this way we support exploration and discovery of information in museum collections. A user study provided insights in characteristics of our target user group, and showed how novice and expert users employ their background knowledge and implicit interest in order to elicit their art preference in the museum collections.

This paper describes ongoing work exploring aspects of personalized access to and presentation of virtual museum collections. The project demonstrator illustrates an interactive approach to collecting data about museum visitors in terms of their interests in and preferences about artefacts from the Rijksmuseum collection. This data is stored in user profiles used further to recommend routes through the museum and to guide the users towards artefacts related to their interests and preferences. The overall goal of the project is to explore different users' characteristics and personalize users' museum experiences within the Rijksmuseum virtual and physical collections.

We have developed a set of small interactive throw pillows containing intelligent touch-sensing surfaces, in order to explore new ways to model the environment, participants, artefacts, and their interactions, in the context of expressive non-verbal interaction. We present the overall architecture of the environment, describing a model of the user, the interface (the interactive pillows and the devices it can interact with) and the context engine. We describe the representation and process modules of the context engine and demonstrate how they support real-time adaptation. We present an evaluation of the current prototype and conclude with plans for future work.

User-adaptive information filters can be a tool to achieve timely delivery of the right information to the right person, a feat critical in crisis management. This paper explores interaction issues that need to be taken into account when designing a user-adaptive information filter. Two case studies are used to illustrate which factors affect trust and acceptance in user-adaptive filters as a starting point for further research. The first study deals with user interaction with user-adaptive spam filters. The second study explores the user experience of an art recommender system, focusing on transparency. It appears that while participants appreciate filter functionality, they do not accept fully automated filtering. Transparency appears to be a promising way to increase trust and acceptance, but its successful implementation is challenging. Additional observations indicate that careful design of training mechanisms and the interface will be crucial in successful filter implementation.

In previous work, we proposed a unified approach for describing multimodal human-computer interaction and interaction constraints in terms of sensual, motor, perceptual and cognitive functions of users. In this paper, we extend this work by providing formalised vocabularies that express human functionalities and anatomical structures required by specific modalities. The central theme of our approach is to connect these modality representations with descriptions of user, device and environmental constraints that influence the interaction. These descriptions can then be used in a reasoning framework that will exploit formal connections among interaction modalities and constraints. The focus of this paper is on specifying a comprehensive vocabulary of necessary concepts. Within the context of an interaction framework, we describe a number of examples that use this formalised knowledge.

A Web browser provides a uniform user interface to different types of information. Making this interface universally accessible and more interactive is a long term goal still far from being achieved. Universally accessible browsers require novel interaction modalities and additional functionalities, for which existing browsers tend to provide only partial solutions. Although functionality for Web accessibility can be found as open source and free software components, their reuse and integration is complex because they were developed in diverse implementation environments, following standards and conventions incompatible with the Web. To enable the integration of existing partial solutions within a mainstream Web browser environment, we have developed a middleware infrastructure, AMICO:WEB. This enables browser access to a wide variety of open source and free software components. The main contribution of AMICO:WEB is in enabling the syntactic interoperability between Web extension mechanisms and a variety of integration mechanisms used by open source and free software components. It also bridges the semantic differences between the high-level world of Web XML-based APIs and the low-level APIs of the device-oriented world. We discuss the design decisions made during the development of AMICO:WEB in the context of Web accessibility, using two typical usage scenarios: one describing a disabled user using a mainstream Web browser with additional interaction modalities; another describing a non-disabled user browsing in a suboptimal interaction situation.

This paper describes how the combination of recommendation and Semantic Web techniques can enhance user interaction with digital cultural heritage collections. The interface that this paper describes adapts user interaction to individual interest in both artworks and abstract concepts related to them. This paper presents an overview of several studies that evaluate a recommender system for museum collections. The findings and requirements derived from them form a basic framework for personalized exhibition tours that present multiple cultural artifacts as a structured and cohesive whole.

Multimedia systems typically contain digital documents of mixed media types, which are indexed on the basis of strongly divergent metadata standards. This severely hamplers the inter-operation of such systems. Therefore, machine understanding of metadata comming from different applications is a basic requirement for the inter-operation of distributed Multimedia systems. In this document, we present how interoperability among metadata, vocabularies/ontologies and services is enhanced using Semantic Web technologies. In addition, it provides guidelines for semantic interoperability, illustrated by use cases. Finally, it presents an overview of the most commonly used metadata standards and tools, and provides the general research direction for semantic interoperability using Semantic Web technologies.

We report the results of a user study that captures knowledge on how cultural heritage experts search for information. We use a qualitative study technique with participants from four cultural heritage institutions in the Netherlands who were interviewed and asked to answer questionnaires about their daily work. Our goal is to acquire knowledge of their information seeking needs and the information sources they use. The paper provides an analysis and discussion of the issues that experts frequently face when searching for information.

This document gives an overview on the state-of-the-art of multimedia metadata formats. Initially, practical relevant vocabularies for developers of Semantic Web applications are listed according to their modality scope. In the second part of this document, the focus is set on the integration of the multimedia vocabularies into the Semantic Web, that is to say, formal representations of the vocabularies are discussed.

Autocompletion is a commonly used interface feature in diverse applications. Semantic Web data has, on the one hand, the potential to provide new functionality by exploiting the semantics in the data used for generating autocompletion suggestions. Semantic Web applications, on the other hand, typically pose extra requirements on the semantic properties of the suggestions given. When the number of syntactic matches becomes too large, some means of selecting a semantically meaningful subset of suggestions to be presented to the user is needed. In this paper we identify a number of key design dimensions of autocompletion interface components. Our hypothesis is that a one-size-fits-all solution to autocompletion interface components does not exist, because different tasks and different data sets require interfaces corresponding to different points in our design space. We present a fully configurable architecture, which can be used to configure autocompletion components to the desired point in this design space. The architecture has been implemented as an open source software component that can be plugged into a variety of applications. We report on the results of a user evaluation that confirms this hypothesis, and describe the need to evaluate semantic autocompletion in a task and application-specific context.

A web browser provides a uniform user interface to different types of information. Making this interface universally accessible and more interactive is a long term goal still far from being achieved. Universally accessible browsers require novel interaction modalities and additional functionalities, for which existing browsers tend to provide only partial solutions. Although functionality for web accessibility can be found as open-source and free software components, their reuse and integration is complex because they were developed in diverse implementation environments, following standards and conventions incompatible with the web. To enable the integration of existing partial solutions within a mainstream web browser environment, we have developed a middleware infrastructure, AMICO:WEB. This enables browser access to a wide variety of open source and free software components. The main contribution of AMICO:WEB is in enabling the syntactic interoperability between web extension mechanisms and a variety of integration mechanisms used by open-source and free software components. It also bridges the semantic differences between the high-level world of web XML-based APIs and the low-level APIs of the device-oriented world. We discuss the design decisions made during the development of AMICO:WEB in the context of web accessibility, using two typical usage scenarios: one describing a disabled user using a mainstream web browser with additional interaction modalities; another describing a non-disabled user browsing in a suboptimal interaction situation.

MPEG-7 can be used to create complex and comprehensive metadata descriptions of multimedia content. Since MPEG-7 is defined in terms of an XML schema, the semantics of its elements has no formal grounding. In addition, certain features can be described in multiple ways. MPEG-7 profiles are subsets of the standard that apply to specific application areas and that aim to reduce this syntactic variability, but they still lack formal semantics. We propose an approach for expressing the semantics explicitly by formalizing the constraints of various profiles using ontologies and logical rules, thus enabling interoperability and automatic use for MPEG-7 based applications. We have implemented VAMP, a full semantic validation service that detects any inconsistencies of the semantic constraints formalized. Another contribution of this paper is an analysis of how MPEG-7 is practically used. We report on experiments about the semantic validity of MPEG-7 descriptions produced by numerous tools and projects and we categorize the most common errors found.

Many Semantic Web applications provide access to their resources through text-based search queries, using explicit semantics to improve the search results. This paper provides an analysis of the current state of the art in semantic search, based on 35 existing systems. We identify different types of semantic search features that are used during query construction, the core search process, the presentation of the search results and user feedback on query and results. For each of these, we consider the functionality that the system provides and how this is made available through the user interface.

A generic facet browser including temporal and spatial visualization. The existing MultimediaN Eculture demo is updated in two ways: improved visualization of the semantic timeline and added a natural language sentence generator for cluster headers. Relation search interface is also implemented.

A webpage for SMaRT cluster (http://www.smart-network.eu/), and works as a webmaster for Vereniging Werkgemeenschap Informatiewetenschap (http://www.informatiewetenschap.org/)

oMAP automatically aligns OWL ontologies.

AMICO is a generic platform for rapid application development with open-source and free software components and services available at: http://amico.sourceforge.net/.

NewsML Demonstrator converts NewsML data and IPTC NewsCodes into RDF and SKOS.

The authors propose an infrastructure for rapidly prototyping applications from open source software components. The Adaptable Multi-Interface Communicator infrastructure (AMICO) is based on ideas of middleware platforms for component integration, but it focuses on pragmatic aspects of OSS integration, often absent from many existing integration platforms. The authors also identify the key requirements of middleware for rapid prototyping with OSS components and illustrate their approach through two examples in complex scenarios.

In recent years, many research activities have focused on design that aims to produce universally accessible systems, taking into account special needs of various user groups. These special needs are associated with many user factors, such as impairments of speech, hearing or vision, cognitive limitations, aging, as well as with various environmental factors. Fields that address this problem, such as Usability, Universal Accessibility, Universal Design, or Inclusive Design have been developed as relatively independent domains, but they share many aspects with other human-computer interaction (HCI) disciplines. However, researchers and practitioners are often not aware of interconnections among concepts of universal accessibility and "ordinary" HCI. In view of this situation, in this article we show there is a fundamental connection between multimodal interface design and universal accessibility, and that awareness of these links can help both disciplines. Researchers from these areas may use different terminology, but the concepts they use often have essentially the same meaning. We propose a unified conceptual framework where these areas can be joined.

This paper describes ongoing work of a project aimed at exploiting Semantic Web techniques to support searching and annotating large cross-institutional digital-heritage collections. The project demonstrator contains multiple collections and multiple vocabularies. The architecture is fully based on Web standards. We show novel search and presentation techniques which make use of interoperability between the collections and between the vocabularies.

Semantic descriptions of non-textual media available on the web can be used to facilitate retrieval and presentation of media assets and documents containing them. While technologies for multimedia semantic descriptions already exist, there is as yet no formal description of a high quality multimedia ontology that is compatible with existing (semantic) web technologies. We explain the complexity of the problem using an annotation scenario. We then derive a number of requirements for specifying a formal multimedia ontology before we present the developed ontology, COMM, and evaluate it with respect to our requirements. We provide an API for generating multimedia annotations that conform to COMM.

The main objective of the MultimediaN E-Culture project is to demonstrate how novel semantic-web and presentation technologies can be deployed to provide better indexing and search support within large virtual collections of cultural-heritage resources. The architecture is fully based on open web standards, in particular XML, SVG, RDF/OWL and SPARQL. One basic hypothesis underlying this work is that the use of explicit background knowledge in the form of ontologies/vocabularies/thesauri is in particular useful in information retrieval in knowledge-rich domains.

Facet browsing has become popular as a user friendly interface to data repositories. The Semantic Web raises new challenges due to the heterogeneous character of the data. First, users should be able to select and navigate through facets of resources of any type and to make selections based on properties of other, semantically related, types. Second, where traditional facet browsers require manual configuration of the software, a semantic web browser should be able to handle any RDFS dataset without any additional configuration. Third, hierarchical data on the semantic web is not designed for browsing: complementary techniques, such as search, should be available to overcome this problem. We address these requirements in our browser, /facet. Additionally, the interface allows the inclusion of facet-specific display options that go beyond the hierarchical navigation that characterizes current facet browsing. /facet is a tool for Semantic Web developers as an instant interface to their complete dataset. The automatic facet configuration generated by the system can then be further refined to configure it as a tool for end users. The implementation is based on current Web standards and open source software. The new functionality is motivated using a scenario from the cultural heritage domain.

Facet browsing has become popular as a user friendly interface to data repositories. We extend facet browsing of Semantic Web data in four ways. First, users are able to select and navigate through facets of resources of any type and to make selections based on properties of other, semantically related, types. Second, we address a disadvantage of hierarchy-based navigation by adding a keyword search interface that dynamically makes semantically relevant suggestions. Third, the interface of our browser, /facet, allows the inclusion of facet-specific display options that go beyond the hierarchical navigation that characterizes current facet browsing. Fourth, the browser works on any RDFS dataset without any additional configuration. These properties make /facet an ideal tool for Semantic Web developers that need a instant interface to their complete dataset. The automatic facet configuration generated by the system can then be further refined to configure it as a tool for end users. The implementation is based on current Web standards and open source software. The new functionality we provide is motivated using a scenario from the cultural heritage domain

Multimedia in all forms (images, video, graphics, music, speech) is exploding on the Web. The content needs to be annotated and indexed to enable effective search and retrieval. However, recent standards and best practices for multimedia metadata don't provide semantically rich descriptions of multimedia content. On the other hand, the World Wide Web Consortium's (W3C's) Semantic Web effort has been making great progress in advancing techniques for annotating semantics of Web resources. To bridge this gap, a new W3C task force has been created to investigate multimedia annotations on the Semantic Web. This article examines the problems of semantically annotating multimedia and describes the integration of multimedia metadata with the Semantic Web. (Editor's note by John R. Smith).

The Semantic Web envisions a Web that is both human readable and machine processible. In practice, however, there is still a large conceptual gap between annotated content repositories on the one hand, and coherent, human readable Web pages on the other. To bridge this conceptual gap, one needs to select the appropriate content from the repository, structure and order the material, and design a Web page that effectively conveys the selected content and the chosen structure. In addition to this conceptual gap, there is also a technological gap. On one side of this gap, we find the semantic-oriented technology deployed to build annotated content repositories. This includes RDF, RDF Schema and OWL. On the other side of the gap is the syntax-oriented technology deployed to build Websites. This includes XML, XSLT, CSS, XHTML and SMIL. In this paper, we discuss the conceptual relationships between the world of explicit metadata semantics and the world of Web presentations and their underlying syntactic formats. We also explore to what extent this gap can be bridged automatically, and how current Web technologies can be used to support this process.

This paper explores the notion of experience in the context of dynamic and interactive environments, such as web-based musea, where neither the individual user requirements nor the requested material can be predicted in advance. A definition of experiences for the particular context is introduced on which the analysis of the what (events), why (context) and the how (presentation) are based. The paper tries to identify the essential aspects of representation for the three main fields of investigation, namely content and expression for the event; goal, task action and role for the context, and the influence of event and context for the presentation. The aim is that the system can find satisfactory solutions for upcoming questions (e.g. based on the content of an image), misunderstandings (rearrangement of the material) or non-understanding (creation of a new sequence). The intent of the paper is to provide a first step towards dynamic and adaptive knowledge structures that facilitate conceptual presentations.

We describe our experimental rhetoric engine Vox Populi that generates biased video-sequences from a repository of video interviews and other related audio-visual web sources. Users are thus able to explore their own opinions on controversial topics covered by the repository. The repository contains interviews with United States residents stating their opinion on the events occurring after the terrorist attack on the United States on the 11th of September 2001. We present a model for biased documentary statements, such as interviews, and explain in detail how this model facilitates the automatic generation of rhetorical arguments on a micro-level. We outline the required representations of relevant rhetorical structures and the way they can be processed. The processesa re described via examples generated by our experimental engine. The first example shows how to logically counter an opinion using semantics contained in the audio tracks from the database, while the second example describes the generation of an emotional counterargument using visual material.

In this paper, we report our experiences with the use of SemanticWeb technology for annotating digital video material.Web technology is used to transform a large, existing video ontology embedded in an annotation tool into a commonly accessible format. The recombination of existing video material is then used as an example application, in which the video metadata enables the retrieval of video footage based on both content descriptions and cinematographic concepts, such as establishing and reaction shots. The paper focuses on the practical issues of porting ontological information to the Semantic Web, the multimedia-specific issues of video annotation, and requirements for Semantic Web query and access patterns. It thereby explicitly aims at providing input to the two new W3C Semantic Web Working Groups (Best Practices and Deployment; Data Access).

Traditionally, research in applying Semantic Web technology to multimedia information systems has focused on using annotations and ontologies to improve the retrieval process. This paper concentrates on improving the presentation of the retrieval results. First, our approach uses ontological domain knowledge to select and organize the content relevant to the topic the user is interested in. Domain ontologies are valuable in the presentation generation process, because effective presentations are those that succeed in conveying the relevant domain semantics to the user. Explicit discourse and narrative knowledge allows selection of appropriate presentation genres and creation of narrative structures, which are used for conveying these domain relations. In addition, knowledge of graphic design and media characteristics is essential to transform abstract presentation structures into real multimedia presentations. Design knowledge determines how the semantics and presentation structure are expressed in the multimedia presentation. In traditional Web environments, this type of design knowledge remains implicit, hidden in style sheets and other document transformation code. Our second use of Semantic Web technology is to model design knowledge explicitly, and to enable it to drive the transformations needed to turn annotated media items into structured presentations

Time-based, media-centric Web presentations can be described declaratively in the XML world through the development of languages such as SMIL. It is difficult, however, to fully integrate them in a complete document transformation processing chain. In order to achieve the desired processing of data-driven, time-based, media-centric presentations, the text-flow based formatting vocabularies used by style languages such as XSL, CSS and DSSSL need to be extended. The paper presents a selection of use cases which are used to derive a list of requirements for a multimedia style and transformation formatting vocabulary. The boundaries of applicability of existing text-based formatting models for media-centric transformations are analyzed. The paper then discusses the advantages and disadvantages of a fully-fledged time-based multimedia formatting model. Finally, the discussion is illustrated by describing the key properties of the example multimedia formatting vocabulary currently implemented in the back-end of our Cuypers multimedia transformation engine.

This article discusses the state of the art in metadata for audio-visual media in large semantic networks, such as the Semantic Web. Our discussion is predominantly motivated by the two most widely known approaches towards machine-processable and semantic-based content description, namely the Semantic Web activity of the W3C and ISO's efforts in the direction of complex media content modeling, in particular the Multimedia Content Description Interface (MPEG-7). We explain that the conceptual ideas and technologies discussed in both approaches are essential for the next step in multimedia development. Unfortunately, there are still many practical obstacles that block their widespread use for providing multimedia metadata on the Web. Based on a scenario to explain our vision of a media-aware Semantic Web, we derive in Part I a number of problems regarding the semantic content description of media units. We then discuss the multimedia production chain, in particular emphasizing the role of progressive metadata production. As a result we distill a set of media-based metadata production requirements and show how current media production environments fail to address these. We then introduce those parts of the W3C and ISO standardization works that are relevant to our discussion. In Part II of this article, we analyze their abilities to define structures for describing media semantics, discuss syntactic and semantic problems, ontological problems for media semantics, and the problems of applying the theoretical concepts to real world problems. Part II concludes with implications of the findings for future action with respect to the actions the community should take.

The presentation generation area of hypermedia authoring contains different approaches to address the challenge of the presentation creation process. This paper presents the approach in which the author creates a presentation following the five stages process. These five stages reflect various facets in hypermedia authoring.

With the rapid growth of interest in the Internet as a means for accessing multimedia presentations for education, entertainment and commerce, comes a corresponding need for systems to supply automatically generated, personalised presentations. Multimedia is a rich and complex genre of resources and the interrelated effects of content, style and structure ensure that automatic presentation generation is in itself a complicated and challenging task. Integrating a model for user personalisation adds a further layer of complication and ensuring that the requirements of user, supplier and platform are all met is a demanding undertaking. This project investigates the influence of information about a user in the process of generating a multimedia presentation. As a result an architecture taking into account these trade-offs is proposed. To evaluate this combined architecture, a framework has been implemented which adjusts colour choices based on the different influences involved. This paper describes the integration of a user modelling approach within an existing system architecture, discusses some of the issues involved in applying user modelling to multimedia presentation generation and describes the prototype implementation and how it addresses some of these issues.

Much current research on hypermedia generation accepts user input only at the start of an otherwise fully-automated process. However, since multimedia presentation creation is often a complex and creative process, it has multiple phases which would each benefit from human intervention. This paper presents a hypermedia generation model that lets the user influence all phases of this computer-assisted human-guided process. The main focus is on providing extra support for helping the user find relevant media items and combine them meaningfully into a rich and coherent multimedia presentation. Like fully-automated systems, our approach uses explicit knowledge about the presentation's topic domain, narrative structures, hypermedia presentation and distinctions between media modalities. This paper presents a motivating scenario that is used to derive a number of system requirements and to discuss the pros and cons of the presented approach.

The 'Document Web' has established itself through the creation of an impressive family of XML and related languages. In addition to this, the 'Semantic Web' is developing its own family of languages based primarily on RDF. Although these families were both developed specifically for 'the Web', each language family has been developed from different premises with specific goals in mind. The result is that combining both families in a single application is surprisingly difficult. This is unfortunate, since the combination of semantic processing with document processing provides advantages in both directions --- namely using semantic inferencing for more intelligent document processing and using document processing tools for presenting semantic representations to an end-user. In this paper, we investigate this integration problem, focusing on the role of (RDF) semantics in selecting, structuring and styling (XML) content. We analyze the approaches taken by two example architectures and use our analysis to derive a more integrated alternative.

Time-based, media-centric Web presentations can be described declaratively in the XML world through the development of languages such as SMIL. It is difficult, however, to fully integrate them in a complete document transformation processing chain. In order to achieve the desired processing of data-driven, time-based, media-centric presentations, the text-flow based formatting vocabularies used by style languages such as XSL, CSS and DSSSL need to be extended. The paper presents a selection of use cases which are used to derive a list of requirements for a multimedia style and transformation formatting vocabulary. The boundaries of applicability of existing text-based formatting models for media-centric transformations are analyzed. The paper then discusses the advantages and disadvantages of a fully-fledged time-based multimedia formatting model. Finally, the discussion is illustrated by describing the key properties of the example multimedia formatting vocabulary currently implemented in the back-end of our Cuypers multimedia transformation engine.

Multimedia presentation generation has to be able to balance the functional aspects of a presentation that address the information needs of the user, and its aesthetic form. We demonstrate our approach using automatic colour design for which we integrate relevant aspects of colour theory. Colour selection takes the relative importance of form and function into account through the use of weights in the generation process. We do not provide a definition of the relative importance of form versus function, but seek to explore the roles of subjective elements in the generation process.

For humans to gain comprehensive views of large amounts of repository contents, they need to have insight into the relations among information objects. It is a challenge to automatically generate presentations of repository contents, through, for example, search results, which reveal such relations to readers. Such presentations must reflect properties of information objects such that large sets of information objects appear as a coherent whole. An approach to this is generation of discourse structures that convey such properties of information objects in presentations. Semantic Web technology provides a conceptual basis for generation of discourse in Web-based information environments. This paper describes automatic generation of sequence and emphasis in presentations of information objects. It shows generation of object sequences and emphasis in accordance with a user input of relevance of information attributes in our Topia architecture. The resulting presentations allow users to encounter information objects in decreasing order of relevance. This makes it easier to identify relevant information objects among many others, as well as to observe their relations with the other information objects.

Traditionally, research in applying Semantic Web technology to multimedia information systems has focused on using annotations and ontologies to improve the retrieval process. This paper concentrates on improving the presentation of the retrieval results. First, our approach uses ontological domain knowledge to select and organize the content relevant to the topic the user is interested in. Domain ontologies are valuable in the presentation generation process, because effective presentations are those that succeed in conveying the relevant domain semantics to the user. Explicit discourse and narrative knowledge allows selection of appropriate presentation genres and creation of narrative structures, which are used for conveying these domain relations. In addition, knowledge of graphic design and media characteristics is essential to transform abstract presentation structures in real multimedia presentations. Design knowledge determines how the semantics and presentation structure are expressed in the multimedia presentation. In traditional Web environments, this type of design knowledge remains implicit, hidden in style sheets and other document transformation code. Our second use of Semantic Web technology is to model design knowledge explicitly, and to let it drive the transformations needed to turn annotated media items into structured presentations.

First generation Web-content encodes information in handwritten (HTML) Web pages. Second generation Web content generates HTML pages on demand, e.g. by filling in templates with content retrieved dynamically from a database or transformation of structured documents using style sheets (e.g. XSLT). Third generation Web pages will make use of rich markup (e.g. XML) along with metadata (e.g. RDF) schemes to make the content not only machine readable but also machine processable - a necessary pre-requisite to the emphSemantic Web. While text-based content on the Web is already rapidly approaching the third generation, multimedia content is still trying to catch up with second generation techniques. Multimedia document processing has a number of fundamentally different requirements from text which make it more difficult to incorporate within the document processing chain. In particular, multimedia transformation uses different document and presentation abstractions, its formatting rules cannot be based on text-flow, it requires feedback from the formatting back-end and is hard to describe in the functional style of current style languages. We state the requirements for second generation processing of multimedia and describe how these have been incorporated in our prototype multimedia document transformation environment, emphCuypers. The system overcomes a number of the restrictions of the text-flow based tool sets by integrating a number of conceptually distinct processing steps in a single runtime execution environment. We describe the need for these different processing steps and describe them in turn (semantic structure, communicative device, qualitative constraints, quantitative constraints, final form presentation), and illustrate our approach by means of an example. We conclude by discussing the models and techniques required for the creation of third generation multimedia content.

Also for multimedia, there is a need to create presentations in presentation-independent forms and have the final form(s) generated from the base document. We can input our expertise on the specific requirements for multimedia in automatic document generation. In addition, we have experience in processing multimedia documents in a Web context in general and related issues within SMIL 2.0 in particular. We can explain the use of alternative media for devices where text is inappropriate. In addition, different sorts of meta-data can be incorporated into a presentation-independent base document, and device/user-specific presentations can be generated from this base document, using the information captured in the meta-data. In addition, we can explain the advantages of incorporating timing as one of the dimensions of variation in a presentation. For example, adding temporal variations within HTML, using e.g. HTML+SMIL, can allow different types of trade-offs in a presentation.

Hypermedia documents are most often created with a particular presentation environment in mind. This requires the authoring of one document per presentation platform. As pointed out in [], much implementation effort can be avoided by specifying how the same underlying document can be presented in different environments. A style sheet defines a mapping from a source document to a presentation for it. We discuss the existing use of style sheets as applied to text and discuss their application to the case of hypermedia, and in particular how they need to be extended.