date: 03-03-2003 author: Joost description: These are some personal notes I wrote down the last two months. The first bit is mainly about presentation structures the last bit is more conceptual and is about how the use of ontolgies fit in the generation process WARNING! THIS IS NOT A COHERENT STORY objective: extent functionality of Cuypers by making mapping RST - HFO explicit. This is realized by using an ontology describing different types of multimedia items such as image, text, video etc. on the higher levels and specific subclasses of these such as title, caption, header, but also biography, comment, painting photograph etc. This level of detail is domain dependent and might be influenced by an author/domain expert. The higher levels should in principle be reusable between different domains. In addition there is an ontology of (RST)relations and a transformation sheet defining rules which define a mapping from relations,nuclei,satellite to hfo. Because both the relations and and nuclei/satellite are hierarchically organized these rules have a range of appliances comparable to the functionality of style-sheet rules. example: ontology of relations (This is just an example) rst (R1) - nucleus/satellite (R2) - elaboration (R3) - example (R4) - multi-nucleus (R5) - joint (R6) - sequence (R7) 'type' ontology media (A) - image (B) - painting (C) - drawing (D) - video (E) - audio (F) - background music (G) - voice over (H) - text (I) - one-liner (J) - title (K) - caption (L) - header (M) - full-text (N) - biography (O) - comment (P) transformation relation,nuc,sat scope hfo ---------------------------------------------------------------------------------- R1,A,A everything hfo(R1,A,A) R3,J,B images with a one line elaboration nbox(R,J,B) R2,B,H images with a voice over par The terms described in the ontology map to transformation rules which map ontology terms to HFO objects. These rules have names and the mapping is made by a designer. Besides an ontology describing media types there is made a mapping between the terms from the ontology and the RST to hfo, currently the presentation structure is implicitly coded into the rest tree. The rules which transform rst to hfo 'know' that at the top level there is a nucleus title and two satellites: one elaboration text and a sequence of example. The resulting hfo transforms both the underlying structures of the satellites but in addition to that it also determines implicitly the relation between the two satellites which is that they can/should be presented next to each other. presentation structures are domain independent and represent the logic structure (grouping, ordering) in the presentation. Presentation in this context could be a mm-presentation but also a hypermedia document of even a textual document. Examples of a holistic structure for multimedia are mm-presentation, scene, sub scene. For a paper document or book : book, chapter, section, subsection. Presentation structure define the initial parameters for a presentation, such as the dimensions which can be used temporal spatial, possible navigational options, linking. etc. Presentation structures define the possible presentation space. A ps is at presentation time represented by a hfo. Each ps is represented by one hfo, however this can be of different type. For example a hbox (left to right) might be substituted by a vbox (top-bottom). At the atomic level ps and hfo are quite similar since there is not much logical structure in a atomic media item. There are composite hfo and they define structure at a low level. (eg slideshow) this however is not a ps since it assumes implicitly a temporal dimension. a ps has made this choice explicitly and can choose to represent this choice by using a slideshow. A paper document can choose to use a non-temporal grid for the same purpose. We distinguish presentation structures based on holistic structure. Examples of holistic structures are Multimedia presentation, Hypermedia presentation, paper documents, power point presentation etc. What they all have in common are the grouping and, order and priority specifications. The implementation into hfo's are all different though. If we can define a correspondence between the different ps, such as a scene in a mm presentation corresponds to a section in a paper document and a page in a hypermedia document then we might be able to define a kind of master structure applicable for all holistic structures. The hfo generation needs to take into account the possibilities it can use. (eg a temporal dimension in a mm presentation). There are however style issues which make thing complicated. The base layout of a mtv mm presentation is different from a cnn mm presentation. If we have to take both into account on this level the hfo generation becomes too complicated i am afraid. Within the PS structure tree there are only composites and atomics. Atomics are media items without children, composites have children and define a group. There are specialized types of composites such as presentation, slide show, hypermedia document etc. Thoughts about using ontologies to go from a semantic graph to PS. ---------------------------------------------------------------- The Objective of using an ontology to create PS out of a semantic graph is to get as much knowledge explicit as possible. This way we claim we can use domain dependent relations within the process of automatically generating a presentation and still make the algorithms generally applicable. The approach is to define general rules which apply to concepts in the ontology. Because of that the rules are applicable to children of this concept as well. This way an author only needs to specify exceptions from the default behavior which in general requires less effort. The input source we use is a graph consisting of domain relations: Rembrandt paints nachtwacht Rembrandt married saskia nactwacht style chiaroscouro The concepts described in an ontology have besides a parent concept and possible children concepts also slot values. These are 'attributes' which in fact define the concept. For example a person concept can be defined by a first name and last name, a birth date, birth place etc. The allowed values for a slot can be constrained, for example the paints relation requires that the subject is of type painter and that the object is of type painting. In the ontologies depicted below capital cased means CONCEPT and lower cased means slot. We distinguish the following ontologies: [domain ontologies] which consist of an object hierarchy: SUBJECT PERSON firstname lastname date of birth PAINTER style PHOTOGRAPHER OBJECT PAINTING title type BOOK title nr of pages and a relation hierarchy RELATION UNARY RELATION OBJ PERS BINARY RELATION subj obj PERS-PERS RELATION MARRIED date PERS-OBJ RELATION CREATED date PAINTED OBJ-OBJ RELATION PART-OF note that a carefully chosen upper ontology is an important one as it defines the possible abstractions. There is much research being done in the field of upper ontology layers (see CYC SUMO etc.) In fact the object hierarchy and relation hierarchy are merged then. This however doesn't really make a difference and in practice one needs to treat then separately. [narrative structure] PRESENTATION INFORMATIVE BIOGRAPHY name city married PAINTERS_BIOGRAPHY list created MANUAL object CV name experience ARGUMENTATIVE CRITIC [RST] RST NUC-SAT ELABORATION EXAMPLE BACKGROUND CONTRAST MULTINUC JOINT SEQUENCE * Semantic graph - Domain ontology The relations within the semantic graph map to relations in the domain ontology. the 'paints' from 'Rembrandt paints the nachtwacht' maps to 'painted' in the domain ontology for relations. And 'Rembrandt' is an instance of painter in the object hierarchy. * Narrative structure - Domain ontology The narrative structure defines the framework of what we are going to present. These are so called macro structures which define common structures hard-wired in our brains. We know the global structure of a biography and what properties it has. To some extent one can view them as being which values need to be filled in. The slot values are typed by the domain ontology. For example the slot value name within a biography should be constrained to be a value which comes from the person object in the object hierarchy. [[ This is about domain object properties]] * Domain Ontology - RST The domain ontology defines domain relationships between subjects. There are at least in theory an infinite amount of possible semantic relationships. From a presentation perspective one cannot always tell the difference between two presented relationships. For example a 'collegueOf' and 'marriedTo' relationship can both be presented in a similar way, let's say by a smaller picture with a title next to the subject. There are different mappings possible a 'precedes' relation for style periods can be mapped to a 'contrast' RST relation (if one wants to focus on the differences) or on 'background' (to provide context in which this style period could be developed) [[this is about domain relations]] * Narrative Structure - RST We (probably) can deduce general RST patterns from analyzing a number of examples of existing presentation structures such as biographies, CV's, manuals etc. What I mean by an RST pattern are the allowed rst relations. For example a biography is likely to have 'background' and 'elaboration' relationships while a 'contrast' relation is probably not part of a biography. In addition, biographies* are typically flat, which means they tend be focused on a topic and do not allow much derivation. A (historical) overview in contrast, is not very much bound to a specific topic and leaves much room for derivation. As a result the RST tree of a biography is less deep then the one of a historical overview. In addition to the presented relationships object properties are part of a presentation as well. For example within a biography properties like name and address etc. should be presented and therefore they should be part of the RST pattern. [[* with biography I mean here a simple presentation only stating name, address etc. ]] % biographies have elaboration and background relations (* = multiple) % and a name and a date biography(TopicNuc) -> elaboration(TopicNuc,SatA)* background(TopicNuc,SatB)* elaboration(TopicNuc,person_name),elaboration(TopicNuc,person_birthdate) % overviews have multiple levels of background elaboration relations. overview(TopicNuc) -> elaboration(TopicNuc)* background(TopicNuc, SA)* background(_,Nuc) -> elaboration(Nuc,SatA)* background(Nuc, SatB)* elaboration(_,Nuc) -> elaboration(Nuc,SatA)* background(Nuc, SatB)* Note that 'elaboration' is a 'high-level' RST relation within the RST ontology. The rule therefore matches elaboration relations and children of elaboration relation, such as 'example'. The RST paterns are extensible. For example the pattern for a PainterBiography is an extension of the general biography: paintersBio(TopicNuc) -> biography(TopicNuc), elaboration(TopicNuc, painting)* elaboration(_,painting) -> elaboration(painting,title), elaboration(painting,date).. * RST - PS RST is merely an ordering in argumentation and therefore insufficient to generate a presentation from. What needs to be added is: + holistic structure - paper document or MM presentation + grouping - which items are presented in a group, (what is the coherence between the media items) + ordering - what is the order of presentation + priority - what can be left out if necessary I don't believe this information should be part of the narrative structure (completely), it means a biography is always presented in the same way/order which is not the case if we look at real examples of biographies. The narrative is dependent on at least user profile and style. Since I don't have a clear picture how this should be realized I leave it for now. In the mean time we could get the required information from the narrative structure ontology.