The Communicative Device Lynda Hardman I have been collecting bits and bobs for the past few months, and haven't got round to finishing anything. The goal of this piece is basically to gather everything together to record it, but nothing further. What you see is all there is... The next piece "Bridging the Discourse/Graphic Design Gap" should start focussing in on something that we can implement and publish. ===== ~lynda/Research/important-things.txt -rw-r--r-- 1 lynda ins 438 Aug 17 16:28 important-things.txt The Communicative Device layer takes an input of a semantic/rhetorical structure and produces a tree of CommDev's. The context is an important unknown. --- Communicative device Input: CommDevice tree Content = number of media items, the media items themselves (implicit width/height/duration), Rhetorical structure guiding the choice of the device Context: The context is the (tree) of CDs that surround the current CD. It could be that some CDs have a list of "these parents only allowed", and others may be more globally applicable. (E.g. a bulleted list shouldn't have a heading within it.) Output: The HOT: see blue_note http://www.cwi.nl/~media/blue_book/hot-reqs.html My current approximation is: spatial/temporal ordering of items in region on screen. ===== ~lynda/Research/comm-device.txt -rw-r--r-- 1 lynda ins 3180 Aug 17 17:12 comm-device.txt --- What is a communicative device? It is "style", device, discourse structure and domain independent. A particular communicative device can be used for expressing various discourse structures. A particular discourse structure may have a preferred communicative device, but could use one of several. In order to fully express a discourse structure, perhaps several comm devs need to be combined together. Discourse structure Communicative device Time/Space/Link (RST) Sequence Bookshelf All sorts of combinations of space/time/link Slide show S+T (no links) Joint ? (I don't know what joint is) Contrast Want something in space/time (link - but we don't do links yet) It should be able to handle different numbers of media items. bookshelf slide show ----------------- How about not thinking about different levels, but thinking about a single nucleus/satellite relationship, and then determining how these can be combined? I.e. don't talk about heading 1, heading 2, heading 3, but talk about heading and "content", and then how to combine the different approaches. For example, with Martijn van Welie's hierarchy in a website, there are a number of options: horizontal menu, with horizontal drop-down for items in chosen one; horizontal menu, with vertical drop-down for items in chosen one; horizontal menu, with separate area of screen for vertical menu of items corresponding to chosen one; vertical... (?) Show hierarchy/context above (this requires reference to previous levels). --------------- Input: Number of media items the media items themselves (implicit width/height/duration) Output: spatial/temporal ordering of items in region on screen, at duration in presentation. Context: This is the important bit I'm forgetting in the equation. The comm dev is not being used independently, just as no human conversation is without context. Highest context is "infotainment" "educational"? Some comm devs. can be used for title pages (when no context has been set up), others for something in the middle. The slide show also has a context, the "device" that "called" it. In some hierarchy of devices we can use other devices, or even the device itself. (C.f. Martijn's website hierarchy. [[can I use martijn's piece as a second example for proving the "general" point for the slide show case]] Either we think hard about how we build up the context, or we ignore it and assume we have lots of it? (But then everything has to be consistent.) Posters are context-free works. Soaps carry almost infinite amounts of context with them. Symphonies have their own comm dev patterns (4 movements, fast, slow, trio?, faster) Even dance builds up its own context. You know when a dance finishes. Looking at a hypermedia piece you need to know when it is finished. -------------- Film poster (context free) Film ontology: Title, shot from film (image), director, actors, date in cinema, suitable viewing age, awards, URL... "Template" is: use shot as background. Title is important actors awards, director T/S/L is all the boxes I want. ---------------- Either the context is separate, or it overlaps (TSL) with the content. ===== ~lynda/Research/objectives.txt -rw-r----- 1 lynda ins 8660 Aug 16 11:07 objectives.txt This is sort of prior to the previous two notes, and is really trying to work out what the relationships are among everything. --- What can I capture? How can I capture it? How can it be integrated into the generation process? How do we measure "quality". Three axes were semantics, aesthetics and resources. I guess I am busy on the aesthetics axis. I need to be able to determine which things are "more aesthetic" than others. Marc Davis only did semantic retrieval, and "better" was implicitly closer in the semantic world. On the basis of design.txt, how can I capture the designs of (a) something I like (b) something "simple"/newspaper (c) something complex/arty - NDT brochure (d) something "ugly" NOVIB brochure Can I actually take the rules, apply them and see a "NOVIB" or "NDT" result? The dimensions should fall out of design{2}.txt All I need do is make the CSS style sheet. Is this the language I would want? I suspect that CSS provides me with the "syntax" of my style, but not really the "semantics". I have a feeling that colours using the same saturation go together. E.g. higly saturated colours are always used for kids toys. Adult clothes tend to be much less saturated. What is brown in rgb? dark orange? As well as the design, there is also the step to the design. I think we are calling these the communicative devices, but I'm not sure if I have muddied things again. The terms I have are: "related", "important" and "adjunct" which sound fairly semantic to me. My "devices" are more or less the list in style-blank.txt Spatial grouping Temporal grouping Links White space font (times, helvetica) type face (bold italic) font size position "fade picture into background and put text on top" Alignment Align within items - need to look within Lines/underlines to emphasize grouping, break up Layout across (same sort of) screens consistent Contrast - different fonts; light on dark, dark on light; heavier/double rules Columns Columns grouped using broad heading Transitions A mapping is: "more important" is in a larger font size "more important" is upper left or top centered In Kamps's book, he takes a DAG relation and simplifies it until he has lots of binary relations which can be represented with his graphics devices. I'm not sure how his levels correspond to our rhetoric level or our communicative device level. He too solves lots of constraints, although I'm not sure which constraint-solver he uses. I suspect we want something like a communicative device layer, otherwise we are trying to translate rhetorics into graphics devices for all sorts of circumstances. The hypermedia design pattern layer allows us to identify common structures which can then be moved to several platforms. I think I want my whiteboard and a digital camera... What are examples of communicative devices? (Not the bookshelf! :-) How can these be conveyed for different platforms? Can I take a single domain (news??) and analyse existing news sites? Same news on TV, web site, radio? That would be good. How about graphics layout? (Or ASN bank paper info. c.f. web site info.) On TV get one story at a time, and the "context", the headlines, precede the sequence of items. On radio the same is true, except no images. On the Web, the context is given through a number of frames and the user can select which item to read. So now we have a design pattern/communicative device: content list plus items and the temporal spatial linking is: large, highbandwidth screen, no navigation - TV option; large, highbandwidth screen with navigation - Web option; no screen, no navigation- radio option; no screen with navigation - list of items and then choose; small screen without navigation - tickertape; small screen (WAP phone) with navigation - items in menu and user chooses. Title plus content Related topics I guess I could look at all the info, on a BBC/NYT page and label it. Then see how it is organised. Then compare with radio program/TV. (The producers themselves are no doubt much more aware of all this.) But let's say I already have my analysis. Then what? I have to be able to express the left hand side (communicative devices), also the right hand side (some way of describing temporal/spatial/link layout) ((and then add the graphic design stuff.)) But, having expressed the time/space/link "solutions" we need to be able to determine which is the better solution. Otherwise we are left with just a long list of templates which we can select from in some boring way. But then maybe there are variations in the templates depending on user preferences? Does it come down to whether we can generate a style, or whether we are lucky to be able to select a style. I guess aiming for the latter would be easier and certainly would teach us something in the shorter term. So, I set up a list of templates: a description of a device (CCPP?) and then associate a style sheet with it. A CSS style sheet is not enough, since need the space/time/link description as well. [[Is this coincidental that I need space/time/link info. plus style info.?? I suspect not, since the whole AHM idea was of filling 3D space with 3D bricks.]] I don't think I care about the rhetorical level just now. I'd rather just concentrate on the communicative device level - since I just want a lead-in to the graphic design level. (Although I suspect my labels will be rhetorical labels.) Annotate colour screen dumps of a few web sites? Ask whether there are better examples I should look at? Scientific American - web and printed, to contrast better with the work of W&W. -------------------------------------------- To do: Read Thomas's book with a pencil in my hand... Build a BBC news? mockup, then put the same info in a different style (Scientific American), and then graphics designer book. How does this differ from the W&W work? What have I learned? Is this not just applying different style sheets. I have to avoid repeating that. I prefer the web/web-smil/paper/radio/TV version. (Read Jim's article!) And what about the PhD thesis I returned? Is this the rhetorics I'm looking for? ------------------ 23-06-01 The MMM00 paper could be turned into a journal paper. I would throw in many more visual examples of the communicative devices. Also, make the distinctions between multi-nuclear and nuclear-satellite more clear. Perhaps even putting nuclear-satellite first. I don't understand Table 1, instead I would have a table of the combinations of time-space, time-link, time-time etc. and what they can be used to express. I think what is lacking is a link to real examples. Showing that this is actually used in an existing (professionally created) example. It's like explaining grammars with sentences like "The cat sat on the mat." and no-one will believe you that Shakespeare uses the same grammars. What we _really_ need is some way of saying which comm. device should be used in which circumstance. That would be the real contribution. If we know that then we know the "better quality" solutions, and Joost can work out his trade-offs. I want to compare Hypermedia Design patterns with Lloyd's "central node" HCD (see notes on p8 of paper). The "problem" with pattern is that you can't invent them. They are by definition extracted from existing uses. When is something perceived to be more of less sequential? ...4 5 6 ... is to me more of a sequence than 1 2 3 4 5 6 7 8 9 He also mentions that in some cases evidence can be red and elaboration blue. These are style conventions on top of the underlying communicative device. ------------------ 26-01-01 Had a completely inspirational talk with Martijn. I'm not sure that he has solved any of my problems directly, but he is covering the ground (with navigation solutions) exactly where I'm looking. I need to read his patterns chapter and study an article he showed me on the Web. http://www.welie.com/ http://www.welie.com/patterns http://www.welie.com/articles I think what I could do is study the patterns he has, and try to come up with ways of formalising the context and the solutions. If that is possible, then I can annotate his (XML) patterns and either have him add in the metadata into the XML, or put together some RDF to point to his XML patterns. There are images too, but maybe something can be extracted from them as well. ------------------ Don't forget to look up where W&W are cited for inspiration for relevant work. "Find similar articles" via google (NL?). What happened to article with Jim? Did it get published? BBC News analysis. ===== ~lynda/Research/design2.txt -rw-r----- 1 lynda ins 12814 Jun 20 2001 design2.txt Notes on graphics design books. --- Graphics Design, hopefully extended to temporal design... Robin Williams The Non-Designer's Design Book Proximity Related items grouped together What does related mean? c.f. Frank vH's work on visualization algorithm (we have 7 +/- 2 items) Some form of distance measure based on domain ontology? Grouping can be spatial and/or temporal. Links are good when proximity is not needed. All this depends on what we mean by related. In terms of the domain ontology or the discourse structure? White space is important, although Tufte doesn't like white space (on paper). Margins on media items. Alignments - it is important to use a grid in generated design since it is needed for consistency. Is it useful to have time where nothing happens too? I doubt it, since time remains a precious resource for the user. Space, however, on the computer screen can be "wasted" with no consequences, except that of the time trade-off (to get to other information). Transition information? Break up information into groups. Treat each group in the same way. p 19 Make difference in use of space between groups. Groups on same level use same amount of space. Use different type faces for different importance of information. More important bigger. So we need to know what important is too. Related is in terms of the subject domain, important must depend on the user, since nothing in the domain is "important" is it? More generalised and specialised concepts perhaps? Where should the most important item go? Central? Upper left? Depends on grid? E.g. title is very important (level 1) name of piece is important (level 2) description of piece is not very important (level 4) time of performance is a bit important (level 3) Place of performance is vaguely important (level5) and price of tickets even less so (level 6). Maybe there is even another level. There aren't 10 levels (but keep a look out - how about 7 +/- 2 importance levels in one piece?) In an article normally have a title, 3 numbered headings, one unnumbered heading (i.e. 4th level), plus paragraph text. Total 6 levels. How to encode importance? Size of font and bold. Level 1 - v. large and bold Level 2 - large and bold Level 3 - normal size and bold Level 4 - normal size Level 5 - Smaller and bold Level 6 - Smaller Look at publications for examples (NRC, Scientific American, interactions, etc.). Italics sometimes used. On second thoughts I wouldn't make level 4 unbold and then level 5 bold. Maybe level 5 normal size and italic? Other tips. Make pictures large, fade them into the background and put text on top of them. Check out new-style CWI site! How would I redesign the concert series for less space and a temporal presentation? Least important info. via a link (link name "concert held at" - need to deduce this from domain ontology, urgh). Still want to keep 3 concerts on single page, but link to details of each concert (the smaller text). Given the graphics design already, fairly easy to transform it. If in danger of getting too many groups on a page (e.g. more than 7+2) then need to regroup. Perhaps by changing level of information? Can tell what the groups are by squinting eyes - an algorithm that can distinguish between large space and small spaces. Alignment Always align. (also temporaly = rhythm?) Never centre. Always align left or align right, and re-use alignments wherever possible. Align to content if possible - e.g. make title font size go right across the page. Break alignment with figures, or even underlines, p41 Use underlines to emphasize and break up. Any way of choosing right against left alignment? Something to do with style "BBC", Designer book, text book? Repetition With headings, and level of importance, everything that is needed is there. Be consistent with layout across screens. (grid) Take an image and repeat it, change slightly and repeat more. Don't change screen layout unless you change "scenes" - based on information grouping? Contrast Different type faces. Light on dark text vs. dark on light. Extra rules. Underline under light on dark text. With rules of different thicknesses, make sure it is clear that there are different thicknesses. If there are many columns, then unify with broad heading. Notes: Can't do bold or italic within text, since we don't know what it's saying. Similarly, we can't align within items. Scott McCloud Understanding Comics p70 panel-to-panel transitions moment to moment Same object(s), different time points. Perhaps made redundant if actually use time to do this, but may be something you would want to do if you want to slow time down. action to action Frames from a film laid out side to side, some narrative progression. Same object(s) en scene. subject to subject Part of same scene, but objects change. Reader needs to associate frames together. scene to scene end of one scene, beginning of next. With hypermedia can make of use of slow fade out and then fade in transitions. aspect to aspect Some build up is being made of a place, or a mood. E.g. christmas (tree, Santa Claus), summer (sun, sunglasses, clear sky) drunken party (empty beer cans & cigarette ends, drunken guy + fed-up girl) time can seem to stand still non sequitur Exactly what we need to avoid! But even with non sequitur, frames displayed next to each other will acquire some sort of association, even if it were not intentional. (See weirdo sequence p73.) 15% of transitions are scene to scene, meaning that a scene consists of about 6 panels (the comic was 95 panels long). In Japanese comics there are more instances of moment-to-moment and aspect-to-aspect, because in the East "being there" wins over "getting there" in the West. And the point that a lot can happen in the empty space between panels: the blood in the gutter.... [ Do all these relate only to narrative as a discourse structure? Can they be used with logical argument (rhetoric?)? ] p94 time frames The shape of the panels. [Are we bound to rectangles? All the low-level window routines are (were?). Can we ignore this? Does it help to have strange shapes?] Tall and thin, Different widths. Panels in panels. [Going back to the Williams book, the panel borders just may be invisible, i.e. all you retain is alignment.] Width of guttering - wider implies more time has passed? Left to right is past to present to future. (Western only...) With space, past, present and future can all be seen. With time (e.g. film, real life) you only ever get to see the present. [So what would make it useful to see different times at once? There is a trade-off between having more white space (Williams likes white space, what about Tufte - he hated _wasted_ space, but I don't remember whether he liked white space) and having more info. Williams implied that you can keep same amount of info and create useful white space.] Same object superimposed over different parts of same background. Then background divided into panels. (p115) But already on p95, time moves left to right across the panel. P 100, time and space are the same. [In multimedia, space can be used for time, but time can also be used for space - Lloyd's/Maja's FIETS matrix.] Summary (p101): time can be controlled through content of panel, number of panels (repetition implies longer), closure between panels, width of panel. Content of panel can run out of bounds of panel and occupy more space, and hence time - timelessness. (p103) p118 Chapter 5 Living in Line. Basic Design and Layout Alan Swann 1987 This book emphasizes the creativity of the designer and tries to get the reader to explore the dimensions of the design space. A completely different approach from Robin Williams, and for ou purposes less useful (since very little is made explicit). Having said this, it is useful to look at the different illustrations and state whether they could have been automatically generated or not. Many not because they depend on a non-rectangular image and then positioning text around the image. p58 New York poster Heading above flag, secondary text nestled in between statue and flag. Perhaps with a colour analysis of the image you could try things like this. p60 Photo of cut apple Actually making variations in content of image. Fuzzy focus around edge (could be done by software), multiple images of apple, black and white, sepia-tinted b&w and vignetted in a circle. If a certain style has been chosen then could apply some of these techniques to a whole series of images, making the images themselves more part of the same design. p62 United shines to Florida Large, bold font, striking colours in picture. A colour analysis could tell us that the colours are bold. Texture analysis that the image is "striking". Barysh-nikov in Russia Photo of dancer placed under large classic lettering. Emphasis on style - lettering and stylish photo. Fisher-Price reinvents the wheels 3 different images with text wrapped round them and heading printed on top. I think this style has become dated. Was fashionable because new computer-based tools allowed it. I think current fashion is much more aligned (a la Robin Williams). This is good for us. p64 Lots about calculating margins and columns and golden section. Perhaps useful later if we can't produce good looking grids. p66 Use of large square in one colour and inner square with differing colour. Could use this for unifying a series of images with different colours, although need to watch out for different colour "maps". Decoration is used at the border between the two. (Dutch) total design cover mixing text and lines and letters and non-horizontal positioning. No way we could do this.... (black, blue, white, yellow) p68 Two image-dominated posters. Some IBM product boxes with same background colour and grid layout. Subtle different uses of colours. Same fonts, same layout grid. Perfect example of what we could achieve - and just as boring. Not quite actually, the colours of the images are all in the same style - bright, striking patterns. p71 chocolate boxes Shape, font, layout identical. The colours of the boxes vary. However, they obviously all belong to the same set. Warm colours. Are there existing collections of colours that fit a style? Microsoft windows has collections for different desktop settings. I could do some collecting from the book. p72 colours for cheese advert - brown, blue, grey/green (soft colours). Maybe some position of saturation and brightness is what I'm looking for? And just the hue is varied? Crisp packets - striking green, red, yellow, blue ("standard" kids colours) I think they are highly saturated. Nice stedelijk museum ad for '60 '80 attitudes|concepts|images Two different alignments, set slightly askew of each other, alongside an image of a triangle. Book cover, left aligned fonts, one line, white and blue text. No image. p74&76 Letters are included in image also image is included in a letter ("O" in "HORST"). Tricky for us to do, unless we select from template-like solutions. Same font, same background but different pictures still looks very much part of the same product. (lamp boxes) Surround text block with a number of images. p78 Cut out block within image and place text within it. Have image as minor component and use text as if it were an image. (Cavalry book title page.) p80 Have text border follow external or internal image border. (giraffe neck) Easy if you know where the light/dark boundaries are in the image. Letters placed on manuscript as if they were notes. We couldn't do this. Align text on predefined shape. E.g. around a circle, and put in circular image. p88 Allow image to go through text, breaking all "alignment rules". Collect two or three "templates" together as part of style. Section title page, main body page, in between page. See FitToWeb flier. Align edge of text externally with internal border in image. (Left edge text aligned with border between 2 hands.) Different colour background under different blocks. p90 Have everything, but one thing, aligned. In other words, unalign deliberately. (Skipped the fonts part.) p100 Colour Somehow collect the colours that go together on these pages. HSV values using gnome-terminal color selector. "COLOR" 5: ( 37, 0.61, 0.93) ( 20, 0.61, 0.93) (260, 0.34, 0.83) (202, 0.79, 0.85) (178, 0.43, 0.68) Shampoo bottles ( 36, 0.36, 0.43) (119, 0.46, 0.58) ( 40, 0.56, 0.86) 7 boxes on p71 Ugly NOVIB colours! To be continued from p 102? =====