============================= = K-Space Technical Meeting = = Athens, 27-28 April 2006 = ============================= Authors: George, Raphael # participants: 33 CWI:George and Raphael General impression: useful and succesful meeting, (mostly) fruitful discussions, cleared some technical issues. Produced a lot of Action Points. Led to better partner integration via face-to-face dicsussions. But also made evident that for many areas there is lack of expertise, at this point at least. There were times that only 1-2 people out of 10-15 would have opinions/suggestions about specific issues! INS2-point of view: i think we contributed substantially. I also think that due to that we may have more Action Points than would be contractually proportional! But not something that cannot be handled, probably.. WP4.1 and WP5.1: cross-discussion --------------------------------- 1. The MM ontology and MPEG-7: We decide to model in RDF+OWL: - Structural descriptors: SegmentType branch of MPEG-7 - Audio + Visual parts ? wait for the output of the WP3-WP4 meeting - Open issue for the textual descriptors ? - What about the administrative part (creator ? author ? media provenance ? rights ?) => equivalent to Dublin Core, VRA ? - What about the Semantic Part ? Translate the 5 top concepts (Agent, Object, Place, Event, ...) The ontology will be manually created ... How to deal with the semantic part (of MPEG7)? - Take the semantic concepts from MPEG7 and link them to an upper ontology (e.g. DOLCE) in the K-Space ontology. This way there can be a link to the original MPEG7 descriptor. - Do 1-1 mapping of concepts - Map MPEG7 descriptors to concepts. Link concepts to existing ontology concepts (equivalent). Some ontologies based on MPEG7 already exist (Hunter, Garcia). Garcia is fully automatic, so this will need reengineering. However, parts of existing ontology could maybe be reused. This is really an engineering task, so all approaches (some manual, some automatic) should be combined. We go for a very modular approach and develop independant islands of the ontology 2. Other features / language extension Fuzzy extensions: - Requirements survey - How to integrate with plain RDF/OWL - Which WP does the reasoning ? Datatypes: - What shall be done with datatypes Structuring of Metadata: - Referencing part of multimedia document - Describing the spatio-temporal structure of multimedia documents - Describing the logical structure of multimedia documents Other features not yet considered: - Implement the Named Graph extension 3. Domain ontologies - Place holder in the MM ontology - Link with the structure module of the ontology (how to reference part of multimedia material) - it would be a good idea to pick a domain ontology as a test bed early on, in order to try in practice the link/use of domain ontologies. Preferably on a domain on which we have test data (football domain chosen) 4. The annotation tool - Annotate documents versus find prototypes for concepts (populate the ontology) - Base the annotation on a-box (use some specific instances) or t-box of the ontology? - Do not adopt a methodology that relies on complex datatypes, unless we ensure it can be supported. - The annotation tool should be able to: . plugin extraction tool to populate instances of the ontology . allow users manual annotation - Unclear to which WP this falls to. Probably WP6, which builds the software framework architecture. - Annotation tool will have to: . integrate different feature analysis tools. . be integrated in the general framework - be able to plug in a manual annotation tool/interface in this architecture afterwards - Specify functionality of the tool. . Survey (maybe with WP3 people) different extraction tools, in order to find common features. . Maybe one way to go would be to use tool output in MPEG7 as input for the annotation tool. 5. Conclusion / Action Points - MM Ontology modules: . module 0: top level / integration => KU, CWI, UEP . module 1: visual descriptors from MPEG-7 => KU, ITI . module 2: audio descriptors from MPEG-7 => KU, ? . module 6: linguistic descriptors (text / audio) => DFKI, ? . module 3: multimedia descriptors from MPEG-7 => split into the different modalities ... see the SegmentType branch => CWI . module 4: structuring the MM content => KU, ITI, CWI, JRS . module 5: administrative metadata => CWI, ITI, UEP => about the multimedia material (author, title, see Dublin Core + VRA) => about the provenance of the metadata (metadata about the metadata) . module 7: semantic module (Agent, Object, Event, Place) => CWI, KU => linking with the domain ontologies - Annotation tool: . Functional study of manual annotation tool => ITI, CWI, JRS (2006/06/15) => includes a survey of the existing tools + what functionalities are desired . Decision of what functionalities should be implemented in a short term period (2006/06/30) => choice of an architecture (re-using or not existing tools) to have them WP5.2 and WP5.3: cross-discussion --------------------------------- 1. Data formats - Decide whether to support XML ? - JRS reponsible for WP3.4, which provides the output format for the whole WP3 in MPEG-7 ! - There is an initiative to make an MPEG-7 query language (CBS = Content-based Search) ... - Choice is to support RDF store. WP3 creates MPEG7 metadata, this is mapped to the ontology created in WP4 and stored in the RDF store provided by WP5. Mapping to be discussed in WP4, as this is identified as a WP4 issue. 2. APIs - What do you expect from the API ? Who will use the API ? - Users: . WP3: . Browse domain ontologies . [3.3 Automatic speech recognition using complementary sources], GET, DFKI, TUB . End user (Semantic based interaction): . Through querying tool . Navigation through metadata . Querying requirements of the End User . INA + CWI to provide real world queries and draw some scenarios 2006/05/20 . ALL: decide which scenarios should be adressed 2006/05/30 . WP 4 Reasoning . Plugin some reasoners . Annotation tool Action Point: KU start up cross WP discussion on API, decision by 2006-06-30 3. Reasoning Support - To which extent shall the repository support reasoning? . RDF(S)/OWL . Language extensions . Fuzzy => ITI claims to have developped a fuzzy reasoner . datatype extensions => see OWL 1.1 ? 4. Distributed store - For the 2nd part of the project (Month 18-36), the task should deal with distributed repositories ... too early. However consensus at this point is that we should target p2p repositories. Possible candidate: Sesame/SWAP. WP4 Technical meeting --------------------- 1. Task 4.1 - Development of the multimedia ontology - Development of an annotation tool Multimedia Ontology: (8 modules) - Module 0: upper ontology [UEP responsible] . give input about the various modeling model: VRA ... FRBR ... to UEP! 2006/06/30 . give input about the existing upper ontologies (DOLCE, SUMO, Cyc, ??) 2006/07/15 - Module 1: visual descriptors [ITI responsible] . determine set of needed MPEG-7 descriptors 2006/06/30 . start from the VDO ontology 2006/09/31 - Module 2: audio descriptors [?? responsible] . contact TUB regarding collaboration on audio descriptors . DFKI should take care of the speech part . implementation of the ontology 2006/09/31 - Module 3: linguistic descriptors [DFKI responsible] . implementation of the ontology 2006/09/31 - Module 4: multimedia descriptors [CWI, KU] . determine set of needed MPEG-7 descriptors 2006/06/30 . implementation of the ontology 2006/09/31 - Module 5: structure [KU, JRS, CWI, ITI] . links between the media parts / sructural part 2006/06/30 . implementation of the ontology 2006/09/31 - Module 6: administrative [UEP responsible] . reuse part of MPEG-7 + other ? 2006/06/30 . implementation of the ontology 2006/09/31 - Module 7: semantics [KU responsible] . reuse part of MPEG-7 + other ? 2006/06/30 . implementation of the ontology 2006/09/31 Annotation tool: - DFKI provides example data + domain for annotation 2006/05/31 - ... 2. Task 4.2 AP: investigate correlation between T4.5 and T4.2 (QMUL, ITI, CWI – 30/6) . T4.5 manually correct bad initial automatic annotations 3. Task 4.3 CWI is not involved 4. Task 4.4 Mining is an under-developed skill for most partners, so not much collaboration or initial contribution was identified. Candidate applications: - Mining from image collections . Corel - Mining data from trec vid collection . Mostly news (Arabic, Chinese, …) . How to link them - BBC rushes collection . Mining for similar patterns - Mining football video – DFKI collection . Audio info to identify events . Video info to identify events . One player in various actions . Similar patterns of games - Personal collections Not really our area of expertise, but personal collection mining sounds interesting (to me, at least :)) - Issues to decide: . Choose applications (end of may) . Feature extractors (audio, video, text) . Mining techniques . Mining system AP: CWI to clarify contribution (May 15th) AP: QU to send draft of SotA report AP: ALL select applications to pursue 4. Task 4.5 QMUL is already working on a web-based application in this field. It is an image classifier, based on biologically inspired image classification algorithms and incorporating user relevance feedback module. That sounds interesting. Note: QU has also done similar (user relevance feedback) work, so this was identified as a potential collaboration. Technologies used: Apache, JSP, Struts, MySQL Action Points: - Initial infrastructure setup (QMUL) 22/5 - Initial gui implementation (QMUL) - Integration of biologically inspired image classification algorithms (??) - Implementation and integration of SVM classifier (QMUL GU CWI DCU) - Partner feedback (ALL) - Implementing the feedback from partners and final system available for demonstration (QMUL): 30/10/2006 - CWI to clarify contribution 12/5 - ALL provide input for the SotA 15/5 The goal would be to have an initial implementation of techniques for intelligent image classification and relevance feedback and a demonstrator for the pre-review meeting (M11) CWI contribution (unofficially): Interface (presentation) for end of August WP5 Technical meeting --------------------- Language extensions Uncertainty ITI: fuzzy ontology and rule language . Extend ontology language (which) . Extend rule language (which) ITI working on DL . Extend ontology language (owl-lite) . Extend rule language (swrl) . AP: ITI, UEP 2 decide on collaboration on reasoning (end of May) UEP: overview of approaches . Fuzzy rdf(s) . Fuzzy owl . BayesOWL . PR-OWL . Proposal: do a feature survey Criteria: expressiveness, decidability, implementations, complexity, reasoning support, integration with existing languages Expected outcome: overview of features of language, use cases, recommendation of approach + extensions In case of W3C participation, possibility to join e.g. OWL 1.1, other groups End of year: OWL-Lite (AP ITI) End of next year: OWL-DL Structuring . KU: provide easy access to meta data using a kind of nested named graphs . AP CWI pointer to Sesame doc . AP KU to revise the Sesame approach and decide on how to continue work on nested named graphs. . AP KU, ITI to decide on collaboration on provenance (to WP4) CWI – referencing part of mm document (to WP4) CWI – spatio-temporal description of mm document structure (to WP4) CWI – logical description of mm document structure Ontology linking (to WP4) CWI: description of mm documents using several ontologies Datatypes Allow for better reasoning with datatypes in OWL ITI status . Knowledge Web deliverable Jeff Pan. We may refer to these results . AP KU, ITI dealing with MPEG7 datatype extensions, reasoning support (esp. vectors). Mid July . JRS: create link between MPEG7 and OWL . Postponed for year 2: query format for mm Rules AP UEP Report analogous to uncertainty Metadata store - AP Finalize survey – end of May. Work on the survey Semantics-based interaction with MM During a manual annotation phase - Browse / access semantic metadata repo (start discussion with JRS) During search phase - How to display the result of a given phase Raphael presented Alia + Lynda use case (feature for layout design). Chance for collaboration with WP3 (feature analysis), WP4 Possible collaboration – INA + DFKI + CWI - Align semantically rich metadata to a large corpus of videos - Provide pre-structuring / navigation part in a collection Workplan Initial use cases were due 20/4 Clarify and expand these use cases - What else? Opinions of other partners - Discussion on mailing list - Expectation: have a set of concrete scenarios by 30/6 Realization - Feasibility to have one scenario implemented for the pre-review (M11) - Image search interface with QMUL? Keyword based query vs. structured query - Ontology browser / KB browser - Faceted browser? Use more semantics Issues Implementation requires: - Mm data - Semantically annotated (annotation tool) - Properly stored (storage) - Queryable (query language) - Dependencies with other WP Deliverables - Report specification of Semantic Retrieval (M12) Milestones - First implementation and evaluation of Semantic Retrieval (M12) AP CWI to contact DCU 2 clarify DCU use case (8/5)