FoundIt Project

Foundit Project Overview

The explosive growth of digital multimedia repositories has created challenging new problems with regard to the indexing, access and retrieval of data. Given the sheer volume of stored data, enabling the user to access information based on its semantic content is rapidly becoming a vital necessity without which the search-process gets bogged down by an avalanche of irrelevant hits. Considerable progress notwithstanding, it is fair to say that content-based access is still in its infancy and fully-automated robust content-based retrieval remains an elusive goal.

These difficulties are particularly forbidding in the case of image-databases. Although in some instances the search for appropriate images can be conducted relying exclusively on keywords, there are a number of (economically) important applications where keywords are non-existent or of limited use and the search is primarily vision-driven. Examples that spring to mind are:

• Catalogues of merchandise with strong decorative component: Consumer goods such as clothes, interior decoration (wallpaper, tiles, flooring, drapes), furniture, etc., for which the selection is based on (among other things) visual appreciation.
• Vision-oriented expert systems:
  
  • Assisted diagnostics and identification: e.g. in microscopy, patho-histology, geology, dermatology, forensics, etc;
    
  • Automatic inspection: e.g. total quality-control for textiles, scratches or imperfections in paint-jobs, etc.
    
  • Monitoring copyright on designs: e.g. depositing trademarks, designs, etc.
    
• Video-abstracting: Automatic extraction of salient frames from video.

This non-exhaustive list illustrates the wide range of specialised digital libraries for which there is a pressing need for content-based image retrieval (CBIR) and this project aims to contribute significantly to the solution. Notice how our insistence on special purpose libraries (i.e. libraries that contain image-material of a more or less well-defined nature and pertaining to professional or commercial goals) is an important difference with other approaches that set themselves the more ambitious but, by necessity, fuzzier goal of locating unconstrained images (e.g. snapshots of "happy families walking on the beach") in an unconstrained medium (such as the internet). In the trade-off between domain-generality and retrieval-specificity we will focus our attention firmly on applications that live within a restricted domain but have nevertheless significant economic value (items in the above-quoted list are cases in point). Note furthermore that the envisaged end-users can be either professionals (e.g. designers accessing a library of designs to repurpose content), or lay-people (e.g. a modal customer accessing a digital catalogue within the context of E-commerce).

To understand our take on these issues, it is important to appreciate the precise nature of the difficulties we are facing. In particular, one has to realise that the non-textual part of visual content is encoded in terms of a (limited) set of numerical features. This means that retrieval heavily depends on (i) the choice of features used to represent an image in the abstract feature-space, and (ii) the metric in that feature-space that quantifies the visual (dis)similarity between two images in terms of their distance in feature-space. Finding a satisfactory answer to any of these questions is highly non-trivial for the following two reasons. First, there is no well-established or canonical feature-framework for encoding the visual information encapsulated in an image. (After all, how does one mathematically distinguish the perceptual or esthetic satisfaction evoked by a van Dyck and a Magritte?). In particular, different sets of features might yield different retrieval results. Second, the very same image can mean entirely different things to different people. In fact, one could argue that content - as its first cousin, beauty - is in the eye of the beholder as it is intricately tied up with the user's underlying intentions and goals. These will obviously differ from person to person, but may vary in time even for the same individual. So, in abstract terms this means that both the choice of features and the way in which they are combined in a metric may vary widely for different users and even over time.

All this underscores the fact that, at least with the current state of technology, it seems foolhardy to attempt full automation of such search procedures and the most successful CBIR search-engines are the ones that keep the human in the loop. By this one means that rather than trying to emulate human visual prowess -- a feat that would make it possible for the system to operate unsupervised -- it seems preferable to concentrate our efforts instead on designing intelligent interfaces that prompt the user for feedback and then try and capitalize on it to expedite the search-action. Within the context of relevance-driven CBIR, a typical image-retrieval session therefore proceeds by iterating the following basic steps:

1. Using some internal model of the user's preferences, the search-engine draws a representative sample from the database and presents this selection to the user for inspection;
   
2. Upon inspection the user provides some feedback, thereby indicating which images or features are particularly relevant with respect to his goals;
   
3. The system uses this feedback to update its internal model of the user's preferences and returns to step 1.
   

The questions we need to address are therefore: (i) what is an easy and transparent way to interact with an interface, and (ii) how can this information be used to make the search more reliable and efficient.

Overview of the project's main objectives:

1. To develop an CBIR-interface that allows the user to give relevance feedback in a natural and intuitively transparent fashion, thus making this technology more efficient for professional users, and opening it up to the much wider audience of non-expert mainstream users;
2. To develop an underlying inference-engine based on parametric forms of machine learning that can harness this qualitative feedback to construct probabilistic models that predict more accurately the relevance of each image. As a consequence the system is able to present the user, at each iteration step, with an updated and more relevant selection drawn from the database;
3. To demonstrate the viability of a search-engine that combines the interface and inference-engine on a number of challenging and economically important applications where fast and accurate CBIR represents significant added value. More specifically, we intend to focus on:
   • Textile design and retrieval for repurposing of design;
   • Stock management and planning for decorative designs (e.g. wallpaper);
   • E-commerce application : electronic catalogue for interior decorating.
4. To disseminate knowledge about the potential of state-of-the-art CBIR-technology to economic sectors (such as the textile or fashion industry) that are traditionally less IST-inclined but nevertheless stand to benefit considerably from a judicious assimilation of these recent developments.