Bossa: Extended BoW Formalism for Image Classification
In image classification, the most powerful statistical learning approaches are based on the Bag-of-Words paradigm. In this article, we propose an extension of this formalism. Considering the Bag-of-features, dictionary coding and pooling steps, we propose to focus on the pooling step. Instead of using the classical sum or max pooling strategies, we introduced a density function-based pooling strategy. This flexible formalism allows us to better represent the links between dictionary codewords and local descriptors in the resulting image signature. We evaluate our approach in two very challenging tasks of video and image classification, involving very high level semantic categories with large and nuanced visual diversity.

I’ve also had two papers accepted on our counterpart national conference, SIBGRAPI. The first is the work of the Ph.D. student Ana Lopes and her R.A. Elerson Santos (supervised by Prof. Arnaldo Araújo and co-supervised by Prof. Jussara Almeida; I give her some technical and nontechnical support every now and then). It concerns the use of transfer learning of concepts from (static) image datasets to video datasets in order to recognize human actions. We show that learning the concepts present on the Caltech256 dataset allow a classifier to obtain improved results on the challenging “in the wild” human action Hollywood2 dataset.

Transfer Learning for Human Action Recognition
To manually collect action samples from realistic videos is a time-consuming and error-prone task. This is a serious bottleneck to research related to video understanding, since the large intra-class variations of such videos demand training sets large enough to properly encompass those variations. Most authors dealing with this issue rely on (semi-) automated procedures to collect additional, generally noisy, examples. In this paper, we exploit a different approach, based on a Transfer Learning (TL) technique, to address the target task of action recognition. More specifically, we propose a framework that transfers the knowledge about concepts from a previously labeled still image database to the target action video database. It is assumed that, once identified in the target action database, these concepts provide some contextual clues to the action classifier. Our experiments with Caltech256 and Hollywood2 databases indicate: a) the feasibility of successfully using transfer learningtechniques to detect concepts and, b) that it is indeed possible to enhance action recognition with the transferred knowledge of even a few concepts. In our case, only four concepts were enough to obtain statistically significant improvements for most actions.

The second is the work of my Ph.D. student Marcelo Coelho and his R.A. Cássio dos Santos Jr. (again his main supervisor is Prof. Arnaldo de Araújo). It concerns the clean-up of noisy SIFT features of street-view images (urban façades). We have found out that subspace clustering, a non-supervised technique, is able to isolate clusters of useful and non-useful SIFT features for the task of retrieving a target image. The challenge is identifying a priori which cluster is the relevant one. This work compares and contrasts two subspace clustering techniques: FINDIT (based on dimension voting) and MSSC (based on a fuzzy mean-shift).

Subspace Clustering for Information Retrieval in Urban Scene Databases
We present a comprehensive study of two important subspace clustering algorithms and their contribution to enhance results from the difficult task of matching images taken of the same object using different devices at different conditions. Our experiments were done on two distinct databases containing urban scenes which were tested using state-of-the-art matching algorithms. After initial evaluation of both datasets by that procedure, clustering algorithms were applied to them. An exhaustive comparison was performed in every cluster found and a significant amelioration in the results was obtained.

I’ll put a link to the preprints as soon as I have they become available.

We propose MONORAIL, an indexing scheme for very large multimedia descriptor databases. Our index is based on the Hilbert curve, which is able to map the high-dimensional space of those descriptors to a single dimension. Instead of using several curves to mitigate boundary effects, we use a single curve with several surrogate points for each descriptor. Thus, we are able to reduce the random accesses to the bare minimum. In a rigorous empirical comparison with another method based on multiple surrogates, ours shows a significant improvement, due to our careful choice of the surrogate points.

I am particularly proud of this paper, not only because of the method itself, but also because of the experimental design we propose for the validation. I have been studying for more than a year the topics of Design of Experiments, statistical tests and validation. This is the first of a crop of publications that are employing those rigorous evaluation tools, which, though commonplace in other fields, are still seldom used in Computer Sciences.

Here is the abstract:

“In Content-based Image Retrieval (CBIR), accurately ranking the returned images is of paramount importance, since users consider mostly the topmost results. The typical ranking strategy used by many CBIR systems is to employ image content descriptors, so that returned images that are most similar to the query image are placed higher in the rank. While this strategy is well accepted and widely used, improved results may be obtained by combining multiple image descriptors. In this paper we explore this idea, and introduce algorithms that learn to combine information coming from different descriptors. The proposed learning to rank algorithms are based on three diverse learning techniques: Support Vector Machines (CBIR-SVM), Genetic Programming (CBIR-GP), and Association Rules (CBIR-AR). Eighteen image content descriptors (color, texture, and shape information) are used as input and provided as training to the learning algorithms. We performed a systematic evaluation involving two complex and heterogeneous image databases (Corel e Caltech) and two evaluation measures (Precision and MAP). The empirical results show that all learning algorithms provide significant gains when compared to the typical ranking strategy  in which descriptors are used in isolation. We concluded that, in general, CBIR-AR and CBIR-GP outperforms CBIR-SVM. A fine-grained analysis revealed the lack of correlation between the results provided by CBIR-AR and the results provided by the other two algorithms, which indicates the opportunity of an advantageous hybrid approach.”

I will be travelling to Philadelphia on late March to present the poster. I am very excited about this upcoming trip to the United States, where I am to meet several friends and colleagues, but at the same time, worried about the radicalization of air security rules and the exaggeration of perception of threats. Have we got so scared to die that we decided instead not to live ?

Thematically, the conference has a broad scope, centered around the representation, processing, analysis, storage and retrieval of documents. My main research topic concerns the retrieval of multimedia documents, and is somewhat at the fringe of the conference theme. Nevertheless, people seemed genuinely interested and I’ve got many useful insights and suggestions.

* * *

I have just arrived at Paris, where I will meet my former Ph.D. supervisor Prof. Matthieu Cord, among other colleagues. I intend to advance our research on high-dimensional multimedia indexing and large scale multimedia retrieval. I am also giving a talk about my current research pursuits at the ETIS labs, on Cergy-Pontoise, next Tuesday, September 22^nd.

If you use Google Calendar you can save the date by clicking below:

Searching by similarity is a critical operation on many systems, and thus has attracted the attention of many disciplines in Computer Sciences, including Computational Geometry, Machine Learning, Multimedia and, of course, Databases. To perform efficiently, similarity search requires the support of indexing, which suffers from the infamous “curse of the dimensionality”. In this tutorial we will introduce the challenges of indexing and searching high-dimensional data, and present the most recent tools available to “tame the curse”. At the end, the audience will have a good grasp of the current state of the art, the most promising research trends and the challenges still faced by the technology.

The tutorials, as I understand, are open to all participants on the conference. Mine will be held on Wednesday, October 7th from 14h40 to 18h20, with a 20′ coffee-break. If you use Google calendar, you can save the date by clicking on the button below.

* * *

I’ve unintentionally let an awful lot of of time pass since my last post — the move to Campinas (and to UNICAMP) has been wonderful, but also laborious. I thought that after moving across countries three times, moving across states would be a piece of cake, but it seems that, no matter the distance, moving is always a lot of hassle!

EDIT 11/11/09: The tutorial presentation, for the moment without narrative, is available on my talks and courses page.

One, co-authored with my French partners Dr. David Picard and Prof. Matthieu Cord, is about the difficult problem of enforcing geometric consistency in vote-counting based CBIR when there are too many outliers — a situation we encounter routinely in our iTowns project. Here’s the title and abstract:

Geometric Consistency Checking for Local-Descriptor Based Document Retrieval — In this paper, we evaluate different geometric consistency schemes, which can be used in tandem with an efficient architecture, based on voting and local descriptors, to retrieve multimedia documents. In many contexts the geometric consistency enforcement is essential to boost the retrieval performance. Our empirical results show however, that geometric consistency alone is unable to guarantee high-quality results in databases that contain too many non-discriminating descriptors.

The other, co-authored with my Brazilian colleagues Flávio Bertholdo and Prof. Arnaldo Araújo, proposes a new method for contrast enhancement in degraded historical documents, which takes into account the structure of the the document:

Layout-Aware Limiarization for Readability Enhancement of Degraded Historical Documents — In this paper we propose a technique of limiarization (also known as thresholding or binarization) tailored to improve the readability of degraded historical documents. Limiarization is a simple image processing technique, which is employed in many complex tasks like image compression, object segmentation and character recognition. The technique also finds applications on itself: since it results in a high-contrast image, in which the foreground is clearly separated from the background, it can greatly improve the readability of a document, provided that other attributes (like character shape) do not suffer. Our technique exploits statistical characteristics of textual documents and applies both global and local thresholding. Under visual inspection on experiments made in a collection of severely degraded historical documents, it compares favorably with the state of the art.

DocEng 2009 will be held in Munich, Germany on September 15–18.

EDIT 23/07: The preprints are now available in my publications page.

Here’s the abstract:

“Local descriptors have been extensively used in CBIR systems, where their robustness to intense geometric and photometric transformations allows the identification of a target object/image with great reliability. However, due to their excessive discriminating power, their application to the retrieval of complex categories is challenging. The introduction of the technique of ‘visual dictionaries’ (also known as ‘dictionary of visual terms’) is an important step towards the conciliation between the robustness of local descriptors and the flexibility of generalization needed by complex queries. As a bonus, we become able to employ advanced retrieval techniques which were so far available only for textual data.

The importance of local descriptors in the current practice of CBIR can hardly be overestimated, and the introduction of the ‘visual dictionary’ technique has allowed to employ them on complex category search, a context where they were previously considered too discriminating to work effectively. In this tutorial we will address all the conceptual components of a powerful framework — the CBIR system architecture itself, the local descriptors and the techniques which allow their advanced application. At the end, the audience will have a good idea of the current state of art, the most promising research trends and the challenges still faced by the technology.”

The tutorial will take 3 hours and will be given either in English or Portuguese, according to the composition of the classroom.

EDIT 11/11/2009: The tutorial survey is available on my publications page.

Here’s the abstract:

“In this paper we address the subject of large multimedia database indexing for content-based retrieval. We introduce multicurves, a new scheme for indexing high-dimensional descriptors. This technique, based on the simultaneous use of moderate-dimensional space-filling curves, has as main advantages the ability to handle high-dimensional data (100 dimensions and over), to allow the easy maintenance of the indexes (inclusion and deletion of data), and to adapt well to secondary storage, thus providing scalability to huge databases (millions, or even thousands of millions of descriptors). We use multicurves to perform the approximate k nearest neighbors search with a very good compromise between precision and speed. The evaluation of multicurves, carried out on large databases, demonstrates that the strategy compares well to other up-to-date k nearest neighbor search strategies. We also test multicurves on the real-world application of image identification for cultural institutions. In this application, which requires the fast search of a large amount of local descriptors, multicurves allows a dramatic speed-up in comparison to the brute-force strategy of sequential search, without any noticeable precision loss.”

The paper and some related material can be found in my publications page.

Anyway, here’s the abstract: “In this paper we introduce a system for the identification of visual documents. Since it stems from content-based document indexing and retrieval, our system does not need to rely on textual annotations, watermarks or other metadata, which can be missing or incorrect. Our retrieval system is based on local descriptors, which have been shown to provide accurate and robust description. Because of the high computational costs associated to the matching of local descriptors, we propose Projection KD-Forest: an indexing technique which allows efficient approximate k nearest neighbors search. Experiments demonstrate that the Projection KD-Forest allows the system to provide prompt results with negligible loss on accuracy. The Projection KD-Forest also compares well when contrasted to other strategies of k nearest neighbors search.”

I guess that the dream of all conference organisers is to amass hundreds of participants and to have an acceptance rate as close to zero as possible. This edition of DocEng was nothing like that — I think that we were 60 or 70 counting everyone, authors and participants. Yet, in terms of exchange of scientific ideas, it was one of the best conferences I’ve ever participated.

The paper, and related material, can be found in my publications page.