Master thesis proposal 2010-2011
Spatio Temporal Interest Points on Meshes


4Dfeature1
 4Dfeature2
 4Dfeature3
3D model of a dancer 3D model of the dancer at a different instant. Spatio-temporal trajectories.

Advisors

Edmond Boyer Franck Hétroy
LJK/PERCEPTION LJK/EVASION
E-mail: Edmond.Boyer@inrialpes.fr E-mail: Franck.Hetroy@inrialpes.fr
Tel. : 04 76 61 53 54 Tel. : 04 76 61 55 04

Context

Recent results obtained in computer vision with multiple camera systems allow to recover dense information on shapes and their motions. This ability to perceive shapes in motion brings a new and rich domain for research investigations on the analysis of moving objects, in particular human motions. In order to enable such analysis several challenges need to be faced. First relevant indices that describe shapes and their dynamic evolutions must be found and second compact representations must be build for analysis purposes. The purpose of this work is to particularly focus on the first challenge and on how to define features that encode both shape and motion information.

Objectives

Assume that a shape model: a 3D mesh, and its motion over time: the vertex displacement fields, are available. The objective is to identify particular features on the mesh both in space and time. Several works already exist that identify features on a static mesh based on purely geometric properties [NN07] or photometric and geometric properties [ZBVH09]. In a way similar to 2D features in images[Laptev05], these spatial interest points on meshes can be extended to spatio-temporal interest points by considering the time dimension within a temporal window. The interest is to detect the locations on an evolving mesh that well describe both shape and motion aspects.

The work will include the following tasks:

  1. Review works on 2D features[Lowe04], 3D mesh features [ZBVH09,NN07] and spatio-temporal interest points [Laptev05,NY10].
  2. Implement the extension of 2D spatio-temporal features to 3D.
  3. Thoroughly evaluate the performances of the local detector/descriptor with respect to: invariance to space scale and time scale, shape invariance, robustness, repeatability, among others. Use the resulsts to improve 4D features.
  4. Investigate comparisons and recognition of object motions based on the proposed descriptor.

Pre-requisites

Required skills include computer vision, linear algebra, numerical analysis. Good programming skills are mandatory.

Keywords: computer vision, segmention, image segmentation, multiple view segmentation.

References

  1. [ZBVH09] Surface Feature Detection and Description with Applications to Mesh Matching, Andrei Zaharescu, Edmond Boyer, Kiran Varanasi and Radu Horaud, in CVPR 2009.
  2. [NN07] Scale-dependent 3D geometric features. J. Novatnack and K. Nishino. In ICCV, 2007.
  3. [Lowe04] Distinctive image features from scale-invariant keypoints. D. G. Lowe. IJCV, 60(2):91~110, 2004.
  4. [Laptev05] On Space-Time Interest Points, Ivan Laptev. IJCV 64(2/3), 107~123, 2005.
  5. [NY10] A SURF-based Spatio-Temporal Feature for Feature-fusion-based Action Recognition, Akitsugu Noguchi, Keiji Yanai, Third Workshop on HUMAN MOTION Understanding, Modeling, Capture and Animation, ECCV 2010.