3D Local Shape Descriptors

Nick Pears, Marcelo Romero, Tom Heseltine

Home | PhD applicants | Research | Publications | Google Scholar | Teaching | Bio

Reliable localisation of facial landmarks is an essential precursor to many face processing algorithms, such as face recognition. In our work, we have devised both new descriptors and new techniques for 3D facial landmarking and 3D facial pose normalisation.

Examples of our new surface descriptors.

In the figure below left, a colour-mapped radial basis function (RBF) respresents a 'signed distance to surface' (blue is inside the head, red outside). The zero-isosurface representing the facial surface is coloured turquoise. The yellow points on a sphere represent a spherical sampling of the RBF function, where the sphere centre is at a point on the facial surface. A 2D histogram of RBF signed 'distance to surface' (DTS) values, for a range of sphere radii, forms a pose invariant surface descriptor that is robust to missing data, due to the natural interpolation properties of the RBF function. (Since it is defined everywhere in 3D, irrespective of missing data). We call this descriptor a spherically sampled RBF (SSR) histogram or, colloquially, a 'balloon image', as shown below right, since one can imagine a spherical balloon being inflated in size and periodically sampling the RBF function as it does so. We have employed this descriptor to reliably locate the facial nose-tip.

In the figure below, a set of space curves are shown, where a set of concentric spheres centered on the nose-tip intersects with the facial surface (the zero isosurface of the RBF function). We call these 'isoradius contours' as every point on a given contour is the same Euclidean distance from the nose tip. From any contour, we can extract a curvature signal, which represents how the tangential curvature changes as we travel around the nose-tip on a sphere. Irrespective of head pose, the curvature signals are very similar, modulo a phase change. Hence correlation of curvature signals across a pair of different poses can recover the relative pose of a subject's head. The curvature signals for a variety of subject head poses are shown in the figure below right.

In the figure below left, we have the axis of a set of concentric cylinders aligned with the nose tip and inner eye corner. At these sample points, we evaluate the RBF 'signed distance to surface' function and we can accumulate these values into a histogram, as shown below right. We call this a 'cylindrically sampled RBF' (CSR) descriptor. Such descriptors can be used as attributes for edges in graph matching approaches.

Suppose we are trying to verify the nose-tip and inner-eye corners, as a triplet of facial landmarks. In this case, we use a sampling pattern based on the barycenters of triangular subdivisions of the patch, as shown in the figure below left. The RBF function is sampled at a variety of heights about and below the triplet's plane. The RBF 'distance to surface' (DTS) values can be constructed into a feature vector, if the values are scanned out in some particular order, as shown by the numbering in the figure. The data can also be used in schemes in which the point triplet is unoriented and, in such cases, we can accumulate DTS values into a pose-invariant shape histogram, as shown in the figure, below right. These 'point triplet' ideas are presented in detail in Marcelo Romero's PhD thesis (published 2010).

Selected publications.

  1. From 3D Point Clouds to Pose-Normalised Depth Maps
      N. E. Pears, T. Heseltine and M. Romero.
      Int. Journ. Computer Vision, Volume 89, Issue 2 (2010), Pages 152-176.
      Special issue on 3D object retrieval.
      [PDF] [DOI][BibTeX]

  2. Point-Pair Descriptors for 3D Facial Landmark Localisation
      M. Romero and N. E. Pears
      Proc 3rd IEEE Int. Conf on Biometrics: Theory, Applications and Systems 2009 (BTAS'09), Washington, USA.
      [PDF] [DOI][BibTeX]

  3. RBF Shape Histograms and their Application to 3D Face Processing
      N. E. Pears
      Proc. 8th IEEE Int. Conf. on Automatic Face and Gesture Recognition (FG'08)', September 17-19, 2008 - Amsterdam, The Netherlands.
      [PDF] [DOI][BibTeX]

  4. Isoradius Contours: New Representations and Techniques for 3D Face Matching and Registration
      N. E. Pears and T. Heseltine
      Proc. 3rd Int. Symposium on 3D Data Processing, Visualization and Transmission (3DPVT'06), pp 176-183, Chapel Hill, NC, USA
      [PDF] [DOI][BibTeX]

BACK to Nick Pears' Research Projects page.