Smartphone Interaction with Registered Displays (Touch Projection)

Nick Pears, Patrick Olivier, Dan Jackson, Leigh Herbert

Home | PhD applicants | Research | Publications | Google Scholar | Teaching | Bio

In this project, we use a camera-equipped mobile phone to view a PC screen (or public display) and we can control many interactions between the two devices. The essence of the idea is that, after estimating the planar projective mapping (homography) between the image on a mobile and a (possibly distant) display, any 'click' on the mobile's touch screen can be projected onto the correct corresponding point on the distant display. Of course, a wireless communication link is required so that the mobile can send its image to the display's PC, along with the point 'clicked' on its touch screen. Our idea has been reportred in a 'New Scientist' article in January 2008. This report made the top ten articles in the ACM TechNews alert, and it has been reported in many press articles and on many websites around the world.

More recent work by Boring et al (CHI 2010) has coined the term Touch Projection for this idea, as a user's touch on the mobile's screen is projected to another device's screen, wherever the particular interaction is occuring (see Touch projector website).

Video demonstrations


The key publication is number 2 below. Paper 1 describes a new markerless registration scheme, based on scale invariant features such as SURF and SIFT. Paper 3 is the original publication on the `display registration' idea.

  1. Smartphone Interaction with Registered Displays
      N. E. Pears, P. Olivier and D. Jackson.
      IEEE Pervasive Computing, vol 8, no 2 (April-June 2009), pp 14-21.
      Special issue on Smarter Phones.
      [PDF] [DOI][BibTeX]

  2. Display Registration for Device Interaction
      N. E. Pears, P. L. Olivier and D. Jackson
      3rd Int. Conf. on Computer Vision Theory and Applications 2008 (VISAPP'08), 22 - 25 March, 2008, Funchal, Madeira, Portugal.
      [PDF] [BibTeX]

  3. Mobile Device and Intelligent Display Interaction via Scale-Invariant Image Feature Matching.
      L. Herbert, N. E. Pears, D. Jackson and P. Olivier
      1st Int. Conf. Pervasive and Embedded Computing and Communication Systems (PECCS 2011)
      [PDF] [BibTeX]

BACK to Nick Pears' Research Projects page.