Sweep and Point & Shoot: Phonecam-Based
Interactions for Large Public Displays – Ballagas, Rohs, Sheridan, 2005


This article explores two different methods of interacting with large public displays. The first method involves tagging each area on the display with a visual code (something similar to QR codes), which they call “point and shoot”. The other method developed, which they dubbed “sweep”, uses the mobile phone’s camera as an optical tracking device. The authors then conducted a within-subjects experiment comparing the two new methods with a traditional 5-way joystick found on the Nokia 6600. 10 subjects were used (7 right-handed, 6 male, 5 were 26-35 years of age, 4 were 17-25 years of age, 1 over 45 years of age, 6 were from the UK, 9 were from Europe, 1 non-European).


The display is divided into different areas, with each area given a visual code, which is initially hidden from the screen. Users select the desired item/area by photographing the desired area of the screen. The codes appear when the user initiates a photo capture on the cell phone (IE. pressing the shutter button); once a successful selection is confirmed by the system, the codes are hidden again.


  • the coding system is designed such that the coordinate system is “invariant to perspective distortion”.
  • Tests show that P&S is not significantly different from using just the joystick to control a cursor
  • visual tags can be on real-world objects as well


  • flashing the coordinate on the screen whenever a user wishes to select an object makes the system visually unappealing, and currently does not easily scale to support multiple users.
  • testing shows that P&S had a significantly higher rate of error than both the joystick and “Sweep” method
  • current implementation did not have good “anti-shaking” camera system, so only people with fairly steady hands can accurately use the system
  • users must utilize the phone’s screen as a viewfinder


The Sweep method also uses the phone’s camera, but instead of requiring high-resolution images to accurately capture the visual codes, Sweep uses the camera to simply detect motion by sampling successive images, as if the phone was an optical mouse and the entire world was a mousepad ( PRETTY COOL – perry). In their implementation, the system only tracks motion when the 5-way joystick is held down – their idea being that releasing the joystick is analogous to taking the mouse off the mousepad. They characterize the joystick as a “clutch”.


  • Sweep is only concerned with relative motion, so it works in 3-dimensions, and can point anywhere in the users’ surroundings
  • ability to use it in 3-D also reduces fatigue over long-term use
  • users don’t have to look at the phone’s screen, but instead focus on the cursor on the public display


  • high latency (if the phone is slow)
  • significantly higher time to task completion than both P&S and joystick methods
  • Student Newman-Keuls analysis showed that error rate is significantly higher than joystick method

Overall findings and observations:

  • one of the users commented that one-handed operation of the Nokia 6600 was like “trying to hold onto a bar of soap”
  • fatigue we not an important factor, but several subjects complained of painful thumbs
  • users rated P&S higher than Sweep in every category of the survey: Overall operation, reliability, response time, and physical effort required for operation
  • screen size matters when choosing between Point-and-Shoot and Sweep: P&S good for when they are close to the display, Sweep otherwise
  • P&S might be scalable to multiple users if the codes were displayed in a non-visible spectrum such as infrared

Remarks on/related to article:

  • would thumbs hurt as much if they reversed the usage for Sweep – press down on the thumbstick only when they want to pause the system? or would that be too unintuitive?
  • use of QR codes – can we use QR codes in our registration system for user-initiated bluetooth pairings? the display can embed info like MAC address and a randomly-generated pairing code
  • how would Sweep work if the surrounding area as mostly uniform? or are they relying on low framerates of phone cameras to generate motion blurs?
  • the Nokia 660 is a GIGANTIC phone…the test results might be very different if they had used a Razr instead