Exiting the cleanroom: on ecological validity and ubiquitous computing is a paper on prototyping, observation, controlled evaluation and field experiments of Ubicomp. There are three class of ubicomp applications: peripheral displays, mobile applications, and applications that integrate physical and digital interactions. Idiosyncrasies with ubicomp applications are due to the need for applications to be in many places, do many tasks, work across many devices. The authors did fieldwork with 28 ubicomp developers to understand the problems in ubicomp.

Peripheral displays

  • Developers said that determining how study participants used the information in the peripheral display is a challenge.
  • Developers complained about having to decide between too many design options. (The paper does not explain why peripheral display developers specifically have too many choices)
  • There is a need for tools that allow applications to multiple outputs.
  • Unobtrusive evaluation is difficult if displays are deployed in real world like homes. Collecting quantitative data is difficult.
  • The best way to evaluate the success of a peripheral display is by conducting situated, long-term deployment.
  • Context of Use Evaluation of Peripheral Displays (CUEPD) method – captures context of use through user scenario building, enactment, and reflection.

Mobile Applications

  • Evaluation is difficult. Literatures examples show the difficulty in understanding how participants use their cell phones.
  • Prototypes have to developed in multiple platforms. Not all phones implement Java JSR spec completely.
  • Planning field studies involving mobile phones difficult due to many mobile operators, plans, devices.
  • In real world scenarios, users are constantly interrupted while using their mobile applications, this is hard to replicate in the lab.
  • Researchers have used diary studies and interviews to understand picture sharing usage.

Integrating physical and digital interactions

  • This subset refers to ubicomp applications that take the data from sensors like camera and process it.
  • Primary challenge is abstracting physical input.
  • Paper Prototyping used for exploration by developers.
  • Developers expressed need for better tools

Challenges in evaluating ubicomp

  • Ambiguity in sensed data and need to mitigate error
  • Sparse data. There are too many unique situations which are difficult to evaluate while prototyping
  • Critical mass: many applications need critical mass to be successful
  • Evaluating obstructively is difficult
  • Tools that can support realistic environments

Authors recommend the following techniques to overcome the problem. (I’m just listing them, the paper explains them in depth)

  1. Situated techniques
  2. Remote evaluators (privacy concerns??)
  3. Diary study
  4. Lightweight prototypes
    • Paper prototypes
    • Wizard of oz
    • Looks like