We present the first real-world dataset and quantitative evaluation of visual attention of mobile device users in-situ, i.e. while they use their devices during everyday routine. Understanding user attention is a core research challenge in mobile HCI but previous approaches relied on usage logs or self-reports that are only proxies and consequently do not reflect attention completely nor accurately. Our evaluations are based on iContact, a new 32-participant dataset containing around 472 hours of video snippets recorded over more than two weeks in real life using the front-facing camera as well as associated usage logs, interaction events, and sensor data. Using a method for automatic eye contact detection, we are first to quantify the highly dynamic nature of everyday visual attention allocation across users, mobile applications, and usage contexts. We discuss key insights from our analyses that highlight the potential and inform the design of future mobile attentive user interfaces.

Quantification of human attention is key to several tasks in mobile human-computer interaction (HCI), such as predicting user interruptibility, estimating noticeability of user interface content, or measuring user engagement. Previous works to study mobile attentive behaviour required special-purpose eye tracking equipment or constrained users' mobility. We propose a novel method to sense and analyse visual attention on mobile devices during everyday interactions. We demonstrate the capabilities of our method on the sample task of eye contact detection that has recently attracted increasing research interest in mobile HCI. Our method builds on a state-of-the-art method for unsupervised eye contact detection and extends it to address challenges specific to mobile interactive scenarios. Through evaluation on two current datasets, we demonstrate significant performance improvements for eye contact detection across mobile devices, users, or environmental conditions. Moreover, we discuss how our method enables the calculation of additional attention metrics that, for the first time, enable researchers from different domains to study and quantify attention allocation during mobile interactions in the wild.

With an ever-increasing number of mobile devices competing for our attention, quantifying when, how often, or for how long users visually attend to their devices has emerged as a core challenge in mobile human-computer interaction. Encouraged by recent advances in automatic eye contact detection using machine learning and device-integrated cameras, we provide a fundamental investigation into the feasibility of quantifying visual attention during everyday mobile interactions. We identify core challenges and sources of errors associated with sensing attention on mobile devices in the wild, including the impact of face and eye visibility, the importance of robust head pose estimation, and the need for accurate gaze estimation. Based on this analysis, we propose future research directions and discuss how eye contact detection represents the foundation for exciting new applications towards next-generation pervasive attentive user interfaces.

Common calibration techniques for head-mounted eye trackers rely on markers or an additional person to assist with the procedure. This is a tedious process and may even hinder some practical applications. We propose a novel calibration technique which simplifies the initial calibration step for mobile scenarios. To collect the calibration samples, users only have to point with a finger to various locations in the scene. Our vision-based algorithm detects the users’ hand and fingertips which indicate the users’ point of interest. This eliminates the need for additional assistance or specialized markers. Our approach achieves comparable accuracy to similar markerbased calibration techniques and is the preferred method by users from our study. The implementation is openly available as a plugin for the open-source Pupil eye tracking platform.

When compared to image recognition, object detection is a much more challenging task because it requires the accurate real-time localization of an object in the target image. In interaction scenarios, this pipeline can be simplified by incorporating the users’ point of regard. Wearable eye trackers can estimate the gaze direction, but lack own processing capabilities. We enable mobile gaze-aware applications by developing an open-source platform which supports mobile eye tracking based on the Pupil headset and a smartphone running Android OS. Through our platform, we offer researchers and developers a rapid prototyping environment for gaze-enabled applications. We describe the concept, our current progress, and research implications.

Nowadays, humans are surrounded by many complex computer systems. When people interact among each other, they use multiple modalities including voice, body posture, hand gestures, facial expressions, or eye gaze. Currently, computers can only understand a small subset of these modalities, but such cues can be captured by an increasing number of wearable devices. This research aims to improve traditional human-human and human-machine interaction by augmenting humans with wearable technology and developing novel user interfaces. More specifically, (i) we investigate and develop systems that enable a group of people in close proximity to interact using in-air hand gestures and facilitate effortless information sharing. Additionally, we focus on (ii) eye gaze which can further enrich the interaction between humans and cyber-physical systems.

Many real-life scenarios can benefit from both physical proximity and natural gesture interaction. In this paper, we explore shared collocated interactions on unmodified wearable devices. We introduce an interaction technique which enables a small group of people to interact using natural gestures. The proximity of users and devices is detected through acoustic ranging using inaudible signals, while in-air hand gestures are recognized from three-axis accelerometers. The underlying wireless communication between the devices is handled over Bluetooth for scalability and extensibility. We present (1) an overview of the interaction technique and (2) an extensive evaluation using unmodified, off-the-shelf, mobile, and wearable devices which show the feasibility of the method. Finally, we demonstrate the resulting design space with three examples of multi-user application scenarios.

When people are introduced to each other, exchanging contact information happens either via smartphone interactions or via more traditional business cards. Crowded social events make it more challenging to keep track of all the new contacts. We introduce HandshakAR, a novel wearable augmented reality application that enables effortless sharing of digital information. When two people share the same greeting gesture (e.g., shaking hands) and are physically close to each other, their contact information is effortlessly exchanged. There is no instrumentation in the environment required, our approach works on the users' wearable devices. Physical proximity is detected via inaudible acoustic signals, hand gestures are recognized from motion sensors, the communication between devices is handled over Bluetooth, and contact information is displayed on smartglasses. We describe the concept, the design, and an implementation of our system on unmodifed wearable devices.

We describe ubiGaze, a novel wearable ubiquitous method to augment any real-world object with invisible messages through gaze gestures that lock the message into the object. This enables a context and location dependent messaging service, which users can utilize discreetly and effortlessly. Further, gaze gestures can be used as an authentication method, even when the augmented object is publicly known. We developed a prototype using two wearable devices: a Pupil eye tracker equipped with a scene camera and a Sony Smartwatch 3. The eye tracker follows the users’ gaze, the scene camera captures distinct features from the selected real-world object, and the smartwatch provides both input and output modalities for selecting and displaying messages. We describe the concept, design, and implementation of our real-world system. Finally, we discuss research implications and address future work.

Indoor localization is an important topic for context aware applications. In particular, many applications for wireless devices can benefit from knowing the location of a user. Despite the huge effort from the research community to solve the localization problem, there is no widely accepted solution for localization in an indoor environment. In this paper we focus on constrained devices and propose an extremely lightweight indoor localization system that can be scaled to different devices, from smart phones to smart glasses and other devices. We devise a simple yet effective WiFi-based system with low computational complexity, which does not need any additional special infrastructure nor map or an internet connection. Our system relies on IEEE 802.11 Received Signal Strength Indicator (RSSI) values and a dead reckoning module to collect walking trajectories which are further clustered and compressed to build a sensor map. The key novelty of our work is a merging algorithm that can fuse multiple sensor maps. We evaluate our system in a real world scenario and we show that using the map produced by our merging algorithm we achieve room-level accuracy. Our system is also comparable to state of the art systems, despite the lightweight approach.

This paper proposes a probabilistic model for automated reasoning for identifying the lane on which the vehicle is driving on. The solution is based on the visual information from an on-board stereo-vision camera and a priori information from an extended digital map. The visual perception system provides information about on-the-spot detected lateral landmarks, as well as about other important traffic elements such as other vehicles. The proposed extended digital map provides lane level detail information about the road infrastructure. An Object-Oriented Bayesian Network is modeled to reason about the lane on which the ego-vehicle is driving on using the information from these two input systems. The probabilistic approach is suitable because of the uncertain and inaccurate nature of the sensorial information. Due to the need of lateral painted landmarks, the method is dedicated to the segment of roads linked to an intersection.

This paper proposes a method for achieving accurate ego-vehicle global localization with respect to an approaching intersection; the method is based on the data alignment of the information from two input systems: a Sensorial Perception system, on-board of the ego-vehicle, and an a priori digital map. For this purpose an Extended Digital Map is proposed that contains the detailed information about the intersection infrastructure: detailed landmarks accurately measured and positioned on the map. The data alignment mechanism is thus based on superimposing the sensorial detected landmarks with the corresponding, correctly positioned map landmarks stored in the new Extended Digital Map. The data Alignment Algorithm requires as input, beside the information from the two input systems, the ego-vehicle driving lane. This information is inferred by using a probabilistic approach in the form of a Bayesian Network; the uncertain and noisy character of the sensorial data require such a probabilistic approach in the quest of the ego-lane.