Ad Hoc UI: On-the-Fly Transformation of Everyday Objects Into Tangible 6DOF Interfaces for AR

Interaction with AR content is often limited on smartphones and AR glasses due to the lack of reliable hand tracking or dedicated 6DoF controllers. The narrow field-of-view screen also makes it challenging to discover digital content -- more severely when input is limited to on-device touch. We aim to improve usability in AR interaction on lightweight AR devices by allowing users to seamlessly anchor input and digital content to uninstrumented everyday objects that are discoverable outside the boundaries of the screen. Recent advances in AR/VR technologies, real-time environmental tracking capabilities, and lightweight machine learning algorithms, make it possible to track most everyday objects with minimal acquisition time without dedicated fiducial markers. Leveraging extensive research in Tangible User Interfaces, this creates new opportunities for opportunistic interfaces using everyday objects. This paper explores the design space of opportunistic interfaces and interaction techniques to transform everyday objects into interactive widgets instantly. We present a design space identifying key aspects and challenges to explore for opportunistic interfaces. We further introduce Ad hoc UI, a prototyping toolkit that empowers users with on-the-fly tracking of everyday objects, 6DoF actions, audio activation, and mid-air gestures. We provide an open-source implementation to help AR researchers and practitioners effortlessly integrate opportunistic interfaces into their AR experiences and amplify the expression of their creative vision.

Note: the full-version of the paper is still in submission and we hope to release our toolkit in 2022Q4 upon approval.

Publications

teaser image of Opportunistic Interfaces for Augmented Reality: Transforming Everyday Objects Into Tangible 6DoF Interfaces Using Ad Hoc UI

Opportunistic Interfaces for Augmented Reality: Transforming Everyday Objects Into Tangible 6DoF Interfaces Using Ad Hoc UI

Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.
Keywords: augmented reality, everyday objects, tangible user interface, 3D user interface, 6 DoF, spatial interaction, markerless tracking, tangible interaction, hand gestures


Videos

Opportunistic Interfaces for Augmented Reality: Transforming Everyday Objects Into Tangible 6DoF Interfaces Using Ad Hoc UI


Talks

Cited By

Stay In Touch