Reading is a fundamental human activity that forms the basis of communication, entertainment, and learning. It spans a wide range of mediums, from handwritten notes and printed books to digital screens and environmental signage. The future of AI relies heavily on understanding the user's physical context. Smart glasses, like the Meta Ray-Ban, offer a promising form factor that bridges visual AI with the real world, enabling personalized, contextual experiences. In this context, the ability to recognize when the user is reading is essential for developing AI that is truly personalized and contextually aware. This capability allows devices to understand what a user has read and where they can assist, enhancing the user's interaction with the world.

A significant challenge in this domain is the scarcity of data for large-scale, real-world reading recognition. Previous reading datasets were collected in controlled settings, where participants read lengthy texts on large screens with front-facing eye-tracking cameras. This approach limits the scale, diversity, and realism of the data. Moreover, effective reading recognition requires leveraging multi-modal information. Simply having text in the field of view does not guarantee that the user is reading it, making it difficult to to solve using visual information alone. Similarly, a horizontal eye gaze pattern does not always indicate reading activity. Additionally, any solution must be efficient enough for real-time, always-on operation, given the practical constraints of all-day wearable devices. Current Optical Character Recognition (OCR) technology demands high-resolution images, resulting in significant power consumption. However, reading recognition can act as an efficient proxy, triggering OCR only when the user is actively engaged with reading material.

Today, we are excited to announce the release of Reading Recognition in the Wild, a dataset created with Project Aria glasses. This dataset features 100 hours of egocentric video, along with eye gaze, head pose, audio, IMU, magnetometer, and barometer data, all captured from real-world scenarios using the Project Aria sensor suite. This dataset is not just a collection of data; it is a testament to our commitment to advancing thought leadership in the AI community, particularly in the realm of egocentric AI.

Achieving reading recognition makes it feasible to keep a record of a user's reading interactions, building a contextually aware AI. This capability opens doors to several applications, such as reading assistant tools for children with learning difficulties and people with low vision. It can also track whether a user has read crucial information, like signs while driving, and measure attention and distraction during tasks.

Additionally, our dataset and method can be extended to classify different types of reading, a topic of interest in cognitive studies of reading comprehension. Unlike previous studies limited to controlled environments, our dataset allows for reading mode and medium classification in unconstrained settings, providing valuable experimental results in this direction.