Topic

Digital Twin Catalog: The field of 3D reconstruction is central in the computer vision community and aims to solve fundamental problems around reconstructing 3D objects from 2D images. State of the art techniques today fall short of the needs for real-world applications, largely limited by the datasets available. Project Aria’s Digital Twin Catalog aims to motivate the 3D reconstruction community to reach a new level of quality and realism by providing a large and highly detailed set of 3D object models, corresponding source capture data, and reconstruction algorithms for researchers to use.

30-minute Break

Surreal's HOT3D dataset for hand-object interaction unerstanding:
We introduce HOT3D, a publicly available dataset for egocentric hand and object tracking in 3D. The dataset offers over 833 minutes (more than 3.7M images) of multi-view RGB/monochrome image streams showing 19 subjects interacting with 33 diverse rigid objects, multi-modal signals such as eye gaze or scene point clouds, as well as comprehensive ground truth annotations including 3D poses of objects, hands, and cameras, and 3D models of hands and objects.

Surreal's Nymeria dataset for egocentric human motion understanding: Future AR/VR technology is the era of human-centric contextualized AI computing. A cornerstone in this paradigm is to understand one’s own body motion and action. To accelerate research in this field, this talk introduces the Nymeria dataset. Nymeria is the world's largest collection of human motion in the wild, capturing diverse people performing diverse activities across diverse locations. It is first of its kind to record body motion using multiple egocentric multimodal devices, all accurately synchronized and localized in one single metric 3D world. Nymeria is also the world's largest dataset with motion-language descriptions, featuring hierarchical in-context narration. To demonstrate the potential of the Nymeria dataset, this talk also discusses how we build state-of-the-art algorithms to solve egocentric body tracking, motion synthesis and action recognition.

Surreal's EFM3D dataset for 3D egocentric foundation models: The advent of wearable computers enables a new source of context for AI that is embedded in egocentric sensor data. This new egocentric data comes equipped with fine-grained 3D location information and thus presents the opportunity for a novel class of spatial foundation models that are rooted in 3D space. To measure progress on what we term Egocentric Foundation Models (EFMs) we establish EFM3D, a benchmark with two core 3D egocentric perception tasks. EFM3D is the first benchmark for 3D object detection and surface regression on high quality annotated egocentric data of Project Aria. We propose Egocentric Voxel Lifting (EVL), a baseline for 3D EFMs. EVL leverages all available egocentric modalities and inherits foundational capabilities from 2D foundation models. This model, trained on a large simulated dataset, outperforms existing methods on the EFM3D benchmark.

Ego-Exo4D Overview: This session provides a overview of the Ego-Exo4D dataset. Ego-Exo4D is a diverse, large-scale multimodal multiview video benchmark dataset centered around simultaneously-captured egocentric and exocentric video of skilled human activtities. The multimodal nature of the dataset is unprecedented: the video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera poses, IMU, and multiple paired language descriptions -- including a novel "expert commentary" done by coaches and teachers and tailored to the skilled-activity domain.

Egocentric Recording in the Home: How to collect and annotate unscripted recordings using the Aria Research Kit. This short talk will focus on new capabilities when using the Aria toolkit for unscripted long term recordings in participants homes.

Building a Procedural Wearable Assistant with ARIA: Current benchmarks on Egocentric Vision lack of principle benchmark including real dual-agent conversation. A personal assistant should be able to see what the user is doing and relate vision to language to contextualize questions such as “what is this?” or “what should I do now?”. In this talk, I will discuss the acquisition of a novel dual-agent multimodal dataset using ARIA glasses and the ARIA Research Kit. I will present a data acquisition and annotation pipeline that leverages the ARIA SDK and the MPS services. Finally, I will provide an overview of the acquired dataset, which includes procedural videos captured in various scenarios with the collaboration of both trainees and experts.

LaMar Aria: We benchmark crowd-sourced 3D localization and mapping for AR, a core technology for the grounding and persistence of digital content in the real world. Devices record egocentric multi-modal data that exhibits specific challenges and opportunities not found in existing benchmarks. Compared to existing mobile and AR devices, Project Aria makes it possible to capture such data at large scale with accurate ground truth.