Keynote Talk:

Building Models of Reality for Mixed Reality, Contextual AI and Robotics

Project Aria and the Digital Twin Catalog

• Introduction of DTC dataset
  • Dataset composition
  • New online dataset explorer
  • Open-source SOTA 3D object reconstruction baseline methods
• Introduction of Aria Gen 2

Capturing Reality End-to-End: Fast, Texture-Ready Digital Twins with Dynamics and Articulation

Building usable 3D digital twins demands methods that work with realistic inputs and produce simulation-ready assets. This talk traces a practical pipeline that starts with Dynamic Gaussians Mesh (DG-Mesh)—the first approach to extract time-consistent, high-fidelity meshes with cross-frame correspondences from a casual video of a non-rigid scene. We then move to LARM, which needs only a handful of viewpoints to recover detailed geometry, textures, and joint skeletons for complex articulated objects, and to PartUV, which turns noisy reconstructions into clean, part-aligned UV maps for texturing. Finally, we show how ManiSkill 3 ingests these assets and drives GPU-parallel physics at tens of thousands of frames per second, closing the loop from monocular capture to large-scale simulation.

4D Digital Twins from Videos in the Wild

Creating 3D digital twins from real-world visual data remains a major challenge, particularly in dynamic environments or when observations are sparse. In this talk, I’ll present our recent work on a streaming 3D reconstruction framework that recovers geometry from sparse-view inputs, in-paints missing regions, and handles the reconstruction of dynamic scenes online. I’ll then introduce approaches that jointly perform reconstruction and tracking, moving toward more complete and coherent digital twins of complex, real-world environments.

Feed-forward digital twins

Traditional 3D reconstruction and photogrammetry systems make use of iterative optimisation techniques such as bundle adjustment and fitting neural radiance fields or 3D Gaussian Splatting. In this talk, I will discuss new, powerful neural networks that can solve these tasks in a feed-forward manner. I will first introduce VGGT, a transformer network that can perform 3D reconstruction similar to COLMAP, but do so quickly, reliably, and, most importantly, by utilising only off-the-shelf components, with no optimisation involved in post-processing. I will then discuss Twinner, a network that can reconstruct not only the 3D shape of objects, but also their materials and scene illumination. The latter allows the model to be trained on real multi-view data.

Neural Relightable Assets

Representing, capturing and generating 3D objects has been a very active research area. However, many of the methods either bake the lighting (i.e., only solve the "novel view synthesis" problem) or assume restrictive shading models. I will offer some ways to capture and represent fully relightable 3D assets that do not make assumptions on specific shading models, and can be inserted and rendered in new scenes under new lighting conditions.

Physics-based inverse rendering

Abstract: Inferring the shape and material of an object is a crucial ingredient for building digital twins of real-world objects. To this end, recent advances in physics-based differentiable rendering have enabled analysis-by-synthesis solutions capable of taking complex light transport effects (e.g., soft shadows and inter-reflections) into consideration. In this talk, I will present some of our recent works in this direction and discuss many remaining challenges.

Discussion Panel - 3D Digital Twin R&D across Robotics, Vision, and Graphics

This session brings together experts from all three disciplines to explore current challenges and future directions in digital-twin research. Audience questions will be collected in advance via Google Form, allowing the panel to address the most pressing topics. We’ll reserve the final 15 minutes for spontaneous, on-site questions to encourage lively, real-time interaction.