Multi-THuMBS: Multi-person Tracking of 3D Human Meshes Beyond Video Shots

Jeongwan On1, Muhammad Salman Ali1, Muneeb A. Khan1, Sunwoo Park1, Inwoong Moon1, Hyung Jin Chang2, Jaekwang Kim3, Seong Jong Ha3, Seungryul Baek1

1UNIST Vision and Learning Lab, UNIST    2University of Birmingham    3CJ Corporation

ECCV 2026 (accepted)

arXiv Video
TL;DR Multi-THuMBS preserves identity-consistent multi-person 3D human mesh tracks across abrupt shot changes by aligning boundary frames in a shared 3D space.

Interactive Results

Interactive viewer Add the Viser client and recording files for each sample.
Sample 01

Abstract

Tracking multi-person 3D human meshes from in-the-wild videos is challenging due to complex interactions, frequent occlusions, and severe truncation in unconstrained environments. While recent approaches have improved robustness against these issues, they largely overlook the critical challenge prevalent in real-world footage: frequent shot changes. Abrupt viewpoint transitions can break identity tracking and temporal trajectory consistency, and existing shot-aware mesh tracking methods remain largely limited to single-person scenarios. To address this gap, Multi-THuMBS reconstructs the boundary frames around each shot change in a shared 3D space using a 3D scene prior, then registers human meshes in that space to preserve per-person identity and motion consistency across shots. Experiments show improvements in 3D mesh recovery, camera pose estimation, and identity tracking over prior methods, yielding temporally coherent multi-person reconstructions across shot boundaries.

Overview

Given a video split by a shot boundary, Multi-THuMBS first estimates human meshes and camera poses for all frames. It then builds a A shared 3D space from the boundary frames, B aligns meshes across shots, C propagates camera and mesh trajectories, D links identities using spatial, pose, and appearance cues, and applies E temporal smoothing.

Overview of the Multi-THuMBS method pipeline.

Acknowledgements

This work is supported by NRF grants (No. RS-2025-00521013 20%, No. RS-2025-02216916 10%) and IITP grants (No. RS-2020-II201336 Artificial intelligence graduate school program(UNIST) 10%; No. RS-2025-25442824 AI Star Fellowship Program(UNIST) 10%), funded by the Korean government (MSIT). This work is supported by CJ Corporation 50%.

BibTeX

Coming soon!