Multi-THuMBS: Multi-person Tracking of 3D Human Meshes Beyond Video Shots

Jeongwan On1, Muhammad Salman Ali1, Muneeb A. Khan1, Sunwoo Park1, Inwoong Moon1, Hyung Jin Chang2, Jaekwang Kim3, Seong Jong Ha3, Seungryul Baek1

1UNIST Vision and Learning Lab, UNIST 2University of Birmingham 3CJ Corporation

UNIST University of Birmingham CJ Corporation

ECCV 2026

arXiv Video

TL;DR Multi-THuMBS preserves identity-consistent multi-person 3D human mesh tracks across abrupt shot changes by aligning boundary frames in a shared 3D space.

Interactive Results

Drag to rotate, scroll or pinch to zoom, and pick a sequence below.

Loading the 3D viewer… Large 3D recordings can take a moment to load.
Sample 01

Abstract

A monocular video with a shot change goes in; identity-consistent 3D human mesh trajectories in a shared 3D space come out.

Tracking multi-person 3D human meshes from in-the-wild videos is challenging due to complex interactions, frequent occlusions, and severe truncation in unconstrained environments. While recent approaches have improved robustness against these issues, they largely overlook the critical challenge prevalent in real-world footage: frequent shot changes. Abrupt viewpoint transitions can break identity tracking and temporal trajectory consistency, and existing shot-aware mesh tracking methods remain largely limited to single-person scenarios. To address this gap, Multi-THuMBS reconstructs the boundary frames around each shot change in a shared 3D space using a 3D scene prior, then registers human meshes in that space to preserve per-person identity and motion consistency across shots. Experiments show improvements in 3D mesh recovery, camera pose estimation, and identity tracking over prior methods, yielding temporally coherent multi-person reconstructions across shot boundaries.

Contributions

  1. Multi-person tracking across shots

    To our knowledge, the first method to extend multi-person 3D human mesh tracking to multi-shot videos. A learned 3D scene prior builds one shared 3D space for adjacent shots, where each person's trajectory is connected.

  2. Geometry-aware re-identification

    People are matched across the cut by 3D distance, pose, and appearance together, so identities hold even when the viewpoint and appearance change abruptly.

  3. Global spatiotemporal optimization

    Joint trajectory smoothing and a cross-camera reprojection loss stabilize poses, correct residual misalignment, and keep motion consistent across shots.

Why Shot Changes Are Hard

Films, broadcast sports, and edited online videos are made of many shots. Current multi-person trackers assume the camera moves continuously within one shot, so at every cut they lose identities and misplace people in 3D.

PromptHMR on sitcom clips: one person gets a different identity color after the cut, and another lands in a different 3D location.
A state-of-the-art method (PromptHMR) at a shot change. Case 1: the same person gets a different identity (mesh color) after the cut. Case 2: the same person is placed in a completely different 3D location. In both cases the number of visible people also changes.

Method

Given a video split by a shot boundary, Multi-THuMBS first estimates human meshes and camera poses for all frames. It then builds a A shared 3D space from the boundary frames, B aligns meshes across shots, C propagates camera and mesh trajectories, D links identities using spatial, pose, and appearance cues, and applies E temporal smoothing.

Overview of the Multi-THuMBS method pipeline.

A 150-frame 1920×1080 video takes about 10 minutes on a single NVIDIA RTX 3090.

Quantitative Results

Multi-shot benchmarks built by inserting shot boundaries into the multi-view sequences of EgoHumans, EgoBody, and Harmony4D. Lower is better unless marked ↑. The best value in each column is bold.

3D human pose

DatasetMethodW-MPJPE ↓WA-MPJPE ↓MPJPE ↓MPVPE ↓Accel ↓
EgoHumansMultishot474.1287.4347.1408.263.15
GVHMR404.8204.7287.4371.459.0
PromptHMR1778.2440.9285.3364.174.3
HSfM†544.2187.7263.1294.348.7
Ours279.0166.0228.3262.227.3
EgoBodyMultishot185.1144.1147.1166.517.9
GVHMR174.0133.3108.0147.114.7
PromptHMR1228.1395.399.0133.917.4
HSfM†113.196.3113.3123.210.9
Ours99.272.872.094.96.0
Harmony4DMultishot248.0231.6511.2609.537.1
GVHMR244.9166.7244.7334.129.6
PromptHMR1746.3399.8675.7746.066.8
HSfM†372.0178.4225.6257.628.3
Ours221.0116.9215.9278.317.4

W-MPJPE aligns only the first frame and measures trajectory consistency; WA-MPJPE aligns the whole trajectory; Accel measures temporal smoothness. HSfM† is HSfM adapted to the multi-shot, multi-person setting.

Identity switches and camera pose

MethodIdentity switches (IDs) ↓Camera pose (ATE) ↓
EgoHumansEgoBodyHarmony4DEgoHumansEgoBodyHarmony4D
VGGT–––1.41.31.4
PromptHMR10.401.298.002.31.62.3
HSfM†3.870.201.582.82.93.2
KPR2.540.051.19–––
Pose2ID4.620.721.32–––
Ours (dist.)1.660.000.54–––
Ours0.970.000.460.70.10.7

IDs counts identity switches across the shot boundary; ATE is the absolute trajectory error of the estimated cameras. Ours (dist.) uses only the 3D distance term for re-identification. – means not reported.

Edited real videos

MethodPCK* ↑Jitter ↓FS ↓
PromptHMR62.7162.4423.11
Ours90.731.510.7

AVA, Friends, and The Big Bang Theory, which contain real shot changes and no 3D ground truth. PCK* is cross-shot PCK; FS is foot sliding.

Robustness to harder scenes

Line charts: PromptHMR error is far higher than ours at every difficulty level and grows on harder splits, while ours stays low.
W-MPJPE of PromptHMR and ours on Easy, Medium, and Hard splits by number of people, camera pose change, human motion, and occlusion ratio. The baseline’s error grows on harder scenes, especially with large camera changes, while ours stays low.

Qualitative Results

Comparisons with PromptHMR and appearance-based re-identification at shot boundaries.

Video results

Challenging cases

Side-by-side results of PromptHMR and ours on a crowded sitcom scene and on close-up shots, in 2D and in global 3D.
Case 1: when the number of visible people changes drastically, PromptHMR loses track of individuals while ours keeps them. Case 2: under severe truncation from a close-up, PromptHMR predicts wrong camera poses while ours recovers accurate cameras and global trajectories.

Global trajectories

Global trajectories on EgoHumans and AVA: PromptHMR trajectories slide and jump, ours stay compact.
PromptHMR shows sliding and abrupt jumps at the cut; ours produces smoother, physically plausible motion with less foot sliding (red dotted trajectories).

Cross-shot re-identification

Identity assignments of Pose2ID, KPR, and ours before and after a shot change; ours keeps all identities correct.
Colors show predicted identities; correct and wrong associations are marked in blue and red. Appearance-based methods (Pose2ID, KPR) often fail under abrupt visual changes, while ours keeps identities consistent.
More results
Additional results on sitcom and AVA clips before and after shot changes.
Additional results on (a) sitcoms and (b) AVA.
Additional results on EgoHumans and Harmony4D before and after shot changes.
Additional results on (a) EgoHumans and (b) Harmony4D.
Additional results on EgoBody before and after shot changes.
Additional results on EgoBody.
Reconstructions for the first frame of each shot in an EgoBody video with more than two shots.
Extension to more than two shots on EgoBody: the first frame of each shot with its reconstructed meshes.

Ablation Study

Removing each component on EgoHumans.

MethodW-MPJPE ↓MPJPE ↓MPVPE ↓Accel ↓ATE ↓
w/o cam389.7230.1264.733.71.40
w/o post311.2241.6298.347.90.77
w/o stage491.9368.1359.433.92.75
w/o align882.7422.4392.134.91.40
Ours278.8228.3262.127.30.77

Acknowledgements

This work is supported by NRF grants (No. RS-2025-00521013 20%, No. RS-2025-02216916 10%) and IITP grants (No. RS-2020-II201336 Artificial intelligence graduate school program(UNIST) 10%; No. RS-2025-25442824 AI Star Fellowship Program(UNIST) 10%), funded by the Korean government (MSIT). This work is supported by CJ Corporation 50%.

BibTeX

@inproceedings{on2026multi,
  title={Multi-THuMBS: Multi-person Tracking of 3D Human Meshes Beyond Video Shots},
  author={On, Jeongwan and Ali, Muhammad Salman and Khan, Muneeb A and Park, Sunwoo and Moon, Inwoong and Chang, Hyung Jin and Kim, Jaekwang and Ha, Seong Jong and Baek, Seungryul},
  booktitle={European Conference on Computer Vision},
  pages={1--18},
  year={2026},
  organization={Springer}
}