BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting

Jeongwan On, Kyeonghwan Gwak, Gunyoung Kang, Junuk Cha, Soohyun Hwang, Hyein Hwang, Seungryul Baek

Ulsan National Institute of Science and Technology (UNIST)

CVPR 2025

TL;DR BIGS reconstructs two hands and an unknown object from a monocular video as 3D Gaussians, sharing one hand Gaussian across both hands and using a diffusion prior to recover object parts hidden by occlusion.

Video

Abstract

Reconstructing 3Ds of hand-object interaction (HOI) is a fundamental problem that can find numerous applications. Despite recent advances, there is no comprehensive pipeline yet for bimanual class-agnostic interaction reconstruction from a monocular RGB video, where two hands and an unknown object are interacting with each other. Previous works tackled the limited hand-object interaction case, where object templates are pre-known or only one hand is involved in the interaction. The bimanual interaction reconstruction exhibits severe occlusions introduced by complex interactions between two hands and an object. To solve this, we first introduce BIGS (Bimanual Interaction 3D Gaussian Splatting), a method that reconstructs 3D Gaussians of hands and an unknown object from a monocular video. To robustly obtain object Gaussians avoiding severe occlusions, we leverage prior knowledge of pre-trained diffusion model with score distillation sampling (SDS) loss, to reconstruct unseen object parts. For hand Gaussians, we exploit the 3D priors of hand model (i.e., MANO) and share a single Gaussian for two hands to effectively accumulate hand 3D information, given limited views. To further consider the 3D alignment between hands and objects, we include the interacting-subjects optimization step during Gaussian optimization. Our method achieves the state-of-the-art accuracy on two challenging datasets, in terms of 3D hand pose estimation (MPJPE), 3D object reconstruction (CDh, CDo, F10), and rendering quality (PSNR, SSIM, LPIPS), respectively.

Contributions

Bimanual 3DGS pipeline

Reconstructs the 3D interaction of two hands and an unknown object from a monocular video with 3D Gaussian splatting.
Two-step optimization

Hand and object Gaussians are first optimized separately for reliable shapes, then jointly so their contact is taken into account.
Priors against occlusion

An SDS loss with a diffusion prior fills in unseen object surfaces, and one canonical hand Gaussian shared by both hands accumulates hand information.

Why Bimanual Interaction Is Hard

Reconstructing hands and objects is essential for AR/VR, robotics, and human-computer interaction, but most methods assume a known object or a single hand. With two hands and an unknown object seen from one camera, no existing pipeline covered the whole problem.

Templates do not scale

Many methods need a pre-scanned object template, which typically restricts them to 10 to 20 known shapes.
One hand, or many views

Category-agnostic methods mostly handle one hand. HOLD is NeRF-based and slow to fit and render; MANUS needs multi-view images.
Severe occlusion

When two hands hold an object, occlusion patterns become dramatic and many hand and object pixels are never observed.

Method

Given a monocular video, BIGS first reconstructs initial hand meshes and an object point cloud. In the single-subject optimization, hand and object Gaussians are optimized separately in a canonical space through TriplaneNet features and MLPs. A single right-hand Gaussian is mirrored to serve both hands, and an SDS loss with a text-to-image diffusion prior reconstructs object surfaces the camera never sees. The interacting-subjects optimization then refines the hand translations with a contact term so the hands and the object stay aligned.

BIGS pipeline: pre-processing, single-subject optimization with TriplaneNet, MLPs and an SDS loss, then interacting-subjects optimization with a contact term. — Pre-processing gives initial hand and object meshes. Single-subject optimization learns canonical hand and object Gaussians through TriplaneNet and MLPs, with textual inversion and an SDS loss for the object. Interacting-subjects optimization then refines the hands with a 3D contact term.

Quantitative Results

Lower is better unless marked ↑. The best value in each column is bold.

Two hands: ARCTIC

Method	Contact		Hand / object
Method	CD_l ↓	CD_r ↓	CD_o ↓	F10 ↑	MPJPE_l ↓	MPJPE_r ↓
HLoc	N/A	N/A	4.24	40.57	N/A	N/A
HaMeR	N/A	N/A	N/A	N/A	27.23	24.37
HOLD	105.92	123.54	2.07	63.92	27.13	24.70
Ours	46.11	31.28	1.28	83.93	24.63	24.35

CD_l and CD_r are hand-relative Chamfer distances (cm²) for contact; CD_o (cm²) and F10 (F-score at 10 mm, %) measure object shape; MPJPE (mm) measures hand pose. N/A: the method does not estimate that quantity. Setting of the ECCV 2024 HANDS workshop challenge.

One hand: HO3D v3

Setting	Method	Contact	Hand / object
Setting	Method	CD_r ↓	CD_o ↓	F10 ↑	MPJPE_r ↓
Full view	HOMan	78.2	N/A	N/A	32.0
	iHOI	75.8	3.8	41.7	38.4
	DiffHOI	68.8	4.3	43.8	32.3
	HOLD	11.3	0.4	96.5	24.2
	Ours	11.1	0.3	96.7	23.9
Limited view	HOLD	95.63	3.39	74.90	24.96
Limited view	Ours	21.70	1.24	84.02	24.22

Limited view keeps 25% of the frames of each sequence, so the object is never seen from all sides. HOLD drops sharply in this setting, while ours stays close to its full-view accuracy.

Rendering quality

Setting	Method	PSNR ↑	SSIM ↑	LPIPS ↓
Two-hand	HOLD	12.83	0.66	0.32
Two-hand	Ours	24.87	0.96	0.05
One-hand	HOLD	16.20	0.74	0.21
One-hand	Ours	24.51	0.92	0.07

Our 3DGS-based method renders better than the NeRF-based HOLD in both the two-hand and one-hand settings.

Qualitative Results

Comparisons with HOLD, and rendering with novel poses.

3D meshes and rendered images

Animatable hands and objects

The reconstructed hands and object rendered from three cameras with new hand and object poses. — Once built, the Gaussians render the interaction from novel cameras (rows) with novel hand and object poses.

More results

Rendered hand-object sequence from eight cameras with novel poses. — Animatable hand-object sequences rendered with novel poses of hand, object, and camera.

Another rendered hand-object sequence from eight cameras with novel poses. — Another sequence rendered with novel poses of hand, object, and camera.

Ablation Study

Turning each component on in turn, on ARCTIC.

Components				Contact		Hand / object
L_SDS	L_contact	L_smt	Share	CD_l ↓	CD_r ↓	CD_o ↓	F10 ↑	MPJPE_l ↓	MPJPE_r ↓
				96.56	117.72	1.36	81.78	27.22	25.08
			✓	96.55	117.72	1.36	81.78	27.01	24.42
		✓	✓	83.14	96.95	1.36	81.78	24.63	24.35
	✓	✓	✓	50.78	35.57	1.36	81.78	24.63	24.35
✓	✓	✓	✓	46.11	31.28	1.28	83.93	24.63	24.35

Share (one canonical Gaussian for both hands) and L_smt (temporal smoothness) improve hand accuracy (MPJPE).
L_contact improves contact reconstruction: CD_l 83.14 → 50.78 and CD_r 96.95 → 35.57.
L_SDS improves both contact and object reconstruction: CD_o 1.36 → 1.28 and F10 81.78 → 83.93.

Acknowledgements

This work is supported by NRF grant (No. RS-2025-00521013 50%) and IITP grants (No. RS-2020-II201336 Artificial intelligence graduate school program (UNIST) 15%; No. RS-2021-II212068 AI innovation hub 15%; No. RS-2022-II220264 Comprehensive video understanding and generation with knowledge-based deep logic neural network 20%), funded by the Korean government (MSIT).

BibTeX

@InProceedings{On_2025_CVPR,
    author    = {On, Jeongwan and Gwak, Kyeonghwan and Kang, Gunyoung and Cha, Junuk and Hwang, Soohyun and Hwang, Hyein and Baek, Seungryul},
    title     = {BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {17437-17447}
}