MoPort: A View-Aware 2D Interface for Motion Language Models

Two motions, a throw and a clap, rendered as SMPL bodies on a checkered floor. Camera frustums around the motions mark in-distribution and out-of-distribution views, and dashed circles show the 2D skeleton each camera observes. MoPort returns the correct caption from all views.

We introduce MoPort, a view-aware 2D interface that enables pretrained motion language models to take 2D observations as input. MoPort gives correct captions under two in-distribution (ID) and two out-of-distribution (OOD) randomly sampled cameras. Dashed circles show the 2D joints observed by each camera.

Abstract

Motion language models (MoLMs) encode 3D human motion as discrete tokens for language-based understanding. Deploying them on monocular video raises two problems. Pose detection yields 2D joint sequences, leaving a dimensional gap from the 3D motion used in training. Moreover, the 2D observation changes with the camera viewpoint, while the depth lost in projection makes the 2D pose ambiguous across viewpoints. We propose MoPort, a view-aware 2D interface that enables MoLMs to take 2D observations as input. We introduce Paired View Alignment (PVA), which gives two views of the same motion a shared 3D teacher target and directly constrains the consistency of their latents, so that 2D observations from different viewpoints are mapped to a consistent semantic representation. Camera-Modulated Encoding (CME) estimates the camera extrinsics from the 2D sequence and uses them to modulate encoding, providing the 2D latents with geometric information about the viewpoint. MoPort achieves R@1 of 0.5674 on HumanML3D and 0.3286 on KIT-ML, outperforming all evaluated baselines. On HumanML3D, its R@1 decreases only from 0.5763 on in-distribution viewpoints to 0.5585 on out-of-distribution viewpoints. We validate portability on three downstream MoLMs. Furthermore, on the real-world RGB video dataset Fit3D, MoPort achieves 80.36% accuracy. Code will be released upon acceptance.

Video Presentation

A two-minute narrated overview. All results shown are stored outputs: the stair-descent captions of Figure 3, the 3D reconstructions of Figure 10, the camera estimate of Figure 4, and zero-shot outputs on an AIST++ dance clip and a Fit3D clip, where the AIST++ answers come from a closed-set probe in which MG-MotionLLM scores candidate captions. The PVA and CME animations are schematic. The narration is synthesized with Kokoro TTS; English captions are available in the player.

Method

MoPort framework. (a) Paired View Alignment: a shared MoPort encodes two 2D projections of one 3D motion and aligns them with latent and token targets from a pretrained 3D encoder. (b) Camera-Modulated Encoding: a camera head estimates the extrinsics, Gram-Schmidt gives a rotation that maps rays into the 3D reference frame, and an MLP turns its R6D representation into FiLM parameters. (c) Inference: YOLOv8x and ViTPose-H extract 2D tracks that MoPort maps to the native tokens of a pretrained MoLM.

(a) A shared MoPort encodes different 2D projections of one 3D motion and aligns them with latent and token targets from a pretrained 3D encoder; the view loss pulls the two latents together, and sampled cameras supervise the camera head. (b) The camera head estimates the extrinsics, Gram-Schmidt gives the rotation that maps the rays into the 3D reference frame, and an MLP maps its R6D representation to multi-scale FiLM parameters. (c) At inference, YOLOv8x and ViTPose-H extract 2D tracks, which MoPort maps to the native tokens of a pretrained MoLM.

MoPort maps 2D joint sequences directly into the native latent and token spaces of a pretrained MoLM, without reconstructing 3D motion or retraining the downstream model. Paired View Alignment (PVA) projects each training motion into two independently sampled views. Both observations share the latents of the pretrained 3D encoder and its native tokens as targets, and a view loss directly pulls their latents together. Camera-Modulated Encoding (CME) estimates the camera extrinsics from the 2D sequence, rotates the viewing ray of every joint into a shared 3D reference frame, and modulates the encoder with FiLM conditioned on the estimated rotation. At deployment, MoPort receives only the 2D sequence and the known intrinsics, and the native codebook turns its latents into input tokens for the unchanged language model.

Results

Comparison on HumanML3D

Method	Input	R@1 ↑	R@3 ↑	MM-D ↓	CIDEr ↑	ID R@1 ↑	OOD R@1 ↑
GT3D	Privileged	0.5924	0.8643	2.5513	0.6457	-	-
MotionBERT	2D→3D	0.0845	0.2056	7.8712	0.0738	0.0985	0.0705
MotionAGFormer	2D→3D	0.0529	0.1399	8.7388	0.0540	0.0609	0.0448
D3DP-P-Agg	2D→3D	0.0343	0.0972	9.6378	0.0455	0.0356	0.0330
WHAM	RGB→3D	0.2559	0.4696	5.4994	0.2305	0.3182	0.1936
GVHMR	RGB→3D	0.2419	0.4429	5.8737	0.2216	0.2829	0.2009
TRACE	RGB→3D	0.0816	0.1864	8.8676	0.0862	0.0950	0.0683
SAM-3D-Body	RGB→3D	0.0878	0.1972	8.5260	0.0845	0.1043	0.0713
2D-Interface	2D joints	0.4537	0.7071	3.7025	0.3673	0.5118	0.3957
MoPort	2D joints	0.5674	0.8378	2.7274	0.6119	0.5763	0.5585

All routes use pretrained MG-MotionLLM. Overall metrics average two ID and two OOD views. Trained interfaces report means over three training seeds. Bold marks the best value among non-privileged methods. MoPort recovers 95.8% of the GT3D R@1, and from ID to OOD views its R@1 falls by only 3.1%, versus 22.7% for 2D-Interface.

Captions from four viewpoints

A stair-descent motion observed from two ID and two OOD cameras. MoPort describes walking down stairs from all four views; 2D-Interface describes walking forward and stopping from every view; GVHMR and MotionBERT captions change with the camera.

MoPort describes the stair descent from all four views, while other routes miss the descent or change with the camera. MoPort and 2D-Interface receive 2D joints, GVHMR reconstructs 3D motion from RGB video, and MotionBERT lifts 2D joints to 3D; all routes use MG-MotionLLM. Amber marks correct stair-descent descriptions, and red marks incorrect actions.

Supervised adaptation on KIT-ML

Method	Input	R@1 ↑	R@3 ↑	MM-D ↓	CIDEr ↑
GT3D	Privileged	0.3771	0.7197	3.0579	1.2407
MotionBERT	2D→3D	0.0892	0.2231	8.5425	0.2154
WHAM	RGB→3D	0.1612	0.3641	7.0163	0.4893
GVHMR	RGB→3D	0.1524	0.3294	7.7564	0.3650
2D-Interface	2D joints	0.3145	0.6070	4.3737	0.9973
MoPort	2D joints	0.3286	0.6257	4.1658	1.0328

All routes share one MG-MotionLLM fine-tuned on KIT-ML and follow native variable-length sequence processing.

Ablation of PVA and CME

Variant	HumanML3D R@1 ↑	KIT-ML R@1 ↑
2D-Interface (baseline)	0.4537	0.3145
PVA-only	0.5045	0.3173
CME-only	0.5137	0.3185
MoPort	0.5674	0.3286

Within each dataset, all variants share the MG-MotionLLM of the main comparison. On HumanML3D, each component adds a comparable gain on top of the other.

Camera estimates

Two motions with ground-truth and estimated camera frustums. Left: an ID view of a vertical jump. Right: an OOD view of a walk, turn, and return.

MoPort estimates extrinsics in ID and OOD views. Left: ID case, a person jumping vertically. Right: OOD case, a person walking, turning around, and walking back. Green and amber frustums are the ground-truth and estimated cameras, overlapping in khaki. Body color darkens over time.

Portability across pretrained MoLMs

Pretrained MoLM	GT3D R@1	MoPort R@1	Retained
MotionGPT	0.5229	0.5129	98.1%
TM2T-512	0.5279	0.5120	97.0%
MG-MotionLLM	0.5924	0.5674	95.8%

HumanML3D, averaging two ID and two OOD views; GT3D is scored once per motion. TM2T-512 shares the 3D encoder and codebook of MotionGPT and reuses its MoPort interface without retraining.

Real-video adaptation on Fit3D

Input route	Class accuracy (%) ↑	Template EM (%) ↑
GT3D	78.57	75.00
2D-Interface	71.43	69.64
MoPort	80.36	78.57

All routes share the same fine-tuned MG-MotionLLM. Class accuracy selects one of 28 exercise classes; EM denotes exact match with the class template. The test views are real cameras rather than sampled ones.

Zero-shot real videos

Real-video examples. Left: a rotated AIST++ dance view with MoPort's answers about hopping, foot support, and the travel path. Right: two Fit3D segments with MoPort's captions.

Left: on a rotated AIST++ view, MoPort answers questions about repeated hopping, alternating foot support, and a curved travel path. Right: two Fit3D segments are captioned separately; the first follows the rise to standing, and the second captures the downward motion but adds an unsupported hand location (dark amber). Skeletons are the ViTPose-H tracks fed to MoPort.

Citation

@article{tan2026moport,
  title   = {MoPort: A View-Aware 2D Interface for Motion Language Models},
  author  = {Tan, Lixing and Xia, Qing and Guo, Yuting and Li, Shuai and Hao, Aimin},
  journal = {arXiv preprint arXiv:2611.01541},
  year    = {2026}
}