Selected Projects
Open-source research systems from our group and collaborators — spanning embodied AI & robotics, autonomous driving, generative models, and efficient multimodal foundation models.
# equal contribution* corresponding author
-
Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces
A fully continuous multimodal flow framework that encodes each modality as continuous hyperchunks in one ordered, chunk-causal stream — unifying understanding, generation, and flexible mixed sequences under a single interface.
-
StableVQ: Practical Guidelines for Stable Vector-Quantized Tokenizer Training
A separation-of-concerns framework — Dynamic STE, Region VQ Loss, and Decoupled Schedule — that turns stable VQ tokenizer training into a principled property, reaching 100% codebook utilization with no new parameters.
-
Bridge3D: Enabling Vision-Language-Action Models to See and Act in 3D
Integrates implicit and explicit 3D geometry guidance into pretrained 2D VLA models — VGGT-feature fusion for 3D perception plus an explicit 3D semantic field conditioning action denoising — reaching 84.8% average success on RoboTwin 2.0 and +11.7pp real-world over Spatial Forcing.
-
AdaRoboVLG: Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
A task-adaptive Vision-Language-Grasp framework that decouples physical grasp synthesis from task-dependent understanding, supporting generalizable grasping across different robotic hands via composable foundation-model priors.
-
DreamWAM: Beyond RGB Future Prediction for World Action Models
A world action model that reformulates future prediction as structured world modeling beyond RGB, representing future states through complementary views of appearance, motion, geometry, and semantics for robust manipulation.
-
Faster-WAM: Efficient Inference-Time Future Conditioning for Robust World Action Models
A world action model that computes future representations once and selectively reuses them during action denoising, achieving robust out-of-distribution generalization with substantially reduced inference latency.
-
EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass
Adapts SAM 3 into a single-pass open-vocabulary segmenter — prompt-free mask generation with end-to-end Attentional Aggregation classification — improving accuracy over vanilla SAM 3 while accelerating inference by up to 338×.
-
ReWorld: Learning Better Representations for World Action Models
The first representation learning framework for autonomous-driving world action models, explicitly optimizing the latent world-to-action pathway without external encoders or teacher models at only +0.3% training cost.
-
Moebius: 0.2B Lightweight Image Inpainting Framework with 10B-Level Performance
A 0.2B lightweight image inpainting framework that delivers 10B-level performance.
-
MotionVLA: Injecting Geometric Motion into Vision-Language-Action Model
Injects geometric motion into a VLA as compact, queryable trajectory-field tokens from strictly past frames, giving the policy a time-continuous motion history for long-horizon manipulation.
-
RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework
A generator-discriminator framework for closed-loop driving: a diffusion generator proposes diverse trajectory candidates while an RL-optimized discriminator reranks them by long-term driving quality, reducing the collision rate by 56% over strong diffusion planners.
-
UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
A unified driving vision-language-action model based on Mixture-of-Transformers, decoupling driving understanding, scene perception, and action planning into specialized experts coordinated through masked joint attention.
-
-
Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planning
A VLM-E2E driving policy that explicitly aligns the VLM's high-level decisions with the E2E policy's low-level planning, via consistency-oriented three-stage training and hierarchical reinforcement learning in 3DGS environments.
-
OmniTrack: General Motion Tracking via Physics-Consistent Reference
A two-stage humanoid motion-tracking framework that decouples physical feasibility from general tracking — a privileged generalist policy generates dynamics-consistent reference motions that the control policy then learns to track — enabling hour-long stable real-world tracking of acrobatic motions like flips and cartwheels.
-
-
DriveLaW: Unifying Planning and Video Generation in a Latent Driving World
A driving world model that unifies video generation and motion planning by injecting the video generator's latent representation directly into a diffusion planner, achieving state-of-the-art results on both nuScenes video prediction and NAVSIM planning.
-
-
VTP: Towards Scalable Pre-training of Visual Tokenizers for Generation
A unified visual tokenizer pre-training framework that jointly optimizes contrastive, self-supervised, and reconstruction objectives, unlocking a new scaling law where generation performance scales with tokenizer pre-training compute, parameters, and data.
-
InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
An efficient vision-language model that combines linear attention for compact long-term memory with sparse attention for precise visual perception, supporting unlimited-input streaming with a constant memory footprint.
-
4DLangVGGT: 4D Language-Visual Geometry Grounded Transformer
The first Transformer-based feed-forward framework for 4D language grounding, jointly integrating geometric perception (StreamVGGT) and language alignment (Semantic Bridging Decoder) — no per-scene optimization, generalizing across dynamic scenes.
-
-
-
TransLight: Image-Guided Customized Lighting Control with Generative Decoupling
Image-guided customized lighting control: generative decoupling transfers the lighting effects of any reference image onto user content, enabling text-free, precise relighting.
-
LENS: Learning to Segment Anything with Unified Reinforced Reasoning
Segment anything with unified reinforced reasoning: a multimodal model that reasons over complex referring expressions and produces precise segmentation masks, trained end-to-end with reinforcement learning.
-
ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving
A reinforced cognitive driving framework that unifies driving understanding and planning by combining a cognition-enhanced VLM with a diffusion planner, further reinforced by DiffGRPO for safer and more comfortable trajectories.
-
GroundingSuite: Measuring Complex Multi-Granular Pixel Grounding
A comprehensive pixel grounding framework: an automated VLM-based annotation pipeline (4.5× faster than GLaMM), a 9.56M-sample training set with diverse referring expressions, and a 3,800-instance evaluation benchmark.
-
OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models
The first linear-model-based unified multimodal understanding and visual generation model — competitive with only 2M training data, up to 119.2× speedup and 63% GPU memory reduction on long-sequence generation vs Transformer counterparts.
-
LightningDiT & VA-VAE: Taming the Reconstruction-Generation Dilemma in Latent Diffusion Models
Aligns the VAE latent space with vision foundation models (VA-VAE) and pairs it with an enhanced DiT baseline (LightningDiT) — FID 1.35 with 0.28 rFID on ImageNet-256, converging 21.8× faster than the original DiT (FID 2.11 in 64 epochs). CVPR 2025 Oral.
-
GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding
Self-supervised 3D spatial understanding: aligning Gaussian-based 3D representations with foundation-model features, enabling open-vocabulary 3D perception without manual 3D annotations.
-
-
-
-
XS-VID: A Large-Scale Benchmark for Small Object Detection and Tracking in Videos
A large-scale video benchmark of extremely small objects across diverse day/night scenes, paired with YOLOFT — a detector that rethinks optical-flow feature alignment for tiny-object detection and tracking.
-
DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention
Brings gated linear attention into the diffusion backbone with minimal parameter overhead — beating DiT at 256² while scaling far better at high resolutions: DiG-S/2 is 2.5× faster with 75.7% less GPU memory than DiT-S/2 at 1792², and DiG-XL/2 is 1.8× faster than DiT with FlashAttention-2 at 2048².
-
VADv2: End-to-End Autonomous Driving via Probabilistic Planning
End-to-end vectorized autonomous driving that outputs a probabilistic distribution over actions learned from large-scale driving demonstrations and samples one action to control the vehicle.
-
EVA-X: A Foundation Model for General Chest X-ray Analysis with Self-Supervised Learning
The first X-ray self-supervised ViT foundation model capturing both semantic and geometric information, spanning 20+ chest diseases with leading results on 11+ detection tasks and strong few-shot capability.
-
MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning
A dual masked-image-modeling pre-training paradigm for multi-view driving videos, supervising pseudo-3D features via volumetric differentiable rendering — improving end-to-end planning (-9% collision), BEV segmentation (+8.7% IoU), 3D detection (+3.5% mAP) and HD map construction (+1.4% mAP) on nuScenes.
-
YOLO-World: Real-Time Open-Vocabulary Object Detection
Real-time open-vocabulary detection: YOLO enhanced with vision-language modeling via the re-parameterizable RepVL-PAN and region-text contrastive pre-training, reaching 35.4 AP at 52 FPS on LVIS zero-shot.
-
-
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
Real-time rendering of dynamic scenes by modeling 4D scenes with Gaussian splatting and a compact spatio-temporal representation, achieving high-quality novel-view synthesis at interactive frame rates.
-
WeakTr: Exploring Plain Vision Transformer for Weakly-Supervised Semantic Segmentation
Explores plain ViT for weakly-supervised semantic segmentation: end-to-end CAM generation by adaptively fusing self-attention maps with learned head weights, plus online retraining with a gradient-clipping decoder — 78.5% mIoU on VOC12, 51.1% on COCO14.
-
-
-
ByteTrack: Multi-Object Tracking by Associating Every Detection Box
Simple yet effective multi-object tracking: by associating every detection box instead of only high-scoring ones, ByteTrack recovers true objects from low-score detections and filters out background, setting the standard tracker integrated in YOLOv8 and beyond.
-
-
