Interpretability (& other areas) for Multimodal Models

💡 This post is initially focused on interpretability for multimodal models, while later a lot of papers in other fields are included, just for convenience. Methods Interpretability for MLLMs survey A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models Sparks of Explainability Recent Advancements in Explaining Large Vision Models Awesome LMMs Mechanistic Interpretability probing Probing Multimodal Large Language Models for Global and Local Semantic Representations representation Zoom in: An introduction to circuits Multimodal Neurons in Artificial Neural Networks Interpreting CLIP’s Image Representation via Text-Based Decomposition Interpreting the Second-Order Effects of Neurons in CLIP CLIP不同层 Multimodal Neurons in Pretrained Text-Only Transformers Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers? Understanding Video Transformers via Universal Concept Discovery circuit **(causal tracing) Understanding Information Storage and Transfer in Multi-modal Large Language Models Automatic Discovery of Visual Circuits Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP SAE Case study: Interpreting, manipulating, and controlling clip with sparse autoencoders Towards multimodal interpretability: Learning sparse interpretable features in vision transformers Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery visualization VITAL: More Understandable Feature Visualization through Distribution Alignment and Relevant Information Flow Visualizer！简化你的Vision Transformer可视化！ (DVT) Denoising Vision Transformers Token Activation Map to Visually Explain Multimodal LLMs LVLM-Intrepret: An Interpretability Tool for Large Vision Language Models Transformer Interpretability Beyond Attention Visualization others **Towards interpreting visual information processing in vision-language models demo (dogit lens) Laying the Foundations for Vision and Multimodal Mechanistic Interpretability & Open Problems Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Models tools VLM-Lens information flow Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs **Cross-modal Information Flow in Multimodal Large Language Models *From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks *What’s in the Image? A Deep-Dive into the Vision of Vision Language Models The Narrow Gate: Localized Image-Text Communication in Vision-Language Models Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference analyses on MLLMs Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training Lost in Embeddings: Information Loss in Vision–Language Models Words or Vision: Do Vision-Language Models Have Blind Faith in Text? Forgotten Polygons: Multimodal Large Language Models are Shape-Blind Vision Transformers Need Registers On the rankability of visual embeddings Interpretability for Diffusion Models survey awesome-generative-explainability representation general The hidden language of diffusion models Denoising diffusion autoencoders are unified self-supervised learners Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry localization Localizing Knowledge in Diffusion Transformers Precise Parameter Localization for Textual Generation in Diffusion Models Stable flow: Vital layers for training-free image editing Unraveling MMDiT Blocks: Training-free Analysis and Enhancement of Text-conditioned Diffusion submission (Revisiting Block-wise Interactions of MMDiT for Training-free Improved Synthesis) review motion (for video gen models) Video Diffusion Models are Training-free Motion Interpreter and Controller Emergent Temporal Correspondences from Video Diffusion Transformers inference Temporal Concept Dynamics in Diffusion Models via Prompt-Conditioned Interventions Towards Understanding the Working Mechanism of Text-to-Image Diffusion Model modules positional encoding FreeFlux: Understanding and Exploiting Layer-Specific Roles in RoPE-Based MMDiT for Versatile Image Editing Exploring Position Encoding in Diffusion U-Net for Training-free High-resolution Image Generation attention What the DAAM: Interpreting Stable Diffusion Using Cross Attention circuit Localizing and editing knowledge in text-to-image generative models On Mechanistic Knowledge Localization in Text-to-Image Generative Models Circuit Mechanisms for Spatial Relation Generation in Diffusion Transformers SAE Emergence and Evolution of Interpretable Concepts in Diffusion Models SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders Tide: Temporal-aware sparse autoencoders for interpretable diffusion transformers in image generation steering vector Video Unlearning via Low-Rank Refusal Vector Decoding Vision Transformers: the Diffusion Steering Lens learning dynamics Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models visualization * others VideoEraser: Concept Erasure in Text-to-Video Diffusion Models Radial Attention: O(nlogn) Sparse Attention with Energy Decay for Long Video Generation Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation Model Ac3d: Analyzing and improving 3d camera control in video diffusion transformers application-guided Emergent Correspondence from Image Diffusion Moaw: Unleashing Motion Awareness for Video Diffusion Models Make It Count: Text-to-Image Generation with an Accurate Number of Objects Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference Other fields of MLLMs visual pretraining Scaling Language-Free Visual Representation Learning spatial models (VIRAL) Visual Representation Alignment for Multimodal Large Language Models (LaVer) Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models analyses Understanding How Positional Encodings Work in Transformer Model Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ViT+LLM > ViT) Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder ! Learning Visual Composition through Improved Semantic Guidance (prompt-based) Things not Written in Text: Exploring Spatial Commonsense from Visual Signals (prompt-based) Does CLIP Bind Concepts? Probing Compositionality in Large Image Models Probing the Role of Positional Information in Vision-Language Models cognitive linguistics / positional metaphor I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors repo The Neural Basis of Metaphor Comprehension: Evidence from Left Hemisphere Degeneration 论 Langacker 的 CRPM 图的合理性与解释力 方位隐喻构式 (Metaphors we live by) 英汉“上–下”方位隐喻对比研究综述 Language and Space 方位概念”前后“的空间隐喻研究 blogs 英汉空间隐喻拓展义的对比研究 REC (看 related work) Exploring Spatial Language Grounding Through Referring Expressions (文本引导) ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension attention Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs Lost in the middle: How language models use long contexts Efficient streaming language models with attention sinks (delimiters) Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection MagicPIG: LSH Sampling for Efficient LLM Generation Label words are anchors: An information flow perspective for understanding in-context learning ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features POS “闭门造车”之多模态思路浅谈（三）：位置编码 社区供稿 | 图解RoPE旋转位置编码及其特性 Base of RoPE Bounds Context Length Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding NoPE Transformer Language Models without Positional Encodings Still Learn Positional Information Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings The Impact of Positional Encoding on Length Generalization in Transformers (NoPE limits) Length Generalization of Causal Transformers without Position Encoding Long context Extending Context Window of Large Language Models via Positional Interpolation YaRN: Efficient Context Window Extension of Large Language Models hallucination survey Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models Hallucination of Multimodal Large Language Models: A Survey *Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs Interpreting and editing vision-language representations to mitigate hallucinations Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference Debiasing Multimodal Large Language Models token compression survey When Tokens Talk Too Much: A Survey of Multimodal Long-Context Token Compression across Images, Videos, and Audios methods Token Merging: Your Vit But Faster (CDPruner) Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs *AdaFV: Rethinking of Visual-Language alignment for VLM acceleration (FasterVLM) [CLS] Attention is All You Need for Training-FreeVisual Token Pruning: Make VLM Inference Faster Sparsevlm: Visual token sparsification for efficient vision-language Model Inference (FastV) An image is worth 1/2 tokens after layer 2: Plug-and-PLay Acceleration for VLLM Inference LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token *Inference Optimal VLMs Need Only One Visual Token but Larger Models TokenPacker: Efficient Visual Projector for Multimodal LLM Matryoshka Multimodal Models Matryoshka Query Transformer for Large Vision-Language Models FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registers Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding active perception Please refer to this post. Datasets & Benchmarks general VQA: Visual Question Answering repo download [(VQA v2.0) Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering] (GQA) GQA:ANewDataset for Real-World Visual Reasoning and Compositional Question Answering [website]https://cs.stanford.edu/people/dorarad/gqa/index.html (TextVQA) Towards VQA Models That Can Read website repo MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models repo MMBench: Is Your Multi-modal Model an All-around Player? spatial (ARO) When and why vision-language models behave like bags-of-words, and what to do about it? (Whatsup) What’s “up” with vision-language models? Investigating their struggle with spatial reasoning [repo]https://github.com/amitakamath/whatsup_vlms (VSR) Visual Spatial Reasoning [repo]https://github.com/cambridgeltl/visual-spatial-reasoning SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data (SpatialMQA) Can Multimodal Large Language Models Understand Spatial Relations? SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation (COMFORT) Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models (MindCube) Spatial Mental Modeling from Limited Views How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions video Action100M: A Large-scale Video Action Dataset hallucination (POPE) Evaluating object hallucination in large vision-language models repo (CHAIR) Object hallucination in image captioning repo (OpenCHAIR) Mitigating Open-Vocabulary Caption Hallucinations website Models LLM foundation models Qwen3 Technical Report Qwen2 Technical Report The Llama 3 Herd of Models gpt-oss Introducing gpt-oss | OpenAI OpenAI开源模型gpt-oss-120b的妙妙小观察 关于gpt-oss那些值得关注的点 (Transformer) Attention is all you need modules embedding Over-tokenized transformer: Vocabulary is generally worth scaling residual Hyper-connections LayerNorm QK Norm and the Curious Case of Logit Drift MLLM foundation models survey (BradyFu) Awesome-Multimodal-Large-Language-Models (yfzhang114) Awesome-Multimodal-Large-Language-Models Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution Qwen2.5-VL Technical Report 万字长文图解Qwen2.5-VL实现细节 (DFN) Data Filtering Networks Qwen3-VL Technical Report Seed1.5-VL Technical Report Kimi K2.5: Visual Agentic Intelligence LLaVA系列 (LLaVA) Visual Instruction Tuning (LLaVA-1.5) Improved Baselines with Visual Instruction Tuning InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (InternVL 1.5) How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites (InternVL 2.5) Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (BLIP-2) BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Flamingo: a Visual Language Model for Few-Shot Learning modules ViT repr Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs Eagle: Exploring the design space for multimodal llms with mixture of encoders arch basic (VIT) An Image is Worth 16x16 Words: Transformers for Image Recognition fat Scale (CLIP) Learning Transferable Visual Models From Natural Language Supervision (SigLIP) Sigmoid Loss for Language Image Pre-Training (PACL) Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning others Differentiable Hierarchical Visual Tokenization (Quadtree) Accelerating Vision Transformers with Adaptive Patch Sizes VLM arch Fuyu-8B: A Multimodal Architecture for AI Agents (SAIL) The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer 字节 SAIL 论文解读 resolution Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution Swin Transformer: Hierarchical Vision Transformer using Shifted Windows DualFocus: A Unified Framework for Integrating Positive and Negative Descriptors in Text-based Person Retrieval self-supervised learning (SimCLR) A Simple Framework for Contrastive Learning of Visual Representations (MoCo) Momentum Contrast for Unsupervised Visual Representation Learning Beit: Bert pre-training of image transformers (MAE) Masked Autoencoders Are Scalable Vision Learners (iBOT) Image BERT: A New Vision-Language Pre-training Framework (BYOL) Bootstrap Your Own Latent A New Approach to Self-Supervised Learning (DINO) Emerging Properties in Self-Supervised Vision Transformers DINOv2: Learning Robust Visual Features without Supervision DINOv3 Self-Supervised Learning 超详细解读 (目录) 万字长文超详解读之DINO全系列—视觉表征对比学习的高峰 万字长文超详解之DINO-V3（DINO全系列之补充篇） DINO&DINO v2：颠覆自监督视觉特征表示学习 generative models The Principles of Diffusion Models video-generation-survey 视觉生成超详细解读 (目录) blogs Stable Diffusion技术路线发展历程回顾 Understanding Diffusion Models: A Unified Perspective 浅谈扩散模型的有分类器引导和无分类器引导 扩散模型与能量模型，Score-Matching和SDE，ODE的关系 Difussion Model、Flow Matching 与 Rectified Flow 浅析 SD3的采样上篇——Flow Matching SD3的采样下篇——Rectified Flow 深入解析Flow Matching技术 一文通透流匹配Flow Matching：作为扩散模型的变体，广泛应用于文生图和具身动作模型去噪中(含Rectified Flow的详解) 生成模型大道至简｜Rectified Flow基础概念｜代码 FLUX.1概要——原SD核心团队推出的最强文生图 FLUX.1 原理与源码解析 Stable Diffusion 3 论文及源码概览 basic (VAE) Auto-Encoding Variational Bayes vae和重参数化技巧 Autoregressive image generation without vector quantization (VQGAN) Taming transformers for high-resolution image synthesis VQGAN 论文与源码解读：前Diffusion时代的高清图像生成模型 (DDPM) Denoising Diffusion Probabilistic Models (DDIM) Denoising Diffusion Implicit Models (Classifier-guided) Diffusion Models Beat GANs on Image Synthesis (Classifier-free) Classifier-free diffusion guidance Generative Modeling by Estimating Gradients of the Data Distribution Score-Based Generative Modeling through Stochastic Differential Equations (EDM) Elucidating the Design Space of Diffusion-Based Generative Modelsßßß (GLIDE) GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (Imagen) Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (DALL-E) Zero-Shot Text-to-Image Generation (DALL-E-2) Hierarchical Text-Conditional Image Generation with CLIP Latents (DALL-E-3) Improving Image Generation with Better Captions (LDM) High-resolution image synthesis with latent diffusion models (ControlNet) Adding Conditional Control to Text-to-Image Diffusion Models (UViT) All are Worth Words: A ViT Backbone for Diffusion Models (DiT) Scalable Diffusion Models with Transformers SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers (MDT) Masked diffusion transformer is a strong image synthesizer (MaskDiT) Fast training of diffusion models with masked transformers SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (SDXL Turbo) Adversarial Diffusion Distillation Flow Matching for Generative Modeling Flow Matching Guide and Code (Rectified Flow) Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow (SD3, MMDiT) Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (VAR) Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (LlamaGen) Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation modules VAE (RAE t2i) Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders (iREPA) What matters for Representation Alignment: Global Information or Spatial Structure? (RAE) Diffusion Transformers with Representation Autoencoders (VA-VAE) Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (REPA-E) REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers (REPA) Representation alignment for generation: Training diffusion transformers is easier than you think attention survey A Survey of Efficient Attention Methods: Hardware-efficient, Sparse, Compact, and Linear Attention kv cache (VSA) Faster Video Diffusion with Trainable Sparse Attention (NSA) Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention MoBA: Mixture of Block Attention for Long-Context LLMs linear attention 线性注意力简史：从模仿、创新到反哺 algorithms acceleration RayFlow: Instance-Aware Diffusion Acceleration via Adaptive Flow Trajectories Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis Mean Flows for One-step Generative Modeling On Distillation of Guided Diffusion Models Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference Consistency Models Progressive distillation for fast sampling of diffusion models blogs 扩散模型算法加速之一：扩散模型加速 image models FLUX.2: Frontier Visual Intelligence FLUX.2: Analyzing and Enhancing the Latent Space of FLUX – Representation Comparison Qwen-Image Technical Report Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer image editing Prompt-to-Prompt Image Editing with Cross Attention Control InstructPix2Pix: Learning to Follow Image Editing Instructions Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry Diffusion Model-Based Image Editing: A Survey stylization AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning video survey The Dawn of Video Generation: Preliminary Explorations with SORA-like Models A Survey on Video Diffusion Models blogs awesome-video-generation 扩散模型各个方向都有哪些经典文献值得一读（附链接）？ Diffusion Models for Video Generation models VideoGPT: Video Generation using VQ-VAE and Transformers ViViT: A Video Vision Transformer Video Diffusion Models Imagen Video: High Definition Video Generation with Diffusion Models Make-A-Video: Text-to-Video Generation without Text-Video Data Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (Gen-1) Structure and Content-Guided Video Synthesis with Diffusion Models VDT: General-purpose Video Diffusion Transformers via Mask Modeling Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning (PixelDance) Make Pixels Dance: High-Dynamic Video Generation VideoPoet: A Large Language Model for Zero-Shot Video Generation (Video-LDM) Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (SVD) Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets Lumiere: A Space-Time Diffusion Model for Video Generation Latte: Latent Diffusion Transformer for Video Generation (Sora) Video generation models as world simulators Open-Sora: Democratizing Efficient Video Production for All Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer Hunyuanvideo: A systematic framework for large video generative models HunyuanVideo 1.5 Technical Report Wan: Open and advanced large-scale video generative models Seedance 1.0: Exploring the Boundaries of Video Generation Models (Veo3 test) Video models are zero-shot learners and reasoners LongCat-Video Technical Report long video train on short videos bidirectional Streamingt2v: Consistent, dynamic, and extendable long video generation from text causal Context Forcing: Consistent Autoregressive Video Generation with Long Context Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation LoL: Longer than Longer, Scaling Video Generation to Hour Pretraining Frame Preservation in Autoregressive Video Memory Compression Rolling Forcing: Autoregressive Long Video Diffusion in Real Time LongLive: Real-time Interactive Long Video Generation Self-Forcing++: Towards Minute-Scale High-Quality Video Generation Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (CausVid) From slow bidirectional to fast autoregressive video diffusion models Diffusion forcing: Next-token prediction meets full-sequence diffusion MAGI-1: Autoregressive Video Generation at Scale train on long videos bidirectional direct coarse-to-fine training-free Fifo-diffusion: Generating infinite videos from text without training memory management StoryMem: Multi-shot Long Video Storytelling with Memory video editing AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks TokenFlow: Consistent Diffusion Features for Consistent Video Editing STABLEV2V: Stablizing Shape Consistency in Video-to-Video Editing VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing RL Flow-GRPO: Training Flow Matching Models via Online RL DanceGRPO: Unleashing GRPO on Visual Generation (ReFL) Imagereward: Learning and evaluating human preferences for text-to-image generation 3D / 4D Birth and Death of a Rose audio AST: Audio Spectrogram Transformer (DAC) High-Fidelity Audio Compression with Improved RVQGAN Synchformer: Efficient Synchronization from Sparse Cues v2a HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation unified generation and comprehension survey Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities models Chameleon: Mixed-Modal Early-Fusion Foundation Models VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model Show-o: One single transformer to unify multimodal understanding and generation Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation Janus-Pro Janus+Janus-Pro 论文解析 JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation code (UniFluid) Unified Autoregressive Visual Generation and Understanding with Continuous Tokens Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation DreamLLM: Synergistic multimodal comprehension and creation (MetaQuery) Transfer between Modalities with MetaQueries Metamorph: Multimodal understanding and generation via instruction tuning Exploring MLLM-Diffusion Information Transfer with MetaCanvas (BAGEL) Emerging Properties in Unified Multimodal Pretraining Emu: Generative Pretraining in Multimodality (Emu2) Generative Multimodal Models are In-Context Learners Emu3: Next-Token Prediction is All You Need NExT-GPT： Any-to-Any Multimodal LLM BLIP3-o: A Family of Fully Open Unified Multimodal Models—Architecture, Training and Dataset Blip3o-next: Next frontier of native image generation OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models Univideo: Unified understanding, generation, and editing for videos **Unified Multimodal Model as Auto-Encoder NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation analysis Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation other relevant papers AToken: A Unified Tokenizer for Vision blogs 2025 年，生成和理解多模态大模型一些思考 evaluation 图像生成常用指标：IS score 和 FID score world models survey Awesome World Models blogs 3D/4D World Model（WM）近期发展的总结和思考 VLA 对VLA的RL最新进展梳理 VLA扩充数据来源方法进展梳理 JEPA JEPA：自主机器智能的“蛋糕胚”——JEPA模型发展脉络梳理（第一弹） (I-JEPA) Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (V-JEPA) Revisiting feature prediction for learning visual representations from video V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning repo LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures world simulator A Summary (LingBot-World) Advancing Open-source World Models “Embodied AI” A Survey on Vision-Language-Action Models: An Action Tokenization Perspective navigation Navigation World Models physical reasoning Denoising Hamiltonian Network for Physical Reasoning

Methods#

Interpretability for MLLMs#

Interpretability for Diffusion Models#

Other fields of MLLMs#

Datasets & Benchmarks#

general#

spatial#

video#

hallucination#

Models#

LLM#

MLLM#

self-supervised learning#

generative models#

world models#

Methods

Interpretability for MLLMs

Interpretability for Diffusion Models

Other fields of MLLMs

Datasets & Benchmarks

general

spatial

video

hallucination

Models

LLM

MLLM

self-supervised learning

generative models

world models