Interpretability (& other areas) for MLLMs and diffusion models

💡 This post is initially focused on interpretability for multimodal models, while later a lot of papers in other fields are included, just for convenience.

Methods

Interpretability for MLLMs

• survey
  • A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models
  • Sparks of Explainability Recent Advancements in Explaining Large Vision Models
  • Awesome LMMs Mechanistic Interpretability
• probing
  • Probing Multimodal Large Language Models for Global and Local Semantic Representations
• representation
  • Zoom in: An introduction to circuits
  • Multimodal Neurons in Artificial Neural Networks
  • Interpreting CLIP’s Image Representation via Text-Based Decomposition
  • Interpreting the Second-Order Effects of Neurons in CLIP
  • CLIP不同层
  • Multimodal Neurons in Pretrained Text-Only Transformers
  • Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers?
  • Understanding Video Transformers via Universal Concept Discovery
• circuit
  • Pathways of Visual Information Flow in Vision-Language Models
  • **(causal tracing) Understanding Information Storage and Transfer in Multi-modal Large Language Models
  • Automatic Discovery of Visual Circuits
  • Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP
• SAE
  • Case study: Interpreting, manipulating, and controlling clip with sparse autoencoders
  • Towards multimodal interpretability: Learning sparse interpretable features in vision transformers
  • Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery
• visualization
  • Interpreting Object-level Foundation Models via Visual Precision Search
  • VITAL: More Understandable Feature Visualization through Distribution Alignment and Relevant Information Flow
  • Visualizer！简化你的Vision Transformer可视化！
  • (DVT) Denoising Vision Transformers
  • Token Activation Map to Visually Explain Multimodal LLMs
  • LVLM-Intrepret: An Interpretability Tool for Large Vision Language Models
  • Transformer Interpretability Beyond Attention Visualization
  • Decoding Vision Transformers: the Diffusion Steering Lens
• modules
  • attention
    • What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation
• others
  • Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation
  • **Towards interpreting visual information processing in vision-language models
    • demo
  • (dogit lens) Laying the Foundations for Vision and Multimodal Mechanistic Interpretability & Open Problems
  • Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space
  • Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Models
• tools
  • VLM-Lens
• information flow
  • Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs
  • **Cross-modal Information Flow in Multimodal Large Language Models
  • *From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks
  • *What’s in the Image? A Deep-Dive into the Vision of Vision Language Models
  • The Narrow Gate: Localized Image-Text Communication in Vision-Language Models
  • Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models
  • Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference
• analyses on MLLMs
  • Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
  • Lost in Embeddings: Information Loss in Vision–Language Models
  • Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
  • Forgotten Polygons: Multimodal Large Language Models are Shape-Blind
  • Vision Transformers Need Registers
  • On the rankability of visual embeddings
• learning dynamics
  • A Theoretical Understanding of shallow Vision Transformers: Learning, Generalization, and Sample Complexity

Interpretability for Diffusion Models

• survey
  • awesome-generative-explainability
• representation
  • general
    • Interpreting Physics in Video World Models
      • V-JEPA 系列
    • The Hidden Language of Diffusion Models
    • FreeU: Free Lunch in Diffusion U-Net
    • Denoising diffusion autoencoders are unified self-supervised learners
    • Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry
  • localization
    • Localizing Knowledge in Diffusion Transformers
    • Precise Parameter Localization for Textual Generation in Diffusion Models
    • Stable flow: Vital layers for training-free image editing
    • Unraveling MMDiT Blocks: Training-free Analysis and Enhancement of Text-conditioned Diffusion
      • submission (Revisiting Block-wise Interactions of MMDiT for Training-free Improved Synthesis)
      • review
  • motion (for video gen models)
    • Video Diffusion Models are Training-free Motion Interpreter and Controller
    • Emergent Temporal Correspondences from Video Diffusion Transformers
  • attention
    • Cross-Attention Head Position Patterns Can Align with Human Visual Concepts in Text-to-Image Generative Models
    • Understanding Attention Mechanism in Video Diffusion Models
  • inference
    • Demystifying Video Reasoning
    • Video Models Reason Early: Exploiting Plan Commitment for Maze Solving
    • Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling
    • Temporal Concept Dynamics in Diffusion Models via Prompt-Conditioned Interventions
    • *Towards Understanding the Working Mechanism of Text-to-Image Diffusion Model
    • All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds
• modules
  • positional encoding
    • FreeFlux: Understanding and Exploiting Layer-Specific Roles in RoPE-Based MMDiT for Versatile Image Editing
    • *Exploring Position Encoding in Diffusion U-Net for Training-free High-resolution Image Generation
  • attention
    • Attention of a Kiss: Exploring Attention Maps in Video Diffusion for XAIxArts
    • What the DAAM: Interpreting Stable Diffusion Using Cross Attention
• probing
  • Gen4U: Unifying Video Generation and Understanding via Diffusion
  • Interpreting Physics in Video World Models
• circuit
  • Localizing and editing knowledge in text-to-image generative models
  • On Mechanistic Knowledge Localization in Text-to-Image Generative Models
  • Circuit Mechanisms for Spatial Relation Generation in Diffusion Transformers
• SAE
  • Emergence and Evolution of Interpretable Concepts in Diffusion Models
  • Tide: Temporal-aware sparse autoencoders for interpretable diffusion transformers in image generation
  • SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders
• steering vector
  • Video Unlearning via Low-Rank Refusal Vector
  • Decoding Vision Transformers: the Diffusion Steering Lens
• learning dynamics
  • Biased Generalization in Diffusion Models
  • Bigger Isn’t Always Memorizing: Early Stopping Overparameterized Diffusion Models
  • Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training
  • An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models
• visualization
  • AttentionBender: Manipulating Cross-Attention in Video Diffusion Transformers as a Creative Probe
• others
  • The Spacetime of Diffusion Models: An Information Geometry Perspective
  • VideoEraser: Concept Erasure in Text-to-Video Diffusion Models
  • Radial Attention: O(nlogn) Sparse Attention with Energy Decay for Long Video Generation
  • Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation Model
  • Ac3d: Analyzing and improving 3d camera control in video diffusion transformers
• application-guided
  • Physics-Driven Spatiotemporal Modeling for AI-Generated Video Detection
  • Emergent Correspondence from Image Diffusion
  • Moaw: Unleashing Motion Awareness for Video Diffusion Models
  • Make It Count: Text-to-Image Generation with an Accurate Number of Objects
  • Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference

Other fields of MLLMs

• spatial
  • models
    • Learning to Draw ASCII Improves Spatial Reasoning in Language Models
    • (VIRAL) Visual Representation Alignment for Multimodal Large Language Models
    • (LaVer) Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
    • Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
    • SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
    • SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models
  • analyses
    • Understanding How Positional Encodings Work in Transformer Model
    • Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models
    • Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas
    • (ViT+LLM > ViT) Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder
    • ! Learning Visual Composition through Improved Semantic Guidance
    • (prompt-based) Things not Written in Text: Exploring Spatial Commonsense from Visual Signals
    • (prompt-based) Does CLIP Bind Concepts? Probing Compositionality in Large Image Models
    • Probing the Role of Positional Information in Vision-Language Models
  • cognitive linguistics / positional metaphor
    • I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors
      • repo
    • The Neural Basis of Metaphor Comprehension: Evidence from Left Hemisphere Degeneration
    • 论 Langacker 的 CRPM 图的合理性与解释力
    • 方位隐喻构式 (Metaphors we live by)
    • 英汉“上–下”方位隐喻对比研究综述
    • Language and Space
    • 方位概念”前后“的空间隐喻研究
    • blogs
      • 英汉空间隐喻拓展义的对比研究
  • REC
    • (看 related work) Exploring Spatial Language Grounding Through Referring Expressions
    • (文本引导) ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension
• hallucination
  • survey
    • Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models
    • Hallucination of Multimodal Large Language Models: A Survey
  • *Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
  • Interpreting and editing vision-language representations to mitigate hallucinations
  • Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs
  • Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference
  • Debiasing Multimodal Large Language Models
• token compression
  • survey
    • When Tokens Talk Too Much: A Survey of Multimodal Long-Context Token Compression across Images, Videos, and Audios
  • methods
    • Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models
    • When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs
    • (ToMe) Token Merging: Your Vit But Faster
    • (CDPruner) Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
    • DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models
    • *AdaFV: Rethinking of Visual-Language alignment for VLM acceleration
    • (FasterVLM) [CLS] Attention is All You Need for Training-FreeVisual Token Pruning: Make VLM Inference Faster
    • Sparsevlm: Visual token sparsification for efficient vision-language Model Inference
    • (FastV) An image is worth 1/2 tokens after layer 2: Plug-and-PLay Acceleration for VLLM Inference
    • LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
    • *Inference Optimal VLMs Need Only One Visual Token but Larger Models
    • TokenPacker: Efficient Visual Projector for Multimodal LLM
    • Matryoshka Multimodal Models
    • Matryoshka Query Transformer for Large Vision-Language Models
    • FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression
    • FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registers
    • Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding

Datasets & Benchmarks

general

• Humanity’s Last Exam
• Multimodal Benchmarks
• (ARC-AGI) On the measure of intelligence
  • website
• BabyVision: Visual Reasoning Beyond Language
  • Hugging Face
• VQA: Visual Question Answering
  • repo
  • download
• [(VQA v2.0) Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering]
• (GQA) GQA:ANewDataset for Real-World Visual Reasoning and Compositional Question Answering
  • [website]https://cs.stanford.edu/people/dorarad/gqa/index.html
• (TextVQA) Towards VQA Models That Can Read
  • website
  • repo
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
  • repo
• MMBench: Is Your Multi-modal Model an All-around Player?

small-object

• JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles
• ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use

Coding & GUI

• Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

spatial

• (ARO) When and why vision-language models behave like bags-of-words, and what to do about it?
• (Whatsup) What’s “up” with vision-language models? Investigating their struggle with spatial reasoning
  • [repo]https://github.com/amitakamath/whatsup_vlms
• (VSR) Visual Spatial Reasoning
  • [repo]https://github.com/cambridgeltl/visual-spatial-reasoning
• SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
• (SpatialMQA) Can Multimodal Large Language Models Understand Spatial Relations?
• SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation
• (COMFORT) Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities
• ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
• (MindCube) Spatial Mental Modeling from Limited Views
• How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective
• Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions
• GST-Bench: Can VLMs Develop Global Spatial Awareness from Video?

video

• understanding
  • Action100M: A Large-scale Video Action Dataset
  • Longvideobench: A benchmark for long-context interleaved video-language understanding
  • Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
• generation
  • VBench: Comprehensive Benchmark Suite for Video Generative Models
  • VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness
  • MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
  • physics
    • (Physics-IQ) Do generative video models understand physical principles?
    • (PhyGenBench) Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation
      • data
    • VideoPhy: Evaluating Physical Commonsense for Video Generation
      • repo
    • VIDEOPHY-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation
      • Hugging Face

hallucination

• (POPE) Evaluating object hallucination in large vision-language models
  • repo
• (CHAIR) Object hallucination in image captioning
  • repo
• (OpenCHAIR) Mitigating Open-Vocabulary Caption Hallucinations
  • website

Training

Learning Dynamics

• Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Optimization

• Scaling Diffusion Transformers Efficiently via µP
• Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

Data

• Scaling Domain Data Repetition in LLM Pretraining

Infra

• The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  • PDF on Z-Library
• 腾讯混元AI Infra如何优化Hy3 Preview：一次大模型推理性能提升的技术拆解

Models

LLM

• foundation models
  • survey
    • LLM Architecture Gallery
  • general
    • Qwen3 Technical Report
    • Qwen2 Technical Report
    • The Llama 3 Herd of Models
    • gpt-oss
      • Introducing gpt-oss | OpenAI
      • OpenAI开源模型gpt-oss-120b的妙妙小观察
      • 关于gpt-oss那些值得关注的点
      • OpenAI GPT-OSS 系列的模型解读 —— Attention Sink
    • (Transformer) Attention is all you need
• architecture
  • Embedding
    • Over-tokenized transformer: Vocabulary is generally worth scaling
  • Residual
    • Hyper-connections
    • Frac-Connections: Fractional Extension of Hyper-Connections
    • mHC: Manifold-Constrained Hyper-Connections
    • Attention Residuals
      • 谈谈Kimi Attention Res的工作
    • Mixture-of-Depths Attention
    • Hyperloop Transformers
  • LayerNorm
    • QK Norm and the Curious Case of Logit Drift
  • Attention
    • survey
      • attention survey (repo)
    • general
      • Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
      • 缓存与效果的极限拉扯：从MHA、MQA、GQA到MLA
      • deepseek技术解读(1)-彻底理解MLA（Multi-Head Latent Attention）
      • Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance
      • MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
      • Lost in the middle: How language models use long contexts
      • Efficient streaming language models with attention sinks
      • (delimiters) Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
      • EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
      • MagicPIG: LSH Sampling for Efficient LLM Generation
      • Label words are anchors: An information flow perspective for understanding in-context learning
      • ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
    • theoretical
      • Stabilizing Transformer Training by Preventing Attention Entropy Collapse
    • linear attention
      • introduction
        • flash-linear-attention
        • 线性注意力简史：从模仿、创新到反哺
        • 详细分析一下Attention, Sparse or Linear?
        • 线性注意力机制(杲哥 wiki)
        • Understanding LSTM Networks
      • methods
        • Morphing into Hybrid Attention Models
        • Scaling Linear Attention with Sparse State Expansion
        • Qwen3-Next源码解析：一文看清架构中的核心点实现
        • (KDA) Kimi Linear: An Expressive, Efficient Attention Architecture
          • Kimi Linear：混合线性注意力架构全面超越全注意力
      • analyses
        • Temporal superposition and feature geometry of RNNs under memory demands
        • A Systematic Analysis of Hybrid Linear Attention
        • Bridging the Divide: Reconsidering Softmax and Linear Attention
    • sparse attention
      • survey
        • A Survey of Efficient Attention Methods: Hardware-efficient, Sparse, Compact, and Linear Attention
      • (NSA) Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
        • 【手撕NSA】DeepSeek新作-原生稀疏注意力-超长文(附代码)
      • MoBA: Mixture of Block Attention for Long-Context LLMs
      • InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
      • Generating Long Sequences with Sparse Transformers
    • attention sink
      • A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
      • How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
      • What are you sinking? A geometric approach on attention sink
      • Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
      • The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
      • Interpreting the Repeated Token Phenomenon in Large Language Models
    • Positional encoding
      • blogs
        • Qwen2-VL-> Qwen3-VL中的MRoPE演进
        • Blogs by CyPaul Space
          • 抛砖引玉：浅谈ROPE位置编码模式下，q、k的分布（均值与方差）对注意力远程衰减的影响
          • 再探RoPE（二）：为什么RoPE + Bias能在远程衰减和长度外推上发挥重要作用？
          • 再探RoPE（三）：从群论视角统一理解位置编码——为什么RoPE几乎是唯一的选择，以及远程衰减究竟从何而来
        • Using group theory to explore the space of positional encodings for attention
        • “闭门造车”之多模态思路浅谈（三）：位置编码
        • 社区供稿 | 图解RoPE旋转位置编码及其特性
      • Revisiting Multimodal Positional Encoding in Vision-Language Models
      • Base of RoPE Bounds Context Length
        • Transformer升级之路：18、RoPE的底数选择原则
      • Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models
      • Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
      • NoPE
        • Transformer Language Models without Positional Encodings Still Learn Positional Information
        • Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
        • The Impact of Positional Encoding on Length Generalization in Transformers
        • (NoPE limits) Length Generalization of Causal Transformers without Position Encoding
      • Long context
        • Extending Context Window of Large Language Models via Positional Interpolation
        • YaRN: Efficient Context Window Extension of Large Language Models
  • MLP
    • activation function
      • 为什么所有主流LLM都使用SwiGLU？
      • 拆解 LLM 中的 SwiGLU
    • moe
      • MoE环游记：1、从几何意义出发
      • MoE环游记：3、换个思路来分配
      • MoE环游记：5、均匀分布的反思
      • A Closer Look into Mixture-of-Experts in Large Language Models

self-supervised learning

• blogs
  • Self-Supervised Learning 超详细解读 (目录)
  • 万字长文超详解读之DINO全系列—视觉表征对比学习的高峰
  • 万字长文超详解之DINO-V3（DINO全系列之补充篇）
  • DINO&DINO v2：颠覆自监督视觉特征表示学习
• papers
  • A Cookbook of Self-Supervised Learning
  • image
    • Perception Encoder: The best visual embeddings are not at the output of the network
    • Scaling Language-Free Visual Representation Learning
    • (SimCLR) A Simple Framework for Contrastive Learning of Visual Representations
    • (MoCo) Momentum Contrast for Unsupervised Visual Representation Learning
    • Beit: Bert pre-training of image transformers
    • (MAE) Masked Autoencoders Are Scalable Vision Learners
    • (iBOT) Image BERT: A New Vision-Language Pre-training Framework
    • (BYOL) Bootstrap Your Own Latent A New Approach to Self-Supervised Learning
    • (DINO) Emerging Properties in Self-Supervised Vision Transformers
    • DINOv2: Learning Robust Visual Features without Supervision
    • DINOv3
  • video
    • InternVideo-Next: Towards World Understanding Video Models
    • VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
    • JEPA
      • blogs
        • JEPA：自主机器智能的“蛋糕胚”——JEPA模型发展脉络梳理（第一弹）
      • papers
        • (I-JEPA) Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
        • (V-JEPA) Revisiting feature prediction for learning visual representations from video
        • V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
          • repo
        • V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
        • LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
        • A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures
        • (EB-JEPA) When Does LeJEPA Learn a World Model?
      • analysis
        • Intuitive physics understanding emerges from self-supervised pretraining on natural videos
  • others
    • Video Generation Models are General-Purpose Vision Learners

MLLM

• foundation models
  • survey
    • (BradyFu) Awesome-Multimodal-Large-Language-Models
    • (yfzhang114) Awesome-Multimodal-Large-Language-Models
  • VLMs
    • Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
    • Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
    • Qwen2.5-VL Technical Report
      • 万字长文图解Qwen2.5-VL实现细节
      • (DFN) Data Filtering Networks
    • Qwen3-VL Technical Report
    • Qwen3.5: Towards Native Multimodal Agents
    • Qwen3.6-Plus: Towards Real World Agents
    • Qwen3.7-Plus: Multimodal Agent Intelligence
    • Qwen3.8-Max: A New Bar for Coding and Cowork
    • Seed1.5-VL Technical Report
    • Kimi K2.5: Visual Agentic Intelligence
    • LLaVA系列
    • (LLaVA) Visual Instruction Tuning
    • (LLaVA-1.5) Improved Baselines with Visual Instruction Tuning
    • LLaVA-OneVision: Easy Visual Task Transfer
    • LLaVA-OneVision-2: Towards next-generation perceptual intelligence
    • InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
    • (InternVL 1.5) How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
    • (InternVL 2.5) Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
    • (BLIP-2) BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
    • Flamingo: a Visual Language Model for Few-Shot Learning\
  • Omni
    • blogs
      • Codec、RVQ、Dual AR、Thinker-Talker——深入 Qwen3-Omni 与 S2 Pro 的 Omni 模型推理流程
    • Qwen3.5-Omni Technical Report
    • Qwen3-Omni Technical Report
      • code
      • huggingface
    • Qwen2.5-Omni Technical Report
    • Mini-omni: Language models can hear, talk while thinking in streaming
• architecture
  • vision
    • basic
      • (VIT) An Image is Worth 16x16 Words: Transformers for Image Recognition fat Scale
      • (CLIP) Learning Transferable Visual Models From Natural Language Supervision
      • (SigLIP) Sigmoid Loss for Language Image Pre-Training
      • (PACL) Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning
      • EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
      • EVA-CLIP: Improved Training Techniques for CLIP at Scale
    • representation
      • DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs
        • Qwen3-VL 架构使用了 DeepStack 策略
      • Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
      • Eagle: Exploring the design space for multimodal llms with mixture of encoders
    • video
      • (ELVA) Breaking the Encoder Barrier for Seamless Video-Language Understanding
      • codec-based
        • How Codec Choice Becomes the Bottleneck in AI Video Pipelines
        • Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
        • OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence
        • CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling
          • Sayan Deb Sarkar’s Talk
        • Efficient Motion-Aware Video MLLM
        • Dense Video Understanding with Gated Residual Tokenization
        • Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization
        • Lei’s Blog
    • resolution
      • Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
      • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
      • DualFocus: A Unified Framework for Integrating Positive and Negative Descriptors in Text-based Person Retrieval
    • others
      • Differentiable Hierarchical Visual Tokenization
      • (Quadtree) Accelerating Vision Transformers with Adaptive Patch Sizes
  • vision-language
    • Fuyu-8B: A Multimodal Architecture for AI Agents
    • (SAIL) The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer
      • 字节 SAIL 论文解读
  • attention
    • VideoRoPE: What Makes for Good Video Rotary Position Embedding?
    • VideoNSA: Native Sparse Attention Scales Video Understanding

generative models

• The Principles of Diffusion Models

• video-generation-survey

• 视觉生成超详细解读 (目录)

• blogs

  • DDPM推导
  • Stable Diffusion技术路线发展历程回顾
  • Understanding Diffusion Models: A Unified Perspective
  • 浅谈扩散模型的有分类器引导和无分类器引导
  • 扩散模型与能量模型，Score-Matching和SDE，ODE的关系
  • Difussion Model、Flow Matching 与 Rectified Flow 浅析
  • SD3的采样上篇——Flow Matching
  • SD3的采样下篇——Rectified Flow
  • 深入解析Flow Matching技术
  • 一文通透流匹配Flow Matching：作为扩散模型的变体，广泛应用于文生图和具身动作模型去噪中(含Rectified Flow的详解)
  • 生成模型大道至简｜Rectified Flow基础概念｜代码
  • FLUX.1概要——原SD核心团队推出的最强文生图
  • FLUX.1 原理与源码解析
  • Stable Diffusion 3 论文及源码概览

• basic

  • (VAE) Auto-Encoding Variational Bayes
    • vae和重参数化技巧
  • Autoregressive image generation without vector quantization
  • (VQGAN) Taming transformers for high-resolution image synthesis
    • VQGAN 论文与源码解读：前Diffusion时代的高清图像生成模型
  • (DDPM) Denoising Diffusion Probabilistic Models
  • (DDIM) Denoising Diffusion Implicit Models
  • (Classifier-guided) Diffusion Models Beat GANs on Image Synthesis
  • (Classifier-free) Classifier-free diffusion guidance
  • Generative Modeling by Estimating Gradients of the Data Distribution
  • Score-Based Generative Modeling through Stochastic Differential Equations
  • (EDM) Elucidating the Design Space of Diffusion-Based Generative Models
  • (GLIDE) GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
  • (Imagen) Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
  • (DALL-E) Zero-Shot Text-to-Image Generation
  • (DALL-E-2) Hierarchical Text-Conditional Image Generation with CLIP Latents
  • (DALL-E-3) Improving Image Generation with Better Captions
  • (LDM) High-resolution image synthesis with latent diffusion models
    • Stable Diffusion 的 UNet 和 Attention
    • Latent Diffusion Models (LDMs) 模型学习笔记
  • (ControlNet) Adding Conditional Control to Text-to-Image Diffusion Models
  • (UViT) All are Worth Words: A ViT Backbone for Diffusion Models
  • (DiT) Scalable Diffusion Models with Transformers
  • SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers
  • (MDT) Masked diffusion transformer is a strong image synthesizer
  • (MaskDiT) Fast training of diffusion models with masked transformers
  • Autoregressive Image Generation without Vector Quantization
  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
  • (SDXL Turbo) Adversarial Diffusion Distillation
  • Flow Matching for Generative Modeling
  • Flow Matching Guide and Code
  • (Rectified Flow) Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
  • (SD3, MMDiT) Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
  • (VAR) Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
  • (LlamaGen) Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation

• architecture

  • VAE
    • (RAE t2i) Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
    • (iREPA) What matters for Representation Alignment: Global Information or Spatial Structure?
    • (RAE) Diffusion Transformers with Representation Autoencoders
    • (VA-VAE) Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models
    • (REPA-E) REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers
    • (REPA) Representation alignment for generation: Training diffusion transformers is easier than you think
    • (REG) Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think
  • attention
    • (VSA) Faster Video Diffusion with Trainable Sparse Attention
    • Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuning

• algorithms

  • acceleration
    • (DMD) One-step Diffusion with Distribution Matching Distillation
    • RayFlow: Instance-Aware Diffusion Acceleration via Adaptive Flow Trajectories
    • Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis
    • Mean Flows for One-step Generative Modeling
    • On Distillation of Guided Diffusion Models
    • Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
    • Consistency Models
    • Progressive distillation for fast sampling of diffusion models
    • blogs
      • 扩散模型算法加速之一：扩散模型加速
  • RL
    • Qwen-Image-2.0-RL Technical Report
    • Improving Video Generation with Human Feedback
    • Flow-GRPO: Training Flow Matching Models via Online RL
      • FlowGRPO 解读
    • DanceGRPO: Unleashing GRPO on Visual Generation
    • (ReFL) Imagereward: Learning and evaluating human preferences for text-to-image generation
    • DiffusionNFT: Online Diffusion Reinforcement with Forward Process

• image

  • models
    • FLUX.2: Frontier Visual Intelligence
      • FLUX.2: Analyzing and Enhancing the Latent Space of FLUX – Representation Comparison
    • Qwen-Image Technical Report
    • Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer
  • general
    • PixelDiT: Pixel diffusion transformers for image generation
    • Back to basics: Let denoising generative models denoise
  • image editing
    • Prompt-to-Prompt Image Editing with Cross Attention Control
    • InstructPix2Pix: Learning to Follow Image Editing Instructions
    • Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
    • Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models
    • Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry
    • Diffusion Model-Based Image Editing: A Survey
  • stylization
    • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

• video

  • survey
    • The Dawn of Video Generation: Preliminary Explorations with SORA-like Models
    • A Survey on Video Diffusion Models
    • Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling
  • blogs
    • awesome-video-generation
    • 扩散模型各个方向都有哪些经典文献值得一读（附链接）？
    • Diffusion Models for Video Generation
  • models
    • VideoGPT: Video Generation using VQ-VAE and Transformers
    • ViViT: A Video Vision Transformer
    • Video Diffusion Models
    • Imagen Video: High Definition Video Generation with Diffusion Models
    • Make-A-Video: Text-to-Video Generation without Text-Video Data
    • Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
    • (Gen-1) Structure and Content-Guided Video Synthesis with Diffusion Models
    • VDT: General-purpose Video Diffusion Transformers via Mask Modeling
    • Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning
    • (PixelDance) Make Pixels Dance: High-Dynamic Video Generation
    • VideoPoet: A Large Language Model for Zero-Shot Video Generation
    • (Video-LDM) Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
    • (SVD) Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
    • Lumiere: A Space-Time Diffusion Model for Video Generation
    • Latte: Latent Diffusion Transformer for Video Generation
    • (Sora) Video generation models as world simulators
    • Open-Sora: Democratizing Efficient Video Production for All
    • Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models
    • CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers
    • CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
    • Hunyuanvideo: A systematic framework for large video generative models
    • HunyuanVideo 1.5 Technical Report
    • Wan: Open and advanced large-scale video generative models
    • Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
    • Wan-Streamer v0.2: Higher Resolution, Same Latency
    • Video =World + Event Stream
    • Seedance 1.0: Exploring the Boundaries of Video Generation Models
    • Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model
    • Seedance 2.0: Advancing video generation for world complexity
    • (Veo3 test) Video models are zero-shot learners and reasoners
    • LongCat-Video Technical Report
    • (LingBot-Video) Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence
  • long video
    • train on short videos
      • bidirectional
        • Streamingt2v: Consistent, dynamic, and extendable long video generation from text
      • causal
        • Context Forcing: Consistent Autoregressive Video Generation with Long Context
        • Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation
        • Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation
        • LoL: Longer than Longer, Scaling Video Generation to Hour
        • Pretraining Frame Preservation in Autoregressive Video Memory Compression
        • Rolling Forcing: Autoregressive Long Video Diffusion in Real Time
        • LongLive: Real-time Interactive Long Video Generation
        • Self-Forcing++: Towards Minute-Scale High-Quality Video Generation
        • Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
        • (CausVid) From slow bidirectional to fast autoregressive video diffusion models
        • Diffusion forcing: Next-token prediction meets full-sequence diffusion
        • MAGI-1: Autoregressive Video Generation at Scale
    • train on long videos
      • bidirectional
        • direct
        • coarse-to-fine
    • training-free
      • Fifo-diffusion: Generating infinite videos from text without training
    • memory management
      • StoryMem: Multi-shot Long Video Storytelling with Memory
  • video editing
    • AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks
    • TokenFlow: Consistent Diffusion Features for Consistent Video Editing
    • STABLEV2V: Stablizing Shape Consistency in Video-to-Video Editing
    • VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing
  • physics
    • survey
      • Generative physical AI in vision: A survey
    • model arch
      • Learning How the World Evolves: Extrapolative Video World Models via Latent Dynamics Reasoning
      • Bridging Vision and Language: Modeling Causality and Temporality in Video Narratives
      • (DiT-Mem) Learning Plug-and-play Memory for Guiding Video Diffusion Models
      • ProPhy: Progressive Physical Alignment for Dynamic World Simulation
      • WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation
      • (Phys-AR) Reasoning physical video generation with diffusion timestep tokens via reinforcement learning
    • training-based
      • WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation
      • VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models
      • (DiffPhy) Think Before You Diffuse: LLMs-Guided Physics-Aware Video Generation
      • NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics
      • What about gravity in video generation? Post-Training Newton’s Laws with Verifiable Reward
      • VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
      • Teaching video diffusion model with latent physical phenomenon knowledge
      • Physmaster: Mastering physical representation for video generation via reinforcement learning
      • Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning
      • FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance
      • Geometry forcing: Marrying video diffusion and 3d representation for consistent world modeling
    • training-free
      • Visual prompt engineering for video models
      • (WWReward) Inference-time Physics Alignment of Video Generative Models with Latent World Models
      • VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System
      • (Ask-A-Video) Controlling Video Generation with Vision Language Models
      • Phyt2v: Llm-guided iterative self-refinement for physics-grounded text-to-video generation
      • VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior
      • PhyRPR: Training-Free Physics-Constrained Video Generation
      • GPT4motion: Scripting physical motions in text-to-video generation via blender-oriented gpt planning
      • VISTA: A Test-Time Self-Improving Video Generation Agent
    • data
      • Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise
      • (PhysHPO) Hierarchical fine-grained preference optimization for physically plausible video generation
    • analysis
      • How Far Is Video Generation from World Model: A Physical Law Perspective
      • Intuitive physics understanding emerges from self-supervised pretraining on natural videos
    • bench
      • (Physics-IQ) Do generative video models understand physical principles?
      • VideoPhy: Evaluating Physical Commonsense for Video Generation
      • (PhyGenBench) Towards world simulator: Crafting physical commonsense-based benchmark for video generation
      • [VBench]
      • *Likephys: Evaluating intuitive physics understanding in video diffusion models via likelihood preference

• 3D / 4D

  • Birth and Death of a Rose

• audio

  • AST: Audio Spectrogram Transformer
  • (DAC) High-Fidelity Audio Compression with Improved RVQGAN
  • Synchformer: Efficient Synchronization from Sparse Cues
  • v2a
    • HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation

• unified generation and comprehension

  • survey
    • Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities
  • models
    • Chameleon: Mixed-Modal Early-Fusion Foundation Models
    • VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation
    • TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation
    • MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding
    • Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
    • Show-o: One single transformer to unify multimodal understanding and generation
    • Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action
    • Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
    • Janus-Pro
      • Janus+Janus-Pro 论文解析
    • JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
      • code
    • (UniFluid) Unified Autoregressive Visual Generation and Understanding with Continuous Tokens
    • Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation
    • SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
    • DreamLLM: Synergistic multimodal comprehension and creation
    • (MetaQuery) Transfer between Modalities with MetaQueries
    • Metamorph: Multimodal understanding and generation via instruction tuning
    • Exploring MLLM-Diffusion Information Transfer with MetaCanvas
    • (BAGEL) Emerging Properties in Unified Multimodal Pretraining
    • Emu: Generative Pretraining in Multimodality
    • (Emu2) Generative Multimodal Models are In-Context Learners
    • Emu3: Next-Token Prediction is All You Need
    • NExT-GPT： Any-to-Any Multimodal LLM
    • BLIP3-o: A Family of Fully Open Unified Multimodal Models—Architecture, Training and Dataset
    • Blip3o-next: Next frontier of native image generation
    • OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models
    • Univideo: Unified understanding, generation, and editing for videos
    • **Unified Multimodal Model as Auto-Encoder
    • NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation
    • LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
    • Image Generators are Generalist Vision Learners
    • Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation
    • Continuous Latent Diffusion Language Model
    • Unified Multimodal Autoregressive Modeling with Shared Context—Visual Tokenizer is Key to Unification
    • (SenseNova-Vision) Vision as Unified Multimodal Generation
    • Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes
  • unified RL
    • UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation
  • analysis
    • Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models
    • Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation
  • other relevant papers
    • AToken: A Unified Tokenizer for Vision
  • blogs
    • 2025 年，生成和理解多模态大模型一些思考

• evaluation

  • 图像生成常用指标：IS score 和 FID score

world models

• survey
  • Awesome World Models
• blogs
  • 3D/4D World Model（WM）近期发展的总结和思考
• resources
  • stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation
    • repo
• VLA
  • survey
    • A Survey on Vision-Language-Action Models: An Action Tokenization Perspective
  • blogs
    • 对VLA的RL最新进展梳理
    • VLA扩充数据来源方法进展梳理
  • papers
    • GE-Sim 2.0: A Roadmap Towards Comprehensive Closed-loop Video World Simulators for Robotic Manipulation
    • VLAFlow: A Unified Training Framework for Vision-Language-Action Models via Co-training and Future Latent Alignment
    • Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments
    • Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models
    • Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation System
    • Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation
• world models
  • A Summary
  • Wonder: Video World Model Done Better
  • (LingBot-World) Advancing Open-source World Models
  • Navigation World Models
• physical reasoning
  • Denoising Hamiltonian Network for Physical Reasoning