AgentFlareScholar · multimodal foundation models
Multimodal Foundation Models: Research Digest
10 foundational papers with 2,727 combined citations, plus the latest arXiv preprints.
AISA Scholar API · arXiv
You're reading the full report — free for humans and search engines.
The same content ships as a structured JSON dataset (normalized facts, sources and
citation arrays, no HTML parsing) so AI agents can consume it directly — metered per fetch via the open
HTTP 402 payment protocol.
crawlerAutoPrice=true · verify at https://esa.aisa.one/api/v1/access/verify
What’s new on arXiv
Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs
Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung, Oh Hyun-Bin, Hyeonggon Ryu, David Harwath, Tae-Hyun Oh
Audio-visual large language models (AV-LLMs) have made strong progress in multimodal understanding, yet most treat audio as monaural semantic content, limiting their ability to reason about where sounds originate and how they relate to the visual scene. Recent spatial audio-visual studies address parts of this problem, but often focus on specific abilities, such as spatial correspondence or direction and distance reasoning. In this work, we introduce HLR-AVSceneQA, a comprehensive benchmark with a hear-localize-reason hierarchy that evaluates whether models can recognize what is heard, localize sound sources in egocentric and allocentric views, and use these localized sources to reason about how they relate to the visual scene. We further introduce HLR-LLM, which augments a strong audio-visual foundation model with a binaural spatial audio branch alongside its monaural semantic audio pathway. We train HLR-LLM with a curriculum that incorporates final-stage chain-of-thought supervision for multi-step spatial reasoning. Experiments show that existing AV-LLMs remain limited in spatial grounding and relational reasoning, while HLR-LLM substantially improves spatial audio-visual scene understanding without compromising semantic audio-visual perception.
MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data
Robert Moseley, Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian, Salman Asif
Large multimodal models have achieved remarkable progress by scaling model capacity and training on massive paired multimodal data. However, this paradigm introduces a fundamental bottleneck: aligned multimodal datasets are expensive to curate, difficult to scale across diverse modality combinations, and increasingly constrain further progress. We introduce Modality Self-Play (MSP), a training framework for multimodal foundation models based on proxy-mediated modality composition. Instead of relying on paired cross-modal data, MSP uses anchor-supporting proxy textual descriptions of unseen modalities as semantic bridges during training. Given an observed modality (e.g., image or audio), the model generates proxy descriptions for complementary unseen modalities, which are then encoded and aligned with the observed modality through learned bridges, enabling cross-modal interaction without real paired non-text supervision. Our central hypothesis is that proxy text, combined with contrastively trained pretrained encoders, provides sufficient semantic structure for multimodal composition to emerge. We show that MSP enables zero-shot modality composition: models trained only with a real anchor modality and accompanying proxy descriptions can integrate and reason over multiple modalities at inference time despite never observing real paired non-text data during training. Empirically, MSP achieves state-of-the-art or competitive performance on multiple multimodal benchmarks. Ablation studies further demonstrate that learned bridges improve alignment, while anchor-supporting proxy descriptions enable effective cross-modal composition. MSP removes the need for real paired non-text (e.g., audio-vision) supervision, while continuing to rely on modality-text pairs, pretrained contrastive encoders, and modality-distribution bridge calibration.
MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining
Yanran Ruan, Zhixiang Wei, Shiyao Hong, Xiaoxiao Ma, Yuxuan Gu, Liming Ye, Huaian Chen, Yi Jin
Understanding fluorescence microscopy images is fundamental to automating biological research. However, existing methods typically learn predefined concepts from closed-set data, limiting transfer across tasks and biological contexts. In this work, we introduce MicroCLIP, a vision-language model that learns generalizable cellular and subcellular representations from natural language supervision for open-set image classification and retrieval. To pretrain MicroCLIP, we construct MicroPool-2M, a dataset of approximately 2.5 million fluorescence microscopy image-text pairs spanning diverse cellular structures, cell lines, and imaging conditions. Aligning microscopy images with biological concepts in natural language is challenging because most images lack descriptive captions and biological knowledge annotations. We therefore progressively establish image-text associations at three levels to provide MicroCLIP with richer semantic supervision: (1) category-level alignment links images to standardized labels defined by subcellular structure-cell-line combinations; (2) knowledge-level alignment incorporates biological priors derived from these labels to capture relationships across species and cell types; and (3) detail-level alignment uses a multimodal large language model to recaption and verify image-text pairs at scale, emphasizing fine-grained visual characteristics. Extensive experiments demonstrate that MicroCLIP achieves leading performance in zero-shot classification and text-image retrieval on microscopy images. It retrieves images with specific cellular characteristics from thousands of candidates and generalizes to unseen biological concepts without additional supervision. These results highlight MicroCLIP's potential as a foundation model for fluorescence microscopy.
HairFlow: Modeling 3D Hair Strands with Multi-Modal Conditioned Diffusion
Yating Wang, Kui Wu, Ran Yi, Zixuan Lu, Tongtong Wang, Zhongtian Zheng, Lizhuang Ma, Yuefan Shen
Modeling 3D hair strands is challenging due to complex hairstyle geometry, scarce high-quality training data, and incomplete or noisy input observations. Foundation models offer broader hairstyle priors, but their outputs are not directly usable for strand-based rendering and simulation. Existing approaches require costly per-instance optimization to fit hair priors to meshes from foundation models and 2D orientations from specialized predictors trained on limited hair data, restricting efficiency and generalization. We present HairFlow, a multimodal conditioned diffusion framework for efficient and controllable hair strands modeling. HairFlow conditions strand generation on two complementary representations: a coarse mesh that constrains global shape and multi-view sketches that specify local strand flow. HairFlow combines an Orientation VAE with a conditioned diffusion transformer to generate volumetric fields that are traced into explicit strands. To overcome the lack of paired sketch-strand training data, we introduce an automatic synthesis pipeline that generates geometrically aligned multi-view sketches with diverse styles representative of inference-time inputs. Experiments show that HairFlow generates plausible 3D strands that faithfully follow the input conditions across diverse hairstyles, substantially reduces inference time, and supports flexible multimodal control.
UniMetric: Universal Metric Scale Recovery for Visual Geometry Foundation Models
Lei Yang, Yuner Zhang, Ziyang Liu, Ziying Song, Kai WU, Yiyang Gao, Jiangtao Su, Yiming Li
Recent advances in multimodal foundation models, exemplified by GPT-6 Astra, raise the prospect of general-purpose embodied reasoning and planning. Using Internet videos for embodied pretraining calls for reliable physical-scale annotations. Yet visual geometry foundation models often recover scene structure and camera motion without reliable metric scale. We show that uncertain size priors of familiar objects can jointly constrain a reconstruction's shared scale. Building on this insight, we present UniMetric, a universal and training-free metric scale recovery framework for geometry foundation models, without additional sensors or task-specific metric supervision. Specifically, Metric Prior Acquisition elicits object height intervals from a vision-language model. Semantic-Geometric Alignment measures reconstructed object heights along the estimated support-surface normal in each frame and converts the priors into scale intervals. Global Scale Recovery reconciles these intervals through a convex objective, yielding a jointly feasible scale when they overlap and a unique compromise otherwise. The estimated sequence-level scale is applied uniformly to reconstructed points and camera translations. Controlled experiments with real VLM priors show that combining observations improves scale accuracy with exact relative object heights. To evaluate metric accuracy on GFM reconstructions without ground-truth scale alignment, we introduce MetricBench, covering twelve embodied, driving, and roadside subsets with sensor-derived ground truth. Across nine geometry foundation models, UniMetric reduces mean depth Abs Rel, reconstruction Chamfer distance, and trajectory error by 54.2%, 83.8%, and 65.6%, respectively, improving every model on all three tasks at the suite level. We release the UniMetric code and MetricBench data.
A Survey of Flow Matching in Reinforcement Learning
Nabuat Zaman Nahim, Fairoz Nower Khan, Peizhong Ju
Flow Matching (FM) has recently emerged as a principled and efficient generative modeling framework for reinforcement learning (RL), enabling expressive, multimodal policy parameterizations via deterministic probability transport. Compared to diffusion-based policies that rely on stochastic denoising chains, FM uses sampling based on ordinary differential equations (ODEs), with learned velocity fields, which can substantially reduce inference latency and simplify the incorporation of RL objectives. As research in flow-based RL rapidly accelerates across offline continuous control, online fine-tuning, and foundation model alignment, the literature has become highly fragmented. In this survey, we provide a comprehensive taxonomy of flow-matching approaches in reinforcement learning. We organize the literature along two axes: the target distribution being modeled (e.g., action policies, value critics, transition dynamics) and the mechanism of RL signal integration (e.g., energy-weighted regression, flow-based policy gradients, and group relative policy optimization). Furthermore, we survey emerging frontiers such as discrete and non-Euclidean action spaces, provide a systematic comparative analysis against Gaussian and diffusion baselines, and outline critical open problems. Ultimately, this survey serves as a foundational roadmap for the next generation of generative reinforcement learning and alignment.
Toward a Multimodal Foundation Model for Plasma State Reconstruction
Bin Xia, Chuanfei Dong, Boting Li, Yilan Qin, Zikang Xie, Sung Hyun Son, Adam J. Stanier, Jongsoo Yoo, Hantao Ji, Ahmed Diallo, Han Ding, J…
Plasma experiments observe coupled physical states through sparse, heterogeneous, and incomplete diagnostics. We present a proof of concept toward a multimodal foundation model for plasma state reconstruction during magnetic reconnection, motivated by the Facility for Laboratory Reconnection Experiments (FLARE). Using 250 synthetic VPIC simulations at fixed reference electron beta β_e0=0.2, with 225 training runs and 25 disjoint validation runs, we train a 5.5M-parameter residual 3D U-Net to reconstruct three magnetic-field components and electron density from incomplete observations. The model combines spatial-only pooling, which preserves all 24 temporal frames, with global spatiotemporal self-attention and rotary positional encoding. Independently masking magnetic and density modalities across random, grid, block, and temporal patterns enables sparse sensing, cross-modal completion, density super-resolution, and temporal completion with a single set of weights and no task-specific retraining. With density fully hidden, just 0.1% magnetic spatial visibility reduces median density NRMSE from approximately 0.9 to 0.1. Conversely, observed density improves current-density reconstruction when magnetic data are absent or extremely sparse. With no magnetic observations, roughly 1% density visibility approaches the density reconstruction accuracy achieved with fully observed magnetic fields, although current structure remains more dependent on direct magnetic measurements. Full magnetic conditioning also stabilizes completion of missing future density frames. These results demonstrate bidirectional cross-modal predictive coupling within this simulation regime. Generalization across plasma regimes or to experimental measurements remains untested.
TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision
Hao Zhu, Heyu Sun, Shuo Jin, Jiayu Xiao, Feng Dai
The recent Segment Anything Model 3 (SAM3) extends powerful promptable segmentation from spatial cues to concepts, raising the prospect of a single foundation model for universal image segmentation. However, this promise often fails to translate into reliable performance on downstream segmentation tasks. In this work, we empirically identify concept recognition, rather than mask quality, as the primary bottleneck: SAM3 accurately delineates the objects it recognizes yet struggles to robustly determine which concepts are present. Driven by this, we present textitunderlineText underlineAnchor re-underlineGrounding (TAG), a simple yet effective framework that rectifies the concept recognition of SAM3 on the target domain without dense annotations. Specifically, TAG introduces an Anchor Residual Pathway comprising image-conditioned, low-rank adapters to re-ground text anchors in the target distribution and modulate subsequent text-conditioned interactions. Guided solely by image-level category labels, TAG jointly optimizes binary cross-entropy with Soft Boundary Ranking objectives to calibrate presence confidence and distinguish present concepts from confusable negatives. Building upon this, we devise Confidence-Weighted Distillation to couple rich semantic context of a multimodal large language model with fine-grained spatial priors of SAM3, thereby achieving effective calibration in the absence of manual annotations. Extensive experiments demonstrate that TAG unlocks SAM3's segmentation potential with only 3.1M trainable parameters, achieving impressive performance across diverse scenarios, and notably, even surpassing leading mask-supervised methods on several metrics. hrefhttps://github.com/MiSsU-HH/TAG_SAM3Our code will be publicly available at GitHub faGithub.
Foundational papers
- 2022
- 2023
- 2024
- 2024
- 2024
- 2025Intern-s1: A scientific multimodal foundation model118 citations
- 2025Mio: A foundation model on multimodal tokens35 citations
- 2026-10
Multimodal foundation models: From specialists to general-purpose assistants
C Li, Z Gan, Z Yang, J Yang, L Li, L Wang et al.
… we limit the scope of multimodal foundation models to the vision and vision-language domains. Recent survey papers on related topics include (i) image understanding models such as …
Towards artificial general intelligence via a multimodal foundation model
N Fei, Z Lu, Y Gao, G Yang, Y Huo, J Wen, H Lu et al.
… a foundation model pre-trained with huge multimodal data, … to pre-train our foundation model by self-supervised learning … Particularly, with the developed modelinterpretability tools, …
Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models.
Z Chen, L Xu, H Zheng, L Chen, A Tolba et al.
… Adhering to the hierarchical structure of the outline, with a focus on large language models and multimodal models as primary keywords, we meticulously curated a collection of …
Hemm: Holistic evaluation of multimodal foundation models
PP Liang, A Goindani, T Chafekar et al.
… and study progress in multimodal foundation models, given the range of … of Multimodal Models (HEMM) to systematically evaluate the capabilities of multimodal foundation models …
Advances in multimodal adaptation and generalization: From traditional approaches to foundation models
H Dong, M Liu, K Zhou, E Chatzi et al.
… , which represent traditional multimodal settings with varying … foundation models including (d) unimodal domain adaptation and generalization assisted by multimodal foundation models…
A survey of resource-efficient llm and multimodal foundation models
M Xu, W Yin, D Cai, R Yi, D Xu, Q Wang, B Wu et al.
… To support the growth of these large models in a scalable … topics from cutting-edge model architectures and training/serving … posed by large foundation models and to potentially inspire …
Intern-s1: A scientific multimodal foundation model
L Bai, Z Cai, Y Cao, M Cao, W Cao, C Chen et al.
… In contrast, our method updates all model parameters jointly during multimodal CPT process. Let M denote a Transformer-based model parameterized by θ that is capable of processing …
Mio: A foundation model on multimodal tokens
ZM Wang, K Zhu, C Xu, W Zhou, J Liu et al.
… MIO, a novel foundation model built on multimodal tokens, ca… models (LLMs) and multimodal large language models (MM-… four modalities using causal multimodal modeling. MIO un…
Internvideo2: Scaling foundation models for multimodal video understanding
Y Wang, K Li, X Li, J Yu, Y He, G Chen, B Pei et al.
… Compared with only using the multimodal model in UMT and VideoPrism, our strategy makes the vision encoder multimodalfriendly as well as enhances its temporal sensitivity for …
Vip5: Towards multimodal foundation models for recommendation
S Geng, J Tan, S Liu, Z Fu, Y Zhang
… the development of a multimodal foundation model (MFM) … To achieve this, we introduce multimodal personalized … -efficient training method for foundation models, which involves …
How this page is made. Foundational papers and citation counts come from academic search via the AISA Scholar API, with abstracts read from each paper’s page; recent preprints and monthly volumes come from the public arXiv API (metadata CC0). The digest is written by a language model from those abstracts only — every theme links the papers it draws on, and figures not in the inputs are removed.
Sources & citations
- Multimodal foundation models: From specialists to general-purpose assistants 530 citations
- Towards artificial general intelligence via a multimodal foundation model 490 citations
- Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models. 166 citations
- Hemm: Holistic evaluation of multimodal foundation models 42 citations
- Advances in multimodal adaptation and generalization: From traditional approaches to foundation models 70 citations
- A survey of resource-efficient llm and multimodal foundation models 281 citations
- Intern-s1: A scientific multimodal foundation model 118 citations
- Mio: A foundation model on multimodal tokens 35 citations
- Internvideo2: Scaling foundation models for multimodal video understanding 845 citations
- Vip5: Towards multimodal foundation models for recommendation 150 citations
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs arXiv · 2026-10-15
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data arXiv · 2026-10-14
- MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining arXiv · 2026-10-13
- HairFlow: Modeling 3D Hair Strands with Multi-Modal Conditioned Diffusion arXiv · 2026-10-12
- UniMetric: Universal Metric Scale Recovery for Visual Geometry Foundation Models arXiv · 2026-10-12
- A Survey of Flow Matching in Reinforcement Learning arXiv · 2026-10-12
- Toward a Multimodal Foundation Model for Plasma State Reconstruction arXiv · 2026-10-10
- TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision arXiv · 2026-10-10