The recent developments in the research area of multimodal large language models (MLLMs) and vision-language models (VLMs) have shown significant advancements in understanding and generating complex visual and textual data. Key innovations include the integration of fine-grained concept annotations, the use of synthetic data for training, and the introduction of novel training paradigms that enhance both understanding and generation capabilities. Notably, there is a growing emphasis on improving the syntactic and semantic understanding of text within VLMs, as well as enhancing the models' ability to handle complex, multi-object scenes and spatial relationships. Additionally, the field is seeing a shift towards more efficient and scalable data generation processes, which are crucial for maintaining the high performance of these models as they grow in complexity. The notable papers in this area include those that introduce frameworks for multimodal understanding and generation, such as ILLUME and SynerGen-VL, which demonstrate the potential for unified models that can both understand and generate content effectively. Other significant contributions include the development of benchmarks and datasets, such as LAION-SG and CompreCap, which are essential for evaluating and advancing the state-of-the-art in this rapidly evolving field.
Advancing Multimodal Understanding and Generation in Vision-Language Models
Sources
Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
Sometimes I am a Tree: Data Drives Unstable Hierarchical Generalization
CompCap: Improving Multimodal Large Language Models with Composite Captions
A polar coordinate system represents syntax in large language models
HMGIE: Hierarchical and Multi-Grained Inconsistency Evaluation for Vision-Language Data Cleansing
Evaluating Hallucination in Text-to-Image Diffusion Models with Scene-Graph based Question-Answering Agent
Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models
Chimera: Improving Generalist Model with Domain-Specific Experts
Measuring Grammatical Diversity from Small Corpora: Derivational Entropy Rates, Mean Length of Utterances, and Annotation Invariance
LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations
The Narrow Gate: Localized Image-Text Communication in Vision-Language Models
ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance
VP-MEL: Visual Prompts Guided Multimodal Entity Linking
Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
FinFlier: Automating Graphical Overlays for Financial Visualizations with Knowledge-Grounding Large Language Model
ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models
The Pitfalls of Memorization: When Memorization Hurts Generalization
Barking Up The Syntactic Tree: Enhancing VLM Training with Syntactic Losses
Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models
Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models
Generate Any Scene: Evaluating and Improving Text-to-Vision Generation with Scene Graph Programming
Can We Generate Visual Programs Without Prompting LLMs?
LAION-SG: An Enhanced Large-Scale Dataset for Training Complex Image-Text Models with Structural Annotations
Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning
Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions
ViUniT: Visual Unit Tests for More Robust Visual Programming
Causal Graphical Models for Vision-Language Compositional Understanding
SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding