The field of multimodal intelligence is rapidly advancing, with a significant focus on enhancing the capabilities of multimodal large language models (MLLMs) and vision-language models (VLMs) through innovative approaches. A key trend is the exploration of Next Token Prediction (NTP) as a unifying framework for understanding and generation tasks across different modalities. This approach is being extended to improve the fine-grained understanding of visuals, optimize task performance, and reduce computational overhead through quantization techniques. Additionally, there is a growing interest in leveraging compositional generalization for medical imaging and accelerating MLLM inference through efficient token processing strategies. The development of methods for selecting high-impact training data and improving autoregressive visual generation further underscores the field's commitment to enhancing model efficiency and performance.

Noteworthy Papers

• Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey: Introduces a taxonomy for multimodal learning through NTP, covering tokenization, model architectures, and more.
• Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment: Proposes TPO to enhance MLLMs with visual tasks, showing significant performance improvements.
• MBQ: Modality-Balanced Quantization for Large Vision-Language Models: Presents MBQ for VLMs, improving task accuracy and computational efficiency.
• On the Compositional Generalization of Multimodal LLMs for Medical Imaging: Explores CG in MLLMs for medical imaging, demonstrating its potential for generalization.
• ST$^3$: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming: Offers a framework for faster MLLM inference without retraining.
• PTQ4VM: Post-Training Quantization for Visual Mamba: Introduces PTQ4VM for efficient quantization of Visual Mamba models.
• ICONS: Influence Consensus for Vision-Language Data Selection: Develops ICONS for selecting compact training datasets, maintaining high performance.
• Improving Autoregressive Visual Generation with Cluster-Oriented Token Prediction: Presents IAR for enhanced visual generation within the LLM framework.