Advancements in Efficiency and Adaptability of Large Pre-trained Models

The recent developments in the field of machine learning and artificial intelligence are characterized by a strong focus on enhancing the efficiency, adaptability, and performance of large pre-trained models (LPMs) and large language models (LLMs). Innovations are particularly evident in the areas of model pruning, multi-objective optimization, continual learning, and parameter-efficient fine-tuning. These advancements aim to address key challenges such as computational resource demands, catastrophic forgetting, and the need for models to adapt to new tasks without extensive retraining.

A significant trend is the development of more sophisticated pruning techniques that not only reduce model size but also maintain or even improve performance on downstream tasks. Multi-objective optimization in deep learning is gaining traction, offering solutions that balance conflicting objectives across various applications. Continual learning strategies are being refined to better retain knowledge and adapt to new information, mitigating the issue of catastrophic forgetting. Additionally, parameter-efficient fine-tuning methods are emerging as a cost-effective way to adapt foundation models to specific tasks without the need for extensive computational resources.

Noteworthy papers include:

  • MultiPruner: Introduces a multidimensional pruning strategy that enhances zero-shot accuracy and model compression ratios.
  • DNA 1.0 Technical Report: Presents a bilingual language model with state-of-the-art performance on Korean and English tasks.
  • Control LLM: Proposes a method for continuous pre-training and fine-tuning that preserves existing knowledge while integrating new information.
  • Neural Contextual Reinforcement Framework: Enhances the logical coherence and structural consistency of text generated by LLMs.
  • Meta-Sparsity: A framework for learning optimal sparse structures in multi-task networks through meta-learning.
  • Learning Versatile Optimizers on a Compute Diet: Advances in learned optimizers that achieve strong meta-generalization with reduced computational resources.
  • Architectural Fusion Through Contextual Partitioning: Introduces a novel approach to parameterized knowledge integration in LLMs.
  • Spurious Forgetting in Continual Learning of Language Models: Investigates the phenomenon of spurious forgetting and proposes a freezing strategy to improve continual learning.
  • How to Complete Domain Tuning while Keeping General Ability in LLM: Addresses catastrophic forgetting through adaptive layer-wise and element-wise regularization.
  • Parameter-Efficient Fine-Tuning for Foundation Models: A comprehensive survey on PEFT techniques applied to diverse foundation models.

Sources

MultiPruner: Balanced Structure Removal in Foundation Models

DNA 1.0 Technical Report

Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond

Control LLM: Controlled Evolution for Intelligence Retention in LLM

Neural Contextual Reinforcement Framework for Logical Structure Language Generation

Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning

Learning Versatile Optimizers on a Compute Diet

Architectural Fusion Through Contextual Partitioning in Large Language Models: A Novel Approach to Parameterized Knowledge Integration

Spurious Forgetting in Continual Learning of Language Models

How to Complete Domain Tuning while Keeping General Ability in LLM: Adaptive Layer-wise and Element-wise Regularization

Parameter-Efficient Fine-Tuning for Foundation Models

Built with on top of