Efficient Model Adaptation and Knowledge Preservation

The recent advancements in parameter-efficient fine-tuning (PEFT) methods for large-scale models are significantly reshaping the landscape of multi-modal and vision-language models. Researchers are increasingly focusing on strategies that preserve the pre-trained representation space while efficiently adapting models to new tasks. This trend is evident in the development of techniques like prefix-tuning, which excels at maintaining the original model's representation space, and dual low-rank adaptation, which addresses the challenge of catastrophic forgetting in continual learning scenarios. Additionally, innovations in sparse tuning and visual Fourier prompt tuning are providing solutions to the memory and performance degradation issues associated with large-scale model adaptation. These methods not only enhance the adaptability of models but also ensure that the models retain their generalizability and efficiency. Notably, the integration of Fourier transforms into prompt tuning and the use of sparse orthogonal parameters in continual learning are particularly groundbreaking, offering new paradigms for model adaptation and knowledge retention. Overall, the field is moving towards more sophisticated, yet parameter-efficient, methods that balance performance, memory usage, and the preservation of pre-trained knowledge.

Sources

Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models

Dual Low-Rank Adaptation for Continual Learning with Pre-Trained Models

Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Prediction

Visual Fourier Prompt Tuning

Expanding Sparse Tuning for Low Memory Usage

SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models

Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models

Sparse Orthogonal Parameters Tuning for Continual Learning

Layer-Adaptive State Pruning for Deep State Space Models

Two-Stage Pretraining for Molecular Property Prediction in the Wild

MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba

Built with on top of