Efficient and Practical Optimization in Large Language Models

Current Trends in Large Language Model Optimization

Recent advancements in the field of Large Language Models (LLMs) have predominantly focused on optimizing memory efficiency, improving model editing techniques, and enhancing pruning strategies. The general direction of the field is moving towards more efficient and practical methods that reduce computational costs and memory demands without compromising model performance. Innovations in subspace optimization, layer-wise pruning, and sequential editing are paving the way for more scalable and deployable LLMs, particularly for edge devices with limited resources.

Noteworthy Developments:

  • SubZero: Introduces a low-rank perturbation method for memory-efficient fine-tuning, significantly reducing variance in gradient estimates.
  • AlphaPruning: Utilizes Heavy-Tailed Self-Regularization Theory to allocate layerwise sparsity ratios more effectively, achieving high sparsity without performance loss.
  • O-Edit: Proposes an orthogonal subspace editing approach that minimizes interference between successive knowledge updates, enhancing sequential editing performance.

Sources

SubZero: Random Subspace Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

FAME: Towards Factual Multi-Task Model Editing

AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix

Subspace Optimization for Large Language Models with Convergence Guarantees

O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing

DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models

Model Balancing Helps Low-data Training and Fine-tuning

Built with on top of