The field of transformer model compression is rapidly advancing, with a clear trend towards more efficient, high-performing models that can be deployed in resource-constrained environments. Recent developments have focused on innovative pruning strategies, knowledge distillation, and novel compression techniques that maintain or even enhance model performance while significantly reducing size and computational costs. These methods include strategic fusion of pruning signals, dynamic pruning at any desired ratio, CUR matrix decomposition for weight approximation, merging of similar parameter groups, and shared weight strategies for similar channels. Additionally, there's a growing interest in understanding and controlling the internal mechanisms of transformers to facilitate reasoning-based compositional generalization, which could lead to more interpretable and generalizable models. The field is also seeing advancements in Neural Architecture Search (NAS) for Vision Transformers, with methods that automate the design of efficient neural architectures through structured pruning and parameter prioritization.

Noteworthy Papers

• Strategic Fusion Optimizes Transformer Compression: Introduces strategic fusion of pruning signals and knowledge distillation, achieving near-optimal performance and improved accuracy-to-size ratios across datasets.
• Adaptive Pruning of Pretrained Transformer via Differential Inclusions: Proposes a dynamic pruning method that allows for pruning at any desired ratio within a single stage, offering greater flexibility and customization.
• CURing Large Models: Compression via CUR Decomposition: Presents CURing, a novel compression method based on CUR matrix decomposition, significantly reducing model size with minimal performance loss.
• Merging Feed-Forward Sublayers for Compressed Transformers: Demonstrates a novel approach to model compression by merging similar parameter groups, maintaining original performance while reducing parameter count.
• SuperSAM: Crafting a SAM Supernetwork via Structured Pruning and Unstructured Parameter Prioritization: Introduces a new method for ViT NAS search-space design, resulting in subnetworks that are smaller yet outperform the original model.
• Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers: Investigates the influence of complexity control strategies on transformers' ability to learn reasoning rules, with broad applicability across tasks.
• SWSC: Shared Weight for Similar Channel in LLM: Proposes an LLM compression method based on shared weight for similar channels, effectively ensuring performance under low-precision conditions.