Efficient Neural Network Optimization and Adaptation

Advances in Neural Network Pruning and Architecture Search

Recent developments in the field of neural network optimization have seen significant advancements in pruning techniques and neural architecture search (NAS). The focus has been on improving computational efficiency, reducing model size, and enhancing generalization capabilities, particularly under distribution shifts.

Pruning Innovations: The field has seen a shift towards more structured and gradient-aware pruning methods, which aim to maintain or even improve model accuracy while significantly reducing the number of parameters. These methods often leverage reinforcement learning or fixed-rate strategies to determine optimal pruning distributions across network layers, ensuring that the most critical components are preserved.

NAS Developments: NAS has progressed towards more efficient and data-free evaluation methods, with a focus on reducing the computational overhead associated with architecture performance evaluation. Techniques such as zero-shot NAS and gradient-based search mechanisms are being employed to accelerate the search process and improve the quality of selected architectures.

Noteworthy Papers:

  1. FGGP: Fixed-Rate Gradient-First Gradual Pruning - Introduces a novel gradient-first pruning strategy that outperforms state-of-the-art methods in various settings.
  2. Zero-Shot NAS via the Suppression of Local Entropy Decrease - Proposes a data-free proxy for architecture evaluation, significantly reducing computation time while maintaining high accuracy.
  3. RL-Pruner: Structured Pruning Using Reinforcement Learning - Utilizes reinforcement learning to optimize pruning distribution, achieving effective model compression and acceleration.
  4. Ghost-Connect Net: A Generalization-Enhanced Guidance For Sparse Deep Networks Under Distribution Shifts - Enhances network adaptability to distribution shifts by introducing a companion network for connectivity-based pruning.

Sources

FGGP: Fixed-Rate Gradient-First Gradual Pruning

Sample and Computationally Efficient Robust Learning of Gaussian Single-Index Models

Guiding Genetic Programming with Graph Neural Networks

Learning Morphisms with Gauss-Newton Approximation for Growing Networks

Zero-Shot NAS via the Suppression of Local Entropy Decrease

RL-Pruner: Structured Pruning Using Reinforcement Learning for CNN Compression and Acceleration

Permutative redundancy and uncertainty of the objective in deep learning

Exploring the loss landscape of regularized neural networks via convex duality

Searching Latent Program Spaces

Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergence

Complexity-Aware Training of Deep Neural Networks for Optimal Structure Discovery

Ghost-Connect Net: A Generalization-Enhanced Guidance For Sparse Deep Networks Under Distribution Shifts

Built with on top of