Federated Learning

Report on Current Developments in Federated Learning

General Direction of the Field

The field of Federated Learning (FL) is currently witnessing a surge in innovative approaches aimed at addressing the inherent challenges of data heterogeneity and model heterogeneity. These challenges arise due to the decentralized nature of FL, where data is distributed across multiple clients with varying distributions and characteristics. The recent advancements focus on enhancing the efficiency, accuracy, and convergence of FL models by introducing novel techniques that better handle the heterogeneity of data and models.

One of the primary directions in the field is the integration of advanced statistical and machine learning techniques to mitigate the effects of data heterogeneity. This includes the use of Neural Tangent Kernels (NTK) to improve the expressiveness of model updates, adaptive normalization methods to standardize weights and features, and distribution balancing techniques to align heterogeneous data distributions. These methods aim to enhance the robustness and generalizability of FL models across diverse datasets and network topologies.

Another significant trend is the development of personalized FL approaches that cater to the unique needs of individual clients or domains. This includes the use of Latent Diffusion Models (LDM) for synthesizing personalized data, multi-domain prototype-based fine-tuning for domain adaptation, and the application of James-Stein Estimators for multi-domain FL. These personalized approaches not only improve model performance but also reduce communication costs and privacy risks.

Additionally, there is a growing emphasis on addressing the challenges of partial-modality missing data and the drift of Batch Normalization (BN) statistics in FL. Novel frameworks like FedMAC and BN-SCAFFOLD are being proposed to tackle these issues by introducing cross-modal aggregation, contrastive regularization, and efficient variance reduction techniques. These advancements are crucial for improving the practicality and scalability of FL in real-world applications.

Overall, the field is moving towards more sophisticated and adaptive solutions that leverage advanced statistical methods, personalized learning, and efficient communication strategies to overcome the inherent challenges of FL. These developments are poised to significantly advance the state-of-the-art in decentralized machine learning, making it more robust, efficient, and applicable to a wider range of real-world scenarios.

Noteworthy Papers

  • NTK-DFL: Introduces a synergy between NTK-based evolution and model averaging, significantly boosting accuracy and convergence in heterogeneous settings.
  • ANFR: Combines weight standardization and channel attention to enhance model performance under data heterogeneity, with minimal computational overhead.
  • FedMAC: Proposes a novel framework with contrastive-based regularization to handle partial-modality missing data, outperforming baselines by up to 26%.
  • FedStein: Enhances multi-domain FL through the James-Stein Estimator, achieving accuracy improvements exceeding 14% in certain domains.
  • FedBiP: Personalizes Latent Diffusion Models for heterogeneous one-shot FL, substantially outperforming other methods in challenging datasets.
  • MPFT: Introduces multi-domain prototype-based federated fine-tuning, significantly improving both in-domain and out-of-domain accuracy with reduced communication costs.

Sources

NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernel

Addressing Data Heterogeneity in Federated Learning with Adaptive Normalization-Free Feature Recalibration

PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation

FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization

BN-SCAFFOLD: controlling the drift of Batch Normalization statistics in Federated Learning

FedStein: Enhancing Multi-Domain Federated Learning Through James-Stein Estimator

FedBiP: Heterogeneous One-Shot Federated Learning with Personalized Latent Diffusion Models

FedL2G: Learning to Guide Local Training in Heterogeneous Federated Learning

Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning

Built with on top of