Efficiency and Privacy in Federated LLMs

Federated Learning and Privacy in Large Language Models

Recent advancements in the field of Large Language Models (LLMs) have predominantly focused on enhancing efficiency, privacy, and robustness in federated learning environments. The general direction of the field is moving towards developing more sophisticated methods for parameter-efficient fine-tuning, model pruning, and privacy-preserving techniques, all while maintaining or improving model performance. Innovations in federated learning frameworks, such as the introduction of novel aggregation functions and adaptive mask expansion techniques, are paving the way for more efficient and secure model updates across distributed clients. Additionally, the integration of auxiliary models, like language models, into privacy-enhancing strategies is showing promising results in mitigating gradient inversion attacks and protecting sensitive data.

Noteworthy developments include:

  • A federated learning framework for pruning LLMs that introduces a novel $\ell_0$-norm aggregation function.
  • A method for approximating data ablations in LLMs through modular training and merging, significantly improving amortized training efficiency.
  • A gradient attack algorithm tailored to spatiotemporal data that utilizes an auxiliary language model to enhance reconstruction accuracy.
  • A subspace regularization method on LoRA structure that effectively balances model capacity and degree of forgetting.
  • A novel approach to model merging in federated continual learning that ensures alignment of model responses across tasks and clients.

Sources

FedSpaLLM: Federated Pruning of Large Language Models

Iterative Methods via Locally Evolving Set Process

MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models

Scalable Data Ablation Approximations for Language Models through Modular Training and Merging

Extracting Spatiotemporal Data from Gradients with Large Language Models

Subword Embedding from Bytes Gains Privacy without Sacrificing Accuracy and Complexity

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

LoRA-C: Parameter-Efficient Fine-Tuning of Robust CNN for IoT Devices

On the Vulnerability of Text Sanitization

Closed-form merging of parameter-efficient modules for Federated Continual Learning

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

LEGO: Language Model Building Blocks

GeoLoRA: Geometric integration for parameter efficient fine-tuning

PSY: Posterior Sampling Based Privacy Enhancer in Large Language Models

Built with on top of