Report on Current Developments in Privacy-Preserving Machine Learning
General Direction of the Field
The field of privacy-preserving machine learning (PPML) is witnessing a significant shift towards more efficient and practical methods for training and fine-tuning large-scale models while ensuring data privacy. Recent advancements are focusing on integrating differential privacy (DP) with parameter-efficient fine-tuning (PEFT) techniques, thereby addressing the dual challenges of computational efficiency and data confidentiality. This integration is particularly relevant in the context of large language models (LLMs), where the sheer size of the models necessitates innovative approaches to reduce computational overhead without compromising on performance or privacy.
One of the key trends is the exploration of low-rank adaptation methods, such as LoRA (Low-Rank Adaptation), which have shown promise in reducing the number of trainable parameters during fine-tuning. These methods are being re-examined through the lens of differential privacy, revealing that they implicitly provide privacy guarantees by injecting noise into the training process. This discovery opens up new possibilities for leveraging low-rank adaptation in privacy-sensitive applications without the need for explicit DP mechanisms.
Another notable development is the introduction of differentially private algorithms for bilevel optimization, a problem class that has gained traction in various machine learning applications. These algorithms offer a way to achieve privacy guarantees without the computational burden of Hessian computations, making them suitable for large-scale settings.
Furthermore, the integration of homomorphic encryption (HE) with PEFT techniques is emerging as a powerful approach to ensure both parameter efficiency and model privacy. This combination allows for collaborative fine-tuning with encrypted data, thereby protecting the confidentiality of the model and the training data.
Noteworthy Papers
On the Implicit Relation Between Low-Rank Adaptation and Differential Privacy: This paper provides a theoretical foundation for understanding how low-rank adaptation methods like LoRA implicitly provide privacy guarantees, bridging the gap between efficiency and privacy in model fine-tuning.
PrivTuner with Homomorphic Encryption and LoRA: This work introduces a novel framework that combines parameter-efficient fine-tuning with privacy-preserving technologies, demonstrating significant reductions in energy consumption while maintaining strong privacy protections.
Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?: This paper proposes a new training framework that achieves full-rank training while maintaining low-rank constraints, offering a potential solution to the trade-off between memory efficiency and model performance.
