Report on Recent Developments in Collaborative Edge Inference and Federated Learning

General Direction of the Field

The recent advancements in the fields of collaborative edge inference and federated learning (FL) are significantly shaping the landscape of distributed machine learning, particularly in resource-constrained environments such as mobile devices, edge computing, and the Internet of Things (IoT). The primary focus of these developments is on optimizing the distribution of computational tasks across multiple nodes, thereby enhancing efficiency, reducing latency, and minimizing communication overhead.

Key Innovations and Trends:

1. Dynamic Resource Allocation and Path Learning:

   • A notable trend is the shift from static to dynamic resource allocation strategies. Researchers are now exploring methods to learn optimal network paths and DNN (Deep Neural Network) partitions in real-time, considering unknown network parameters and potential security threats. This dynamic approach allows for more adaptive and efficient task distribution, which is crucial for improving the performance of collaborative edge inference systems.

2. Heterogeneity Management in Federated Learning:

   • Addressing the heterogeneity in both statistical and system aspects has become a central theme in FL. Novel frameworks are being developed to cluster devices based on their capabilities and data characteristics, thereby optimizing model training and reducing communication bottlenecks. These frameworks aim to balance the trade-off between training efficiency and model accuracy, especially in environments with diverse data distributions and varying computational resources.

3. Hybrid Learning Approaches:

   • The integration of horizontal and vertical federated learning is gaining traction. These hybrid approaches leverage the strengths of both paradigms—horizontal for data parallelism and vertical for feature parallelism—to enhance the overall learning process. This hybridization is particularly beneficial in scenarios where data samples and features vary across devices, offering a more comprehensive solution to the challenges of non-IID (non-independent and identically distributed) data.

4. Early Exit Strategies for Distributed Inference:

   • Early exit strategies are being employed to optimize the inference process in distributed DNNs. By dynamically determining the complexity of samples and offloading them to the appropriate layer of the DNN, these strategies reduce inference latency and computational costs. This approach is particularly effective in natural language processing (NLP) tasks, where the complexity of samples can vary significantly.

5. Scalable Second-Order Federated Learning:

   • Second-order FL algorithms, which leverage curvature information for faster convergence, are being refined to address the scalability issues associated with large-scale models. Innovations in sparse Hessian estimation and over-the-air aggregation are making these algorithms more feasible for practical deployment, offering significant reductions in communication and energy costs.

6. Joint Resource Allocation Strategies:

   • A comprehensive review of joint resource allocation strategies in federated edge learning highlights the importance of optimizing multiple resources simultaneously. These strategies aim to enhance system efficiency, reduce latency, and improve resource utilization, while also contributing to privacy preservation by minimizing communication requirements.

Noteworthy Papers

• Learning the Optimal Path and DNN Partition for Collaborative Edge Inference: Introduces a novel bandit algorithm, B-EXPUCB, which demonstrates superior performance in dynamic path and DNN layer assignment, addressing unknown network parameters and security threats.

• ParallelSFL: A Novel Split Federated Learning Framework Tackling Heterogeneity Issues: Proposes an effective clustering strategy to optimize model training efficiency and accuracy in heterogeneous edge systems, significantly reducing traffic consumption and speeding up model training.

• Distributed Inference on Mobile Edge and Cloud: An Early Exit based Clustering Approach: Develops a method that significantly reduces inference cost while maintaining minimal accuracy drop, leveraging early exit strategies in distributed DNNs for NLP tasks.

These papers represent significant advancements in their respective areas, offering innovative solutions to the challenges of collaborative edge inference and federated learning in resource-constrained environments.