Conception jointe de l'entrainement et de l'allocation des ressources réseau pour l'apprentissage automatique distribué

Joint Training Design and Network Resource Allocation for Distributed Machine Learning

• Auteur / Autrice : Tiago Da silva barros
• Direction : Frédéric Giroire
• Type : Projet de thèse
• Discipline(s) : Informatique
• Date : Inscription en doctorat le 30/09/2022
• Etablissement(s) : Université Côte d'Azur
• Ecole(s) doctorale(s) : École doctorale Sciences et technologies de l'information et de la communication (Nice ; 1992-....)
• Partenaire(s) de recherche : Laboratoire : I3S - Informatique, Signaux et Systèmes de Sophia-Antipolis

Mots clés

Mots clés libres

Apprentissage automatique

Réseaux de communication

Allocation de ressources

Machine learning

Resource allocation

Communication networks

Résumé

Voir le résumé en anglais.

In the last years, Distributed Machine Learning (DML) architectures have been gaining momentum. This is due to the practical limitations of the deployment of centralized learning onto the cloud-to-the-edge continuum environment. In the centralized paradigm, training data are collected from different sources and uploaded to a remote central data server where training phase will take place once all source data consolidated into a unique dataset. Eventually, in the inference phase, the so-trained Machine Learning (ML) model be executed in this central server to answer to client requests demanding its inference service. However, this centralized operation implies delays, communication overhead and potential privacy issues [1]. Different DML approaches have been proposed to cope with these problems. In all of them, the ML model is distributed: model copies or model bricks are scattered onto the different network devices to be placed closer to end-users and data collection devices. These DML-based proposals differ in how the model eventually gets distributed. Literature has considered so far two main scenarios: 1. Distributed training. Here, the training phase is moved from the central server towards devices closer to the data collecting. Each one of these devices performs a training using its own local data and sends the local-learnt model (e.g., the model parameters) to the central server. The central server creates a new global model by aggregating the local-learnt ones. The process is repeated till convergence to a stable global model, whose copies are distributed to the local devices. 2. Distributed inference. In this case, the inference phase is performed in a distributed manner, while the training phase is normally centralized as in classical ML. That means that a model (or several of them) are trained in a central server with existing datasets before deployment onto the cloud-edge environment. Hence, the problem is to select the network devices where to place these pre-trained models, i.e., to decide how to allocate network resources to the models for the inference given rise to the inference service provisioning problem. The distributed training scenario (traditionally represented by Federated Learning, FL [1]), has been recently a subject of great interest exploring from different FL algorithms (FedAvg [2], FedProx,[2]) up to different synchronization procedure (sharing weights [2], gradient updates [2], logits [3]]). On the contrary, the Distributed Inference case is much less investigated and it is gaining attraction at this moment under different problematics, as Distributed (Convolutional) Neural Networks [4] or Inference Delivery Networks (IDN) [5]. In the former, the different stages of a same inference model are placed onto different network nodes, similarly as virtualized network function are placed to constitute a service chain [6]. In the latter, the whole inference models are allocated onto different nodes, in a similar way to how contents are placed in Content Delivery Network (CDN) [7]. We could state the problems solved in both approaches as follows: - Distributed Training consists of optimally designing the training procedure (and the model to train), assuming that network resource allocation is fix. - Distributed Inference consists of optimally deciding the network resource allocation, assuming training is done (and the model is fix). THESIS GOAL: Then, the goal of this thesis is to combine both approaches in training phase by jointly deciding the network resource allocation and the distributed training procedure parameters to maximize model performance and minimize resource consumption. In other words, we aim to tune both degrees of freedom: (i) training design, and; (ii) resource allocation (inference provisioning) at the same time. Let be a certain inference service that we want to train and put into operation in a distributed manner onto a certain cloud-edge infrastructure. We have two decision sets: (i) a set of alternate training procedures (each one with different resources consumption); and, (ii) a set of alternate network resource allocations (each one with different model inference performance), we chose the pair <training procedure, resource allocation> that maximizes inference performance and minimizes resource consumption. This approach will bring the next benefits: 1. In training phase: the distributed training procedure (and its parameters) are selected based on several alternate resource allocations, not based on only one fix. 2. In inference phase: the model to deploy has been already trained in a distributed manner, instead allocating model trained in a different scenario: a centralized one. Pre-requisites if any: Languages: - Python language (absolutely) - Deep Learning libraries (like TensorFlow [6], Keras, rllab, OpenAI Gym) (recommended) Theory: - Machine Learning, Data Science, particularly Neural Networks theory (recommended) - Classical optimization theory (Linear Programming, Dual Optimization, Gradient Optimization, Combinatorial Optimization, Graph Theory) (recommended) Technology: - Computer networking (recommended)