%0 Conference Proceedings
%T Efficient and Systematic Partitioning of Large and Deep Neural Networks for Parallelization
%+ Laboratoire d'Informatique Fondamentale d'Orléans (LIFO)
%+ Huawei Technologies France [Boulogne-Billancourt]
%A Wang, Haoran
%A Li, Chong
%A Tachon, Thibaut
%A Wang, Hongxing
%A Yang, Sheng
%A Limet, Sébastien
%A Robert, Sophie
%< avec comité de lecture
%( Euro-Par 2021: Parallel Processing
%B European Conference on Parallel Processing EuroPar 2021
%C Lisbon, Portugal
%I Springer International Publishing
%C Cham
%3 Lecture Notes in Computer Science
%V 12820
%P 201-216
%8 2021
%D 2021
%R 10.1007/978-3-030-85665-6_13
%K Distributed algorithm
%K Distributed machine learning
%K Neural network partitioning
%Z Computer Science [cs]/Logic in Computer Science [cs.LO]Conference papers
%X Deep neural networks (DNNs) are playing an increasingly important role in our daily life. Since the size of DNNs is continuously growing up, it is highly important to train them effectively by distributing computation on multiple connected devices. The efficiency of training depends on the quality of chosen parallelization strategy. Being able to find a good parallelization strategy for a DNN in a reasonable amount of time is not trivial. Previous research demonstrated the possibility to systematically generate good parallelization strategies. However, systematic partitioning still suffers from either a heavy preprocessing or poor quality of parallelization. In this paper, we take a purely symbolic analysis approach by leveraging the features of DNNs like dense tensor balanced computation. We propose the Flex-Edge Recursive Graph and the Double Recursive Algorithm, successfully limiting our parallelization strategy generation to a linear complexity with a good quality of parallelization strategy. The experiments show that our solution significantly reduces the parallelization strategy generation time from hours to seconds while maintaining the parallelization quality.
%G English
%L hal-03526611
%U https://univ-orleans.hal.science/hal-03526611
%~ UNIV-ORLEANS
%~ INSA-GROUPE
%~ INSA-CVL