Distributed Deep Learning Systems – A Comparative Study on Different DDL Frameworks

Abstract

Deep Learning, a subcategory of machine learning has gained immense success because of the recent advancements in distributed deep learning systems. The concept of distributed deep learning scales well with the available computational methods and this in turn helps to increase the overall productivity. Using this methodology, productivity can be fueled by distributing networks and tasks over a number of devices. This methodology is useful when a large data model cannot be stored on a single machine. In such cases, DDL techniques like data parallelism and model parallelism are proven to be successful for distributing data across several machines.

In this paper, we will be discussing the different distributed deep learning frameworks

available. TensorFlow, PyTorch, Singa and Caffe will be the main frameworks that will be analysed . The paper highlights the current scenario in the distributed deep learning domain and unfolds areas where there can be progress.