Distributed Keras Engine, Make Keras faster with only one line of code.

Chimera: bidirectional pipeline parallelism for efficiently training large-scale models.

sensAI: ConvNets Decomposition via Class Parallelism for Fast Inference on Live Data

SHADE: Enable Fundamental Cacheability for Distributed Deep Learning Training

Ok-Topk is a scheme for distributed training with sparse gradients. Ok-Topk integrates a novel sparse allreduce algorithm (less than 6k communication volume which is asymptotically optimal) with the decentralized parallel Stochastic Gradient Descent (SGD) optimizer, and its convergence is proved theoretically and empirically.

🚨 Prediction of the Resource Consumption of Distributed Deep Learning Systems

Decentralized Asynchronous Training on Heterogeneous Devices

Eager-SGD is a decentralized asynchronous SGD. It utilizes novel partial collectives operations to accumulate the gradients across all the processes.

WAGMA-SGD is a decentralized asynchronous SGD based on wait-avoiding group model averaging. The synchronization is relaxed by making the collectives externally-triggerable, namely, a collective can be initiated without requiring that all the processes enter it. It partially reduces the data within non-overlapping groups of process, improving the…

Distributed deep learning framework based on pytorch/numba/nccl and zeromq.

Collection of resources for automatic deployment of distributed deep learning jobs on a Kubernetes cluster

Horovod Tutorial for Pytorch using NVIDIA-Docker.

A foundational repository for setting up distributed training jobs using Kubeflow and PyTorch FSDP.

An implementation of a distributed ResNet model for classifying CIFAR-10 and MNIST datasets.

A blockchain based neural architecture search project.

Simultaneous Multi-Party Learning Framework

Implemented training strategies to help improve bottlenecks and to improve the training speed while maintaining the quality of our GANs.