Fold3D: Rethinking and Parallelizing Computational and Communicational Tasks in the Training of Large DNN Models

Fanxin Li, Shixiong Zhao, Yuhao Qing, Xusheng Chen, Xiuxian Guan, Sen Wang, Gong Zhang, Heming Cui. Fold3D: Rethinking and Parallelizing Computational and Communicational Tasks in the Training of Large DNN Models. IEEE Trans. Parallel Distrib. Syst., 34(5):1432-1449, May 2023. [doi]