Decentralized Training of Foundation Models in Heterogeneous Environments

Binhang Yuan, Yongjun He, Jared Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Ré, Ce Zhang 0001. Decentralized Training of Foundation Models in Heterogeneous Environments. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. 2022. [doi]

Authors

Binhang Yuan

This author has not been identified. Look up 'Binhang Yuan' in Google

Yongjun He

This author has not been identified. Look up 'Yongjun He' in Google

Jared Davis

This author has not been identified. Look up 'Jared Davis' in Google

Tianyi Zhang

This author has not been identified. Look up 'Tianyi Zhang' in Google

Tri Dao

This author has not been identified. Look up 'Tri Dao' in Google

Beidi Chen

This author has not been identified. Look up 'Beidi Chen' in Google

Percy Liang

This author has not been identified. Look up 'Percy Liang' in Google

Christopher Ré

This author has not been identified. Look up 'Christopher Ré' in Google

Ce Zhang 0001

This author has not been identified. Look up 'Ce Zhang 0001' in Google