Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints - researchr publication authors

researchr

You are not signed in
Sign in
Sign up

Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani 0001, Neil Houlsby. Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. [doi]

This author has not been identified. Look up 'Aran Komatsuzaki' in GoogleThis author has not been identified. Look up 'Joan Puigcerver' in GoogleThis author has not been identified. Look up 'James Lee-Thorp' in GoogleThis author has not been identified. Look up 'Carlos Riquelme Ruiz' in GoogleThis author has not been identified. Look up 'Basil Mustafa' in GoogleThis author has not been identified. Look up 'Joshua Ainslie' in GoogleThis author has not been identified. Look up 'Yi Tay' in GoogleThis author has not been identified. Look up 'Mostafa Dehghani 0001' in GoogleThis author has not been identified. Look up 'Neil Houlsby' in Google

runs on WebDSL