Self-Distillation for Further Pre-training of Transformers

Seanie Lee, Minki Kang, Juho Lee 0001, Sung Ju Hwang, Kenji Kawaguchi. Self-Distillation for Further Pre-training of Transformers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. [doi]

Authors

Seanie Lee

This author has not been identified. Look up 'Seanie Lee' in Google

Minki Kang

This author has not been identified. Look up 'Minki Kang' in Google

Juho Lee 0001

This author has not been identified. Look up 'Juho Lee 0001' in Google

Sung Ju Hwang

This author has not been identified. Look up 'Sung Ju Hwang' in Google

Kenji Kawaguchi

This author has not been identified. Look up 'Kenji Kawaguchi' in Google