Scaled ReLU Matters for Training Vision Transformers

Pichao Wang, Xue Wang, Hao Luo, Jingkai Zhou, Zhipeng Zhou, Fan Wang, Hao Li, Rong Jin 0001. Scaled ReLU Matters for Training Vision Transformers. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022. pages 2495-2503, AAAI Press, 2022. [doi]

Abstract

Abstract is missing.