Yihe Dong, Jean-Baptiste Cordonnier, Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Marina Meila, Tong Zhang 0001, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event. Volume 139 of Proceedings of Machine Learning Research, pages 2793-2803, PMLR, 2021. [doi]
Abstract is missing.