1 pages tagged with "transformer"

The Johnson-Lindenstrauss lemma for the brave

December 23, 2020 - 3484 words - 18 mins
If you are interested in dimensionality reduction, chances are that you have come across the Johnson-Lindenstrauss lemma. I learned about it while studying the Linformer paper, which contains a result on dimensionality reduction for the Transformer. Essentially, they prove that self-attention is lo… read more