Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
Data can be represented as a vector in a continuous space, but often it is concentrated in a narrow area. Can it be improved?
Natural Language Processing (NLP)
Natural Language Processing (NLP) has significantly advanced in the last few years. It became hard to keep track with all the changes and technologies in the field. These selected articles will help you to keep-current with NLP.
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
This paper suggests a new computationally efficient method for constructing low-dimensional representation of unlabeled data.
Keep-Current - April 2021 Summary
Self-Supervised Learning keynote talk by Yann LeCunn; Perceiver - computation efficient transformer algorithm; Data Visualization also for summaries and more news in this monthly update.
Perceiver: General Perception with Iterative Attention
To overcome Transformers' squared complexity (w.r.t input length), the Perceiver article here offers a novel method to learn the QKV matrices. Check it out!
Rethinking Attention with Performers
The article suggests a method to lower the Transformer's complexity to a linear order and proves all the arguments also in a rigorous form. The article is not easy to read, but luckily, to understand the main idea, the first 5-6 pages are more than enough.