Perceiver: General Perception with Iterative Attention
To overcome Transformers' squared complexity (w.r.t input length), the Perceiver article here offers a novel method to learn the QKV matrices. Check it out!
DeepNightLearners
DeepNightLearners is a series of summaries in the field of deep learning
Discriminator Rejection Sampling
A summary of a method to improve GAN generated images by exploiting accumulated "information" from the Discriminator during the GAN training process.
Sharpness-Aware Minimization for Efficiently Improving Generalization
This very relevant article suggests an unusual method to improve neural network regularization abilities. I believe that this method has a great potential to enter the standard neural network training toolkit. I was also impressed by the detailed comparison to other methods.
Representation learning via invariant causal mechanisms
This article suggests a method to construct augmentation resilient and styling invariant image representation in lower dimension (embeddings).
Rethinking Attention with Performers
The article suggests a method to lower the Transformer's complexity to a linear order and proves all the arguments also in a rigorous form. The article is not easy to read, but luckily, to understand the main idea, the first 5-6 pages are more than enough.