The recent advancements in the field of self-supervised learning (SSL) for visual representations have seen a significant shift towards leveraging masked modeling techniques. Specifically, Masked Autoencoders (MAEs) and Masked Image Modeling (MIM) have gained prominence due to their ability to generate robust representations without relying on augmentation techniques, which are common in contrastive learning frameworks. These methods align well with the principles of SSL in natural language processing, where masking and reconstruction are central. However, the integration of these techniques into Transformer-based architectures has highlighted the need for regularization to match the performance of convolutional neural network (CNN) counterparts. Novel approaches, such as manifold regularization for MAEs, have been introduced to address this gap, demonstrating improved performance across various SSL methods. Additionally, research has begun to explore the true potential of MIM representations, identifying issues with representation aggregation and proposing solutions that could lead to higher-quality visual representations for high-level perception tasks. These developments suggest a promising future for SSL in visual learning, where the focus is shifting towards optimizing and fully utilizing the capabilities of masked modeling techniques.

Noteworthy papers include 'MAGMA: Manifold Regularization for MAEs,' which introduces a novel regularization loss that significantly enhances MAE performance, and 'Beyond [cls]: Exploring the true potential of Masked Image Modeling representations,' which identifies and addresses critical issues in MIM representation aggregation.