Why Self-Supervised Learning Thrives on Redundant Data [Video]

What’s keeping researchers awake at night? It’s the nagging question of whether AI systems can truly learn like humans simply by looking at data. Maybe self-supervised learning guru Yann LeCun has the answers. 

In a recent discussion on X, LeCun was summoned to talk about the critical role of redundancy in self-supervised learning (SSL). According to him, SSL thrives on data redundancy, enabling it to uncover structure and patterns within the input. 

If the data has redundancy, which means there are repeated or similar parts, SSL can use it to learn useful structures and insights. However, “highly compressed data has no redundancy and appears random. SSL can not learn anything from random data,” said LeCun.