Self-supervised learning
Self-supervised learning is a way for AI to teach itself by turning raw information into its own study guide. Instead of needing a human to label every piece of data, the AI hides parts of the information and tries to guess what is missing, effectively creating its own practice tests to learn patterns.
A machine learning paradigm where the model generates its own supervisory signals directly from the input data. By defining pretext tasks—such as predicting the next token in a sequence or filling in masked segments—the model learns useful representations from unlabeled datasets, which serves as the foundational pre-training phase for LLMs.
A learning framework where the objective function is derived from the structure of the data itself, eliminating the requirement for external human-annotated labels. During pre-training, the model optimizes a surrogate loss function based on internal data dependencies, such as autoregressive next-token prediction or masked language modeling, to learn high-dimensional latent representations.
evolution
- 2008 · historyIntroduction of Autoencoders
Geoffrey Hinton and Ruslan Salakhutdinov popularized deep autoencoders, which learn representations by reconstructing input data without explicit labels.
- 2014 · historyWord2Vec
Tomas Mikolov introduced word embeddings that learn semantic relationships by predicting context words, a foundational self-supervised technique for NLP.
- 2018 · historyBERT and GPT
The release of BERT and GPT demonstrated that massive-scale self-supervised pre-training on unlabeled text could achieve state-of-the-art performance across diverse tasks.
- 2020 · historySimCLR
Google Research introduced a framework for contrastive learning that significantly advanced self-supervised visual representation learning.