Wiki · Reference entry
Pretraining
The first and largest stage of training, where a model learns from an enormous amount of text.
A new hire spends years in school absorbing general knowledge before a single day of training on your company's procedures. Pretraining is the school years. The model reads a vast amount of text and learns to predict what comes next, and that one exercise is where most of what it knows comes from. Later stages, such as fine-tuning and RLHF, shape how that knowledge gets used: the tone, the format, which requests to decline.
Why it matters
Later training shapes how a model uses what it knows. Most of what it knows, including the parts nobody would have chosen, arrived here first.