Training data

Training data is the text, code, images, and other media that a foundation model learns from during pre-training. Modern LLMs are trained on staggering volumes of data - Trillions of tokens drawn from across the public internet and licensed collections - Representing what may be the largest corpora of human-produced text ever assembled. The quality, diversity, and balance of this data shapes almost everything the model knows, believes, and gets wrong.

Scale of training data

What goes into training data

• Common Crawl - A petabyte-scale scrape of the public web, refreshed monthly. Forms the backbone of most pre-training datasets but requires heavy filtering to remove spam, duplicates, and low-quality content.
• Books - Digitised books via BookCorpus, Project Gutenberg, and licensed datasets. Improves long-form coherence and reasoning.
• Code - GitHub repositories, Stack Overflow, documentation. Makes models dramatically better at programming tasks.
• Wikipedia - High-quality factual content in 300+ languages.
• Scientific papers - ArXiv, PubMed, licensed academic journals.
• Synthetic data - AI-generated text used to supplement real data, particularly for tasks where human-labelled data is scarce.

Why data mix matters

The composition of training data directly determines a model's strengths and blind spots. A model trained on more code is better at programming. One with more multilingual text is better at translation. Models trained on medical literature perform better on clinical tasks. Fine-tuning after pre-training can correct some gaps, but it cannot fully compensate for missing knowledge in the base training data. Data quality is increasingly seen as more valuable than raw quantity: carefully curated datasets often outperform larger but noisier ones.

Controversies and challenges

Training data sits at the centre of several live debates:

• Copyright - Authors, news organisations, and code authors have filed lawsuits arguing that scraping their work without permission or compensation violates copyright law. Outcomes will shape how future datasets are built.
• Bias - Training data that over-represents Western, English-speaking perspectives produces models that reflect those biases. This affects everything from which names the model associates with professions to how it renders non-English languages. See alignment for how labs try to correct this.
• Synthetic data risks - Using AI-generated text as training data can create feedback loops where models reinforce their own errors. Researchers call this 'model collapse.'
• Knowledge cutoff - Training data has a fixed end date, after which the model knows nothing about world events. See knowledge cutoff.

A common follow-up question: do your own chats become training data? On AskAI.free they do not - the FAQ explains why conversations are never used for training.