Markdown source

The Base Model is Dead

Conference Context

Session Description

It's a common belief that large language models are trained to be a good model of human web-text, and thus base models are "mirrors" of what we see on the internet. Historically, this was largely true, but no modern base model truly reflects the internet in the way that GPT-3 once did. Instruction data along with synthetic reasoning traces are moving earlier and earlier into the training pipeline, and "mid-training" has emerged as a new stage to accommodate longer datapoints that more concretely resemble downstream capabilities. As a result, pre-training no longer has the goal of creating a linguistic prior, but instead has the additional goals of baking in behavior and more atomic skills into the trained "base" model. Between this shift in what a base model is and the blurring of the lines between the different stages of model training, it's an open question as to what the best approach is here (at least outside the walls of the big labs). But I believe that the role we view the base model playing will continue to shift as we're pulled forward through new phases of model capabilities.

Media Evidence

No related AI Engineer channel video found yet.

Evidence Graph

This evidence graph is generated from currently linked source material: official schedule text, related video pages, cached transcripts, visible slide text, dense/reconstructed slide pages, and AI slide-classification audits.

Media Signals

No linked video, transcript, or slide source has been attached yet.

Agent Reading Notes

Use these signals to refine the synopsis, topic links, people/company context, and method notes. If a source is a related external video rather than an exact official recording, keep it framed as supporting evidence.

Transcript Status

No official session recording transcript was found by exact title match on the AI Engineer YouTube channel during this run.

People

Notes

Synthesis

Synthesized Breakdown

The Base Model is Dead ## Conference Context - Date/time: 2026-06-29 · 1:30pm-1:50pm - Track/room: Data Quality · Track 9 - Speaker(s): Varun Singh - Session type/status: session · confirmed - Track: Data Quality - Room: Track 9 - Session type: session - Status: confirmed ## Session Description It's a common belief that large language models are trained to be a good model of human web-text, and thus base models are "mirrors" of what we see on the internet. Historically, this was largely true, but no modern base model truly reflects the internet in the way that GPT-3 once did. Instruction data along with synthetic reasoning traces are moving earlier and earlier into the training pipeline, and "mid-training" has emerged as a new stage to accommodate longer datapoints that more concretely resemble downstream capabilities. As a result, pre-training no longer has the goal of creating a linguistic prior, but instead has the additional goals of baking in behavior and more atomic skills into the trained "base" model.

Speaker And Company Context

Topics Covered

Derived Links And Source Material

Novel Concepts / Clever Methods

Evidence Boundary

This synthesis is based on the official schedule and linked source pages. It should be revisited when exact session recordings or transcript-backed secondary sources are available.