Slides: Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI
Source Video
Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI
Relationship To World's Fair 2026
These slides are extracted from a public AI Engineer YouTube video connected to World's Fair 2026. Speaker-matched clips are supporting context unless later confirmed as exact session recordings; official livestream recordings are day-level/event-level source material.
Related Scheduled Sessions
- No individual scheduled session mapping has been assigned yet; treat this as an event livestream deck.
Extracted Slides
OCR text:
together.ai
The AI Native Cloud
Road to 5M
Sequence Length
Breaking Memory Barriers in
Context Parallelism
Max Ryabinin
VPRSD,Model Shaping
AlEngineer
EUROPE
OCR text:
Al Native Cloud
bd *
“1. cursor 2 Runware LM aiclol ee COMU Lore A417
i e ty
_ ’ Engineering the future of Al
OCR text:
Why do we want to long-context training?
ra ae ne es CY i}
. * Cee Mey yy)
— =
| AlEngineer |
, 4 a EUROPE
OCR text:
What's stopping us?
Long Context
. * O(NA2) O(N)
ra é Computation Memory
a ad
| AlEngineer |
i . : EUROPE
OCR text:
What's stopping us?
Long Context
aaa a ae
Ps _- oe
‘B88 BES === _ll
_ 4— : Engineering the future of Al
OCR text:
LONE aker- A No 4 1
80
a Model Gi
. * a a. Attn act. Hi
* bd 3% Other ==:
QC
a zo
320
0” Default
Llama 3-8B, 3M tokens, 8xH1O0
a ; |
ye Engineering the future of Al
OCR text:
DeepSpeed Ulysses
|
Pein be 7 — 7 0
aoe “ | a
** 7 a - Full Flash Attention : kt
What if we divide by num_heads dim?
_ a Engineering the future of Al
OCR text:
With Ulysses context parallelism
4 ° Model Gl
Pe PY 2 gq Attn act =
a [ave] " o” Other ==
we = oot
ae E40 i
320
aU
0° ‘Default FSDP FSDP
+ Ulysses
Llama 3-88, 3M tokens, 8xHI1OO
, ww Engineering the future of Al
OCR text:
What if the activations are still too big?
|. Inputs to each Transformer block can't be recomputed quickly
he 2. Still, we can offload to CPU
Ps * Prefetch before we reach the layer. similar to weights for FSOP
a " Ps x first implemented in
* >
Unsloth Gradient
Checkpointing - 4x longer
context windows
- | AlEngineer |
; EUROPE
OCR text:
Untied Ulysses
@ = Intermediate butfers are toa large! (all OK, x2 because of all-to-all)
4 e Let's leverage the multi: head aspect of attention
o— Rages: |§- EB EB SE 3 -s208
8-388 —|2- Eh Gap 88— 3 |-asee-
|
(RC 7 (a)
2100 loome 0 Com
rv ai Engineering the future of Al
OCR text:
Chunk size ablations
a 20: | 840-
* - 15; f a0.
5g * 9 3
* E10: $820:
5° = 810:
Ons 16 3200 FON 'g 16 32,
Chunk size Chunk size
, = Engineering the future of Al
OCR text:
| Al Engineer |
weeds
AI.ENGINEER
Slide-Derived Subjects To Review
Subject extraction uses video title, related session titles/descriptions, transcript context, and OCR text when available. OCR is best-effort and should be reviewed against the embedded slide images.