Context Length and Hardware Scalability
Context windows have exploded from 4k tokens to 10 million in just a few years. Meta’s Llama 4 Scout supports 10M tokens — 78x more than Llama 3’s 128k. Google’s Gemini 3 Pro handles 1M tokens, while Claude 4 offers 1M in beta.
This enables processing entire codebases, hundreds of research papers, or multi-day conversation histories in a single pass. But there’s a problem: context length has outpaced hardware capacity.