We will start by defining the most fundamental building block of LLMs: Language modeling, which dates back to early statistical NLP methods in the 1980s and 1990s and was later popularized with the advent of neural networks in the early 2010s.
In its simplest form, language modeling is essentially about learning to predict the next word in a sentence. This task, known as next-word prediction, is at the core of how LLMs learn language patterns. The model accomplishes this by estimating the probability distribution over sequences of words, allowing it to predict the likelihood of any given next word based on the context provided by the preceding words.