Large Language Models as Predictive Text

What Do Large Language Models Do?

One way to think about large language models is through the predictive text on your phone. You start to write, “I’m sorry I’m . . . ,” and your phone immediately prompts you. Are you “late,” “not feeling well,” or “just getting up”? You may have even played games with this feature, where you and friends challenge each other to write sentences using only predictive text. If Mike starts with, “The thing you need to know about me” and chooses one of three choices at the bottom of the chat box for each word, he gets, “The thing you need to know about me is I need to get a new phone.” This is likely because Mike (like others) often apologizes for his battery dying. If you run all those predictions through a statistical model, you can produce a somewhat peculiar version of something Mike might say by chaining predictions together.

What if, instead of running all the things you texted into that statistical model, you ran a big portion of what people said on the internet into it? A really big portion—like a half a trillion words? Then you started a sentence with something like, “Three causes of rumor are as follows . . .” A sophisticated model would look at the words and phrases people tend to say around that language, such as “major causes of rumor” or “one source of false stories.” It might find that people often say “Uncertainty” next, and that after “Uncertainty,” people tend to say, “When people are unsure of the future.”

Three causes of rumor are as follows. Uncertainty: When people are unsure of the future.

Now we repeat the game and type that in, and we get more. The thing is that these models provide spookily human-like answers to questions, many of which are surprisingly good. When we see it in action, we can forget we’re seeing a predictive text generator:

Me: What are three causes of rumor?

Reply: There are many causes of rumor, but three causes of rumor are as follows. Uncertainty: When people are unsure of the future, they tend to speculate on the present. Lack of information: When good information is not available, people often turn to unreliable sources. Novelty: When rumors feel new, people often wish to be the first to share them.

The reason these answers are both useful and compelling is that they mimic how we speak and write. As we worked on this book, we often started sentences without knowing how they would end. Typing one word or phrase somehow propelled us to the next. If you’ve ever struggled with writer’s block, you know the solution is to write anything on that page. Words lead to other words, and as they come together, new ideas develop. If there is one thing that LLMs seem to be making clear, it’s that a lot of what we do when we write and summarize is more automatic and predictive than we might care to admit.

Of course, we humans can do something that an LLM can’t: we can look at the sentence we just finished and ask whether it really captures what we mean. We can revise based on how we imagine the sentence might be interpreted. We can do this because beneath the activity of writing we have something we want to convey—a model not just of language but of the world, and a set of communicative goals that are bigger than selecting the next word or phrase. Large language models don’t have goals. They don’t have underlying understandings. They have models of what certain people are likely to say after saying other things.1