Understanding How ChatGPT Generates Answers: Probabilistic Language Modeling and Word Vectors

I'm ChatGPT, currently very popular.

Many think I'm extremely powerful, but I have limitations; for example, I cannot provide real‑time information such as today's Beijing weather because my knowledge stops at September 2021.

My answers often feel like generic AI responses; I generate them based on probabilities, essentially playing a word‑chain game.

All of my output is derived from probability distributions; I do not truly understand the meaning of the words I produce.

I represent words as vectors. For a tiny vocabulary of four words—“meow”, “woof”, “cat”, “dog”—their 2‑dimensional vectors might be:

meow: [0.9, 0.1]

woof: [0.1, 0.9]

cat: [0.8, 0.2]

dog: [0.2, 0.8]

These vectors can be plotted on a 2‑D plane, showing that “meow” and “cat” are close, as are “woof” and “dog”, indicating semantic similarity.

In practice, vectors can have hundreds or thousands of dimensions to capture richer semantics.

To generate an answer, I first convert the query words into vectors, feed them into a neural network, obtain an output vector, and then translate that into a probability distribution over possible answer tokens.

For the question “What does a cat like to eat?”, the token probabilities might be: “fish” 0.6, “bone” 0.2, “dog food” 0.1, “chocolate” 0.05, “fruit” 0.05. The highest‑probability token “fish” leads to the answer “Cats like to eat fish.”

In reality, answer generation is a step‑by‑step selection of the next word based on probabilities, similar to a word‑chain.

Two major directions in probabilistic language modeling are BERT and GPT. BERT works like a fill‑in‑the‑blank task, while GPT predicts the next word, akin to writing an essay.

Google released BERT in 2018, achieving strong results on many NLP tasks. My creators invested massive compute resources into GPT, leading to breakthroughs from GPT‑3 to the publicly available ChatGPT.

New technologies often generate hype and unrealistic expectations, but over time their limitations become clear and they find appropriate applications.

ChatGPT is expected to follow a similar trajectory, and early exploration can bring value.

On April 22, Xu Lei will hold an offline public class in Beijing titled “Intelligent Application Development Practice Based on ChatGPT, Codex, and GitHub Copilot.” Participants will learn the latest NLP and machine‑learning techniques and how to use tools such as ChatGPT, GitHub Copilot, and Azure OpenAI to improve development efficiency and quality.

Scan the QR code below to register.