Latent space

What is latent space?

Latent space is the space of numbers an AI model works in. When a model reads a word or looks at a picture, it turns it into a long list of numbers, and that list marks a point in the space. Things the model treats as alike end up close together, so the space works like the model’s map of meaning.

It’s called latent, meaning hidden, because you see only what the model makes of the numbers, not the numbers themselves. When people say a model might reason in neuralese, they mean it would do its thinking in this space, without putting it into words.

Also called: embedding space, representation space
Last updated: 1 October 2026

Vectors, in plain words

A vector is a list of numbers. Two numbers can pin down a place on a map: its latitude and longitude. Three can pin down a spot in a room. Each number is a direction you can move in, and researchers call each direction a dimension.

AI models use much longer lists. GPT-3, which OpenAI described in 2020, turns each piece of a word into a list of 12,288 numbers. People can’t picture a space with 12,288 directions, but it works like the map: lists with similar numbers sit close together, and you can measure how far apart any two are.

Researchers don’t choose what each number means. Training sets them, nudging the numbers again and again toward whatever helps the model do its job, such as guessing the next word. What comes out is an arrangement where words used in similar ways land near each other, though the directions don’t come with labels.

Fig. 1 A word becomes a list of numbers, and the list becomes a point. Here the first two numbers place it on a map, and it lands near the words used like it. A real model’s list has thousands of numbers, so its map has thousands of directions. The numbers and positions here are made up to show the idea.

Directions that mean something

In 2013, Tomas Mikolov and two colleagues at Microsoft Research found that some relationships between words show up as directions in this space. The step from man to woman pointed roughly the same way as the steps from uncle to aunt and from king to queen. So if you start at king, take away man and add woman, you land, in the paper’s words, “very close to ‘Queen.’”

The trick is famous, and it has a catch. The standard way of running it doesn’t let the answer be one of the words you started with. In a 2019 paper, Malvina Nissim and two colleagues at the University of Groningen let those words back in. Accuracy on the standard test fell from 74% to 21%, mostly because the answer came back as one of the starting words: man is to king as woman is to king. The directions are there, but they’re rougher than the famous example suggests.

Fig. 2 The step from man to woman, moved over to king, lands by queen. That’s in two directions. Google’s word vectors have hundreds, and searching them for the word nearest to king − man + woman finds king itself. The standard code leaves out the three starting words, which leaves queen. The drawing follows Mikolov and colleagues’ 2013 figure, and the answers are the ones Nissim and colleagues report.

Where latent space shows up

Word vectors were only the start. Image generators use latent space to save work. Stable Diffusion doesn’t make its pictures pixel by pixel. An encoder first squeezes an image into a smaller block of numbers: a picture 512 pixels square, which takes 786,432 numbers, becomes 16,384. The model learns to make pictures in that smaller space, and a decoder turns the result back into pixels. Its makers called the method latent diffusion, after the space where the work happens.

Some models put different kinds of things in one space. OpenAI’s CLIP, from 2021, learned from 400 million pictures and their captions to place each picture near its own caption, so a photo of a dog and the words “a photo of a dog” end up close together. Stable Diffusion uses CLIP’s text half to turn a prompt into numbers it can follow.

In a chatbot, latent space is where the work gets done. Each piece of the text becomes a vector, and each layer of the model reads those vectors and adds what it has worked out. Researchers at Anthropic call this running total the residual stream, “a communication channel” that “all layers communicate through.” At the top, the last vector becomes a guess at the next word.

What researchers have found inside

If concepts sit along directions, you could find the direction for a concept, read it off, or turn it up. Anthropic’s interpretability team, which studies what goes on inside models, starts from that idea: that a network stores meaningful concepts as directions in its space.

It’s harder than it sounds, because a model can store more concepts than it has directions by letting them overlap a little. Anthropic’s researchers showed how in small test networks in 2022 and called it superposition. How far those toy results carry over to real models was, in their words, “very unclear.”

In 2024 they tried a real one. They trained a second network, a sparse autoencoder, to pull the overlapping concepts in Claude 3 Sonnet apart, and found millions of what they call features. One fired on the Golden Gate Bridge, in text in several languages and in pictures. Turned up, it made the model bring the bridge into most of its replies. For 24 hours anyone could try this Golden Gate Claude: asked how to spend $10, it suggested driving across the bridge and paying the toll. Anthropic says the same method can turn safety-related features up or down, like those for dangerous code or deception.

In 2025 Anthropic looked inside Claude 3.5 Haiku and found the same features standing for the same concepts in English, French and Chinese, which it says suggests “a kind of universal ‘language of thought.’”

These tools see part of the space. Even with 34 million features, the researchers found evidence that what they had uncovered was “an incomplete description of the model’s internal representations.”

Latent space and neuralese

A chatbot that reasons in words leaves latent space at each step. Its last vector, thousands of numbers, comes out as a single word or part of one, and the next step starts from there. AI 2027 described the alternative: passing those vectors back to the model’s early layers, “potentially transmitting over 1,000 times more information.”

That’s what this site calls neuralese: reasoning that stays in latent space. In COCONUT, a method published in 2024, a model feeds its last vector straight back in as its next input instead of writing a word. Recurrent-depth models loop through the same layers, working on the same vectors for longer before they write anything. What is latent reasoning? goes through these methods and how well they work, and Which AI models think in neuralese? tracks which models use them.

It’s also why neuralese worries the people who check AI’s reasoning. Written reasoning can be read. Reasoning in vectors has to be decoded with tools like the ones above, which see only part of the space. As AI 2027 put it, “Unlike English words, these high-dimensional vectors are likely quite difficult for humans to interpret.”

Sources

These are the sources behind this page. We checked them against the original text on 1 October 2026.