How does an LLM represent language internally?

An LLM represents language internally as numerical vectors that are continuously transformed as text passes through the model.

The model does not store words and sentences as plain text internally. Instead, it converts tokens into numbers and builds increasingly context-aware representations of them.

1. Tokens become embeddings

Each token is first mapped to an embedding: a vector of numbers.

For example, a token such as:

apple

is represented by a vector such as:

[0.12, -0.44, 0.81, ...]

The exact numbers are learned during training.

Tokens used in similar contexts often develop related representations.

2. Context changes the representation

The initial embedding is only the starting point.

As the token passes through Transformer layers, its representation is updated based on the surrounding context.

For example:

river bank
bank account

The token bank starts from the same token representation, but the model can produce different contextual representations because the surrounding words are different.

3. Transformer layers refine hidden states

At each layer, self-attention and feed-forward networks transform the vectors.

These intermediate vectors are often called hidden states.

They can encode information related to:

• Token identity
• Position
• Syntax
• Semantic relationships
• References between words
• Broader context

This information is distributed across many dimensions rather than stored in one single location.

4. Knowledge is distributed across parameters and activations

An LLM's learned language patterns are represented partly in its parameters and partly in the temporary hidden states created while processing a specific input.

The parameters contain patterns learned during training, while the hidden states represent the current context.

5. Final representations are used for prediction

For a generative LLM, the final hidden representation is passed to the output layer, which produces scores for possible next tokens.

The model then uses those scores to continue the sequence.

For related concepts, see “What are parameters in an LLM?” and “What is a token in an LLM?”

In short: an LLM represents language as high-dimensional numerical vectors. These vectors are repeatedly transformed by Transformer layers so they capture context, relationships, and patterns needed for prediction.