Understanding LLM Response Generation Principles — Tokenization and Context Window

When you input a question into a large language model, natural-sounding sentences appear on the screen. They are so natural that it feels as if the model reads the entire sentence, understands its meaning like a human, and then retrieves a stored answer from its memory.

However, the actual processing is different. LLMs do not handle the sentences we input as a whole. First, they divide the text into small units called tokens, convert each token into a number, and then calculate the relationships between tokens within a limited context window. Then, they select one token that is most likely to come next in the current context. The selected token is added back to the context, and the same calculation is repeated.

사용자 입력
→ 토큰화
→ 토큰 ID와 벡터 변환
→ 컨텍스트 안의 관계 계산
→ 다음 토큰 확률 예측
→ 토큰 하나 선택
→ 컨텍스트에 추가
→ 종료 조건까지 반복

Understanding this flow can simultaneously explain “why answers vary slightly even for the same question,” “why some content is missed when a long document is provided,” “why earlier instructions are forgotten as conversations get longer,” and “why the number of tokens affects cost and speed.”

LLMs See Tokens, Not Sentences

A token is the basic unit that a model uses to process text. A single token does not always represent a single word. Short, frequently used words might be a single token, but longer words can be split into multiple pieces. Spaces and punctuation marks can also become separate tokens, and depending on the implementation, a space might be included in the next word piece.

Conceptually, you can imagine the following segmentation:

"생성형 AI는 문장을 토큰으로 처리한다"

→ ["생성", "형", " AI", "는", " 문장", "을", " 토큰", "으로", " 처리", "한다"]

This example is for illustrating the principle. The actual results vary depending on the tokenizer and vocabulary dictionary used by the model. Even the same sentence can be tokenized differently by GPT-series, Llama-series, and BERT-series models.

The reason for using subword units, which are smaller than words, is to maintain a balance between the size of the vocabulary dictionary and its expressive power.

If only word units are used, it’s difficult to process new words not in the dictionary, typos, or proper nouns.
If only character units are used, almost all sentences can be expressed, but the input length becomes excessively long.
Subwords group frequently appearing strings into larger units and split rare strings into smaller pieces.

Therefore, even unfamiliar product names or newly coined words are not treated as a single unknown word, but can be expressed as a combination of smaller, already known pieces.

Tokenization Is Not Just Splitting by Spaces

An actual tokenizer is closer to a pipeline involving multiple steps. The Hugging Face Tokenizers documentation describes a typical flow as normalization, pre-tokenization, subword modeling, and post-processing steps.

원문
→ Normalizer
→ Pre-tokenizer
→ Subword model
→ Post-processor
→ Token ID 배열

Normalization

This step involves unifying uppercase and lowercase letters, cleaning up Unicode representations, or converting specific characters into standard forms. Not all models use the same normalization. Some models preserve case, and normalization rules can vary depending on the training objective.

Pre-tokenization

This is the initial step of dividing text based on spaces, punctuation marks, etc. Languages like English, where word boundaries are relatively clear with spaces, and languages like Korean, where particles and endings combine with roots, may show different token distributions.

Subword Segmentation

This involves dividing strings into pieces that exist in the model’s vocabulary dictionary using methods like BPE, WordPiece, Unigram, and SentencePiece. While the detailed goals of these algorithms differ, their common objective is to efficiently represent various strings with a limited vocabulary dictionary.

Adding Special Tokens

Depending on the model, special tokens are added to indicate the beginning and end of a sentence, speaker roles, separators, padding, and so on. While developers see messages in a simple system, user, assistant structure in a chat API, when passed to the model, internal tokens and templates expressing roles and boundaries may be added.

Therefore, it is difficult to accurately determine the actual number of input tokens by simply counting the characters visible on the screen.

Estimating Korean Token Count Based on English Standards Can Be Misleading

Empirical rules like “one English token is approximately X characters” are often introduced, but applying them directly to Korean can lead to significant errors.

In Korean, particles and endings combine with word roots, and the way Hangeul syllables are processed also varies by tokenizer. Even sentences with the same meaning can have significantly different token counts depending on the language and writing system. Code, URLs, long numbers, emojis, and unfamiliar product names can also use more tokens than expected.

사람이 보는 길이
≠ 단어 수
≠ 글자 수
≠ 모델이 계산하는 토큰 수

For systems where cost or context usage is critical, do not rely solely on estimates; instead, measure with a tokenizer compatible with the actual deployed model. If the model changes, the tokenizer and chat template may also change, necessitating regression testing.

Tokens Are Converted into Numbers and Vectors

Once tokenization is complete, each token is converted into an integer ID defined in the vocabulary dictionary.

["생성", "형", " AI"]
→ [18452, 317, 9552]

These numbers do not directly represent the meaning of a word. They are closer to identifiers used to look up tokens in a dictionary. The model queries an embedding table with these IDs to convert them into high-dimensional vectors.

Token ID
→ Embedding vector
→ Transformer layer
→ 문맥을 반영한 표현

The initial embedding is a learned representation of the token itself. As it passes through subsequent Transformer layers, the same token can acquire different internal representations depending on its surrounding context.

For example, the string “은행” (eunhaeng) can mean either a financial institution or a riverbank. The model calculates the relationships with surrounding tokens to reflect which meaning is closer in the current sentence. The core structure playing a key role here is Self-Attention.

The Context Window Is the Workspace a Model Can Refer to for Current Calculations

The context window is the range of tokens a model can refer to when generating a single response. It is often described as an LLM’s memory capacity, but it is more accurate to view it as a limited workspace for processing the current request rather than permanent memory.

In chat-based services, the context does not only include the question the user just entered.

시스템·개발자 지시
+ 이전 사용자 메시지
+ 이전 모델 응답
+ 현재 사용자 질문
+ RAG로 검색한 문서
+ 도구 실행 결과
+ 이미지·파일에서 변환된 입력
+ 새로 생성할 응답 토큰
= 전체 컨텍스트 사용량

Internal formats and special tokens not directly written by the developer also consume some space. If tool definitions or JSON schemas are long, they can also be included in the input tokens.

Let’s assume a simple model with a context limit of 100 tokens.


Component	Tokens Used

—	—
System Instruction	10
Previous Conversation	35
Search Document	25
Current Question	10
Space Reserved for Response	20
Total	100

If you add 20 more tokens for a search document here, it exceeds the total limit. In this case, the system must either reject the request, truncate older conversations, reduce the search document, or decrease the maximum possible response length.

In other words, just because the context window is 100 tokens does not mean you can use all 100 tokens for input. You must also consider the token budget for the output.

Why Earlier Content Is Forgotten in Long Conversations

As conversations get longer, services cannot infinitely pass all messages to the model. When approaching the context limit, the following strategies are used:

Remove the oldest messages.
Summarize past conversations to make them shorter.
Select only highly relevant messages.
Truncate parts of attached documents or tool results.
If the input is too large, fail the request itself.

For example, the truncation option in the OpenAI Responses API distinguishes between automatically removing earlier conversation items when the input exceeds the context limit and returning an error without automatic truncation. While specific policies vary by service and API, the important principle remains the same.

Even if the model appears to remember past conversations, it is often because that content is re-included in the current context, making it available for reference.

If past messages are truncated or poorly summarized, the model cannot directly see that content in its current calculation. This is also why the model might miss output formats or important constraints initially set by the user later in a long conversation.

Being Within the Context Window Does Not Mean Everything Is Equally Well Remembered

A long context limit means that many tokens can be included. It does not mean that all input information is always accurately utilized.

The model calculates how relevant each token is to other tokens using Attention. However, long inputs can contain similar information, irrelevant information, conflicting instructions, and repeated sentences. If important facts are buried among hundreds of thousands of tokens, the model might miss them or treat them as less important.

Problems that can arise with long contexts include:

Missing key information located in the middle of a document.
Inconsistency if instructions in the beginning and end conflict.
Incorrectly associating similar names or numbers.
Being influenced by irrelevant information if search results are too numerous.
Increased cost and first response latency due to increased input tokens.

Therefore, the statement “longer context is always better” is not accurate. Actual quality is influenced not only by the length of the context but also by the selection, arrangement, structure, redundancy, and conflict of information.

Attention Connects Context but Is Not Permanent Storage

The Self-Attention mechanism in a Transformer calculates the relationships between tokens within the current context. Intuitively, it can be understood as the process of creating weights to determine how much each token should refer to other tokens.

Query: 지금 처리하는 토큰이 찾으려는 정보
Key: 각 토큰이 가진 검색 단서
Value: 실제로 전달할 정보

It calculates relevance by comparing the Query at the current position with the Keys of preceding tokens, and then combines the Values using those weights. This calculation is repeated across multiple heads and layers, reflecting various patterns such as grammatical relationships, anaphora, topics, formats, and code structures.

However, Attention is different from database retrieval. It does not precisely query a specific sentence by a key, nor does it permanently store conversation content within the model. It is a mechanism that calculates necessary representations based on the currently input tokens.

Responses Are Generated One Token at a Time

Once context processing is complete, the model calculates scores for all possible tokens that could come next and converts them into a probability distribution.

현재 문맥: "컨텍스트 윈도우는 모델의"

다음 토큰 후보
작업: 31%
기억: 24%
입력: 18%
처리: 11%
기타: 16%

The model selects the next token from this distribution. It can always choose the token with the highest score, or it can select one of several candidates based on sampling settings like Temperature or Top-p.

The selected token is immediately appended to the end of the context.

1단계: "컨텍스트 윈도우는 모델의"
예측: "작업"

2단계: "컨텍스트 윈도우는 모델의 작업"
예측: "공간"

3단계: "컨텍스트 윈도우는 모델의 작업 공간"
예측: "입니다"

This process repeats until an end-of-sequence token is selected, the maximum output length is reached, or a stop condition is met.

This means that LLMs do not complete a long answer all at once and then display it; instead, they use the previously generated content as context again, appending tokens one by one. Therefore, if an incorrect premise is chosen early on, subsequent sentences will naturally follow that premise, potentially creating more elaborate errors.

Reasons for Different Answers to the Same Question

It is rare for only one next token candidate to have a 100% probability. Multiple expressions can be grammatically or semantically natural. If sampling is enabled, different tokens might be selected for the same question, and that difference can change the entire subsequent sentence.

Output is also influenced by the following factors:

System and developer instructions
Previous conversation content and order
Retrieved external documents
Tool execution results
Model version and tokenizer
Temperature and Top-p
Maximum output length
Safety policies and output post-processing

Therefore, an LLM’s response is not so much the result of looking up a fixed answer as it is a path generated from the given context and generation settings.

Token Prediction and Fact Verification Are Different Problems

The model predicts the naturalness of the next token. This process itself does not mean verifying facts from a database.

Because it has learned many facts and patterns during training, it can often produce accurate answers, but it can also generate plausible sentences even in situations without solid evidence. This is one of the fundamental backgrounds for hallucinations.

언어적으로 자연스러운 다음 토큰
≠ 외부 세계에서 검증된 사실

For tasks where accuracy is crucial, external verification methods such as official data retrieval, RAG, calculation tools, code execution, and database queries should be used in conjunction. Furthermore, simply including external data in the context does not automatically guarantee a correct answer. It is necessary to accurately retrieve relevant documents, include sufficient portions, and distinguish between claims from different sources.

RAG Is Not About Making Context Longer, But About Selecting It Better

RAG (Retrieval-Augmented Generation) is a structure that retrieves documents related to a query and adds their content to the model’s context.

사용자 질문
→ 관련 문서 검색
→ 필요한 구간 선택
→ 질문과 함께 컨텍스트 구성
→ LLM 응답 생성

The core of good RAG is not about including many documents. It’s about placing highly relevant information within a limited context so that the model can find the evidence needed for its answer.

Unconditionally including too many search results can lead to the following problems:

Important documents get buried among irrelevant ones.
Conflicting versions of documents are included together.
Token costs and latency increase.
Exposure to indirect prompt injection from malicious documents.

Therefore, document chunk size, number of retrievals, re-ranking, deduplication, source attribution, and relevance filtering are important.

Practical Design for Effective Use of Long Contexts

Keep Important Instructions Short and Clear

Organize core requirements into a separate section instead of burying them within long background explanations.

[목표]
점검 결과를 경영진용으로 요약한다.

[반드시 지킬 조건]
- 확인되지 않은 사실을 만들지 않는다.
- 위험도와 근거를 함께 표시한다.
- 표는 5행 이내로 작성한다.

[참고 자료]
...

Summarize Current Task Status Rather Than Old Conversations

If an agent is performing a long task, it’s better to maintain a structured summary of the current goal, completed steps, reasons for failure, and remaining tasks, rather than continuously accumulating all logs.

Retrieve and Include Only Necessary Parts of Documents

Instead of unconditionally including hundreds of pages, search for sections relevant to the question and provide the source and document version of the search results.

Manage Input and Output Budgets Separately

Filling the input to its limit might not leave enough space for the response. You should allow for buffer, including the expected output length and tool call results.

Observe Actual Token Usage

Record input and output tokens using the usage information from API responses or a compatible tokenizer. Check not only the average but also the upper percentiles of long requests to prevent sudden cost increases or limit overruns.

import tiktoken

encoding = tiktoken.get_encoding("o200k_base")
text = "LLM은 문장을 토큰으로 나누어 처리합니다."

token_ids = encoding.encode(text)

print(token_ids)
print("token count:", len(token_ids))

This example is a simple code to verify the principles of tokenization. Actual API model token calculation may include model-specific encoding, message formats, special tokens, and tool definitions, so official SDKs and model documentation should be consulted together.

Common Misconceptions


Misconception	Closer to Reality

—	—
A token is the same as a word	Words, subwords, spaces, punctuation marks, byte pieces, etc., can all be tokens.
The context window is the model’s permanent memory	It is closer to a limited workspace available for current response calculation.
If it’s within the limit, all content is accurately found	In long inputs, important information can be buried or conflict.
Output is completed all at once	It predicts the next token one by one and adds it back to the context.
Next token prediction is simple autocomplete	It is autoregressive generation that calculates complex contextual representations through large-scale training and multiple Transformer layers.
Natural-sounding sentences are factual	Naturalness and fact verification are separate issues.
If the context is long, RAG is not needed	Retrieval and selection processes are still important for recency, cost, access control, and source tracking.

Connecting the Entire Process Again

LLM response generation can be summarized as follows:

The user’s text, system instructions, previous conversations, and retrieved documents are combined into a single input.
The tokenizer divides the input into subword-centric tokens.
Each token is converted into an integer ID and an embedding vector.
The Transformer calculates the relationships between tokens within the context window using Attention.
A probability distribution for the next token at the last position is created.
One token is selected according to the generation settings.
The selected token is added to the context, and the same calculation is repeated.
When an end-of-sequence token or output limit is reached, the tokens are decoded back into a string and presented as a response.

텍스트
→ 토큰
→ 토큰 ID
→ 임베딩
→ Attention과 Transformer
→ 다음 토큰 확률
→ 토큰 선택
→ 반복 생성
→ 텍스트 응답

Conclusion

The best starting point for understanding LLMs is to temporarily set aside the impression that “the model understands sentences and retrieves answers.”

The model converts text into tokens, calculates the relationships between tokens within the current context window, and then generates the next token one by one. The context window is a limited workspace for these calculations, and the fact that long contexts can be included is different from the fact that their content is accurately utilized.

Knowing this principle changes how prompts are written. It leads to clearly placing important instructions, reducing unnecessary context, summarizing long conversations, selecting only necessary evidence with RAG, and managing input and output token budgets together.

The quality of an LLM is not solely determined by model size. How information is tokenized and placed into the context, and how that limited space is structured, significantly influences the actual response quality.