Q bias and K bias in Attention: Why do softmax results differ?

Attention is an operation that calculates relevance scores between Query and Key, then mixes Value based on those ratios. The linear layers that create Q, K, and V can include not only weights but also bias, which is a learnable vector added independently of the input.

Adding bias to Q and adding the same bias to all K’s may seem similar, but their impact on softmax differs in basic dot-product Attention. A common K bias shifts all scores for a single Query by the same amount, while a Q bias can add different values depending on the Key. Softmax eliminates the common shift of the former but reflects the relative differences of the latter.

Following Q·K·V and KV Cache Explanation, this article verifies this single difference with equations and small numbers. It is not the result of running actual LLMs or PyTorch models, and it targets basic Scaled Dot-Product Attention without positional transformations.

This is a conceptual image generated by AI. It does not represent actual screens or measurement results.

1. First, distinguish what the bias is added to

When the current representation of a token is x, the Q and K used for explanation are as follows:

qᵢ = xᵢWQ + bQ
kⱼ = xⱼWK + bK

bQ and bK are added to the respective projection result vectors. The condition is that tokens passing through the same K linear layer within the same Head receive the same bK. PyTorch’s Linear is also described as a layer that applies weight transformation and adds bias to the input. Depending on the direction of matrix storage, the transpose notation in the documentation may vary, but the point of adding a common bias to a vector remains the same. PyTorch Linear

This is distinct from attention bias added to the score matrix after calculating the dot product of Q and K. Masks or relative position scores can be added to the score matrix later, and just because they have ‘bias’ in their name does not mean they can be grouped as the same term.

The basic calculation is as follows:

sᵢⱼ = qᵢ · kⱼ / √dₖ
pᵢⱼ = exp(sᵢⱼ) / Σⱼ exp(sᵢⱼ)
출력ᵢ = Σⱼ pᵢⱼ vⱼ

dₖ is the dimension of the Key vector. Softmax normalizes the score row for a single Query. Therefore, the term “constant” does not mean the same number for the entire matrix, but rather a number that does not change with the Key within a row where the Query is fixed. Attention Is All You Need, 3.2.1

2. Adding the same vector to all K’s maintains score differences

Let’s write the Key before adding bias as kⱼ⁰. If Query qᵢ is fixed, the following equation holds:

qᵢ · (kⱼ⁰ + bK) / √dₖ
= qᵢ · kⱼ⁰ / √dₖ + qᵢ · bK / √dₖ

The latter term qᵢ · bK / √dₖ does not have a Key position j. Therefore, the same cᵢ is added to all scores in a row. The difference between two Keys remains unchanged.

(s₁ + c) - (s₂ + c) = s₁ - s₂

The same term is canceled out in softmax:

exp(sⱼ + c) / Σₜ exp(sₜ + c)
= exp(c)exp(sⱼ) / [exp(c)Σₜ exp(sₜ)]
= exp(sⱼ) / Σₜ exp(sₜ)

From this, the conclusion is that even if only the common K bias is changed in the given basic calculation, the softmax weights remain the same. This does not mean the scores themselves are the same. If the Query changes, cᵢ can also change, but the property of being a common shift within each row is maintained.

3. Q bias can create different score changes for each Key

If we write the Query before bias as qᵢ⁰, it is as follows:

(qᵢ⁰ + bQ) · kⱼ⁰ / √dₖ
= qᵢ⁰ · kⱼ⁰ / √dₖ + bQ · kⱼ⁰ / √dₖ

The latter term changes depending on kⱼ⁰. Even if the same vector is added to Q, when it is dot-producted with different Keys, the change in each score does not necessarily have to be the same. This is the difference from a common K bias.

If both Q and K have bias, the dot product can be divided into four terms:

(qᵢ⁰ + bQ) · (kⱼ⁰ + bK)
= qᵢ⁰·kⱼ⁰ + bQ·kⱼ⁰ + qᵢ⁰·bK + bQ·bK

qᵢ⁰·bK and bQ·bK do not change with the Key in a row. The additional term that can affect relative scores is bQ·kⱼ⁰. Dividing by the scale √dₖ does not change this distinction.

However, Q bias does not always change the weights. If bQ is 0 or its dot product with all allowed Keys is the same, the additional term also becomes a common shift. The accurate expression is “Q bias can change the relative scores between Keys.”

Relative score changes are read as the difference between two Keys

The additional score difference between the first and second Keys due to Q bias is bQ · (k₁⁰ − k₂⁰) / √dₖ. If this value is 0, the relative scores of those two Keys remain the same; if it is not 0, their relative proportions can change. It is more accurate to dot-product with the difference of the Keys to be compared rather than just stating “Q bias has been added.”

The same conclusion can be read from the basic softmax ratio. If both weights are calculated from finite scores, then p₁ / p₂ = exp(s₁ − s₂). Adding the same number to all scores maintains the difference, while adding different values for each Key can change the difference. The number table below directly confirms this relationship.

4. Direct verification with 2D vectors

Now let’s use small synthetic numbers.

q  = (1, 0)
k₁ = (1, 0), k₂ = (0, 1), k₃ = (-1, 0)
bQ = (0, 1)
bK = (3, -2)
dₖ = 2

Adding Q bias makes the Query (1, 1). The scores of the first and second Keys become equal, and the third maintains a low score. A common K bias adds 3/√2, approximately 2.121320, to all scores when dot-producted with Query (1, 0).

Below is a table of actual results calculated with Python’s standard library, rounded to six decimal places.

Condition	Scores before softmax (s₁, s₂, s₃)	Softmax weights (p₁, p₂, p₃)
No bias	(0.707107, 0, -0.707107)	(0.575975, 0.283995, 0.140029)
Q bias only	(0.707107, 0.707107, -0.707107)	(0.445808, 0.445808, 0.108383)
Common K bias only	(2.828427, 2.121320, 1.414214)	(0.575975, 0.283995, 0.140029)
Both Q and K bias	(1.414214, 1.414214, 0)	(0.445808, 0.445808, 0.108383)

The first and third rows have different scores but the same weights. The second and fourth rows also have the same relationship. The mere presence of a common K bias did not change which Value received more weight. The output of mixing fixed V’s is also calculated with the same weights under these conditions.

The numerical calculation used softmax by subtracting the maximum score. This is also a method to avoid large exponential values by utilizing the same common shift invariance.

def softmax(xs):
    peak = max(xs)
    exps = [math.exp(x - peak) for x in xs]
    return [x / sum(exps) for x in exps]

Running the attached verify_attention_bias.py saves inputs, scores, and weights to results.json.

python3 verify_attention_bias.py --output results.json

All 8 verifications passed in Python 3.9.6. We confirmed the basic scores, common shift of K, softmax preservation, relative score changes of Q, and the results of adding both. It also includes other Query rows, exceptions where Q bias has no effect, and a contrast where different K shifts per token are outside the common shift condition. The allowed comparison error was 1e-12 for both relative and absolute errors, and the maximum difference between basic weights and K bias weights in the synthetic numbers above was actually 0.0.

Figure: Comparison of softmax weights from the synthetic numbers in the main text. Reconstructed based on official documentation and not a performance measurement of an actual model. Official Reference 1 · Official Reference 2

5. Observe the boundaries where this conclusion can be applied

This verification calculated the row-wise softmax after dot-producting Query and Key of the same dimension in a single Head, and adding the same bK to all Keys. Dropout, quantization, GPU kernels, or model training results were not measured. The mathematical invariance of real numbers and errors in floating-point implementations must also be distinguished.

In particular, this conclusion is not directly extended to RoPE (Rotary Position Embedding), which rotates Q·K vectors according to position. If the configuration involves adding a common bias and then rotating differently at each position, it must be recalculated whether the additional term after rotation is still common to all Keys. The RoPE paper explains how positions are reflected by rotation matrices. This code does not include that operation and does not make judgments about RoPE models. RoFormer Paper

Other transformations such as Key-specific additional vectors, Q·K normalization, and relative position terms at the score stage also require checking their application points and equations. Even in PyTorch’s Attention examples, QK dot product, terms added to scores, softmax, dropout, and multiplication with V are distinct stages. PyTorch Scaled Dot-Product Attention

This calculation should not be interpreted as a model change recommendation to “remove K bias from all Transformers.” The confirmed conclusion is a property of limited equations. Rather than what was added to the vector, observing whether the difference between Keys changes in the final score row can distinguish the effects of Q and K bias.