Rotary Positional Embeddings, Worked Through in NumPy

Most explanations of rotary positional embeddings stop at the formula. A query vector gets rotated by an angle that depends on its position, a key vector gets rotated by its own angle, and the dot product between them ends up depending on the distance between the two positions rather than on either position alone. That sentence is correct, and it tells you almost nothing about what actually happens to real numbers. So I sat down with NumPy and a pair of eight-number vectors, rotated them by hand, and watched the dot products move. The numbers below are the actual output, not a sketch of what the output should look like.
The question RoPE answers is older than transformers. A transformer reads a whole sequence at once, so on its own it has no idea which word came first. The first fix was to add a position vector to each word vector, a different learned or sinusoidal offset per position. That works, but it bakes absolute position into the representation: position 7 is a different place from position 70 in a way the model has to learn, and a model trained on short passages has never seen position 4,000 at all. RoPE, from Su and colleagues' 2021 paper "RoFormer: Enhanced Transformer with Rotary Position Embedding", takes another route. It leaves the word vectors alone and rotates them instead. Each pair of dimensions gets turned by an angle proportional to the position, with a different rotation speed per pair. The rotation is a 2D turn, the kind you meet in school geometry, applied to every consecutive pair of numbers in the vector.
The setup: one vector, eight numbers
I used a model width of 8, so there are four pairs to rotate, and the rotation speeds follow the paper's rule: the i-th pair turns at 10000^(-2i/8) radians per position. That gives speeds of 1, 0.1, 0.01 and 0.001 radians per step. The first pair spins fast, the last pair barely moves. That spread is the whole trick for long sequences: the fast pair tells nearby positions apart, the slow pair still changes measurably over thousands of steps.
The two vectors I rotated are nothing special, which is the point. Any pair would do; a cherry-picked one would prove less.
import numpy as np
def rope_angles(d_model, positions):
i = np.arange(d_model // 2)
theta = 10000.0 ** (-2.0 * i / d_model)
return positions[:, None] * theta[None, :]
def apply_rope(x, angles):
x = x.reshape(*x.shape[:-1], -1, 2)
x_even, x_odd = x[..., 0], x[..., 1]
cos, sin = np.cos(angles), np.sin(angles)
out_even = x_even * cos - x_odd * sin
out_odd = x_even * sin + x_odd * cos
return np.stack([out_even, out_odd], axis=-1).reshape(*x.shape[:-2], -1)
The first function builds the angles; the second applies the rotation pair by pair. Nothing here needs a deep learning library. Rotating a vector is high-school trigonometry, and that is part of why the idea spread so fast: the implementation is about fifteen lines, and it slots into the attention computation without changing anything else.
The query and key I used:
q = np.array([1.0, 0.5, -0.75, 0.25, 0.9, -0.4, 0.3, 0.8])
k = np.array([0.6, -0.2, 0.4, 0.9, -0.7, 0.1, 0.5, -0.3])
Without any position at all, their dot product is -0.335. That is the baseline: what the attention score would be if the model had no sense of order.
What a rotation does to a small vector
Before the dot products, one concrete rotation. Take x = [1, 0, 0, 1, 1, 0, 0, 1] and rotate it to position 1. The angles at position 1 are [1, 0.1, 0.01, 0.001], one per pair, and the result is:
x = [ 1. 0. 0. 1. 1. 0. 0. 1. ]
RoPE(x, 1) = [ 0.5403 0.8415 -0.0998 0.995 1. 0.01 -0.001 1. ]
Read it pair by pair. The first pair (1, 0) turns by a full radian, so it becomes (0.5403, 0.8415), the cosine and sine of 1. The second pair (0, 1) turns by 0.1 radians and becomes (-0.0998, 0.995). The last two pairs barely move, because their speeds are 0.01 and 0.001. Two things worth checking by hand: the length is unchanged, 2.0 before and 2.0 after, because a rotation never stretches a vector, and position 0 leaves the vector exactly as it was, since every angle is zero. Both came out as expected. A rotation that changed the length would have been a bug, because attention scores would then drift with position for no reason at all.
The relative-position property, measured
Now the claim that matters. Put the query at position m and the key at position n, and the dot product should depend only on m - n, never on m or n separately. Here is what came out:
m=0 n=0 dot = -0.335000
m=1 n=1 dot = -0.335000
m=3 n=3 dot = -0.335000
m=5 n=5 dot = -0.335000
m=1 n=0 (m-n=1) dot = -1.064937
m=2 n=1 (m-n=1) dot = -1.064937
m=4 n=3 (m-n=1) dot = -1.064937
m=5 n=4 (m-n=1) dot = -1.064937
m=2 n=0 (m-n=2) dot = -1.654841
m=3 n=1 (m-n=2) dot = -1.654841
m=5 n=3 (m-n=2) dot = -1.654841
Same distance, same dot product, to six decimal places, wherever the pair sits in the sequence. That is the property in its raw form: the model can learn "three tokens back" as one thing, instead of learning "position 4 attending to position 1" and "position 9 attending to position 6" as two unrelated facts. It is also why RoPE extrapolates better than added position vectors. A model trained on passages up to 2,000 tokens long has still seen relative distances of 0, 1 and 2 at every position, so those patterns transfer to position 4,000 without retraining. Absolute positions do not transfer like that; position 4,000 is simply a place the model has never been.
The two catches the formula hides
The first catch showed up when I put one query at position 2 and scored it against the same key at positions 0 through 5:
query at position 2, keys at 0..5: [-1.6548, -1.0649, -0.335, -0.0639, -0.428, -1.0196]
The key sitting at the same position as the query scores -0.335. The key one step ahead scores -0.0639, higher. If you assumed RoPE means "nearby tokens always attend more strongly", this breaks the assumption in the first experiment. It does not break RoPE. The dot product mixes two things: how far apart the tokens are, and what the two vectors actually contain. Rotation changes the alignment between them; it never promises that distance zero wins. What it promises is narrower and more useful: for a fixed pair of vectors, the score is a function of distance alone. Content still decides the sign and size. A model that wants "attend to the previous token" has to learn that from the content, with distance as one input among others.
The second catch is about long distances. Same pair, distances 0 to 40:
distance 0: dot = -0.335000
distance 1: dot = -1.064937
distance 2: dot = -1.654841
distance 5: dot = -0.587188
distance 10: dot = -1.620704
distance 20: dot = -1.720099
distance 40: dot = -0.871104
There is no smooth decay here. The score wobbles as distance grows, dipping and recovering, because each pair of dimensions oscillates at its own speed and the sum of four cosines is not a monotone curve. People sometimes describe RoPE as giving attention a natural long-range decay, and for some vectors it looks that way, but it is not a guarantee of the mechanism. It is a property of particular vectors, and this pair is a counterexample in eight numbers. That wobble is one reason stretching a RoPE model to very long contexts takes extra work in practice: techniques with names like NTK-aware scaling and YaRN rescale the rotation speeds so the slow pairs keep distinguishing distant positions instead of wrapping around. The base mechanism stays the same; the speeds get retuned for the longer horizon.
Why this one stuck
RoPE is now the default position scheme in most open-weight models of the last few years, including the LLaMA family and GPT-NeoX. The reasons are visible in the experiment above rather than in the paper's algebra. It costs almost nothing, fifteen lines and no extra parameters to learn. It gives the model relative position for free, which is what attention actually needs: whether two words are next to each other, not where they sit on the page. And it leaves the vectors' lengths alone, so nothing downstream has to adjust for a position-dependent scale.
The part I keep coming back to is how little machinery it takes. Positional information feels like it should need a table, a learned embedding, a whole subsystem. Here it is a rotation, four angles, and a dot product that only cares about the gap. The next time a model surprises you by keeping track of a pronoun twenty sentences back, part of that is this: somewhere inside, two vectors were turned by a few radians, and the angle between them still says how far apart they are.
Sources
- Su, Lu, Pan, Murtadha, Wen, Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding", arXiv:2104.09864 (2021)
- All numbers in this piece are the output of the NumPy program shown above, run on 16 November 2026: rotation speeds 1, 0.1, 0.01, 0.001 radians per step at width 8; same-distance dot products equal to six decimal places; long-range scores as listed.
P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam
Comments
Join the discussionNo comments yet. Be the first to comment.






