inv_freq or freq in RoPE implementation? (Ch05-07 GPT to Llama)

[EricTay1997]

In both converting-gpt-to-llama2.ipynb and converting-llama2-to-llama3.ipynb, precompute_rope_params has this line:

inv_freq = 1.0 / (theta_base ** (torch.arange(0, head_dim, 2)[: (head_dim // 2)].float() / head_dim))

Given that this initializes the vector of $\theta_i$ values, which is used in $cos(\theta_i)$ and $sin(\theta_i)$ calculations, shouldn't this technically be freq and not inv_freq? To support this, Section 4 of this paper, also terms these frequencies.

Perhaps what is confusing is that Hugging Face's implementation also calls these inverse frequencies, while Llama's implementation calls these frequencies.

Please let me know if I'm missing something! Thank you!

[d-kleine]

Hi,

As a fellow user/reader, I had the same question some time ago, see #412. inv_freq is correct here as the actual frequencies are computed as (theta_base ** (torch.arange(0, head_dim, 2)[: (head_dim // 2)].float() / head_dim)).

The inv_freq will be used later when calculating the outer product between the position indices and the inverse frequencies for retrieving the rotation angles with this code:

angles = positions[:, None] * inv_freq[None, :]

Llama uses the same approach with torch.outer(), see here. But yeah, in the Llama implementation, the input should have been named inv_freqs instead of freqs there imho (you could raise an issue or PR on that?)

[EricTay1997]

Hm... Sorry I still don't really understand this!

First, I believe that the frequency of $\cos(x)$ is $\frac{x}{2\pi}$.

Per the RoPE paper, we have that $\theta_i = 10000^{-2(i-1)/d}$. Because of the negative sign in the exponent, I'm inclined to believe that $\theta_i$ is exactly what is computed in 1.0 / (theta_base ** (torch.arange(0, head_dim, 2)[: (head_dim // 2)].float() / head_dim)).

Is there something I'm missing here?

Thank you!

[d-kleine]

I double-checked, and I believe you're correct: These computed values ($10000^{-2(i-1)/d}$) can indeed be considered frequencies, as they determine how quickly each dimension "rotates" across positions. The term 'inverse frequencies' seems to be used informally at times because these values decrease exponentially, becoming smaller for higher dimensions and resulting in slower rotations. This naming likely stems from historical conventions (such as sinusoidal positional encoding) rather than a precise mathematical definition. Perhaps @rasbt could elaborate further on this.

[rasbt]

Thanks for the great discussion here. I think the reason why "inverse" frequency is a popular term is simply because of the fact that it decreases inversely with the index.

Sure, it's also a "frequency" as a general term, but the "inverse" kind of describes the relationship with i.

[EricTay1997]

Thank you for the response!

I think this was most confusing when understanding how we adjust frequencies for longer context windows, e.g. the code excerpt below:

wavelen = 2 * torch.pi / inv_freq

inv_freq_llama = torch.where(
    wavelen > low_freq_wavelen, inv_freq / freq_config["factor"], inv_freq
)

It definitely helps to interpret inv in the way you mentioned!