---
title: "RoFormer: Enhanced Transformer with Rotary Position Embedding"
description: "RoFormer rotates query and key coordinate pairs so their attention score carries relative position. Work through the shared-shift identity, then audit why computable positions and successful long-context generalization are different claims."
canonical_url: "https://fanout.sh/daily/2026-10-23-roformer-rotary-position-embeddings"
md_url: "https://fanout.sh/daily/2026-10-23-roformer-rotary-position-embeddings.md"
last_updated: "2026-10-23"
access: "public"
---

# RoFormer: Enhanced Transformer with Rotary Position Embedding

RoFormer rotates query and key coordinate pairs so their attention score carries relative position. Work through the shared-shift identity, then audit why computable positions and successful long-context generalization are different claims.

## Paper details

- Authors: Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu

- Venue: arXiv 2021 · revised 2023

- Track: AI Research

- Difficulty: Medium

- Reading time: 11 min

- Original paper: https://arxiv.org/abs/2104.09864v5

## What you will learn

- How absolute-position rotations make a query-key comparison depend on relative offset.

- Why paired frequencies preserve the geometric identity without making every score decrease with distance.

- What the mixed benchmark results and long-text training schedule establish about generalization.

## Continue exploring

[Start with soft attention](https://fanout.sh/daily/2026-09-16-neural-translation-soft-attention): Review how similarity scores become a weighted read before changing the position rule inside each score.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md