RoFormer: Enhanced Transformer with Rotary Position Embedding

RoFormer rotates query and key coordinate pairs so their attention score carries relative position. Work through the shared-shift identity, then audit why computable positions and successful long-context generalization are different claims.

Reading focus: How absolute-position rotations make a query-key comparison depend on relative offset. Why paired frequencies preserve the geometric identity without making every score decrease with distance. What the mixed benchmark results and long-text training schedule establish about generalization.

arXiv 2021 · revised 2023. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 11 min read, medium difficulty.