Announcement_8

In long contexts, RoPE provably loses both its locality bias and consistent token relevance, while increasing its base trades off distinguishing positions against distinguishing tokens—see our new preprint RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably.