Zyphra explains recency bias from neighboring-window word mixing
Original titleBecause neighboring windows contain many of the same words, mixing information within each window makes nearby words carry similar inform...
AISummary
Zyphra says that mixing information within overlapping neighboring windows makes nearby words carry similar information inside a model. This creates a built-in clue about how far apart words are, even before training begins, which the post calls recency bias.
Source: Zyphra · x.comPublished · added here