Zyphra finds smaller context windows improve text prediction while cutting training compute
Original titleIn Zyphra’s experiments, models with smaller windows predicted text better while using less compute during training. We believe the stron...
AISummary
In Zyphra's experiments, models with smaller context windows predicted text better while using less training compute. Zyphra believes the stronger recency bias induced by smaller windows outweighs the reduced compute benefit.
Source: Zyphra · x.comPublished · added here