ttok 1.0 switches default tokenizer to GPT-5 family and GPT-6
Original titlettok 1.0
AISummary
Simon Willison released ttok 1.0, changing its default tokenizer from GPT-4 to a newer GPT-5/GPT-6 tokenizer after finding the old default was outdated. OpenAI has not confirmed that GPT-6 shares the GPT-5 tokenizer, but William Liu's experiment found all seven GPT models tested reported 44,794 tokens on a 31-fixture corpus.
Source: Simon Willison · simonwillison.netPublished · added here