AINews Portal

AI Glossary

What is Token?

Models do not process words or characters directly; text is split into tokens — common words, word fragments, or punctuation. A token averages about four characters of English. API pricing, context windows, and speed are all measured in tokens, which is why "per-million-token" pricing is the standard unit.

← All terms