What is a token, practically?

Tokens are the operating unit of language models.

They are not exactly words, characters, or sentences. They are the pieces of text a model reads, prices, stores in context, and generates as output.

That makes tokens more than a technical detail.

They shape how much information can be passed into a model, how long a document can be drafted, how much an analysis costs, and how much evidence can support one answer.

In everyday use, the practical question is not only:

“How many tokens does this model support?”

The better questions are:

• How many documents can be reviewed at once?
• How much source material can support one analysis?
• When does output length become the bottleneck?
• When does retrieval become necessary?
• When does a larger context window create a new workflow, rather than simply a larger prompt?

This is the goal of miniToken: to translate token numbers into practical sense.