https://signal-and-syntax.com/tags/tokenization/Recent content in Tokenization on Signal & SyntaxHugoen-usTue, 11 Nov 2025 06:00:00 -0700https://signal-and-syntax.com/posts/how-large-language-models-tokenize-text/Tue, 11 Nov 2025 06:00:00 -0700https://signal-and-syntax.com/posts/how-large-language-models-tokenize-text/When you type “I love programming” into ChatGPT, you might assume the model reads three words. It doesn’t. It reads somewhere between three and seven tokens, depending on how the text is split. When you ask Claude to count the letters in the word “strawberry,” it often gets it wrong. The reason is simple. Claude never saw the word “strawberry” as a complete unit. It saw tokens like "str", "aw", "berry" and tried to reason about letters it couldn’t directly access.https://signal-and-syntax.com/posts/how-large-language-models-read-code/Mon, 06 Oct 2025 09:00:00 -0700https://signal-and-syntax.com/posts/how-large-language-models-read-code/Developers are accustomed to thinking about code in terms of syntax and semantics, the how and the why. Syntax defines what is legal; semantics defines what it means. A compiler enforces syntax with ruthless precision and interprets semantics through symbol tables and execution logic. But a Large Language Model (LLM), reads code the way a seasoned engineer reads poetry, recognizing rhythm, pattern, and context more than explicit rules. The difference may seem subtle, but it has vast consequences.