https://signal-and-syntax.com/Recent content on Signal & SyntaxHugoen-usWed, 26 Aug 2026 00:00:00 +0000https://signal-and-syntax.com/posts/roadmap/Wed, 26 Aug 2026 00:00:00 +0000https://signal-and-syntax.com/posts/roadmap/If there is a topic in the planned list you want to see sooner, or a topic missing that you think belongs here, get in touch . ◐ In Progress ○ Planned ✓ Published In Progress Implementing a Minimal Transformer in PyTorch. Building the core machinery of a language model from embeddings and attention to training and generation Planned (not ordered) The Cooperative Witness Problem. A piece on the ways language models tend to accept and continue user premises rather than push back on them, and the training dynamics that produce that tendency.https://signal-and-syntax.com/posts/inside-attention-part-1/Sat, 22 Aug 2026 06:00:00 -0700https://signal-and-syntax.com/posts/inside-attention-part-1/The transformer architecture is composed of many repeating transformer layers, or blocks. Each block contains an attention sublayer followed by a feedforward sublayer, wrapped in residual connections and layer normalization. Positional information is added to the input so the model knows what order the tokens came in. The attention sublayer sets the table for the feedforward sublayer: it does the work of looking at other tokens and deciding what information to absorb from them.https://signal-and-syntax.com/posts/minimal-transformer/Wed, 01 Apr 2026 06:00:00 -0700https://signal-and-syntax.com/posts/minimal-transformer/In 2017, Vaswani et al. published “Attention Is All You Need,” a paper that quietly rearranged the entire landscape of machine learning. It introduced the Transformer architecture — a design that has since become the backbone of every major language model you’ve heard of: GPT, BERT, Claude, Gemini, and dozens of others. The paper’s title was a provocation. Attention mechanisms already existed. The claim was that you could throw out recurrence entirely and let attention carry the whole load.https://signal-and-syntax.com/posts/discrete-math-in-large-language-models/Tue, 31 Mar 2026 06:00:00 -0700https://signal-and-syntax.com/posts/discrete-math-in-large-language-models/A recent LinkedIn post from Michael Palmer described how discrete mathematics is the foundation for how computers reason about problems. That thread got me thinking about just how many discrete math concepts show up inside systems that seem purely statistical. LLMs are often described in terms of neural networks, gradient descent, and probability distributions. If you’ve taken discrete mathematics and wondered what it has to do with modern AI, the answer is: more than you’d expect.https://signal-and-syntax.com/posts/how-large-language-models-know-things-they-were-never-taught/Mon, 09 Feb 2026 06:00:00 -0700https://signal-and-syntax.com/posts/how-large-language-models-know-things-they-were-never-taught/When you ask an LLM without web search enabled a question like “What happened in the news this morning?”, the LLM will respond by telling you that it doesn’t have access to current events and suggest you check a more current news source such as Reuters or Google News. Conversely, ask an LLM with web search enabled the same question, and you receive a detailed rundown of breaking stories, political controversies, and sports news from the past 24 hours.https://signal-and-syntax.com/posts/temperature-top-p-creativity-knobs/Wed, 24 Dec 2025 12:00:00 -0800https://signal-and-syntax.com/posts/temperature-top-p-creativity-knobs/Every API call to ChatGPT , Claude , or any other LLM includes two parameters most people either ignore or tweak randomly: temperature and top-p. The defaults work fine for casual use, so why bother understanding them? Because these two numbers fundamentally control how your model thinks. The temperature value determines whether the model plays it safe or takes creative risks while the top-p value decides how many options the model even considers.https://signal-and-syntax.com/posts/edmund-fitzgerald/Sun, 30 Nov 2025 06:00:00 -0700https://signal-and-syntax.com/posts/edmund-fitzgerald/Originally appearing on his 1976 album, Summertime Dream, “The Wreck of the Edmund Fitzgerald” is a powerful ballad written and performed by folk singer Gordon Lightfoot . In 1976, the song hit No. 1 in Canada on the RPM chart, and No. 2 in the United States on the Billboard Hot 100. The lyrics are a masterpiece, but there was one specific line that always stood out to me: “The lake, it is said, never gives up her dead.https://signal-and-syntax.com/posts/birthday-paradox/Fri, 28 Nov 2025 06:00:00 -0700https://signal-and-syntax.com/posts/birthday-paradox/You generate a UUID. It’s 128 bits total, with 122 bits of randomness ( Davis et al., 2024 ). That’s 340 undecillion possible values. Collision-proof, right? Your system generates a million IDs per second. Still safe? What about a billion? As I like to say, common sense and intuition are the enemies of science. Common sense tells you that with 340,000,000,000,000,000,000,000,000,000,000,000,000 possible values, you’d need to generate at least trillions before worrying about duplicates.https://signal-and-syntax.com/posts/hash-collisions/Mon, 17 Nov 2025 06:00:00 -0700https://signal-and-syntax.com/posts/hash-collisions/In 2017, Google researchers generated two different PDF files with identical SHA-1 hashes, finally proving what cryptographers had warned about for years: hash functions don’t create truly unique fingerprints ( Stevens et al., 2017 ). This “SHAttered” attack required 9 quintillion SHA-1 computations, which is the equivalent to 6,500 years of single-CPU computation. The attack cost approximately $45,000 in cloud computing resources, making it accessible to well-funded adversaries but not casual attackers.https://signal-and-syntax.com/posts/how-large-language-models-tokenize-text/Tue, 11 Nov 2025 06:00:00 -0700https://signal-and-syntax.com/posts/how-large-language-models-tokenize-text/When you type “I love programming” into ChatGPT, you might assume the model reads three words. It doesn’t. It reads somewhere between three and seven tokens, depending on how the text is split. When you ask Claude to count the letters in the word “strawberry,” it often gets it wrong. The reason is simple. Claude never saw the word “strawberry” as a complete unit. It saw tokens like "str", "aw", "berry" and tried to reason about letters it couldn’t directly access.https://signal-and-syntax.com/posts/how-large-language-models-handle-context-windows/Mon, 10 Nov 2025 06:00:00 -0700https://signal-and-syntax.com/posts/how-large-language-models-handle-context-windows/When you have a long conversation with a large language model (LLM) such as ChatGPT or Claude , it feels like the model remembers everything you’ve discussed. It references earlier points, maintains consistent context, and seems to “know” what you talked about pages ago. But here’s the uncomfortable truth: the model doesn’t remember anything. It’s not storing your conversation in memory the way a database would. Instead, it’s rereading the entire conversation from the beginning every single time you send a message.https://signal-and-syntax.com/posts/safe-distance-in-traffic/Thu, 23 Oct 2025 09:00:00 -0700https://signal-and-syntax.com/posts/safe-distance-in-traffic/While researching why car insurance rates are so extremely high in Las Vegas, I started thinking about the three-second rule and its validity. As I’ve always heard, the three-second rule refers to how far you should be behind a car in traffic. The idea is that you pick out a fixed roadside marker and you are supposed to pass that marker at least three seconds after the car in front of you.https://signal-and-syntax.com/posts/thermo-water-energy-balance/Sun, 12 Oct 2025 09:00:00 -0700https://signal-and-syntax.com/posts/thermo-water-energy-balance/Every summer, it feels like a small miracle when the pool finally warms up enough to swim. In Nevada, where the air temperature can sit above 100°F (38°C) for weeks, you’d expect the water to keep pace. Yet, somehow, it takes forever to warm, and only a few cool nights can undo all that progress. The same phenomenon shows up in a stick of butter. Butter melts quickly, while margarine stays stubbornly firm even under the same heat.https://signal-and-syntax.com/posts/how-large-language-models-learn/Wed, 08 Oct 2025 06:00:00 -0700https://signal-and-syntax.com/posts/how-large-language-models-learn/When you interact with a large language model (LLM) such as ChatGPT or Claude , the model seems to respond instantly relative to the question’s degree of difficulty. What’s easy to forget is that every word it predicts comes from a long history of learning where billions of gradient steps have slowly sculpted its understanding of language. Large language models don’t memorize text. They optimize it. Behind that optimization lies calculus.https://signal-and-syntax.com/posts/how-large-language-models-think/Tue, 07 Oct 2025 06:00:00 -0700https://signal-and-syntax.com/posts/how-large-language-models-think/When you enter a sentence into a Large Language Model (LLM) such as ChatGPT or Claude , the model does not process words as language. It represents them as numbers. Each word, phrase, and code token becomes a vector — a list of real-valued coordinates within a high-dimensional space. Relationships between meanings are captured not by grammar or logic but by geometry. The closer two vectors lie, the more similar their semantic roles appear to the model.https://signal-and-syntax.com/posts/how-large-language-models-read-code/Mon, 06 Oct 2025 09:00:00 -0700https://signal-and-syntax.com/posts/how-large-language-models-read-code/Developers are accustomed to thinking about code in terms of syntax and semantics, the how and the why. Syntax defines what is legal; semantics defines what it means. A compiler enforces syntax with ruthless precision and interprets semantics through symbol tables and execution logic. But a Large Language Model (LLM), reads code the way a seasoned engineer reads poetry, recognizing rhythm, pattern, and context more than explicit rules. The difference may seem subtle, but it has vast consequences.https://signal-and-syntax.com/posts/python-integer-division-and-modulus/Wed, 17 Sep 2025 13:00:00 -0700https://signal-and-syntax.com/posts/python-integer-division-and-modulus/numbers with // and % for speed and clarity." }} When you need to parse a number, the first instinct is often to convert it to a string and slice it. That works well for data that comes from people — like phone numbers, credit cards, or postal codes — where formatting and leading zeros matter. But when you are working with raw numeric data that is guaranteed to be fixed-width and free of formatting, numeric parsing with integer division (//) and modulus (%) is the better option.https://signal-and-syntax.com/posts/python-sympy-vs-numpy/Fri, 05 Sep 2025 11:45:00 -0700https://signal-and-syntax.com/posts/python-sympy-vs-numpy/Most of us reach for NumPy whenever math shows up in a project. But sometimes, you don’t want approximate answers, you want exact math. That’s when you pull SymPy out of your programmer’s toolkit and get to work. It’s easy to think of SymPy only in academic terms, like running physics simulations where small rounding errors can snowball into nonsense, or checking algebraic identities where a value such as 0.0000001 should really be treated as exactly 0.https://signal-and-syntax.com/posts/five-second-rule/Thu, 04 Sep 2025 06:00:00 -0700https://signal-and-syntax.com/posts/five-second-rule/You know the story: drop a cookie on the kitchen floor, swoop in before five seconds are up, and declare it safe. It is comforting. It is also wrong ( Dawson et al., 2007 ; Miranda & Schaffner, 2016 ). The truth is much more interesting than the myth. Germs do transfer gradually, but they are especially fast at the beginning. That means if you want to know whether your floor-cookie is still edible, you need to think in curves, not in timers.https://signal-and-syntax.com/posts/meeting-diet/Thu, 28 Aug 2025 05:00:00 -0700https://signal-and-syntax.com/posts/meeting-diet/Every week your calendar fills with more meeting invites than you can reasonably handle. Which ones are worth the time and energy, and which should you politely decline? What if there was a way to quantify that choice? The good news: math can help. By modeling your schedule as a 0/1 knapsack problem with two constraints ( Kellerer et al., 2004 ), you can treat meetings like items with value, time cost, and energy cost.https://signal-and-syntax.com/posts/python-dispatch-maps/Fri, 22 Aug 2025 10:38:20 -0700https://signal-and-syntax.com/posts/python-dispatch-maps/Let’s be honest: argument validation code is rarely the proudest part of anyone’s repo. Most of us start with the usual suspects: ❌ The dreaded inverted-V tower of if/else statements ❌ A graveyard of guard clauses scattered line after line Both work fine… until they don’t. Then you’re left maintaining a wall of conditionals that feels like it was designed by a committee of goblins. There’s a better way: dispatch tables!https://signal-and-syntax.com/posts/three-d-packing/Wed, 20 Aug 2025 09:00:00 -0700https://signal-and-syntax.com/posts/three-d-packing/My first full-time programming job was for Holiday on Ice, an international ice show. While I focused mainly on back office systems such as accounting, itinerary, and box office reporting, I knew that one of the biggest technical challenges faced by the show’s crew was efficiently loading trucks for the next city. One day, the controller asked me if I could code a system that took, as input, the trucks’ 3D dimensions and the 3D dimensions (and weight) of every object to be packed.https://signal-and-syntax.com/posts/tfp-chatgpt/Tue, 19 Aug 2025 05:00:00 -0700https://signal-and-syntax.com/posts/tfp-chatgpt/If ChatGPT can write code, summarize legal briefs, and help draft business strategies in seconds, why doesn’t that show up in our productivity statistics? Economists have long relied on a metric called Total Factor Productivity (TFP) to measure technological progress. But in an era of free digital tools and generative AI, TFP looks more like a rearview mirror than a windshield. It tells us a lot about the past, but almost nothing about where the economy is headed.https://signal-and-syntax.com/posts/rain-paradox/Mon, 18 Aug 2025 09:00:00 -0700https://signal-and-syntax.com/posts/rain-paradox/Early in my programming career, I came across a coding challenge that stuck with me for many years: At the time, I didn’t have the skillset or tools to simulate the problem properly. It became one of the first exercises that nudged me toward a lifelong fascination with modeling the real world through code. The problem wasn’t about recursion or memory management. It was about getting wet, and how fast you move through falling rain.https://signal-and-syntax.com/about/Mon, 01 Jan 0001 00:00:00 +0000https://signal-and-syntax.com/about/Most writing about LLMs sits at one of two altitudes: high-level explainers aimed at executives and curious newcomers, or dense research papers aimed at other researchers. There’s less writing in the middle, where working engineers actually operate. Most engineers need to understand enough of what’s happening under the hood to build and debug real systems, but not so much that they feel like they’re replicating a PhD program. Signal & Syntax serves that middle altitude.https://signal-and-syntax.com/bio/Mon, 01 Jan 0001 00:00:00 +0000https://signal-and-syntax.com/bio/My focus is the engineering beneath modern AI systems. After several years of building applications with AI through APIs, my interests have moved below that abstraction layer: understanding how models are architected, trained, evaluated, and served, why they’re designed the way they are, and the engineering tradeoffs behind those decisions. That direction follows a pattern across my forty-year career as a software developer. I’ve primarily built applications, but I’ve repeatedly found myself pushing beyond the application itself and building the systems, abstractions, and tooling behind it: metaprogramming layers, dynamic runtimes, configuration-driven engines, and automation platforms designed to outlive the original feature request.https://signal-and-syntax.com/glossary/Mon, 01 Jan 0001 00:00:00 +0000https://signal-and-syntax.com/glossary/This glossary defines technical terms used throughout Signal & Syntax. ABCDEFGHIJKLMNOPQRSTUVWXYZ AAblation An experimental technique that removes or disables part of a model to determine how that component contributes to its behavior. Abstract syntax tree (AST) A tree representation of the grammatical structure of source code, with nodes representing constructs such as expressions, statements, and declarations. Adam An optimization algorithm that adapts learning rates for individual parameters using estimates of the first and second moments of their gradients.