https://signal-and-syntax.com/tags/softmax/Recent content in Softmax on Signal & SyntaxHugoen-usSat, 22 Aug 2026 06:00:00 -0700https://signal-and-syntax.com/posts/inside-attention-part-1/Sat, 22 Aug 2026 06:00:00 -0700https://signal-and-syntax.com/posts/inside-attention-part-1/The transformer architecture is composed of many repeating transformer layers, or blocks. Each block contains an attention sublayer followed by a feedforward sublayer, wrapped in residual connections and layer normalization. Positional information is added to the input so the model knows what order the tokens came in. The attention sublayer sets the table for the feedforward sublayer: it does the work of looking at other tokens and deciding what information to absorb from them.https://signal-and-syntax.com/posts/temperature-top-p-creativity-knobs/Wed, 24 Dec 2025 12:00:00 -0800https://signal-and-syntax.com/posts/temperature-top-p-creativity-knobs/Every API call to ChatGPT , Claude , or any other LLM includes two parameters most people either ignore or tweak randomly: temperature and top-p. The defaults work fine for casual use, so why bother understanding them? Because these two numbers fundamentally control how your model thinks. The temperature value determines whether the model plays it safe or takes creative risks while the top-p value decides how many options the model even considers.