https://signal-and-syntax.com/tags/induction-heads/Recent content in Induction Heads on Signal & SyntaxHugoen-usSat, 22 Aug 2026 06:00:00 -0700https://signal-and-syntax.com/posts/inside-attention-part-1/Sat, 22 Aug 2026 06:00:00 -0700https://signal-and-syntax.com/posts/inside-attention-part-1/The transformer architecture is composed of many repeating transformer layers, or blocks. Each block contains an attention sublayer followed by a feedforward sublayer, wrapped in residual connections and layer normalization. Positional information is added to the input so the model knows what order the tokens came in. The attention sublayer sets the table for the feedforward sublayer: it does the work of looking at other tokens and deciding what information to absorb from them.