A language model can be read in two ways. Statistically, it is a probability distribution over strings, estimated from finite data and judged by how well it predicts held-out text. Computationally, it is a model of computation—an object in the same lineage as automata and Turing machines—and the question is which algorithms it can implement. These views usually travel together, but they come apart precisely where it matters: on rare, structured, or out-of-distribution inputs, where good average-case prediction says nothing about whether the right algorithm is present. I will begin with a deep dive on this statistical–computational duality and argue that the computational view demands its own mathematical tools—drawn from logic, automata theory, and complexity. The rest of the talk develops a computational account for the unique-hard-attention transformer (UHAT), a standard and analytically tractable abstraction of self-attention whose expressivity bounds are known to transfer to fixed-precision softmax transformers. On expressivity, fixed-precision UHATs recognize exactly the star-free languages—the class simultaneously captured by linear temporal logic (LTL), first-order logic with order, and counter-free automata. This places transformers strictly below recurrent neural networks (RNNs) as recognizers, which capture all regular languages. Raw expressivity, I will argue, is therefore the wrong lens: it declares transformers weaker than architectures they routinely outperform. I then turn to succinctness—how compactly a formalism describes a language—as the sharper measure. The main results: UHATs can be exponentially more succinct than LTL and RNNs (hence also state-space models), and doubly exponentially more succinct than finite automata. There exist languages recognized by polynomial-size transformers whose smallest equivalent LTL formula or RNN is exponentially larger, and whose smallest automaton is doubly exponentially larger. I show that these bounds are tight. Finally, succinctness comes at a cost: I show that basic verification questions for transformers—emptiness, equivalence—are EXPSPACE-complete, so no efficient algorithm can reason about transformer behavior in general.