LLMs in practice have activation data that is relatively well explained by a sparse linear combination of a dictionary of feature vectors. I discuss some mean field theory-flavored work on solutions found by toy models trained on such data, and show how analogs of such structure appear in real LLM activations.