Error-correcting codes have been constructed using combinatorial and algebraic techniques for more than 70 years since Shannon. Recently, deep learning has been used to obtain error-correcting codes using the channel autoencoder architecture, where the encoder and decoder are neural networks, separated by a layer representing the noisy channel. Learning an error-correcting code is a challenging task in a well-studied mathematical domain, and therefore it may provide a ``sandbox'' for the general theory of deep learning, from aspects such as training dynamics and interpretability. For example, are the learned codes interpretable as being similar to classical codes or in terms of other mathematical concepts? We discuss experimental results on the deep-learned error-correcting code Turbo-AE, using various approaches such as influences, discrete optimization and Fourier representation. We also discuss a theoretical question suggested by the experiments. As learning with a fixed network architecture can be viewed as optimization over a set of codes (those representable by some weight setting), one can first consider the general problem of optimization over all codes and try to understand the loss landscape. Some initial results are given on the stationarity of some classical types of codes, using symmetry considerations. Joint work with N. Devroye, A. Mulgund, R. Shekhar, Y. Wei, M. Zefran and Y. Zhou.