This is a two-part talk. In the first part of the talk, I'll introduce a simple but effective framework for bounding the compression error of representing a signal with a compressive parameterization, such as an implicit neural representation. These bounds do not require access to the ground truth signal, are proven to hold (in error norm) for several representative signal parameterization models, are efficient to compute, and are empirically informative about local (pixel-level) compression error. In the second part of the talk, I'll turn to posterior sampling using diffusion models, which are a powerful way to use data-driven priors in inverse problems. While posterior sampling approaches such as DPS often work well, they can fail unexpectedly and catastrophically. We introduce a finite-sample lens on posterior sampling that allows us to probe when, why, and how popular approaches such as DPS fail, including allowing us to observe how hallucinations arise during posterior sampling.