White Paper: Computational Microscopy
IPAM - March 2023
White Paper: Computational Microscopy.
This whitepaper summarizes the activities and outcomes of the long program on “Computational Microscopy” at the Institute of Pure and Applied Mathematics (IPAM) from September 12 to December 16, 2022.
For more than three centuries, lens-based microscopy, such as optical, phase-contrast,
fluorescence, confocal, and electron microscopy, has played an important role in the evolution of
modern science and technology. In 1999, a novel form of microscopy, known as coherent
diffractive imaging (CDI), was developed and transformed our traditional view of microscopy, as
the diffraction pattern of a noncrystalline object was first measured and then directly phased to
obtain a high-resolution image. The well-known phase problem—the usually unavoidable loss of
phase information in the diffraction intensity—was solved by a combination of coherent
illumination and computational algorithms. Over the years, various CDI methods including
plane-wave CDI, ptychography (i.e. scanning CDI), Bragg CDI and Fourier ptychography have
been broadly implemented using synchrotron radiation, X-ray free electron lasers, high harmonic
generation, and optical and electron microscopy. Furthermore, the 2017 Nobel Prize in chemistry
was awarded to Richard Henderson, Joachim Frank, and Jacques Dubochet for developing
cryo-electron microscopy (cryo-EM) for the high-resolution structure determination of
biomolecules in solution. All these groundbreaking developments require the use of advanced
computational algorithms and mathematical tools. This IPAM long program brought together
senior and junior applied mathematicians, physicists, chemists, materials scientists, engineers,
and biologists to discuss and debate on the current status and future perspectivesof modern
microscopy using computation, mathematics, and modeling. The program hosted four workshops
focusing on different aspects of computational microscopy:
fluorescence, confocal, and electron microscopy, has played an important role in the evolution of
modern science and technology. In 1999, a novel form of microscopy, known as coherent
diffractive imaging (CDI), was developed and transformed our traditional view of microscopy, as
the diffraction pattern of a noncrystalline object was first measured and then directly phased to
obtain a high-resolution image. The well-known phase problem—the usually unavoidable loss of
phase information in the diffraction intensity—was solved by a combination of coherent
illumination and computational algorithms. Over the years, various CDI methods including
plane-wave CDI, ptychography (i.e. scanning CDI), Bragg CDI and Fourier ptychography have
been broadly implemented using synchrotron radiation, X-ray free electron lasers, high harmonic
generation, and optical and electron microscopy. Furthermore, the 2017 Nobel Prize in chemistry
was awarded to Richard Henderson, Joachim Frank, and Jacques Dubochet for developing
cryo-electron microscopy (cryo-EM) for the high-resolution structure determination of
biomolecules in solution. All these groundbreaking developments require the use of advanced
computational algorithms and mathematical tools. This IPAM long program brought together
senior and junior applied mathematicians, physicists, chemists, materials scientists, engineers,
and biologists to discuss and debate on the current status and future perspectivesof modern
microscopy using computation, mathematics, and modeling. The program hosted four workshops
focusing on different aspects of computational microscopy:
● Workshop I: “Diffractive Imaging with Phase Retrieval” focused on advanced
computational methods to solve the phase problem using iterative algorithms and deep
learning.
● Workshop II: “Mathematical Advances for Multi-Dimensional Microscopy” focused on
the incorporation of state-of-the-art mathematical and computational methods into
multi-dimensional electron microscopy.
● Workshop III: “Cryo-Electron Microscopy and Beyond” focused on the current
challenges and future perspectives of the cryo-EM field.
● Workshop IV: “Multi-Modal Imaging with Deep Learning and Modeling” focused on the
integration of data acquisition, mathematical modeling, and deep learning in multimodal
microscopy.
computational methods to solve the phase problem using iterative algorithms and deep
learning.
● Workshop II: “Mathematical Advances for Multi-Dimensional Microscopy” focused on
the incorporation of state-of-the-art mathematical and computational methods into
multi-dimensional electron microscopy.
● Workshop III: “Cryo-Electron Microscopy and Beyond” focused on the current
challenges and future perspectives of the cryo-EM field.
● Workshop IV: “Multi-Modal Imaging with Deep Learning and Modeling” focused on the
integration of data acquisition, mathematical modeling, and deep learning in multimodal
microscopy.
In addition to these four workshops, we formed seven working groups, including 1) Simulation
for electron and optical microscopy, 2) Inverse problems in cryo-EM and phase retrieval, 3) AI
& learning theory, 4) Data-driven information extraction from microscopy data, 5) Multimodal
data processing and acquisition, 6) Space-time models, and 7) Geometry in data processing for
microscopy. The working groups met regularly during the program and tackled a number of
outstanding problems in the field. Below we provide the open challenges that we identified in
computational microscopy, the progress that we made at IPAM, and the research directions that
we will continue to investigate in the future rections in the field of electronic structure theory and computational chemistry as well as related fields that were discussed during the program.
for electron and optical microscopy, 2) Inverse problems in cryo-EM and phase retrieval, 3) AI
& learning theory, 4) Data-driven information extraction from microscopy data, 5) Multimodal
data processing and acquisition, 6) Space-time models, and 7) Geometry in data processing for
microscopy. The working groups met regularly during the program and tackled a number of
outstanding problems in the field. Below we provide the open challenges that we identified in
computational microscopy, the progress that we made at IPAM, and the research directions that
we will continue to investigate in the future rections in the field of electronic structure theory and computational chemistry as well as related fields that were discussed during the program.