For real-world engineering problems, computational models play a crucial role in informing predictions in extrapolative regimes where observational data may be unavailable, making meaningful uncertainty quantification essential. While Bayesian inference provides a unifying framework for characterizing and extrapolating uncertainties, standard approaches can underpredict uncertainties when model misspecification is present. Recently, frameworks for prediction-oriented inverse problems have aimed to remedy this shortcoming by directly targeting the predictive performance of inverse problem solutions. However, many open challenges and opportunities remain in applying these approaches to complex systems governed by differential equations, which are highlighted in this talk.