High Performance Computing application codes often contain many tuning parameters and require extensive computational resources. It is desirable to have a machine leaning framework to automate the parameter optimization process with minimal code executions on the actual HPC systems. Another desideratum is a trustworthy digital twin for a physical system, by which we want to quantify the uncertainties of the simulation model with respect to the physical phenomena. For these optimization and uncertainty quantification purposes, a Bayesian statistical learning framework is a powerful tool that can treat the application as a black-box function and use Gaussian Process regression to compute the mean function and the variance in distribution. To this end, we have been developing a public domain software framework called GPTune. For parameter tuning we implemented a number of learning methods including multi-task learning, transfer learning, multi-objective tuning, and multi-fidelity tuning. For uncertainty quantification, we have been developing novel kernels to make them scientific-domain aware, fast linear algebra algorithms to enable large scale GP with millions of data points, and novel Gaussian Process to handle uncertainties in both input data and modeling. We illustrate the versatility of the GPTune software when it is applied to HPC codes ranging from mathematical libraries to complex simulation codes, as well as large-scale scientific apparatuses and instruments.