Deep Learning for Bayesian Optimization of High-Dimensional Scientific Problems*

Bayesian optimization (BO) is a popular algorithm for global optimization of expensive black-box functions (e.g. experiments or derivative-free numerical simulations that are costly or time-consuming), but there are many domains where the function is not completely black-box. For example, the data may have some known structure or symmetries, and the data generation process can yield useful intermediate or auxiliary information. However, the surrogate models typically used in BO, Gaussian Processes (GPs), scale poorly with dataset size and dimensionality and struggle to adapt to specific domains. Here, we propose using a class of deep learning models called Bayesian Neural Networks (BNNs) as the surrogate function, as their representation power and flexibility to handle structured data and exploit auxiliary information enable BO to be applied to complex problems. We demonstrate BO on a number of realistic problems in physics and chemistry, including topology optimization of photonic crystal materials using convolutional neural networks, and chemical property optimization of molecules using graph neural networks. On these complex tasks, we show that BNNs often outperform GPs as surrogate models for BO in terms of sampling efficiency and computational cost.