Information-bottleneck renormalization group for self-supervised representation learning*

While highly successful, most deep learning applications rely on supervised learning which requires a large set of manually labelled data. But labelled data are not always available, and effective learning from unlabelled datasets - self-supervised learning - has the potential to greatly expand the scope of deep learning applications. Here we propose a self-supervised learning method that combines the concepts of the information bottleneck and the renormalization group. More specifically we use the information bottleneck to regularize a coarse-graining procedure by encouraging a representation to discard locally specific information (bottleneck) while retaining the long-wavelength features (implicitly assumed to be relevant for downstream tasks). We use variational and noise contrastive approaches to scale up our method for large systems, and we demonstrate our implementation on datasets from machine learning.