Natural Language Processing for Material Properties Prediction*

Transformer based models have risen in popularity in natural language processing (NLP) due to its ability to contextualize words by capturing their correlations through the attention mechanism. Recognizing correlations between atoms is crucial to determining material properties especially when long-ranged interactions are present; however, many graph-based neural networks built on node-edge representations either rely on a large cutoff radius or an increased number of layers to accommodate long-distance correlations. In this talk, we introduce a transformer-based neural network model for representing crystalline materials. We explored its capacity for capturing long-range interactions through unsupervised pre-training. We further examined the performance of the pretrained model in some downstream property prediction tasks. Our work paves the way towards more computationally efficient architectures for material predictions.