A pipeline for robustly measuring social behavior using deep learning*

Understanding social behavior requires tracking and quantifying animals’ movements as they engage in social interactions. Recently, there has been notable progress in developing deep learning algorithms to track multiple animals in social paradigms. Although these methods perform well when the animals are some distance apart or have brief close contact with each other, they exhibit poor tracking accuracy when the animals spend more time with each other, performing behaviors such as huddling, mutual-grooming, and mating. To improve the tracking of animals within social contexts, where the animals are in close proximity for long periods of time, we implemented a pipeline that combines multiple deep-learning-based tracking methods to obtain detailed and high-accuracy postural trajectories of multiple animals. Tested on a data set of prairie voles - a model organism for the study of social interactions - our pipeline robustly maintains animal identities and greatly increases the accuracy of the posture tracking over applying convolutional neural network methods by themselves. With this improved tracking accuracy, we are able to isolate behaviors that are key for understanding the dynamics of social interactions.