A Deep Learning Network for Disease Classification with Longitudinal Data*

Deep learning approaches to medical image analysis have achieved success in modeling disease progression. However, their application to longitudinal analysis is low and most fail to leverage time-relevant patient data. This work aims to develop a deep learning model to predict disease status and investigate the influence of longitudinal data on model performance. We train a 2-class convolutional neural network (CNN) both with and without a cascaded recurrent neural network (RNN) to investigate the impact of longitudinal features on Alzheimer’s Disease (AD) classification. We use a dataset of 1,260 ¹⁸F-FDG PET scans containing brain images from 87 normal (NC) patients, 30 patients with stable mild cognitive impairment (sMCI), 81 with progressive MCI (pMCI), and 65 with AD across 3 timepoints with 1 year gaps. Performance is also evaluated using a similar dataset of 822 MRI scans. The CNN+RNN model shows improvement over the CNN alone for both (AD vs NC) and (sMCI vs pMCI) classification. Thus adding longitudinal data leads to better identification of the stages of AD than single-timepoint data alone. Saliency maps were also analyzed to explore the most important regions for the network’s decisions.