Prediction of visual impairment in retinitis pigmentosa using deep learning and multimodal fundus images

**Background:** Retinitis pigmentosa (RP) is a common inherited retinal dystrophy with many active clinical trials, but screening is hampered by variability in visual function measures like visual acuity (VA) and visual fields. Confocal scanning laser ophthalmoscopy (cSLO) imaging, including optical coherence tomography (OCT) and infrared (IR) reflectance, offers objective structural biomarkers. However, selecting a single surrogate for VA is challenging due to multiple coexisting retinal changes. This study aimed to develop a deep learning (DL) model to predict visual impairment (VA <20/40) from cSLO images in RP.

**Methods:** This retrospective study used two independent datasets. The Johns Hopkins University (JHU) dataset (2569 images, 462 eyes, 231 patients) was used for 10-fold cross-validation and internal testing. The Amsterdam University Medical Centers (AUMC) dataset (349 images, 166 eyes, 83 patients, all with confirmed genetic mutations) was used for external testing. Inclusion criteria for JHU were clinical RP diagnosis; exclusion criteria were significant media opacities and off-center images. Snellen corrected VA and cSLO images (Spectralis, Heidelberg Engineering) were collected. Three versions of the ResNet-152 neural network were trained via transfer learning: IR only, OCT only, and combined image (CI, containing both IR and OCT). The binary classification task distinguished Snellen VA 20/40 or better from worse than 20/40. Performance was assessed by area under the curve (AUC), precision, and recall.

**Key Results:** In internal testing (JHU), AUCs were 0.83 (IR), 0.87 (OCT), and 0.85 (CI). In external testing (AUMC), AUCs were 0.78 (IR), 0.87 (OCT), and 0.85 (CI). For OCT, precision and recall were 0.76 and 0.76 (internal) and 0.79 and 0.79 (external). The OCT model correctly classified 71% of eyes with serial images (all images from an eye correct). Among 52 images with structural abnormalities (e.g., cystoid macular edema, macular hole), 52% were misclassified by the OCT network. Gradient-based class activation maps showed strong activation near the fovea and ellipsoid zone, indicating biologically meaningful learning.

**Clinical Implications:** This study demonstrates that a DL algorithm can predict visual impairment from OCT images in RP with good accuracy, using a functionally meaningful VA cut-off of 20/40. The algorithm can detect inter-eye and temporal VA differences, suggesting potential for clinical trial screening and monitoring. The OCT-only model performed best, and combining IR did not improve results. Limitations include retrospective design, lack of standardized VA measurement, and reduced performance with concurrent structural abnormalities. Further research should include additional imaging modalities, larger datasets, and regression models for exact VA prediction.