Multimodal Super Resolution with Dual Domain Loss and Gradient Guidance

Spatial resolution plays a crucial role in quantitative assessment of various structures in brain MRI. Super resolution (SR) as a post-processing tool holds promise for restoring the high frequency details lost in a low resolution (LR) acquisition with no additional scan time. Prior multicontrast deep learning SR approaches are mostly in 2D and operate in a pre-upsampling or progressive setting. Here we propose an efficient shallow 3D projection based post-upsampling network for anisotropic SR of brain MRI. The network is optimized using losses in the spatial and frequency domains and a complementary high resolution (HR) input to inform SR of the low resolution (LR) input with tighter integration of features. We investigated the benefit of different feature aggregation strategies such as concatenation and multiplicative attention and gradient guidance from the HR target or the additional HR input. The models were trained and evaluated on diverse datasets and performed comparably with MINet, another recently developed multimodal SR model, with approximately half the number of model parameters. The model generalized well to an external test set; performed satisfactorily on acquired LR MRI volumes despite the LR input being simulated from HR volumes during training and resulted in lower high frequency error norm. From the ablation studies, we note that a multimodal network noticeably improves SR compared to a unimodal network and feature aggregation using concatenation and multiplicative attention performed equally well. We also highlight the leakage of information from the complementary HR input to the SR output volume and the limited value of PSNR and SSIM as evaluation metrics in such cases.