Abstract

Purpose: The structural similarity index measure (SSIM) has become a popular quality metric to evaluate QSM in a way that is closer to human perception than RMS error (RMSE). However, SSIM may overpenalize errors in diamagnetic tissues and underpenalize them in paramagnetic tissues, resulting in biasing. In addition, extreme artifacts may compress the dynamic range, resulting in unrealistically high SSIM scores (hacking). To overcome biasing and hacking, we propose XSIM: SSIM implemented in the native QSM range, and with internal parameters optimized for QSM. Methods: We used forward simulations from a COSMOS ground-truth brain susceptibility map included in the 2016 QSM Reconstruction Challenge to investigate the effect of QSM reconstruction errors on the SSIM, XSIM, and RMSE metrics. We also used these metrics to optimize QSM reconstructions of the in vivo challenge data set. We repeated this experiment with the QSM abdominal phantom. To validate the use of XSIM instead of SSIM for QSM quality assessment across a range of different reconstruction techniques/algorithms, we analyzed the reconstructions submitted to the 2019 QSM Reconstruction Challenge 2.0. Results: Our experiments confirmed the biasing and hacking effects on the SSIM metric applied to QSM. The XSIM metric was robust to those effects, penalizing the presence of streaking artifacts and reconstruction errors. Using XSIM to optimize QSM reconstruction regularization weights returned less overregularization than SSIM and RMSE. Conclusion: XSIM is recommended over traditional SSIM to evaluate QSM reconstructions against a known ground truth, as it avoids biasing and hacking effects and provides a larger dynamic range of scores.