The proposed study is the first pilot study that integrates a cloud-based explainable ar-tificial intelligence system using four techniques, namely, (i) Grad-CAM, (ii) Grad-CAM++, (iii) Score-CAM, and (iv) FasterScore-CAM-based lesion localization using three DenseNet models, namely, DenseNet-121, DenseNet-169, and DenseNet-201. Thus, it compares the methods and explainability of the four different CAM strategies for COVID-19-based CT lung lesion localization. DenseNet-121, DenseNet-169, and DenseNet-201 demonstrated an accuracy performance of 98%, 98.5%, and 99%, respectively. The study incorporated a hybrid DL (ResNet-UNet) for COVID-19-based CT lung segmentation using independent cross-validation and performance evaluation schemes. To validate the lesion, three trained senior radiologists scored the lesion localization on the CT lung data set and then compared it against the heatmap generated by cXAI, resulting in the MAI score. Overall, ~80% of CT scans were above an MAI score of four out of five, demonstrating matching lesion locations using cXAI vs. gold standard, thus proving the clinical applicability. Further, the Friedman test was also performed on the MAI scores by comparing the three radiologists.
The online cloud-based COVLIAS 2.0-cXAI achieves (i) CT lung image segmentation and (ii) generation of four CAM techniques in less than 10 s for one CT slice. The COVLIAS 2.0-cXAI demonstrated reliability, high accuracy, and clinical stability.
Author Contributions:Conceptualization, J.S.S., S.A. and N.N.K.; Data curation, G.L.C., A.C., A.P., P.S.C.D., L.S., A.M., G.F., M.T., P.R.K., F.N., Z.R. and K.V.; Formal analysis, J.S.S.; Funding acquisition, M.M.F.; Investigation, J.S.S., I.M.S., P.S.C., A.M.J., N.N.K., S.M., J.R.L., G.P., D.W.S., P.P.S., G.T., A.D.P., D.P.M., V.A., J.S.T., M.A.-M., S.K.D., A.N., A.S., M.F., A.A., F.N., Z.R., M.M.F. and K.V.;
Methodology, J.S.S., S.A., G.L.C. and A.B.; Project administration, J.S.S. and M.K.K.; Software, S.A.
and L.S.; Supervision, J.S.S., L.S., A.M., M.F., M.M.F., S.N. and M.K.K.; Validation, S.A., G.L.C., K.V.
and M.K.K.; Visualization, S.A. and V.R.; Writing—original draft, S.A.; Writing—review & editing, J.S.S., G.L.C., A.C., A.P., P.S.C.D., L.S., A.M., I.M.S., M.T., P.S.C., A.M.J., N.N.K., S.M., J.R.L., G.P., M.M., D.W.S., A.B., P.P.S., G.T., A.D.P., D.P.M., V.A., G.D.K., J.S.T., M.A.-M., S.K.D., A.N., A.S., V.R., M.F., A.A., M.M.F., S.N., K.V. and M.K.K. All authors have read and agreed to the published version of the manuscript.
Funding:This research received no external funding.
Institutional Review Board Statement:The study was conducted in accordance with the Declaration of Helsinki, and approved by: For Italian Dataset: IRB for the retrospective analysis of CT lung in patients affected by COVID-19 granted by the Hospital of Novara to Alessandro Carriero, Co-author of the research you are designing in the artificial intelligence application in the detection and risk stratification of COVID patients. Ethic Committee Name: Assessment of diagnostic performance of Computed Tomography in patients affected by SARS COVID-19 Infection. Approval Code: 131/20.
Approval: authorized by the Azienda Ospedaliero Universitaria Maggiore della Caritàdi Novara on 25 June 2020. For Croatian Dataset: Ethic Committee Name: The use of artificial intelligence for multislice computer tomography (MSCT) images in patients with adult respiratory diseases syndrome and COVID-19 pneumonia. Approval Code: 01-2239-1-2020. Approval: authorized by the University Hospital for Infectious Diseases “Dr. Fran Mihaljevic”, Zegreb, Mirogojska 8. On 9 November 2020. Approved to Klaudija Viskovic.
Informed Consent Statement: Informed Consent was waived because the research involves anonymized records and data sets.
Data Availability Statement:Not available.
Conflicts of Interest:The authors declare no conflict of interest.
Diagnostics2022,12, 1482 33 of 41
Appendix A
Diagnostics 2022, 12, x FOR PEER REVIEW 33 of 41
acquisition, M.M.F.; Investigation, J.S.S., I.M.S., P.S.C., A.M.J., N.N.K., S.M., J.R.L., G.P., D.W.S., P.P.S., G.T., A.D.P., D.P.M., V.A., J.S.T., M.A.-M., S.K.D., A.N., A.S., M.F., A.A., F.N., Z.R., M.M.F.
and K.V.; Methodology, J.S.S., S.A., G.L.C. and A.B.; Project administration, J.S.S. and M.K.K.; Soft-ware, S.A. and L.S.; Supervision, J.S.S., L.S., A.M., M.F., M.M.F., S.N. and M.K.K.; Validation, S.A., G.L.C., K.V. and M.K.K.; Visualization, S.A. and V.R.; Writing—original draft, S.A.; Writing—re-view & editing, J.S.S., G.L.C., A.C., A.P., P.S.C.D., L.S., A.M., I.M.S., M.T., P.S.C., A.M.J., N.N.K., S.M., J.R.L., G.P., M.M., D.W.S., A.B., P.P.S., G.T., A.D.P., D.P.M., V.A., G.D.K., J.S.T., M.A.-M., S.K.D., A.N., A.S., V.R., M.F., A.A., M.M.F., S.N., K.V. and M.K.K.. All authors have read and agreed to the published version of the manuscript.
Funding: This research received no external funding.
Institutional Review Board Statement: The study was conducted in accordance with the Declara-tion of Helsinki, and approved by: For Italian Dataset: IRB for the retrospective analysis of CT lung in patients affected by COVID19 granted by the Hospital of Novara to Alessandro Carriero, Co-author of the research you are designing in the artificial intelligence application in the detection and risk stratification of COVID patients. Ethic Committee Name: Assessment of diagnostic perfor-mance of Computed Tomography in patients affected by SARS COVID 19 Infection. Approval Code:
131/20. Approval: authorized by the Azienda Ospedaliero Universitaria Maggiore della Carità di Novara on 25 June 2020. For Croatian Dataset: Ethic Committee Name: The use of artificial intelli-gence for multislice computer tomography (MSCT) images in patients with adult respiratory dis-eases syndrome and COVID-19 pneumonia. Approval Code: 01-2239-1-2020. Approval: authorized by the University Hospital for Infectious Diseases “Dr. Fran Mihaljevic”, Zegreb, Mirogojska 8. On 9 November 2020. Approved to Klaudija Viskovic.
Informed Consent Statement: Informed Consent was waived because the research involves anon-ymized records and data sets.
Data Availability Statement: Not available.
Conflicts of Interest: The authors declare no conflict of interest.
Appendix A
Figure A1. ResNet-UNet architecture.
Table A1. Friedman test using DenseNet-121 model on the MAI score from three experts.
XAI Experts Min. 25th
Percentile Med 75th
Percentile Max DF-1 DF-2 p Value F
DenseNet-121
Grad-CAM
Expert 1 2 4 5 5 5
2 2278 <0.00001 171.81 Expert 2 3 4 5 5 5
Expert 3 2.7 4.2 4.6 4.8 5 Grad-CAM++
Expert 1 2 4 5 5 5
2 2278 <0.00001 244.9 Expert 2 3 4 5 5 5
Expert 3 2.8 4.3 4.6 4.8 5
Score-CAM Expert 1 1 5 5 5 5 2 2278 <0.00001 740.1 Figure A1.ResNet-UNet architecture.
Table A1.Friedman test using DenseNet-121 model on the MAI score from three experts.
XAI Experts Min. 25th Percentile Med 75th Percentile Max DF-1 DF-2 pValue F
DenseNet-121
Grad-CAM
Expert 1 2 4 5 5 5
2 2278 <0.00001 171.81
Expert 2 3 4 5 5 5
Expert 3 2.7 4.2 4.6 4.8 5
Grad-CAM++
Expert 1 2 4 5 5 5
2 2278 <0.00001 244.9
Expert 2 3 4 5 5 5
Expert 3 2.8 4.3 4.6 4.8 5
Score-CAM
Expert 1 1 5 5 5 5
2 2278 <0.00001 740.1
Expert 2 3 5 5 5 5
Expert 3 2 4.5 4.7 4.9 5
FasterScore-CAM
Expert 1 1 5 5 5 5
2 2278 <0.00001 1072.54
Expert 2 3 5 5 5 5
Expert 3 2.8 4.5 4.7 4.8 5
Min: minimum; Med: median; Max: maximum; F: Friedman statistics.
Table A2.Friedman test using DenseNet-169 model on the MAI score from three experts.
XAI Experts Min. 25th Percentile Med 75th Percentile Max DF-1 DF-2 pValue F
DenseNet-169
Grad-CAM
Expert 1 2 5 5 5 5
2 2278 <0.00001 432.84
Expert 2 3 4 5 5 5
Expert 3 2.7 4.4 4.6 4.8 5
Grad-CAM++
Expert 1 2 5 5 5 5
2 2278 <0.00001 689.05
Expert 2 3 5 5 5 5
Expert 3 3.2 4.5 4.7 4.8 5
Score-CAM
Expert 1 1 4 5 5 5
2 2278 <0.00001 282.56
Expert 2 3 4 5 5 5
Expert 3 2.8 4.5 4.7 4.8 5
FasterScore-CAM
Expert 1 1 4 5 5 5
2 2278 <0.00001 253.15
Expert 2 3 4 5 5 5
Expert 3 2.7 4.4 4.4 4.8 5
Min: minimum; Med: median; Max: maximum; F: Friedman statistics.
Diagnostics2022,12, 1482 34 of 41
Table A3.Friedman test using DenseNet-201 model on the MAI score from three experts.
XAI Experts Min. 25th Percentile Med 75th Percentile Max DF-1 DF-2 pValue F
DenseNet-201
Grad-CAM
Expert 1 2 5 5 5 5
2 2278 <0.00001 499.3
Expert 2 3 5 5 5 5
Expert 3 2.8 4.5 4.7 4.9 5
Grad-CAM++
Expert 1 2 5 5 5 5
2 2278 <0.00001 1151.78
Expert 2 3 5 5 5 5
Expert 3 2.7 4.6 4.7 4.9 5
Score-CAM
Expert 1 3 5 5 5 5
2 2278 <0.00001 1719.93
Expert 2 3 5 5 5 5
Expert 3 3 4.6 4.7 4.9 5
FasterScore-CAM
Expert 1 3 5 5 5 5
2 2278 <0.00001 1239.82
Expert 2 3 5 5 5 5
Expert 3 2.9 4.6 4.7 4.9 5
Min: minimum; Med: median; Max: maximum; F: Friedman statistics.
References
1. Cucinotta, D.; Vanelli, M. WHO Declares COVID-19 a Pandemic.Acta Biomed.2020,91, 157–160. [CrossRef] [PubMed]
2. WHO Coronavirus (COVID-19) Dashboard. Available online:https://covid19.who.int/(accessed on 24 January 2022).
3. Suri, J.S.; Puvvula, A.; Biswas, M.; Majhail, M.; Saba, L.; Faa, G.; Singh, I.M.; Oberleitner, R.; Turk, M.; Chadha, P.S.; et al.
COVID-19 pathways for brain and heart injury in comorbidity patients: A role of medical imaging and artificial intelligence-based COVID severity classification: A review.Comput. Biol. Med.2020,124, 103960. [CrossRef] [PubMed]
4. Cau, R.; Mantini, C.; Monti, L.; Mannelli, L.; Di Dedda, E.; Mahammedi, A.; Nicola, R.; Roubil, J.; Suri, J.S.; Cerrone, G.; et al. Role of imaging in rare COVID-19 vaccine multiorgan complications.Insights Imaging2022,13, 44. [CrossRef] [PubMed]
5. Saba, L.; Gerosa, C.; Fanni, D.; Marongiu, F.; La Nasa, G.; Caocci, G.; Barcellona, D.; Balestrieri, A.; Coghe, F.; Orru, G.; et al.
Molecular pathways triggered by COVID-19 in different organs: ACE2 receptor-expressing cells under attack? A review.Eur. Rev.
Med. Pharm. Sci.2020,24, 12609–12622. [CrossRef]
6. Onnis, C.; Muscogiuri, G.; Paolo Bassareo, P.; Cau, R.; Mannelli, L.; Cadeddu, C.; Suri, J.S.; Cerrone, G.; Gerosa, C.; Sironi, S.;
et al. Non-invasive coronary imaging in patients with COVID-19: A narrative review.Eur. J. Radiol.2022,149, 110188. [CrossRef]
[PubMed]
7. Viswanathan, V.; Puvvula, A.; Jamthikar, A.D.; Saba, L.; Johri, A.M.; Kotsis, V.; Khanna, N.N.; Dhanjil, S.K.; Majhail, M.; Misra, D.P. Bidirectional link between diabetes mellitus and coronavirus disease 2019 leading to cardiovascular disease: A narrative review.World J. Diabetes2021,12, 215. [CrossRef]
8. Fanni, D.; Saba, L.; Demontis, R.; Gerosa, C.; Chighine, A.; Nioi, M.; Suri, J.S.; Ravarino, A.; Cau, F.; Barcellona, D.; et al.
Vaccine-induced severe thrombotic thrombocytopenia following COVID-19 vaccination: A report of an autoptic case and review of the literature.Eur. Rev. Med. Pharm. Sci.2021,25, 5063–5069. [CrossRef]
9. Gerosa, C.; Faa, G.; Fanni, D.; Manchia, M.; Suri, J.; Ravarino, A.; Barcellona, D.; Pichiri, G.; Coni, P.; Congiu, T. Fetal programming of COVID-19: May the barker hypothesis explain the susceptibility of a subset of young adults to develop severe disease?Eur.
Rev. Med. Pharmacol. Sci.2021,25, 5876–5884.
10. Cau, R.; Pacielli, A.; Fatemeh, H.; Vaudano, P.; Arru, C.; Crivelli, P.; Stranieri, G.; Suri, J.S.; Mannelli, L.; Conti, M.; et al.
Complications in COVID-19 patients: Characteristics of pulmonary embolism.Clin. Imaging2021,77, 244–249. [CrossRef]
11. Kampfer, N.A.; Naldi, A.; Bragazzi, N.L.; Fassbender, K.; Lesmeister, M.; Lochner, P. Reorganizing stroke and neurological intensive care during the COVID-19 pandemic in Germany.Acta Biomed.2021,92, e2021266. [CrossRef]
12. Congiu, T.; Demontis, R.; Cau, F.; Piras, M.; Fanni, D.; Gerosa, C.; Botta, C.; Scano, A.; Chighine, A.; Faedda, E. Scanning electron microscopy of lung disease due to COVID-19-a case report and a review of the literature.Eur. Rev. Med. Pharmacol. Sci.2021,25, 7997–8003. [PubMed]
13. Pan, F.; Ye, T.; Sun, P.; Gui, S.; Liang, B.; Li, L.; Zheng, D.; Wang, J.; Hesketh, R.L.; Yang, L. Time course of lung changes on chest CT during recovery from 2019 novel coronavirus (COVID-19) pneumonia.Radiology2020,13, 200370.
14. Wong, H.Y.F.; Lam, H.Y.S.; Fong, A.H.; Leung, S.T.; Chin, T.W.; Lo, C.S.Y.; Lui, M.M.; Lee, J.C.Y.; Chiu, K.W.; Chung, T.W.; et al.
Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19.Radiology2020,296, E72–E78.
[CrossRef] [PubMed]
15. Smith, M.J.; Hayward, S.A.; Innes, S.M.; Miller, A.S.C. Point-of-care lung ultrasound in patients with COVID-19—A narrative review.Anaesthesia2020,75, 1096–1104. [CrossRef] [PubMed]
16. Tian, S.; Hu, W.; Niu, L.; Liu, H.; Xu, H.; Xiao, S.Y. Pulmonary Pathology of Early-Phase 2019 Novel Coronavirus (COVID-19) Pneumonia in Two Patients With Lung Cancer.J. Thorac. Oncol.2020,15, 700–704. [CrossRef] [PubMed]
17. Ackermann, M.; Verleden, S.E.; Kuehnel, M.; Haverich, A.; Welte, T.; Laenger, F.; Vanstapel, A.; Werlein, C.; Stark, H.; Tzankov, A.; et al. Pulmonary Vascular Endothelialitis, Thrombosis, and Angiogenesis in COVID-19.N. Engl. J. Med.2020,383, 120–128.
[CrossRef]
Diagnostics2022,12, 1482 35 of 41
18. Aigner, C.; Dittmer, U.; Kamler, M.; Collaud, S.; Taube, C. COVID-19 in a lung transplant recipient.J. Heart Lung Transpl.2020,39, 610–611. [CrossRef]
19. Suri, J.S.; Rangayyan, R.M.Recent Advances in Breast Imaging, Mammography, and Computer-Aided Diagnosis of Breast Cancer; SPIE:
Bellingham, WA, USA, 2006.
20. Peng, Q.Y.; Wang, X.T.; Zhang, L.N.; Chinese Critical Care Ultrasound Study, G. Findings of lung ultrasonography of novel corona virus pneumonia during the 2019–2020 epidemic.Intensive Care Med.2020,46, 849–850. [CrossRef]
21. Nillmani; Jain, P.K.; Sharma, N.; Kalra, M.K.; Viskovic, K.; Saba, L.; Suri, J.S. Four Types of Multiclass Frameworks for Pneumonia Classification and Its Validation in X-ray Scans Using Seven Types of Deep Learning Artificial Intelligence Models.Diagnostics 2022,12, 652. [CrossRef]
22. Suri, J.S.; Agarwal, S.; Gupta, S.K.; Puvvula, A.; Biswas, M.; Saba, L.; Bit, A.; Tandel, G.S.; Agarwal, M.; Patrick, A. A narrative review on characterization of acute respiratory distress syndrome in COVID-19-infected lungs using artificial intelligence.Comput.
Biol. Med.2021,130, 104210. [CrossRef]
23. Saba, L.; Biswas, M.; Kuppili, V.; Cuadrado Godia, E.; Suri, H.S.; Edla, D.R.; Omerzu, T.; Laird, J.R.; Khanna, N.N.; Mavrogeni, S.;
et al. The present and future of deep learning in radiology.Eur. J. Radiol.2019,114, 14–24. [CrossRef] [PubMed]
24. Kuppili, V.; Biswas, M.; Sreekumar, A.; Suri, H.S.; Saba, L.; Edla, D.R.; Marinhoe, R.T.; Sanches, J.M.; Suri, J.S. Extreme learning machine framework for risk stratification of fatty liver disease using ultrasound tissue characterization.J. Med. Syst.2017,41, 152.
[CrossRef] [PubMed]
25. Noor, N.M.; Than, J.C.; Rijal, O.M.; Kassim, R.M.; Yunus, A.; Zeki, A.A.; Anzidei, M.; Saba, L.; Suri, J.S. Automatic lung segmentation using control feedback system: Morphology and texture paradigm.J. Med. Syst.2015,39, 22. [CrossRef] [PubMed]
26. Suri, J.S.; Agarwal, S.; Pathak, R.; Ketireddy, V.; Columbu, M.; Saba, L.; Gupta, S.K.; Faa, G.; Singh, I.M.; Turk, M. COVLIAS 1.0:
Lung Segmentation in COVID-19 Computed Tomography Scans Using Hybrid Deep Learning Artificial Intelligence Models.
Diagnostics2021,11, 1405. [CrossRef]
27. Suri, J.S.; Agarwal, S.; Carriero, A.; Pasche, A.; Danna, P.S.C.; Columbu, M.; Saba, L.; Viskovic, K.; Mehmedovic, A.; Agarwal, S.; et al. COVLIAS 1.0 vs. MedSeg: Artificial Intelligence-Based Comparative Study for Automated COVID-19 Computed Tomography Lung Segmentation in Italian and Croatian Cohorts.Diagnostics2021,11, 2367. [CrossRef]
28. Saba, L.; Agarwal, M.; Patrick, A.; Puvvula, A.; Gupta, S.K.; Carriero, A.; Laird, J.R.; Kitas, G.D.; Johri, A.M.; Balestrieri, A.; et al.
Six artificial intelligence paradigms for tissue characterisation and classification of non-COVID-19 pneumonia against COVID-19 pneumonia in computed tomography lungs.Int. J. Comput. Assist. Radiol. Surg.2021,16, 423–434. [CrossRef]
29. Acharya, R.U.; Faust, O.; Alvin, A.P.; Sree, S.V.; Molinari, F.; Saba, L.; Nicolaides, A.; Suri, J.S. Symptomatic vs. asymptomatic plaque classification in carotid ultrasound.J. Med. Syst.2012,36, 1861–1871. [CrossRef]
30. Acharya, U.R.; Faust, O.; Alvin, A.P.C.; Krishnamurthi, G.; Seabra, J.C.; Sanches, J.; Suri, J.S. Understanding symptomatology of atherosclerotic plaque by image-based tissue characterization.Comput. Methods Programs Biomed.2013,110, 66–75. [CrossRef]
31. Acharya, U.R.; Faust, O.; Sree, S.V.; Alvin, A.P.C.; Krishnamurthi, G.; Sanches, J.; Suri, J.S. Atheromatic™: Symptomatic vs.
asymptomatic classification of carotid ultrasound plaque using a combination of HOS, DWT & texture. In Proceedings of the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA, 30 August–3 September 2011; pp. 4489–4492.
32. Acharya, U.R.; Mookiah, M.R.; Vinitha Sree, S.; Afonso, D.; Sanches, J.; Shafique, S.; Nicolaides, A.; Pedro, L.M.; Fernandes, E.F.J.;
Suri, J.S. Atherosclerotic plaque tissue characterization in 2D ultrasound longitudinal carotid scans for automated classification:
A paradigm for stroke risk assessment.Med. Biol. Eng. Comput.2013,51, 513–523. [CrossRef]
33. Molinari, F.; Liboni, W.; Pavanelli, E.; Giustetto, P.; Badalamenti, S.; Suri, J.S. Accurate and automatic carotid plaque characteriza-tion in contrast enhanced 2-D ultrasound images. In Proceedings of the 2007 29th Annual Internacharacteriza-tional Conference of the IEEE Engineering in Medicine and Biology Society, Lyon, France, 22–26 August 2007; pp. 335–338.
34. Banchhor, S.K.; Londhe, N.D.; Araki, T.; Saba, L.; Radeva, P.; Laird, J.R.; Suri, J.S. Wall-based measurement features provides an improved IVUS coronary artery risk assessment when fused with plaque texture-based features during machine learning paradigm.Comput. Biol. Med.2017,91, 198–212. [CrossRef]
35. Saba, L.; Sanagala, S.S.; Gupta, S.K.; Koppula, V.K.; Johri, A.M.; Khanna, N.N.; Mavrogeni, S.; Laird, J.R.; Pareek, G.; Miner, M.;
et al. Multimodality carotid plaque tissue characterization and classification in the artificial intelligence paradigm: A narrative review for stroke application.Ann. Transl. Med.2021,9, 1206. [CrossRef] [PubMed]
36. Agarwal, M.; Saba, L.; Gupta, S.K.; Johri, A.M.; Khanna, N.N.; Mavrogeni, S.; Laird, J.R.; Pareek, G.; Miner, M.; Sfikakis, P.P.
Wilson disease tissue classification and characterization using seven artificial intelligence models embedded with 3D optimization paradigm on a weak training brain magnetic resonance imaging datasets: A supercomputer application.Med. Biol. Eng. Comput.
2021,59, 511–533. [CrossRef] [PubMed]
37. Acharya, U.R.; Kannathal, N.; Ng, E.; Min, L.C.; Suri, J.S. Computer-based classification of eye diseases. In Proceedings of the 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, New York, NY, USA, 30 August–3 September 2006; pp. 6121–6124.
38. Acharya, U.; Vinitha Sree, S.; Mookiah, M.; Yantri, R.; Molinari, F.; Ziele´znik, W.; Małyszek-Tumidajewicz, J.; St˛epie ´n, B.; Bardales, R.; Witkowska, A. Diagnosis of Hashimoto’s thyroiditis in ultrasound using tissue characterization and pixel classification.Proc.
Inst. Mech. Eng. Part H J. Eng. Med.2013,227, 788–798. [CrossRef] [PubMed]
Diagnostics2022,12, 1482 36 of 41
39. Biswas, M.; Kuppili, V.; Edla, D.R.; Suri, H.S.; Saba, L.; Marinhoe, R.T.; Sanches, J.M.; Suri, J.S. Symtosis: A liver ultrasound tissue characterization and risk stratification in optimized deep learning paradigm.Comput. Methods Programs Biomed.2018,155, 165–177. [CrossRef] [PubMed]
40. Acharya, U.R.; Saba, L.; Molinari, F.; Guerriero, S.; Suri, J.S. Ovarian tumor characterization and classification: A class of GyneScan™ systems. In Proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA, USA, 28 August–1 September 2012; pp. 4446–4449.
41. Pareek, G.; Acharya, U.R.; Sree, S.V.; Swapna, G.; Yantri, R.; Martis, R.J.; Saba, L.; Krishnamurthi, G.; Mallarini, G.; El-Baz, A.
Prostate tissue characterization/classification in 144 patient population using wavelet and higher order spectra features from transrectal ultrasound images.Technol. Cancer Res. Treat.2013,12, 545–557. [CrossRef]
42. Shrivastava, V.K.; Londhe, N.D.; Sonawane, R.S.; Suri, J.S. Exploring the color feature power for psoriasis risk stratification and classification: A data mining paradigm.Comput. Biol. Med.2015,65, 54–68. [CrossRef]
43. Shrivastava, V.K.; Londhe, N.D.; Sonawane, R.S.; Suri, J.S. A novel and robust Bayesian approach for segmentation of psoriasis lesions and its risk stratification.Comput. Methods Programs Biomed.2017,150, 9–22. [CrossRef] [PubMed]
44. Shrivastava, V.K.; Londhe, N.D.; Sonawane, R.S.; Suri, J.S. Reliable and accurate psoriasis disease classification in dermatology images using comprehensive feature space in machine learning paradigm.Expert Syst. Appl.2015,42, 6184–6195. [CrossRef]
45. Bayraktaroglu, S.; Cinkooglu, A.; Ceylan, N.; Savas, R. The novel coronavirus pneumonia (COVID-19): A pictorial review of chest CT features.Diagn. Interv. Radiol.2021,27, 188–194. [CrossRef]
46. Verschakelen, J.A.; De Wever, W.Computed Tomography of the Lung; Springer: Berlin/Heidelberg, Germany, 2007.
47. Arrieta, A.B.; Díaz-Rodríguez, N.; Del Ser, J.; Bennetot, A.; Tabik, S.; Barbado, A.; García, S.; Gil-López, S.; Molina, D.; Benjamins, R. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.Inf. Fusion 2020,58, 82–115. [CrossRef]
48. Choi, T.Y.; Chang, M.Y.; Heo, S.; Jang, J.Y. Explainable machine learning model to predict refeeding hypophosphatemia.Clin.
Nutr. ESPEN2021,45, 213–219. [CrossRef] [PubMed]
49. Gunashekar, D.D.; Bielak, L.; Hagele, L.; Oerther, B.; Benndorf, M.; Grosu, A.L.; Brox, T.; Zamboglou, C.; Bock, M. Explainable AI for CNN-based prostate tumor segmentation in multi-parametric MRI correlated to whole mount histopathology.Radiat. Oncol.
2022,17, 65. [CrossRef] [PubMed]
50. Gunning, D.; Aha, D. DARPA’s explainable artificial intelligence (XAI) program.AI Mag.2019,40, 44–58.
51. Sabih, M.; Hannig, F.; Teich, J. Utilizing explainable AI for quantization and pruning of deep neural networks. arXiv2020, arXiv:2008.09072.
52. Apley, D.W.; Zhu, J. Visualizing the effects of predictor variables in black box supervised learning models.J. R. Stat. Soc. Ser. B (Stat. Methodol.)2020,82, 1059–1086. [CrossRef]
53. DenOtter, T.D.; Schubert, J. Hounsfield Unit. InStatPearls; StatPearls Publishing LLC.: Treasure Island, FL, USA, 2022.
54. Adedigba, A.P.; Adeshina, S.A.; Aina, O.E.; Aibinu, A.M. Optimal hyperparameter selection of deep learning models for COVID-19 chest X-ray classification.Intell.-Based Med.2021,5, 100034. [CrossRef]
55. Chhabra, M.; Kumar, R. A Smart Healthcare System Based on Classifier DenseNet 121 Model to Detect Multiple Diseases. In Mobile Radio Communications and 5G Networks; Springer: Berlin/Heidelberg, Germany, 2022; pp. 297–312.
56. Hasan, N.; Bao, Y.; Shawon, A.; Huang, Y. DenseNet convolutional neural networks application for predicting COVID-19 using CT image.SN Comput. Sci.2021,2, 389. [CrossRef]
57. Nandhini, S.; Ashokkumar, K. An automatic plant leaf disease identification using DenseNet-121 architecture with a mutation-based henry gas solubility optimization algorithm.Neural Comput. Appl.2022,34, 5513–5534. [CrossRef]
58. Ruiz, J.; Mahmud, M.; Modasshir, M.; Shamim Kaiser, M.; for the Alzheimer’s Disease Neuroimaging Initiative. 3D DenseNet en-semble in 4-way classification of Alzheimer’s disease. InInternational Conference on Brain Informatics; Springer: Berlin/Heidelberg, Germany, 2020; pp. 85–96.
59. Jiang, H.; Xu, J.; Shi, R.; Yang, K.; Zhang, D.; Gao, M.; Ma, H.; Qian, W. A multi-label deep learning model with interpretable Grad-CAM for diabetic retinopathy classification. In Proceedings of the 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Montreal, QC, Canada, 20–24 July 2020; pp. 1560–1563.
60. Joo, H.-T.; Kim, K.-J. Visualization of deep reinforcement learning using grad-CAM: How AI plays atari games? In Proceedings of the 2019 IEEE Conference on Games (CoG), London, UK, 20–23 August 2019; pp. 1–2.
61. Panwar, H.; Gupta, P.; Siddiqui, M.K.; Morales-Menendez, R.; Bhardwaj, P.; Singh, V. A deep learning and grad-CAM based color visualization approach for fast detection of COVID-19 cases using chest X-ray and CT-Scan images.Chaos Solitons Fractals2020, 140, 110190. [CrossRef]
62. Zhang, Y.; Hong, D.; McClement, D.; Oladosu, O.; Pridham, G.; Slaney, G. Grad-CAM helps interpret the deep learning models trained to classify multiple sclerosis types using clinical brain magnetic resonance imaging.J. Neurosci. Methods2021,353, 109098.
[CrossRef]
63. Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 618–626.
Diagnostics2022,12, 1482 37 of 41
64. Chattopadhay, A.; Sarkar, A.; Howlader, P.; Balasubramanian, V.N. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In Proceedings of the 2018 IEEE winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018; pp. 839–847.
65. Inbaraj, X.A.; Villavicencio, C.; Macrohon, J.J.; Jeng, J.-H.; Hsieh, J.-G. Object Identification and Localization Using Grad-CAM++
with Mask Regional Convolution Neural Network.Electronics2021,10, 1541. [CrossRef]
66. Joshua, E.S.N.; Chakkravarthy, M.; Bhattacharyya, D. Lung cancer detection using improvised grad-cam++ with 3d cnn class activation. InSmart Technologies in Data Science and Communication; Springer: Berlin/Heidelberg, Germany, 2021; pp. 55–69.
67. Joshua, N.; Stephen, E.; Bhattacharyya, D.; Chakkravarthy, M.; Kim, H.-J. Lung Cancer Classification Using Squeeze and Excitation Convolutional Neural Networks with Grad Cam++ Class Activation Function.Traitement Signal2021,38, 1103–1112. [CrossRef]
68. Wang, H.; Wang, Z.; Du, M.; Yang, F.; Zhang, Z.; Ding, S.; Mardziel, P.; Hu, X. Score-CAM: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA, 14–19 June 2020; pp. 24–25.
69. Wang, H.; Du, M.; Yang, F.; Zhang, Z. Score-Cam: Improved Visual Explanations via Score-Weighted Class Activation Mapping.
arXiv2019, arXiv:1910.01279.
70. Naidu, R.; Ghosh, A.; Maurya, Y.; Kundu, S.S. IS-CAM: Integrated Score-CAM for axiomatic-based explanations.arXiv2020, arXiv:2010.03023.
71. Oh, Y.; Jung, H.; Park, J.; Kim, M.S. Evet: Enhancing visual explanations of deep neural networks using image transformations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Virtual, 5–9 January 2021; pp. 3579–3587.
72. Sugeno, A.; Ishikawa, Y.; Ohshima, T.; Muramatsu, R. Simple methods for the lesion detection and severity grading of diabetic retinopathy by image processing and transfer learning.Comput. Biol. Med.2021,137, 104795. [CrossRef]
73. Cozzi, D.; Cavigli, E.; Moroni, C.; Smorchkova, O.; Zantonelli, G.; Pradella, S.; Miele, V. Ground-glass opacity (GGO): A review of the differential diagnosis in the era of COVID-19.Jpn. J. Radiol.2021,39, 721–732. [CrossRef]
74. Wu, J.; Pan, J.; Teng, D.; Xu, X.; Feng, J.; Chen, Y.C. Interpretation of CT signs of 2019 novel coronavirus (COVID-19) pneumonia.
Eur. Radiol.2020,30, 5455–5462. [CrossRef]
75. De Wever, W.; Meersschaert, J.; Coolen, J.; Verbeken, E.; Verschakelen, J.A. The crazy-paving pattern: A radiological-pathological correlation.Insights Imaging2011,2, 117–132. [CrossRef]
76. Niu, R.; Ye, S.; Li, Y.; Ma, H.; Xie, X.; Hu, S.; Huang, X.; Ou, Y.; Chen, J. Chest CT features associated with the clinical characteristics of patients with COVID-19 pneumonia.Ann. Med.2021,53, 169–180. [CrossRef]
77. Salehi, S.; Abedi, A.; Balakrishnan, S.; Gholamrezanezhad, A. Coronavirus Disease 2019 (COVID-19): A Systematic Review of Imaging Findings in 919 Patients.AJR Am. J. Roentgenol.2020,215, 87–93. [CrossRef]
78. Xie, X.; Zhong, Z.; Zhao, W.; Zheng, C.; Wang, F.; Liu, J. Chest CT for Typical Coronavirus Disease 2019 (COVID-19) Pneumonia:
Relationship to Negative RT-PCR Testing.Radiology2020,296, E41–E45. [CrossRef] [PubMed]
79. Cau, R.; Falaschi, Z.; Pasche, A.; Danna, P.; Arioli, R.; Arru, C.D.; Zagaria, D.; Tricca, S.; Suri, J.S.; Karla, M.K.; et al. Computed tomography findings of COVID-19 pneumonia in Intensive Care Unit-patients.J. Public Health Res.2021,10, 2270. [CrossRef]
[PubMed]
80. Yang, X.; He, X.; Zhao, J.; Zhang, Y.; Zhang, S.; Xie, P. COVID-CT-dataset: A CT scan dataset about COVID-19. arXiv2020, arXiv:2003.13865.
81. Shalbaf, A.; Vafaeezadeh, M. Automated detection of COVID-19 using ensemble of transfer learning with deep convolutional neural network based on CT scans.Int. J. Comput. Assist. Radiol. Surg.2021,16, 115–123.
82. Gozes, O.; Frid-Adar, M.; Greenspan, H.; Browning, P.D.; Zhang, H.; Ji, W.; Bernheim, A.; Siegel, E. Rapid ai development cycle for the coronavirus (COVID-19) pandemic: Initial results for automated detection & patient monitoring using deep learning ct image analysis.arXiv2020, arXiv:2003.05037.
83. Solano-Rojas, B.; Villalón-Fonseca, R.; Marín-Raventós, G. Alzheimer’s disease early detection using a low cost three-dimensional densenet-121 architecture. In Proceedings of the International Conference on Smart Homes and Health Telematics, Hammamet, Tunisia, 24–26 June 2020; pp. 3–15.
84. Saba, L.; Suri, J.S.Multi-Detector CT Imaging: Principles, Head, Neck, and Vascular Systems; CRC Press: Boca Raton, FL, USA, 2013;
Volume 1.
85. Murgia, A.; Erta, M.; Suri, J.S.; Gupta, A.; Wintermark, M.; Saba, L. CT imaging features of carotid artery plaque vulnerability.
Ann. Transl. Med.2020,8, 1261. [CrossRef]
86. Saba, L.; di Martino, M.; Siotto, P.; Anzidei, M.; Argiolas, G.M.; Porcu, M.; Suri, J.S.; Wintermark, M. Radiation dose and image quality of computed tomography of the supra-aortic arteries: A comparison between single-source and dual-source CT Scanners.
J. Neuroradiol.2018,45, 136–141. [CrossRef]
87. Bustin, S.A.; Benes, V.; Nolan, T.; Pfaffl, M.W. Quantitative real-time RT-PCR—A perspective.J. Mol. Endocrinol.2005,34, 597–601.
[CrossRef]
88. Dramé, M.; Teguo, M.T.; Proye, E.; Hequet, F.; Hentzien, M.; Kanagaratnam, L.; Godaert, L. Should RT-PCR be considered a gold standard in the diagnosis of COVID-19?J. Med. Virol.2020,92, 2312–2313. [CrossRef]
89. Gibson, U.E.; Heid, C.A.; Williams, P.M. A novel method for real time quantitative RT-PCR.Genome Res. 1996,6, 995–1001.
[CrossRef]
Diagnostics2022,12, 1482 38 of 41
90. Jain, P.K.; Sharma, N.; Giannopoulos, A.A.; Saba, L.; Nicolaides, A.; Suri, J.S. Hybrid deep learning segmentation models for atherosclerotic plaque in internal carotid artery B-mode ultrasound.Comput. Biol. Med.2021,136, 104721. [CrossRef] [PubMed]
91. Jena, B.; Saxena, S.; Nayak, G.K.; Saba, L.; Sharma, N.; Suri, J.S. Artificial intelligence-based hybrid deep learning models for image classification: The first narrative review.Comput. Biol. Med.2021,137, 104803. [CrossRef] [PubMed]
92. Jain, P.K.; Sharma, N.; Saba, L.; Paraskevas, K.I.; Kalra, M.K.; Johri, A.; Nicolaides, A.N.; Suri, J.S. Automated deep learning-based paradigm for high-risk plaque detection in B-mode common carotid ultrasound scans: An asymptomatic Japanese cohort study.
Int. Angiol.2022,41, 9–23. [CrossRef] [PubMed]
93. Suri, J.S.; Agarwal, S.; Elavarthi, P.; Pathak, R.; Ketireddy, V.; Columbu, M.; Saba, L.; Gupta, S.K.; Faa, G.; Singh, I.M.; et al. Inter-Variability Study of COVLIAS 1.0: Hybrid Deep Learning Models for COVID-19 Lung Segmentation in Computed Tomography.
Diagnostics2021,11, 2025. [CrossRef] [PubMed]
94. Skandha, S.S.; Nicolaides, A.; Gupta, S.K.; Koppula, V.K.; Saba, L.; Johri, A.M.; Kalra, M.S.; Suri, J.S. A hybrid deep learning paradigm for carotid plaque tissue characterization and its validation in multicenter cohorts using a supercomputer framework.
Comput. Biol. Med.2022,141, 105131. [CrossRef]
95. Gupta, N.; Gupta, S.K.; Pathak, R.K.; Jain, V.; Rashidi, P.; Suri, J.S. Human activity recognition in artificial intelligence framework:
A narrative review.Artif. Intell. Rev.2022, 1–54. [CrossRef]
96. Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 5–9 October 2015; pp. 234–241.
97. Badrinarayanan, V.; Kendall, A.; Cipolla, R. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation.
IEEE Trans. Pattern Anal. Mach. Intell.2017,39, 2481–2495. [CrossRef]
98. Mateen, M.; Wen, J.; Song, S.; Huang, Z. Fundus image classification using VGG-19 architecture with PCA and SVD.Symmetry 2018,11, 1. [CrossRef]
99. Wen, L.; Li, X.; Li, X.; Gao, L. A new transfer learning based on VGG-19 network for fault diagnosis. In Proceedings of the 2019 IEEE 23rd International Conference on Computer Supported Cooperative Work in Design (CSCWD), Porto, Portugal, 6–8 May 2019; pp. 205–209.
100. Xiao, J.; Wang, J.; Cao, S.; Li, B. Application of a novel and improved VGG-19 network in the detection of workers wearing masks.
J. Phys. Conf. Ser.2020,1518, 012041. [CrossRef]
101. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778.
102. Zhou, T.; Ye, X.; Lu, H.; Zheng, X.; Qiu, S.; Liu, Y. Dense Convolutional Network and Its Application in Medical Image Analysis.
Biomed. Res. Int.2022,2022, 2384830. [CrossRef]
103. De Boer, P.-T.; Kroese, D.P.; Mannor, S.; Rubinstein, R.Y. A tutorial on the cross-entropy method.Ann. Oper. Res.2005,134, 19–67.
[CrossRef]
104. Jamin, A.; Humeau-Heurtier, A. (Multiscale) Cross-Entropy Methods: A Review.Entropy2019,22, 45. [CrossRef] [PubMed]
105. Shore, J.; Johnson, R. Properties of cross-entropy minimization.IEEE Trans. Inf. Theory1981,27, 472–482. [CrossRef]
106. Browne, M.W. Cross-Validation Methods.J. Math. Psychol.2000,44, 108–132. [CrossRef] [PubMed]
107. Refaeilzadeh, P.; Tang, L.; Liu, H. Cross-validation.Encycl. Database Syst.2009,5, 532–538.
108. Powers, D.M. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation.arXiv2020, arXiv:2010.16061.
109. Juba, B.; Le, H.S. Precision-recall versus accuracy and the role of large data sets. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 27 January–1 February 2019; pp. 4039–4048.
110. Yacouby, R.; Axman, D. Probabilistic extension of precision, recall, and f1 score for more thorough evaluation of classification models. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, Punta Cana, Dominican Republic, 16 March 2020; pp. 79–91.
111. Dewitte, K.; Fierens, C.; Stockl, D.; Thienpont, L.M. Application of the Bland-Altman plot for interpretation of method-comparison studies: A critical investigation of its practice.Clin. Chem.2002,48, 799–801, author reply 801–792. [CrossRef]
112. Giavarina, D. Understanding Bland Altman analysis.Biochem. Med.2015,25, 141–151. [CrossRef]
113. Asuero, A.G.; Sayago, A.; Gonzalez, A. The correlation coefficient: An overview.Crit. Rev. Anal. Chem.2006,36, 41–59. [CrossRef]
114. Taylor, R. Interpretation of the correlation coefficient: A basic review.J. Diagn. Med. Sonogr.1990,6, 35–39. [CrossRef]
115. Eelbode, T.; Bertels, J.; Berman, M.; Vandermeulen, D.; Maes, F.; Bisschops, R.; Blaschko, M.B. Optimization for medical image segmentation: Theory and practice when evaluating with dice score or jaccard index. IEEE Trans. Med. Imaging2020,39, 3679–3690. [CrossRef] [PubMed]
116. Zimmerman, D.W.; Zumbo, B.D. Relative power of the Wilcoxon test, the Friedman test, and repeated-measures ANOVA on ranks.J. Exp. Educ.1993,62, 75–86. [CrossRef]
117. Schemper, M. A generalized Friedman test for data defined by intervals.Biom. J.1984,26, 305–308. [CrossRef]
118. Ishitaki, T.; Oda, T.; Barolli, L. A neural network based user identification for Tor networks: Data analysis using Friedman test.
In Proceedings of the 2016 30th International Conference on Advanced Information Networking and Applications Workshops (WAINA), Crans-Montana, Switzerland, 23–25 March 2016; pp. 7–13.
119. Hayes, B. Cloud computing.Commun. ACM2008,51, 9–11. [CrossRef]