A Review On Multimodal Medical Image Fusion Compendious Analysis

Computers in Biology and Medicine 144 (2022) 105253
Contents lists available at ScienceDirect
Computers in Biology and Medicine

journal homepage: www.elsevier.com/locate/compbiomed
A review on multimodal medical image fusion: Compendious analysis of

medical modalities, multimodal databases, fusion techniques and
quality metrics
Muhammad Adeel Azam a, b, Khan Bahadar Khan c, Sana Salahuddin d, Eid Rehman e,
Sajid Ali Khan e, Muhammad Attique Khan f, Seifedine Kadry g, Amir H. Gandomi h, *
a
Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genova, Italy
b
Department of Informatics, Bioengineering, Robotics, and System Engineering, University of Genoa, Italy
c
Department of Information and Communication Engineering, Faculty of Engineering, The Islamia University of Bahawalpur, Pakistan
d
Institute of Physics, The Islamia University of Bahawalpur, Bahawalpur, Pakistan
e
Department of Software Engineering, Foundation University Islamabad, 44000, Pakistan
f
Department of Computer Science, HITEC University Taxila, Taxila, Pakistan
g
Department of Applied Data Science, Noroff University College, Kristiansand, Norway
h
Faculty of Engineering and IT, University of Technology Sydney, Ultimo, NSW, 2007, Australia
A R T I C L E I N F O A B S T R A C T
Keywords: Background and objectives: Over the past two decades, medical imaging has been extensively apply to diagnose
Multimodal medical image fusion diseases. Medical experts continue to have difficulties for diagnosing diseases with a single modality owing to a
Multimodal databases lack of information in this domain. Image fusion may be use to merge images of specific organs with diseases
Fusion techniques
from a variety of medical imaging systems. Anatomical and physiological data may be included in multi-modality
Image fusion quality metrics
image fusion, making diagnosis simpler. It is a difficult challenge to find the best multimodal medical database
with fusion quality evaluation for assessing recommended image fusion methods. As a result, this article provides
a complete overview of multimodal medical image fusion methodologies, databases, and quality measurements.
Methods: In this article, a compendious review of different medical imaging modalities and evaluation of related
multimodal databases along with the statistical results is provided. The medical imaging modalities are orga
nized based on radiation, visible-light imaging, microscopy, and multimodal imaging.
Results: The medical imaging acquisition is categorized into invasive or non-invasive techniques. The fusion
techniques are classified into six main categories: frequency fusion, spatial fusion, decision-level fusion, deep
learning, hybrid fusion, and sparse representation fusion. In addition, the associated diseases for each modality
and fusion approach presented. The quality assessments fusion metrics are also encapsulated in this article.
Conclusions: This survey provides a baseline guideline to medical experts in this technical domain that may
combine preoperative, intraoperative, and postoperative imaging, Multi-sensor fusion for disease detection, etc.
The advantages and drawbacks of the current literature are discussed, and future insights are provided
accordingly.
1. Introduction medical images from various modalities to make a new fused image with
rich information that is reliable for clinical usage. Multimodal Medical
For more than two decades, different medical imaging modalities Image Fusion (MMIF) utilizes images from different sources like X-Rays,
have been used in clinical applications for disease diagnostic purposes. Computed Tomography (CT), Single Photon Emission Computed To
Generally, it is difficult to extract all required information from a single mography (SPECT), Ultrasound (US), Magnetic Resonance Imaging
imaging modality to guarantee clinical precision and strength of the (MRI), Infrared and Ultraviolet, Positron Emission Tomography (PET),
examination for diagnoses. Therefore, multimodal methods combine etc. Images from MRI, X-ray, CT, and US can all show where the lesion is
* Corresponding author.
E-mail address: gandomi@uts.edu.au (A.H. Gandomi).
https://doi.org/10.1016/j.compbiomed.2022.105253
Received 31 December 2021; Accepted 20 January 2022
Available online 3 February 2022
0010-4825/© 2022 Elsevier Ltd. All rights reserved.
M.A. Azam et al. Computers in Biology and Medicine 144 (2022) 105253
located, how large it is, and what it looks like, as well as the morpho IM (Imaging Modalities); It describes the presence of multimodal
logical and structural changes it has induced in nearby tissues. To get medical imaging modalities used in fusion.
insight into a tumor’s biological processes, soft tissue as well as func Quant. A (Quantitative Analysis); whether the survey articles pre
tional information, the use of PET, fMRI, and SPECT is becoming more sented quantitative comparison on different fusion metrics for
common. Functional and structural data of the medical images can be various techniques.
combined to produce more valuable information. Medical image fusion Qual. A (Qualitative Analysis); whether the survey articles presented
plays a critical part in the treatment of the same human organ, allowing qualitative comparison on different fusion imaging modalities for
for more accurate disease monitoring and analysis [1]. Image fusion various techniques.
techniques are widely applicable in the following domains, including D.D (Database Description); It shows whether the review articles
remote sensing, machine learning, satellite surveillance, contrast encapsulated multimodal databases with comprehensive de
enhancement of images, boosting geometric adjustment, and medical scriptions that use in medical fusion.
imaging to significantly improve features that have not been observable T.D (Techniques Description); It represents whether review articles
using a single image, such as malignancies, lesions, cancer cells, etc. [2]. have any theoretical technique description for MMIF.
The growing number of research articles available in magazines, books, D.F (disease-based Fusion); the articles included diseased-based
and journals demonstrates the high interest and importance of multi fusion work in their contribution or not.
modal image fusion. Fig. 1 displays the number of publications in the F. B. S (Fusion Basic steps); whether the articles included general
area of multimodal image fusion per year. The results were obtained steps and rules to perform medical image fusion before going toward
from PubMed, an online database for biomedical subjects [3]. advanced techniques.
The multimodal medical image fusion techniques and classifications M. D. (Metrics Description); is there a proper description of MMIF
have been summarized in various surveys and review articles [2–6]. For performance metrics in survey articles.
instance, Alex et al. [2] cover the various scientific issues addressed in
the area of medical image fusion. Define medical image fusion research We present a review of multimodal medical imaging modalities by
by the techniques, imaging modalities, and organs examined, but the overcoming all the shortcomings of the previous works, thoroughly
multimodal databases and fusion quality assessments metrics were still discussing such modalities, freely accessible multimodal databases,
missing in this survey. Jiao et al. [4] explained multimodal medical classification of medical image fusion techniques, and associated dis
image fusion steps, e.g. image fusion decomposition, and reconstruction, eases. The MMIF approaches are classified into six domains: spatial,
and fusion rules, in detail, comparing six fusion methods and using eight frequency, decision-level fusion type, deep learning, hybrid, and sparse
image fusion quality metrics. The work focuses exclusively on Harvard representation fusion. These five MMIF methods are compared with
medical (AANLIB), but there is still a need to examine more multimodal each other based on fusion quality performance metric results. In
databases for a greater variety of datasets. Additionally, the author did addition, some recent multimodal image fusion articles of different
not address current diseases involved with MMIF in their research. disease detection are also summarized. The arrangement of this review
Fatma et al. [5] introduced the classification of medical image regis article is highlighted in Fig. 2.
tration and highlighted medical image fusion and the current diseases The major contributions of this paper are as follows:
based on fusion work. Bikash et al. [6] provided a detailed comparison
of region-based image fusion techniques using different fusion quality • Classification of medical imaging modalities based on the electro
metrics. The images used in this review were from different sources, magnetic spectrum and invasive/non-invasive methods are shown in
such as multimodal medical images, infrared rays, and visible image Fig. 3 and Table 2, respectively. The categorization based on the
fusion, multi-focus image fusion, etc. The MMIF quality metrics are also energy source used for image acquisition is presented in Fig. 4.
discussed in tabular form. • The detailed comparisons of all online public accessible multimodal
Table 1 summarizes a recent literature review and compares it to our medical databases are presented in Table 3. The frequency distri
current study. We highlight many critical points that must be included in bution of the usage of each database over the last five years is shown
each MMIF evaluation. The table below is based on the following points:
Fig. 1. Multimodal image fusion publications per year obtained from PubMed (1993 to third-quarter year Q3, 2021).
2
Table 1
A recent study of the literature on MMIF, as well as our work.
Ref. Year Contributions I. Quant. Qual. D. T. D. F. B. M.
M A A D D F S D
Dolly JM et al. 2019 Classification: (Feature, Decision, and Pixel level). ✓ ✓

[7] Pros and cons of each technique described.
Huang B. et al. 2020 Classification: (Spatial, Transform, and DL). ✓ ✓ ✓ ✓
[8] Comparatively analysis of MRI/CT and PET/MRI.
NSCT and CNN framework presented.
Tirupal T. et al. 2020 Classification: (Frequency, Spatial and Pixel level). ✓ ✓ ✓ ✓ ✓ ✓
[9] Imaging modalities brief description.
Both Qualitative and statistical evaluations were done.
Hermessi H et al. 2021 Classification: (Feature, Decision, and Pixel level). ✓ ✓ ✓ ✓ ✓ ✓
[10] Well-renowned fusion techniques architecture illustrated.
CT/MRI, MRI/PET, MRI/SPECT visual and statistical comparison.
Comprehensive fusion metrics description.
Techniques summary with limitations.
Sebastian J et al. 2021 Classification: (Spatial, Transform, and DL) ✓ ✓
[11] Advantages and drawbacks of each technique.
All articles included in this review are related to brain imaging.
Diwakar M et al. 2021 A comprehensive review of SWT and DWT techniques. ✓ ✓ ✓ ✓ ✓
[6] A fair comparison of visual and quantitative on Harvard medical dataset.
The few most common fusion metrics are described.
Zhang H et al. 2021 The entire article focuses on DL techniques (CNN, AE, and GAN). ✓ ✓ ✓ ✓
[12] MRI/PET visual results with standard statistical results were presented.
Our work 2022 Classification: (frequency fusion, spatial fusion, decision-level fusion, deep ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
learning, hybrid fusion, and sparse representation).
Five MMIF databases description.
Comprehensive fusion metrics.
Disease-based fusion summarization.
Qualitative and quantitative results on various techniques.
Fig. 2. Arrangement of sections encapsulated in this article.
3
Fig. 3. Biomedical image modalities w.r.t. electromagnetic spectrum.
2. Medical imaging modalities
In the medical field, each imaging modality has unique information

and characteristics. The different medical imaging modalities used for
screening and diagnoses of different diseases of the human body lie in
the entire electromagnetic (EM) spectrum range, as shown in Fig. 3.
Each imaging modality has a different wavelength and frequency and
also shows different characteristics [13]. When EM waves strike an
object, they are scattered, reflected, or absorbed by the object. MRIs
generate a magnetic field that induces the protons in the body to align.
Protons are activated and spin out of phase with the magnetic field when
a radiofrequency current is applied to the patient. The modalities of
X-ray and CT images are highly radiative and harmful to the human
body due to their high-frequency range from 3 × 1016 to 3 × 1019 Hz.
Nuclear imaging modalities, such as PET and SPECT, use gamma rays for
diagnosing functional activity in the organs of a human body. Gamma
rays have a frequency greater than 1019 Hz and wavelength less than 10
Fig. 4. Biomedical image modalities w.r.t source of energy. picometers. X-rays, CT, SPECT, and PET are based on the principle of
ionization, while MRI and US are based on non-ionization [14]. Some
in Fig. 6, and the articles compared in this review using each modern medical imaging is also used nowadays for endoscopic imaging
mentioned database are summarized in Fig. 6 (b). such a While light and Narrowband imaging [15].
• The overall general and basic procedure of medical image fusion In addition, the image acquisition process of each modality is
techniques are presented in Figs. 11 and 12. different. Some use an internal source of energy for image acquisition,
• The classification of medical image fusion techniques based on fre some employ an external source, while others utilize a combined in
quency fusion, spatial fusion, decision-level fusion, deep learning, ternal and external energy source. The classification of different imaging
hybrid fusion, and sparse representation fusion is shown in Fig. 13. modalities based on the energy source used for image acquisition is
• Comparison of visual image results obtained using various MMIF shown in Fig. 4.
approaches implemented in this is shown in Fig. 14. Another classification of medical imaging modality is based on
• The taxonomy of existing works based on different diseases is sum invasive and non-invasive methods, as shown in Table 2. Invasive
marized in Table 6. methods refer to the examination of an organ by inserting any object
• All the possible MMIF quality assessment metrics are explained in into the body via an incision or needle injection, whereas non-invasive
Table 7. methods utilize some kind of radiation or sound waves. The major
• The medical image fusion statistical results of the six different classification of medical imaging modalities is shown in Fig. 5.
methods using one large database, i.e., AANLIB, are tabulated in
Table 8.
2.1. Radiology
Section 2 contains a full discussion of medical modalities. Section 3

For diagnosis and treatment purposes, this type of medical imaging
discusses open access multimodal medical databases. Section 4 discusses
uses ionization and radio waves, gamma rays, and sound waves. X-rays,
the method of medical image fusion and its categorization discussed in
computed tomography (CT), and magnetic resonance imaging (MRI) all
Section 5. Section 6 illustrates the diseases related to each technique.
involve radiation (ionization and radio waves), whereas the US uses
The fusion quality assessments metrics are explained in Section 7, while
sound waves and PET and SPECT use gamma rays for diagnostic reasons.
the review article is concluded in Section 8.
X-ray images, first introduced in 1895, are the oldest imaging
4
Table 2 of protons. Hence, more detailed images consist of many sequences of

Comparison of the most prominent MMIF imaging modalities. slices. Compared to CT images, MRI images contain higher soft-tissue
Medical Abbreviations Invasive/ Characteristics information but require a longer acquisition time. MRI images are also
Modalities Non- used to examine blood vessels, brain and breast tumors, abnormal tis
Invasive sues, spinal injuries, etc. [17]. MRI is usually non-invasive but some
MRI Magnetic Non- MRI images show great times it may be invasive depending on the application and requirement
Resonance Image Invasive details about soft and of imaging, with references in many articles MRI is considered as
abnormal tissues in the brain non-invasive so we classified it as a non-invasive group.
and blood flow. MRI can
identify a small stroke,
Functional MRI (fMRI) is a type of MRI that is used for acquiring
lesion, etc., but cannot functional information of the body, such as brain activity, blood
provide information about oxygenation level, etc. [18]. The US modality utilizes sound waves and
activity in the brain or is used to analyze tumors, identify cancer, perform prostate biopsies,
cancer cell information [17,
and observe the movement of a pre-birth baby in the fetus [2,12].
18].
CT Computed Non- CT images show skull, bone Angiography is an invasive technique, in which a thin long tube is
Tomography Invasive structure, large inserted into an artery via a small incision to reach the targeted area.
abnormalities, stroke, After a contrast agent is injected into this tube, a series of X-ray images
bleeding, and lesions in the are obtained to detect the vessels and examine blood flow in arteries of
brain, but do not provide
great detail about soft tissues
the heart, brain, eye, and hand. SPECT and PET are also invasive tech
[17]. niques, in which radioactive isotopes are injected into a patient’s body
X-Rays _ Non- X-rays are used to diagnose to obtain functional information for diagnosing cancer in the brain or
Invasive the anatomic structure of a liver, analyzing blood flow in soft tissues, and detecting brain and
human body. Normal,
pancreatic tumors [2,12,17].
abnormal, and fractured
bones can easily be seen
from X-rays images [16]. 2.2. Visible light imaging
US Ultrasound Non- The US works on sound
Invasive waves of about 20 kHz to In visible-light imaging, patients are subjected to visible light source
several GHz frequency
ranges. It is low cost and
rays instead of invisible rays, from which color or gray-scale images are
safer compared to X-Rays produced. Dermatology and endoscopy medical images are two impor
and CT images. The US is tant examples of visible light imaging modalities [20]. The dermatology
applicable on both modality is applicable on the skin by using light rays to evaluate skin
anatomical and functional
diseases, such as skin allergies, lesions, infections, and bacterial dis
body organs [2,19].
PET Positron Emission Invasive PET scans reveal activity in eases. The endoscopy modality also uses visible light to examine the
Tomography the brain, as well as sugar internal structure of a human body. These types of images are useful for
activity as energy in the pre-surgery, during surgery, and post-surgery diagnoses. A small optical
brain. They also give camera is usually attached to an endoscopic tube to acquire medical
information about cancer
cells in the brain using
images. Ophthalmology is an application of visible light imaging to
different colors [17,19]. study the structure, function, and diseases of the eyes [21]. Another
SPECT Single Photon Invasive SPECT scans indicate the organ modality is otorhinolaryngology, which is the study of the nose,
Emission Computed most active area of the brain throat, ears, and associated structures of the neck and head.
Tomography by analyzing blood flow.
SPECT can detect diseases
like dementia, seizures, 2.3. Microscopy imaging
clogged blood vessels, etc.
[17,19]. In microscopy modalities, the medical information of small-scale
Endoscopy _ Invasive The internal structure of the biological objects that cannot be obtained from ordinary other modal
human body can be seen
from endoscopy using a
ities is collected [22]. The two major types of microscopes are electron
small camera. The main microscopy and light (optical) microscopy. In an electron microscope,
application of endoscopy is which includes scanning and transmission electron microscopes, the
to examine the internal diagnostic object is exposed to a beam of electrons. In a light (optical)
structure of the stomach
microscope, light rays are passed through the microscope’s lens to
[20].
Microcopy _ Non- Light (optical) microscopy magnify the object and obtain diagnostic medical images. Phase
Invasive and electron microscopy are Contrast Microscopy (PCM), Bright Light Microscopy (BLM), fluores
used to examine biological cence microscopy, and Dark Field Microscopy (DFM) all employ optical
information at the micro microscopes.
level [22].
2.4. Multimodal imaging

modality used in the medical field. The X-ray modality is used to di
agnose the anatomical structure of broken bones [16]. Although CT Individual imaging modalities can only collect limited medical in
images fall under the X-ray imaging domain, X-rays use radiation in one formation. Specifically, anatomical and rigid structure information can
direction, whereas CT rays are emitted into a patient’s body from only be acquired from CT and X-rays images, while structural with some
different angles and distances. CT images are presented in the form of functional information can be collected from MRI images. Moreover,
cross-section areas of an organ and are comprised of many 2D slices of SPECT and fMRI images reveal only detailed functional information.
images, from which 3D CT images can be obtained [17]. CT images are Medical experts often face difficulties in diagnosing diseases when using
used to diagnose bone fractures, locations of tumors, cancer, cardiac a single imaging modality due to the limitation of information. There
tissues, pulmonary embolism, etc. MRI medical images employ a strong fore, the multimodal imaging modalities concept has been introduced,
magnetic field inside the human body that induces different orientations which fuses more than one modality with another via a fusion algorithm
to achieve a detailed resultant image. Popular combinations of
5
Fig. 5. Classification of medical imaging modalities.
multimodal modalities include CT-MRI, MRI-SPECT, MRI-PET, X-ray Dicom image format [25], different imaging modalities, and multiple
and US, MRI-US, among others [23]. The comparison between the most body scanned areas. TCIA contains a total of 22 modalities, in which
common imaging modalities is described in Table 2, indicating the most images are from mammography (MG), followed by CT than MRI
characteristics and invasive/non-invasive classification of each images. This database also provides images from a total of 52 anatomical
modality. body organs, such as the breast, chest, brain, and colon. Harvard Med
ical School provides the WholeBrain Atlas (AANLIB) brain image data
3. Multimodal image databases set, this dataset is online publicly accessible [26–28], AANLIB dataset is
mainly categorized into normal and diseased based brain images.
In this section, we discuss some important multimodal medical Normal brain images are in 2D or 3D, while diseased images are further
datasets that are a crucial initial step of any multimodal medical image classified into sub-brain diseases, including brain stroke and tumors,
fusion technique, particularly for testing and diagnoses. Although many degenerative and infectious diseases, and many other brain-associated
datasets are freely available online for experimental purposes, re diseases. All the images in this database are in GIF file format and
searchers need medical images from different modalities of the same and easy to use. The AANLIB database focuses on brain images and contains
different patients for validation of the fusion algorithm. These images MRI, CT, PET, and SPECT imaging modalities. The “Alzheimer’s Disease
are mostly acquired within the same periods, while other images are Neuroimaging Initiative” (ADNI) also provides Alzheimer’s brain im
taken at different periods for better diagnoses and to evaluate disease ages from three modalities: CT, MRI, and PET. The MIDAS medical
progression/recession. database contains many medical modalities images of various body or
Herein, five free online multimodal databases, namely OASIS, TCIA, gans in different image formats [29]. In MIDAS, the RIRE project
BrainWeb Atlas (AANLIB), ADNI, and MIDAS are discussed in this article designed in 2010 focused only on multimodal medical image processing
and these databases contained thousands of images with diversity of [24].
imaging modalities. These datasets are useful for researchers to analyze The PubMed medical database, an online biomedical search engine
their image fusion algorithms. The overall comparison of these data containing more than 30 million biomedical and life sciences citations,
bases is given in Table 3. These databases were selected because they: 1) was used for gathering statistical results about these datasets. Herein,
are freely available, 2) contain different modalities of images of the same results from the past five years were collected from PubMed. The fre
and different patients, and 3) contain many scanned body organs with quency distribution of multimodal fusion articles using these five med
different diseases and also in different image formats. ical datasets over the previous five years is given in Fig. 6 (a). We
The “Open Access Series of Imaging Studies” (OASIS) database collected only publications that were related to these five public open-
mainly contains medical brain images and three projects, i.e. OASIS-1, access databases; moreover, we used a subfilter search in PubMed to
OAIS-2, and OASIS-3. The OASIS-1 project includes MRI-T1 and T2 identify the most often occurring organ diseases in the human body that
images of a total of 416 subjects, both men and women, ranging 18–96 would be treated with MMIF. However, because these subfilter searches
years of age [24]. The OASIS-1 dataset also contains information about provide comprehensive information, we were able to eliminate a large
Alzheimer’s disease (AD) and 20 nondemented diseased subjects. The number of publications, as mentioned before in Fig. 1.
Cancer Imaging Archive (TCIA) mostly includes medical images in The following keywords were used to search PubMed for the results
6
Table 3 • For the ADNI database, keywords (medical image fusion) AND (ADNI
Multimodal database comparisons for the application of MMIF. OR http://adni.loni.usc.edu/) were used.
Dataset Year Modalities Body Images Format • For the TCIA database, keywords (medical image fusion) AND (tcia
Organ(s) Categories OR cancer imaging archive OR https://www.cancerimagingarchive.
OASIS 2010 MRI and PET Brain OASIS-3 Nifti net/) were searched.
(2000 MR • For the MIDAS database, keywords (medical image fusion) AND
sessions and (RIRE OR https://www.insight-journal.org/) were utilized.
PET three
different
tracers).
The frequency distribution chart of the multimodal databases of the
OASIS-2 (373 articles used in this review is shown in Fig. 6 (b). Furthermore, the ar
MRI sessions ticles in the multimodal image fusion domain examined different body
with organs for clinical diagnoses of specific human organs. The fusion arti
nondemented
cles obtained from PubMed based on different body organs, namely
and
Alzheimer’s Brain, Lungs, Eye/Retina, and Cardiac, are also categorized in Fig. 6 (c).
diseases) Again, the PubMed medical database was utilized along with keywords
OASIS-1 (434 (Multimodal medical image fusion AND (“organ name” OR dataset). It
MRI sessions can be observed that most articles contain information on the Brain,
with
Alzheimer’s
while only 37 articles are on the Eye/Retina.
diseases). Fig. 7 displays some sample images from the Harvard medical
TCIA 2014 Mammography, Brain, Contains a Dicom database [30], and Fig. 8 shows healthy MRI and MRI images of de
X-rays, US, CT, Chest, total of 22 mentia and Alzheimer’s diseases from the OASIS database [24]. The
MRI, PET, Lungs, modalities
sample brain images from the MIDAS database [24] are displayed in
SPECT Breast, and 52
Abdomen, anatomical Fig. 9, and those collected from ADNI [31] are provided in Fig. 10.
Kidney, scanned
Heart, images. 4. Fusion steps
Neck, etc. Multiple
diseases of
human
The multimodal image fusion method is a procedure that integrates
organs. many images from one or various imaging modalities to increase accu
BrainWeb 1995 CT, MRI, PET, Brain Contains GIF racy and quality while preserving the complementary information of the
Atlas SPECT normal and images [32]. Medical image fusion mainly concerns MRI, PET, CT, and
disease-based
SPECT [2]. PET and SPECT modalities deliver images with functional
images (brain
strokes, information of the body, such as details about metabolism, soft tissue
tumors, movement, and blood flow, despite having low spatial resolution. MRI,
degenerative, CT, and US provide high spatial resolution images, lending anatomical
infectious information about the body. Multimodal images are usually obtained by
diseases, etc.)
and also
merging functional images with structural images to yield better infor
normal 3D mation for health specialists to diagnose clinical diseases.
brain images. Medical image registration is used to geometrically align two images,
ADNI 2003 MRI, FMRI, PET Brain Contains MRI - then the image fusion technique is applied by overlapping the two input
and PET
source images to produce a resultant image with excess and comple
imaging
modalities of mentary information [19]. Two points must be satisfied during the
Alzheimer’s image fusion step: 1) all the appropriate medical information that exists
disease. in the input images must be present in the resultant image, and 2) the
MIDAS 2010 CT, MRI, SPECT, Heart, MRI cardiac Dicom fused image should not have any extra information that did not exist in
PET, US Brain, phantom scan Raw,
Bones, images, CT Mhd,
the input images. Fusion can be applied on multi-sensor images obtained
Head, bones, 3D US hdr from a different source of imaging modalities, multi-focus images that
Liver, etc. phantom, CT are usually taken from the same modality, and multimodal images
scan of the widely used in medicine. The MMIF procedure, its various steps, and the
abdominal
multimodal fusion classification and methods are described in this
phantom,
Brain MRI, section.
CT, PET, and In the multimodal fusion process, initially, the researcher selects the
SPECT. body organ of interest. Then, two or more imaging modalities selected to
fuse using the appropriate fusion algorithm. To validate the fusion al
gorithm, performance metrics are required [34]. In the final step, the
of the five mentioned databases over the last five years.
resultant fused image contains more information about the scanned area
of the body organ than the input images. The overall MMIF procedure
• For the OASIS database, keywords (medical image fusion) AND
shown in Fig. 11.
(OASIS OR https://www.oasis-brains.org/) were used, from which
In the initial stage, the input source image is registered by mapping it
18 research articles were extracted.
with the reference image to relate and match the equivalent images
• For the AANLIB database, keywords (medical image fusion) AND
depending on specific features for the image fusion procedure [35]. In
(AANLIB OR Harvard medical OR http://www.med.harvard.edu/
the image decomposition stage, the input images are decomposed into
AANLIB/home.html) were used. It can be observed that most of
sub-images and fusion coefficients using the fusion algorithm. Then,
the articles used this database because it contains a variety of mo
fusion rules are applied to extract important information and multiple
dalities and is easy to use. Most articles on medical image fusion were
features of sub-images that help in the subsequent steps. Finally, the
obtained from this database compared to the other four.
fused image is reconstructed by combining the sub-images using an in
verse algorithm known as image reconstruction [36].
7
Fig. 6. PubMed database statistical results of MMIF (a) Distribution of usage frequency of multimodal databases w.r.t. medical image fusion of recent five years; (b)
Frequency distribution of multimodal databases used in this study and (c) Number of multimodal fusion articles from PubMed w.r.t. human organs in the last
five years.
As seen, the image fusion method consists of various simple stages further characterize into transform fusion techniques and pyramidal
that help to accomplish the objective. Fig. 12 demonstrates the principle fusion methods [32]. Pyramidal techniques are utilized for
steps associated with the image fusion methodology. multi-resolution analysis, while transform fusion methods include
wavelet decomposition, Contourlet (CT), and Curvelet Transform fusion,
5. MMIF techniques classification etc. [33,34].
In the feature-based techniques, the complementary features are
Although there are several image fusion approaches, we concen extracts from input medical images, such as different regions, edges,
trated on six dimensions: frequency fusion, spatial fusion, decision-level dimensions, images segments, and shapes [28]. Many researchers have
fusion, deep learning, hybrid fusion, and sparse representation fusion. demonstrated that the fusion of medical images based on a combination
We begin by providing an overview of pixel-level and feature-level of regions or objects of images gives more significant fusion results over
image fusion to aid in the comprehension of broad categorization. the fusion of images based on pixel-level [56]. The feature level-based
In pixel-level-based fusion methods, images are combined straight methods are divided into region-based and machine learning-based
forwardly utilizing singular pixels to make the fusion decision [27,37]. It fusion. Neural Network, PCNN, K-means and Fuzzy clustering are the
further classified into the frequency domain (section 5.1) and spatial common machine learning techniques. In region-based techniques, the
domain (section 5.2). Spatial domain techniques utilize a basic input images are segmented into sub-images, and then features are
pixel-level strategy, i.e. functioning on the pixel level of images. Since it determined from these segments or regions.
is implement on the initial images, the resulting images exhibit less
spatial distortion and a lower signal-to-noise ratio (SNR). The methods 5.1. Frequency fusion methods
under this domain are Intensity Hue Saturation (IHS), Simple maximum,
Independent Component Analysis (ICA), simple minimum, Principal In these methods, the input images are first converted into the fre
Component Analysis (PCA), weighted average and simple average quency domain by computing Fourier transform (FT), then the fusion
method, etc. In the frequency domain method, the images are first algorithm is applied on the transformed image followed by inverse FT to
convert into the frequency domain via Fourier transform. Then, the obtain the resultant fused image [34]. These techniques are further
fusion process is implement on the frequency quantities, followed by characterized into pyramidal and transform techniques [38]. The py
inverse transformation to obtain the final fused image. This method is ramidal fusion methods include Gaussian, differentiation, Laplacian and
8
Fig. 7. Harvard medical dataset (open-access public database) of normal brain 3D images of slice-60 [30,32]: (a) MR-T1, (b) MR-T2, and (c) PET brain images of
Trans-axial, Sagittal, and Coronal directions (left to right).
Fig. 8. MRI images from OASIS dataset (open-access public database) [24]: (a) non-demented aging disease, (b) high-grade mild dementia, (c) mild dementia disease,
(d) less extreme dementia disease, (e) Alzheimer’s Disease (AD), and (f) healthy control (HC).
morphological pyramid, Filter Subtract Decimate Pyramid (FSD), and methods [41].
slope pyramid [39]. The transform fusion methods pertain to wavelet Qu et al. [42] provided a fusion rule for finding the wavelet modulus
decomposition, CT and Curvelet transform. Moreover, wavelet decom maxima of input images with varying bandwidths and levels. To quan
position techniques includes Discrete Wavelet Transform (DWT), tify the fusion impact, a metric based on MI measurements is computed.
Dual-Tree Complex Wavelet (DT-CWT), Lifting Wavelet (LWT), and The researchers selected CT and MRI brain input images for fusion. The
Redundant Discrete Wavelet (RDWT) methods. Examples of contourlet benefit of this method is the preservation of both components and edge
transform fusion methods are Dual-Tree Complex Contourlet Transform feature information in the fused image. However, this technique failed
(DT-CCT), and Non-Subsampled Contourlet (NSCT) [40]. Additionally, to meet the requirements of shift invariance. This problem was resolved
ridgelet, curvelet, and bandlet are types of curvelet transform fusion by RDWT [43], which demonstrated improved performance over DWT
9
Fig. 9. From MIDAS dataset (open access public database): (a) CT images, (b) T1-weighted MRI images, (c) T2-weighted MRI images, and (d) PET images [24,33].
Fig. 10. From ADNI dataset (open-access public database): (a) raw MRI coronal image, (b) axial diffusion tensor image (DTI), (c) raw axial DTI, and (d) post-processed
MRI image [31].
due to mutual information dependent on non-linear registration and Liu et al. [45] recommended a multimodal image fusion technique
entropy image information. Moustafa et al. [44] introduced four based on Multi-wavelet Transform (MWT) and used PET and CT chest
multimodal image fusion methods, namely Laplacian Pyramid, images for fusion. Yang et al. [46] suggested a new fusion technique
Multi-focus, Wavelet Transform, and Computationally Efficient based on Contourlet transform (CT), where the resultant fused image
Pixel-level Image Fusion (CEMIF), which were applied on liver CT and was acquired by taking the inverse CT of the high pass sub-bands.
MRI images for detection of hepatic lesions. Although this technique demonstrated improved localization,
10
Fig. 11. Overall multimodal medical fusion procedure.
Fig. 12. Medical image fusion step-by-step process.
anisotropy, and multiscale characteristic, it could not give significant multiscale geometric analysis of 3D ST using Contextual Hidden Markov
improvement than other multiscale analysis techniques like wavelet. Model (CHMM). Firstly, 3D ST is used to decompose the input images
Three groups of CT and MRI brain images were fused based on Shearlet into low and high-frequency components. Then, the max fusion rule
transform (ST), which yielded the best sparse directional image since the with local energy is applied on low-frequency sub-bands, and an effec
ST method has no restrictions on the number of orientations for sharing. tive multi-feature fusion rule is implemented for integrating components
Xue-jun et al. [47] suggested another multimodal image fusion method of high-frequency sub-bands. 3D ST provides better decomposition
based on LWT to decompose the input image based on high-low fre performance compared to the 2D MST tools for volumetric medical
quency weight and by various fusion rules. The input CT and MRI of images. Ming et al. [51] presented a new method based on NSST, which
human brain images were considered for fusion. Bhatnagar et al. [48] was used for the decomposition of source medical images into
presented a multimodal image fusion method based on NSCT. The multi-scale, while a “parameter-adaptive pulse-coupled neural network”
medical images were initially converted by adding low-frequency (PA-PCNN) was used to determine high-frequency band in images. An
components with high-frequency components. Two different fusion energy preservation approach applied to determine the low-frequency
rules depending on directive contrast and phase congruency were band of medical images. The resultant fused image was constructed
implemented to fuse high and low-frequency coefficients. At last, the using the inverse NSST technique and four multimodal medical modal
resultant fused image was obtained by taking inverse NSCT. Sahu et al. ities: CT, MRI, PET, and SPECT. Arif et al. [52] suggested a multimodal
[49] proposed a method using Laplacian Pyramid (LP) and DCT, image fusion method based on CT and the Genetic Algorithm (GA). GA is
wherein LP decomposes the input image into a distinct low pass image, utilized to compute the uncertainties and rambling existing in the image
similar to a pyramidal structure. The quality of fused images increased and to optimize the fusion procedure. The methodology verified on
when the pyramidal level increased, leading to improved edges and brain images using MRI, MRA, PET, and SPECT modalities. All the
information. This method also yielded superior fused results in terms of above-cited techniques are summarized in Table 4.
quantitative and qualitative analysis compared to the Daubechies
complex wavelet transform (DCxWT).
5.2. Spatial fusion methods
Xi et al. [50] introduced a Multimodal Medical Volumetric Image
Fusion (MVIF) method, which incorporates many features acquired via
Spatial fusion methods based on the pixels of images, where pixel
11
Fig. 13. Classification of medical image fusion techniques.
values are manipulated to accomplish the desired results. Spatial 5.3. Decision level fusion
domain methods include PCA, IHS, Brovey, High Pass Filtering methods,
ICA, Simple maximum, simple average, and weighted average [19]. Decision Level fusion decides each input image using specific pre
However, the problem with spatial domain methods is that they define criteria and then merges based on the trustworthiness of each
generate spatial distortion in the resultant fused image, which is conclusion into global optimum to make the single fused image. These
considered a negative factor in the fusion process. Stokking et al. [53] types of techniques produce maximum information using certain rules
proposed an HSV model for a fusion of anatomical and functional in defined before the fusion process [56]. Dictionary learning and Bayesian
formation obtained from MRI and SPECT modalities by using a color techniques are the most prevalent methods used in decision fusion. At
encoding pattern. This model performed better than the RGB model and the decision level, usually, there are three approaches integrated to get
allowed rapid, easy, and intuitive retrospective determination of color fused images, these approaches are the following (information theory,
encoding of the functional in the fused image. The proposed method was logical reasoning, and statistics approaches); include joint measures,
implemented on brain images to evaluate two HSV color manipulation Bayesian fusion techniques, hybrid consensus methods, voting, and
techniques. Daneshvar et al. [48] introduced a multi-resolution image fuzzy decision rules. The Bayesian approach is based on probabilities for
fusion technique based on the retinal model, which incorporates a Dif combining data from various sensors, these techniques rely on the Bayes
ference of Gaussian (DoG) operation. The performance results were hypothesis. Nonparametric Bayesian, HWT Bayesian, and DWT Swarm
compared with DWT, HSI, and WT methods. The retina visual-based Optimized are examples of Bayesian techniques [57].
model gives high spectral features in resultant image with less spatial PET and MRI images fused using the nonparametric Bayesian-based
distortion. on sparse representation (SR-NPB) technique provided by Ref. [58]. For
He et al. [54] developed an algorithm that integrates the benefits of both visual and quantitative comparisons, this technique outperformed
both IHS and PCA methods to improve the fused image. PET generates the other three SRs. In terms of execution time, the new approach is
medical images with reasonable color and less spatial resolution, while faster. Twenty sets of PET and MRI scans were used for the experiment in
MRI gives suitable spatial resolution with color appearance. The sug their research. The [59] used the BRATS database that includes the MRI
gested strategy performed better for both human perception and image brain in their fusion process. The authors proposed a
quality assessments criteria. Bashir et al. [55] presented a model based Fractional-BSA-based Bayesian fusion strategy. This approach makes it
on PCA and SWT, which was tested on a variety of medical images. possible to calculate the appropriate Bayesian parameter for fusion.
Results demonstrate that in multimodal fusion, PCA achieved superior Fusing images is made easier by using wavelets generated from the
results with distinctive contrast and brightness levels. SWT seemed to original images using the HWT. In Ref. [60] study, birds swarm opti
give superior performance when the images were from different sources mization was used to develop a novel Bayesian fusion algorithm. Image
like multimodal and multi-sensor images. The above-listed techniques fusion is being developed using the BRATS [61] database. Bayesian
are tabulated in Table 4. fusion is performed by optimizing the source images using the BSA al
gorithm after they have been modified using the Haar-DWT algorithm.
According to the findings of the research, the strategy beat the three
12
Table 4
MMIF technique classification and major contributions in literature.
Research work Fusion Methods Year Multimodal Body Organ Multimodal Fusion Techniques Dataset
combination
Stokking et al. [53] Spatial domain 2001 MRI/SPECT Brain color-encoding technique with HSV –
Daneshvar [83] 2007 MRI/PET Brain Retina based fusion method –
He et al. [54] 2010 MRI/PET Brain IHS and PCA AANLIB
Bashir et al. [55] 2019 X-ray/CT Brain, Leg Principal Components Analysis (PCA) –
MRI/SPECT
Zhizhong [84] 2020 CT/MRI Brain Gradient similar filtering (GSF) –
Qu et al. [42] Frequency Domain 2001 CT/MRI Brain Discrete Wavelet Transform –
Moustafa et al. [44] 2006 CT/MRI Liver Laplacian Pyramid, Multi-focus Technique, Wavelet –
Transform, (CEMIF)
Yang et al. [46] 2008 CT/MRI Brain Contourlet transform (CT) –
Singh et al. [43] 2009 MRI-T1, MRI-T2 Brain Redundant Discrete wavelet transform (RDWT) AANLIB
Liu et al. [45] 2010 CT/SPECT Chest Multi-wavelet transform (MWT) –
Xue-jun et al. [42] 2010 CT/MRI Brain Lifting Wavelet Transform (LWT) –
Bhatnagar et al. [48] 2013 CT and MRI-T1/T2 Brain Non-subsampled contourlet transform (NSCT) AANLIB
Sahu et al. [49] 2014 CT/MRI Brain Laplacian Pyramids –
Xi XX et al. [50] 2017 MRI T1/MRI T2 Brain 3-D Shearlet transform (3D-ST) AANLIB
Ming et al. [51] 2018 MRI/CT/PET Brain Parameter-Adaptive PCNN-NSST ADNI
Arif et al. [52] 2019 MRA/MRI Brain Fast curvelet transform through genetic algorithm AANLIB
PET/SPECT
Srinivasu et al. [85] 2020 MRI/PET/SPECT Brain Empirical wavelet decomposition AANLIB
Qiu et al. [86] 2020 CT/MRI Brain NSCT- separable dictionary learning and Gabor AANLIB
MRT1/MRT2 filtering
Wang et al. [69] Deep Learning 2007 CT/MRI Brain Fuzzy-RBFNN –
Wang et al. [70] 2008 CT/MRI Brain m-PCNN –
Teng et al. [71] 2010 CT/MRI/SPECT Brain Neuro-Fuzzy Logic –
Sivasangumani et al. 2015 CT/MRI Brain Regional firing characteristic PCNN (RFC-PCNN) –
[72]
Liu Y et al. [87] 2016 CT/MRI Brain Convolutional Sparse Representation (CSR) –
Liu et al. [73] 2017 CT/MRI/SPECT Brain CNN AANLIB
Hou et al. [74] 2018 CT/MRI Brain convolutional neural AANLIB
networks and a DCSCM
Liu Y et al. [88] 2019 CT/MRI Brain Convolutional Sparsity-MCA AANLIB
Xia KJ et al. [89] 2019 CT/MRI/PET Brain/ DCNN AANLIB/
Abdom-en Public
Hospital
Jingming et al. [90] 2020 CT/MRI/SPECT/ Brain parameter-adaptive PCNN TCIA/
PET AANLIB
Panigrahy et al. [91] 2020 MRI/SPECT Brain Parameter Adaptive Dual Channel PCNN AANLIB
Daneshvar et al. [75] Hybrid based methods 2010 PET/MRI Brain IHS and Retina-inspired model (RIM) AANLIB
Das et al. [76] 2012 MRI/CT Brain NSCT and PCNN AANLIB
Sharmila [77] 2013 MRI/CT Brain DWT-A-EN-PCA –
Kavitha et al. [78] 2014 PET/SPECT Brain swarm intelligence and neural network AANLIB
MRI
Ramlal [79] 2019 MRI/CT Brain NSCT and SWT AANLIB
Lina et al. [92] 2020 CT/MRI/SPECT/ Homomorphic filter and DWT AANLIB
PET
Zong et al. [93] Sparse representa-ion (SR) 2017 CT/MRI/SPECT/ Brain/Lungs SR of classified image patches AANLIB
methods PET
Jiang et al. [94] 2018 CT/MRI Brain SR- weighted least squares filter AANLIB
Zhu et al. [95] 2018 MRI/PET/SPECT Brain Image decomposition and SR AANLIB
Shahdoosti et al. [96] 2018 CT/MRI/PET/ Brain Sparse Representation Classification (SRC) AANLIB
SPECT
Zhang et al. [97] 2019 MRT1/MRT2/ Brain SR- analysis-synthesis dictionary –
SPECT
Maqsood S et al. [98] 2019 CT/MRI Brain SR-Two-scale Image Decomposition AANLIB
Liu et al. [99] 2020 CT/MRI Brain Joint Sparse –
Representation (JSR)
Anantrasirichai et al. 2020 CT/MRI/OCT/ Brain/Retina Sparse Regularization –
[100] fundus
Li H et al. [101] 2020 CT/MRI Brain low-rank sparse decomposition AANLIB
Das A et al. [62] Decision level 2009 CT/MRI-T1, MRI-T2 Brain evolutionary algorithm AANLIB
Huafeng Li et al. [63] 2018 CT/MRI-T1, MRI-T2 Brain low-rank sparse dictionaries learning AANLIB
Shabanzade F et al. 2019 MRI/PET Brain nonparametric Bayesian AANLIB
[58]
Bhardwaj J et. Al [59] 2020 MRI-T1, MRI-T2 Brain Fractional BSA Bayesian BRATS
Nayak A et al. [60] 2020 MRI-T1, MRI-T2 Brain BSABayesian BRATS
currently used fusion techniques - NSCT, SWT, and SWT-NSCT. information (MI) reveals that the fuzzy clustering strategy outperforms
All images were integrated and matched up initially in Ref. [62] the genetic searching technique using MI. A unique medical image
research. Multiresolution wavelet decomposition of input images gives fusion technique based on low-rank sparse decomposition and dictio
more supplementary information than a single resolution approach nary learning was proposed in Ref. [63] study for de-noising and
provides. A comparison of genetic searching strategies using Mutual enhancing medical imaging. The dictionary-learning model contains
13
terms of regularization with a low rank and a sparse distribution. The 5.5. Hybrid fusion methods
fused image output is constructed by combining the low rank and sparse
components of the source images. Considering that the results of the conventional multimodal image
fusion methods are not satisfactory, the basic idea behind hybrid
methods is to combine two or more fusion techniques, such as spatial or
5.4. Deep learning fusion methods transform fusion and neural network techniques, to improve the fused
image quality and performance. The general advantage of the hybrid
These methods consist of multiple layers, where each layer takes methods is to improve the visual quality and decrease the artifacts and
input from the previous layer. Deep learning contributes to the layered noise in the fused images. Daneshvar et al. [75] proposed a hybrid fusion
structure and suitability of the complex framework architecture for large method based on Retina-Inspired Model (RIM) and HIS fusion methods,
data manipulation [64,65]. The deep learning fusion methods include which can maintain high spatial features and additional functional data.
CCN, Convolution Sparse Representation (CSR) also known as convo The performance and visual results showed that this technique was su
lution sparse coding techniques, and Deep Convolution Neural Networks perior to the Brovey, HIS, and DWT methods. In the proposed method,
(DCCNs). The CNN model of deep learning, which is more popular entropy, discrepancy, mutual information, and averaging gradient were
among all the other techniques [66,67], is trainable and well-tuned to used as fusion quality assessment parameters. Das et al. [76] introduced
learn features of input data in the multilayered architecture framework. an image fusion technique based on NSCT with PCNN. The input med
In CNN, every layer consists of several features maps that hold co ical image was initially decomposed by NSCT, while the low and
efficients known as neurons. In the multiple stages, the features maps high-frequency sub-bands were fused by the fusion rule ’max selection’
connect with every stage using different calculations, including spatial and PCNN, respectively. The spatial frequency in the NSCT domain is
pooling, convolution, and non-linear activation. Another popular deep used as the input to PCNN for the subsequent fusion procedure. The
learning fusion technique is Convolutional Sparse Coding (CSC), which resultant fused image was obtained by taking inverse NSCT.
was first proposed by Zeiler et al. [68] and originates from the decon Sharmila et al. [77] proposed another multimodal image fusion
volutional networks. The main target of this technique is to achieve method based on DWT-Averaging-Entropy-PCA [DWT-Av EN-PCA],
convolutional decomposition of an image under the sparsity constraint. which was compared with other current fusion procedures utilizing
The multistage feature representation of the input image learns from the qualitative and quantitative metrics. Kavitha et al. [78] developed a
deconvolutional networks by developing a hierarchical structure of such fusion method that incorporates swarm intelligence and a neural
decomposition. Then, the input image is reconstructed with the help of network to accomplish a superior fusion output. The image edges were
these multiple decomposition levels in a layered-wise manner. The CSR recognized and improved by utilizing Ant Colony Optimization (ACO).
techniques have yielded promising results in terms of image recon The detected edges were then applied as an input to PCNN. The results
struction and the feature learning approach. demonstrated that the proposed hybrid strategy performed much better
Wang et al. [69] presented a new multimodal image fusion technique than the current computational and hybrid intelligent techniques.
based on the fuzzy radial basis function neural network (Fuzzy-RBFNN) Ramlal et al. [79] proposed an improved hybrid fusion strategy based on
to carry out auto-adaptive image fusion. To train the network, GA was NSCT and SWT. Firstly, the input images were decomposed into various
implemented, and artificially blurred medical images were included in sub-bands utilizing NSCT. Then, SWT was employed to decompose the
sample sets. The experimental outcomes showed that this method was estimation coefficients of NSCT into various sub-bands. Weighted sum
more suitable for blurry input images compared with other traditional modified Laplacian and entropy square of the coefficients were imple
fusion methods. Wang et al. [70] proposed and tested a novel mented as fusion rules with SWT. The fused output image was acquired
multi-channel m-PCNN for multimodal image fusion on four different by taking inverse NSCT.
medical imaging modalities, which exhibited better performance ac The term over-coompetness In recent years, medical image fusion
cording to mutual information criteria. Teng et al. [71] presented a has been progressively used to diagnose different completeness means
fusion technique using a neuro-fuzzy logic and hybrid approach, BP, that the total number of atoms in the dictionary is always greater than
with Least Mean Square (LMS) to train and tune parameters of the the dimension of image signals. Over-completeness gives a sufficient
membership function. The final fused image from neuro-fuzzy logic was amount of atoms for dictionary learning and allows a correct represen
based on a feed-forward neural network reserved texture feature and tation of the signals [80]. As one might expect, SR has attracted note
contained enhanced useful information compared to the BP neural worthy consideration in the exploration field of image fusion [87,88].
network technique (BPNN). Sivasangumani et al. [72] developed a Yang and Li [65] applied the SR technique to the image fusion field for
multimodal image fusion method using regional firing characteristic the first time after such various SR-based image fusion strategies were
PCNN (RFC-PCNN). The proposed technique was well-suited for introduced. The works reporting on the sparse representation of MMIF
increasing medical image fusion quality to determine brain tumors and are provided in Table 4.
minimize the effect of artifacts in the resultant fused image.
Liu et al. [73] presented a multimodal image fusion technique based 5.6. Sparse representation methods
on CNN, in which a Siamese convolutional network was implemented to
obtain a weighted map that combines the pixel activity information from In the sparse representation (SR) method, an over-complete dictio
both input images. The fusion procedure is applied in a multiscale nary is obtained from a sequence of images to achieve a steady and
manner using pyramids, and a local similarity-based technique is used to significant representation of the source images [82]. The basic principle
adaptively adjust the fusion mode for the decomposition of coefficients. of SR representation is based on the treatment of an image signal as a
Hou et al. [74] proposed a fusion method based on CNN and a linear combination of less significant atoms from the pre-trained dic
dual-channel spiking cortical model (DCSCM). Initially, NSST was used tionary learning, where the sparse coefficient shows the significant
to generate low and high-frequency coefficients of the images. Then, features of the input images. Sparsity refers to the fact that only a
low-frequency coefficients were fused and applied as input to the CNN minimal number of atoms are necessary to properly reconstruct a signal,
framework, where a weighted map was obtained by feature maps of the resulting in sparse coefficients.
image and by applying an adaptive selection fusion rule. The To examine the visual performance of each MMIF main approach, we
high-frequency coefficients were selected as an input to DCSCM. Finally, used qualitative findings from the Harvard Atlas [30] database. MRI, CT
the fused image was obtained by inverse NSST. This method showed images were used as source images for visual assessment. All tests were
better performance than some current fusion methods. The described carried out on a Windows 10 computer running MATLAB 2020a and
techniques based on deep learning MMIF are described in Table 4. Python 3.8, with an Intel ® Core I716010U CPU running at 1.8 GHz and
14
Fig. 14. Comparison of visual image results obtained using various MMIF approaches. (a1) and (a2) are CT and MRI source images. (a3-a8) are qualitative fusion
results of GFF [102], LE-NSCT [103], CNN [104], NSST PAPCNN [51], MST SR [81] and N-Fuzzy [105].
16 GB of RAM. Fig. 14 depicts a visual examination of brain fusion area. In the medical field, the information from a single imaging mo
imaging. We compared the visual findings of Li S et al. [102] (Source dality does not provide complete information of human body organs. For
Link-1), Zhu Z et al. [103] (Source Link-2), Zhang Y et al. [104] (Source example, MRI images show only soft-tissue information, while CT im
Link-3), Yin M et al. [51] (Source Link-4), Liu Y et al. [81] (Source ages display bone density information. Thus, MMIF becomes a vital zone
Link-5) and Das S et al. [105] (Source Link-6). The source codes for each of research because of its significance for providing high-quality output
approach are all available online through the source links. The advan images for diagnostics and medical treatment [113]. Some recent
tages and drawback of each fusion techniques summarized in Table 5. disease-based multimodal fusion works are summarized in Table 6.
Lei et al. [114] presented a fusion technique that implements ca
6. Multimodal fusion and recent diseases nonical correlation analysis (CCA) utilizing discriminative feature
learning. Multimodal features and their CCA projections were inter
MMIF has demonstrated exceptional performance for analyzing connected to show each subject, where both individual and shared in
diseases and improving the precision and performance of the diagnostics formation from images of Alzheimer’s disease (AD) were obtained. In
15
Table 5
Comparison of MMIF techniques.
Techniques Advantages Disadvantages
Spatial based • The easiest image fusion process. • The resultant fused image is not promised to be sharp.
• Results in highly concentrated image results produced from the image input. • Image contrast is diminished by the blurring effect.
• Highly simplistic, easy to understand, and enforce. • Usually, spatial domain fusion causes spectral degradation [106].
• Processing and high sharpening capacity are very quick, computationally efficient, • Colors may become distorted and spectrum degradation may occur
and faster [106,107]. [107].
Frequency-based • Performs better than standard fusion methods in terms of reducing the spectral • The final fused image may have a less spatial resolution in this
distortion. process.
• Gives a higher signal-to-noise ratio than an approach based on the pixel level • The fusion algorithm requires a more complicated procedure than
method. pixel-level techniques.
• Enhanced results are produced by multilevel fusion where the image is fused In the • A good fusion technique is needed for a better result.
medical field twice using an appropriate fusion technique. • Generate output images that are more or less identical [108,109].
• Both high spatial resolution and decent quality spectral components are included in
the output image.
• For multi-focus images, strategies provide excellent detailed image quality [108,
109].
Deep learning • Neural networks’ learning environment makes it much easier to optimize the • Based on dynamic processes with complex parameters.
process of image fusion. • Many challenges must be addressed, such as local extremum,
• Various input data increase the focus on fusing high-dimensional data to yield a misidentification, and pace of training convergence.
feasible solution. • Requires more time and hardware specification to train the fusion
• The methodology can be customized to the needs of the application. model.
• Produce excellent results when there is a lot of input data compared to other fusion • Does not give accurate results for small datasets of images [107].
techniques [107].
Hybrid based • Complementary detailed information is achieved from the input images. • Requires detailed knowledge of each technique, otherwise, non-
• Improve clarity, contrast, texture, brightness, and edge information in the fused uniform fusion results are produced.
image. • Has a typically complex and time-consuming fusion process.
• Hybrid techniques prevent difficulties at the pixel level, such as too sensitive noise • Cannot be used for large input datasets [110].
and blurring.
• Minimizes the artifacts in the resultant images [110].
Sparse • SR coefficients are the most significant factors that improve the final fusion • Face two main disadvantages: minimal detail preservation capacity
Representation performance. and high susceptibility to misregistration.
• Retain the visual information better and improve the contrast of the image • Often produces visual artifact results in the reconstructed image
compared to other techniques [107]. [107,111].
• Preserve information related to the structure of images and maintain the extensive
detail of source images [111].
Decision level • The quantity of superfluous and unclear information is reduced. • The method gets increasingly difficult to master as time goes on.
• Increased accuracy by including the information content that is linked to each pixel • In addition, the method becomes more time-consuming and
in the image, which improves than feature level fusion [112]. complicated [112].
Table 6
Some recent disease detection-based multimodal fusion methods.
Reference Disease/disorder/abnormality Year Multi-modality Body organ Fusion algorithms Dataset
Lei et al. [114] Alzheimer’s disease & MCI 2016 MRI/PET Brain Canonical Correlation Analysis (CCA) ADNI
Ahmed et al. [115] AD, MCI disease 2017 MRI/DTI Brain Multiple Kernel Learning ADNI
Chavan et al. [116] Neurocysticercosis (NCC) is a parasite infection 2017 MRI/CT Brain NSRCxWT Radiopaedia
Piccinelli et al. [117] Coronary Artery Disease 2018 PET/CTA Heart Feature extraction using MI –
Rajalingam [118] Astrocytoma Disease 2019 MRI/PET/SPECT Brain Hybrid fusion (DFRWTx AANLIB
DTCWT)
Kaur et al. [119] Degenerative and neoplastic brain tumor 2019 CT/MRI/SPECT Brain SWT and PCA AANLIB
Algarni et al. [120] Tumor detection 2019 CT/MRI Brain CNN –
Xiaoke et al. [121] Alzheimer’s Disease 2020 MRI/PET Brain Feature selection technique with CMC ADNI
Jose J et al. [122] Glioma, Encephalopathy, Mild Alzheimer’s 2021 CT/MRI Brain AISA-NSST AANLIB
/PET
this work, an accuracy of 86.57% for Mild Cognitive Impairment (MCI coronary artery heart disease. In this article, they covered some fusion
vs. NC) disease and 96.93% for AD [vs. normal control (NC)] was ach multimodalities of PET/SPECT and CTA. In an example case, fusion was
ieved. Ahmed et al. [115] suggested a multimodal image fusion method implemented by utilizing left ventricle (LV) features extracted from both
based on Multiple Kernel Learning (MKL) to fuse visual features ob datasets and then optimized results with mutual information methods.
tained from structural MRI and Diffusion Tensor Imaging (DTI) MD Rajalingam et al. [118] developed an MMIF technique by combining the
maps of brain images to differentiate between AD and MCI disease. The discrete fractional wavelet transform (DFRWT) with DTCWT to diagnose
researchers selected both MRI-T1 and DTI image data of 155 total sub neural astrocytoma, which is a type of cancer in the human brain or
jects, where 52 were the Normal Control, 45 had AD, and 58 had MCI. human spinal cord system that develops from cells called astrocytes.
Chavan et al. [116] presented a fusion technique for the diagnosis of Kaur et al. [119] suggested an MMIF fusion model based on SWT and
lesions caused by neurocysticercosis (NCC) infection in the brain. NCC is PCA techniques to analyze degenerative and neoplastic brain tumor
a parasitic disease caused by the tapeworm Taenia solium, which affects disease. The CLAHE technique was implemented for medical image
the central nervous structure of the human brain. In this technique, enhancement information in pre-processing before the fusion process.
“Nonsubsampled Rotated Complex Wavelet Transform” (NSRCxWT)
was used to combine CT and MRI medical images. Piccinelli et al. [117]
presented a review article on multimodal fusion for the analysis of
16
7. Image fusion quality assessment metrics similarity index measure (SSIM), etc. [123]. In the case without ground
truth images, the quality metric is calculated using the source medical
The quality of a fused image is determine by Fusion Quality As images and the resultant fused image. Objective quality evaluation pa
sessments Metrics (FQAMs) following subjective/qualitative and rameters without a reference image include entropy (H), standard de
objective/quantitative methods. Particularly, subjective methods are viation (SD), spatial frequency (SF), a sum of correlation difference
based on visual examination to compare the final fused image with (SCD), cross-entropy (CE), fusion mutual information (FMI), Petrovic
original input images. The examination of the fused image considers metric, Piella metric (Q0, Qw, QE), etc. Some important quality fusion
various parameters like image size, spatial details, color, etc. However, assessments metrics with mathematical expressions are given in Table 7.
these strategies are inconvenient, costly, time-consuming, and trouble To analyses the quantitative result we collected some fusion work from
some in many fusion applications because of the absence of ground truth literature with each technique and compared their statistical fusion re
images that are completely fused [6]. The other method is objective sults using three most common metrics (MI, SD and SSIM). The quan
quality assessment, which employs some evaluation metrics. The titative quality assessment results are arranged in Table 8.
quantitative/objective method further categorized based on whether a
reference image is available or not. The ground truth image is the 8. Conclusions & future insights
reference image for validation of the fusion algorithm. The ground truth
medical image is available in a very rare case or it can be construct The subject of medical image fusion briefly discussed in this recent
manually. survey. To begin with, the study describes the various medical imaging
Objective quality evaluation parameters of the reference image modalities used in the MMIF in detail. Before moving on to more
include mutual information (MI), root mean square error (RMSE), cor advanced MMIF approaches, we described several MMIF databases and
relation coefficient (CC), peak signal to noise ratio (PSNR), structural the main processes with fusion rules workflow. Research on MMIF
Table 7
MMIF quality performance metrics.
FQAMs Formula Description Ref.
MI MIFAB = MIFA + MIFB Mutual information determines the similarity between two images. For better [124]
fusion, its value should be high. Where A and B are two input images and F is the
fused image.
√̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅
SD ∑Q ∑ R The intensity variation of the resultant fused medical image is computed. If its [125]
SD = (f(i.j) − u)2 /QR value is high, then fusion will be better, and vice versa. Where u is the mean value
i=1 j=1
of the resultant image.
EN n−
∑1 The quantity of information in an image is a measure of entropy, and its value [126]
E = − P(Xi )logP(Xi )
i=0 ranges from 0 to 8. The xi of ith point represents the image gray scale value, and P
p(xi) is the normalized histogram of the fusion image’s corresponding is probability. Image fusion results are better for a large entropy value
gray level.
( )
PSNR Imax PSNR computes the ratio of number of intensity levels in the medical image to the [128]
PSNR = 20 x log10 √̅̅̅̅̅̅̅̅̅̅
MSE correlated pixels in the fused medical image. A higher value of PSNR shows
⃒ ⃒
1 ∑ Q ∑R ⃒
⃒
⃒
⃒ superior. The I denote original image while max indicates the maximum pixel gray
MSE = ⃒(I(m, n) − J(m, n))⃒ level of I. MSE represents mean squre error.I and J are original and fused images
QR n=1 n=1 ⃒ ⃒
respectively. For comprehensive description se ref. [127]
UQI σxy 2xy 2σx σy UQI is based on the structural information of the final fused resultant images after [129]
UQI =
σx σy (X)2 + (y)2 σ2x σ2y correlation loss, intensity deformation, and brightness deformation have been
Where σ denotes variance and μ represents average. applied. UQI metrics are inspired by the visual human system. The range of its
value metric is from − 1 to 1. X and Y represent information transformation from
two images.
EI Sx = f*hx, Sy = f*hy Higher image edge intensity indicates high image quality and greater image [130]
Where clarity. The edge intensity of image f can be computed utilizing the Sobel operator
√̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅
(S).
(Sx 2 + Sy 2 )
√̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅
RMSE √
√1 ∑ X− 1 ∑ Y− 1 RMSE computes the quality of the final fused medical image by relating the [131]
RMSE = √ (R(i, j) − F(i, j))2 ground truth or ideal fused medical image. For good fused image results, its value
XY i=o j=0
should be near zero. Where R is input and F denoted fused images, i and j are pixel
horizontal and vertical respectively. The x and y show height and width of images.
SSIM (2FA FB + c1 )(2σFA FB + c2 ) The SSIM assessment metric determines the resemblance between sub-regions of [132]
SSIM(A, B|F) = 2 2
(FA +FB +c1 )(σ2 FA +σ2 FB +c2 ) images FA and FB with images A and B. where σ2 denote variance of input images
Where FA fused image similarity with image A and FB shows similarity and σ is the average of input images A and B.
between fused and image B.
NCC ∑ Normalize Cross Correlation is used to determine feature similarity in both [133]
i,j [(Ai,j − Aij ).( F i,j − Fij )]
̂ ̂
NCC = √̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅ reference images and the resultant fused image. Its value should be high, usually
∑ 2 ∑ ̂ 2
i,j [(A i,j − A ij ) ] i,j [( F
̂ i,j − Fij ) ] close to 1. Aij indicates the average of ground truth images, and ̂
F i,j is the average
of fused images.
CE D(hA ||hF ) + D(hB ||hF ) Cross Entropy measures the resemblance of information content between the [134]
CE(A, B, F) =
2 source reference images and resultant fused images. The value of CE should be low
for better fusion.
Q0, Qw, Q0 (x, y) = Piella’s metric determines prominent information in the resultant image by [135,
)
QE 1 ∑ calculating measurements, i.e. mean luminance, image contrast, and correlation
(λ(w)Q0 (x, f|w) + (1 − λ(w))Q0 (y, f|w) 136]
|w| w∈W coefficient. It considers important information of image edges. The range of this
metric is from 0 to 1, where a value closer to 1 indicates better fusion.
Qw (x, y, f) =
)
∑
C(w)(λ(w)Q0 (x, f|w) + (1 − λ(w))Q0 (y, f|w)
w∈W
QE (x, y, f) = QW (x, y, f).QW (x .y .f )x

′ ′ ′
QXY/F XY ∑N ∑M This evaluation method reveals similarities between edges transmitted during the [137]
QXF (n, m)WA (n, m) + QYF (n, m)WB (n, m)
Q F = n=1 m=1∑N ∑M fusion procedure. The range of this metric is from 0 to 1. X and Y are input images,
i=1 j=1 (WX (i, j) + WY (i, j)) and F is the resultant image.
17
Table 8
Performance evaluation metric comparison.
Methods Fusion level Year Harvard Medical School Dataset (AANLIB)
MRI/CT with PET/SPECT CT with MRI
MI SSIM SD MI SSIM SD
He et al. [54] Spatial domain 2010 2.9546 – – – – –

Zhizhong [84] 2020 – – – – 0.8410 –
Singh [43] Transform domain 2009 – – – 0.790 – –
Bhatnagar [48] 2013 1.5017 0.7972 – 0.9681 0.7795 –
Xi [50] 2017 – – – 2.8001 – 66.26
Arif et al. [52] 2019 5.4121 0.9751 – 3.3121 0.8651 –
Srinivasu [85] 2020 4.4543 0.9915 – – – –
Qiu et al. [86] 2020 – – – 2.2317 – –
Wang [69] Deep Learning 2007 – – – 5.0066 – 68.9896
Liu et al. [73] 2017 – – – 0.8872 0.6457 –
Hou et al. [74] 2018 – – – 4.4628 0.7456 74.3093
Liu [66] 2019 0.6397
Xia [67] 2019 6.344 – 75.422 5.147 – 45.907
Jingming [90] 2020 1.4261 – 109.515 0.8599 – 111.020
Panigrahy [69] 2020 0.7597 0.7597
Daneshvar [75] Hybrid based methods 2010 0.6541 – – – – –
Das et al. [76] 2012 – – – 5.0067 – 68.9896
Sharmila [77] 2013 – – – – – –
Kavitha [78] 2014 7.2647 0.6197 64.922 – – –
Ramlal [79]. 2019 – – – 4.3780 – 58.0671
Rajalingam [60] 2019 4.2991 0.8127 – – – –
Lina [92] 2020 – – – 2.8210 0.777 –
Zong [77] Sparse representa-ion (SR) methods 2017 4.1962 0.7335 68.7610 – – –
Jiang [94] 2018 – – – 4.673 0.625 –
Zhu et al. [95] 2018 2.0520 0.9178 – – – –
Shahdoosti [96] 2018 2.7545 0.8635 – 2.9194 0.7839- –
Maqsood [98] 2019 – – – 4.3580 0.7654 –
Das A et al. [62] Decision level 2009 – – – 3.4005 – –
Huafeng Li et al. [63] 2018 4.0708 – – 4.3575 – –
Shabanzade F et al. [58] 2019 3.0031 – – – – –
Bhardwaj J et. Al [59] 2020 – – – 1.4860 – –
Nayak A et al. [60] 2020 – – – 1.4764 – –
*Where MI and SSIM are reference-based metrics while SD is non-reference-based metrics.
algorithms is summarized in this publication. Comparative image fusion diagnose different diseases efficiently, Previously, significant informa
studies involving frequency fusion, spatial fusion, decision fusion, deep tion was obtained from single modalities, but this may easily be
learning, hybrid fusion, and sparse representation fusion are presented increased by using MMIF from different sensors with imaging equip
in this research. Results and literature evaluation show that most MMIF ment. Works that utilized such fusion techniques for disease diagnoses
reviews do not cover all classification strategies fully, as well as multi are also discussed in this article. Significant fusion quality assessments
modal databases and current diseases that are related to MMIF are ab metrics are highlighted for benchmarking the MMIF methods. In
sent from many of these reviews. In this research, a comparison of conclusion, researchers have a choice of optimal fusion techniques and
several image fusion techniques is presented, along with their advan datasets for benchmarking their results and further developing suitable
tages and disadvantages. Tables 4–6 provide a comparison of several models for diagnosing diseases and analyzing different patterns in
MMIF algorithms based on literature reviews. The quality of the output- medical imaging. However, some existing challenges of MMIF pertain to
fused image is evaluated using a variety of objective criteria in Table 8. miss-registration and artifacts in the resultant fused image. Such issues
DL fusion and Hybrid level approaches were shown to enhance the can be solved by increasing knowledge of the multimodal fusion domain
quantitative outcomes of fusion under this review observation. In and achieving improved diagnoses of more recent diseases that cannot
comparison to other methods of fusing, the results of DL and Hybrid are be detected by a single modality.
more reasonable. There are many factors to consider when deciding
which imaging approach is best for a certain application, therefore it is 8.1. Advantages and drawback of previous techniques
impossible to say which one is the best overall. In this review article, the
following contribution summarized below: Various multimodal medical fusion techniques have covered in this
work. It was found that spatial image fusion methods are ineffective in
• Classification of medical modalities used in MMIF according to the real-world applications. For instance, the PCA, hue intensity saturation,
EM spectrum, source of energy, and acquisition of imaging. and Brovey techniques are computationally efficient, fast, and simple,
• Analysis of five medical databases associated with multimodal im yet they result in illumination variation. The use of principal component
ages to extract statistical results. analysis to fuse images offers a spatial benefit but suffers from spectral
• Discussion of the general procedure of MMIF, which further classi deterioration. While frequency domain methods reduce spectral
fied into six distinct methods: frequency fusion, spatial fusion, de distortion and can perform better than standard fusion methods, they
cision fusion, deep learning, hybrid fusion, and sparse representation have a higher signal-to-noise ratio than pixel-level-based methods.
fusion. Enhanced results can be achieved by multilevel fusion, wherein the
• Comparison of these techniques based on image quality assessment image is fused twice using an appropriate fusion technique. Fused im
metrics. ages may have a less spatial resolution because of this process. This
fusion algorithm is more complex than pixel-level techniques. In liter
In recent years, medical image fusion has been progressively used to ature, the hybrid fusion technique can improve the clarity, contrast,
18
texture, brightness, and edge information of the fused image. This (Source Link.5) https://github.com/yuliu316316/MST-SR-Fusio
technique can also achieve complementary information from input im n-Toolbox (MATLAB)
ages with reduced artifacts. However, a thorough understanding of each (Source Link.6) https://github.com/HarrisXia/image-fusion-zoo
approach is still need to improve the hybrid fusion algorithm, which (MATLAB)
yields non-uniform fusion outcomes, and these techniques are typically
difficult and time-consuming. The other challenge in the hybrid fusion References
technique is that large input datasets should not be use. In the sparse
representation approach, the coefficients are the most important pa [1] B. Rajalingam, R. Priya, R. Scholar, Review of multimodality medical image
fusion using combined transform techniques for clinical application, Int. J. Sci.
rameters that increase the fusion performance. By enhancing the im Res. Comput. Sci. Appl. Manag. Stud. IJSRCSAMS 7 (3) (2018) [Online].
age’s contrast and retention of visual information, information about Available: www.ijsrcsams.com.
image structure and a high level of detail in source images can pre [2] A.P. James, B.V. Dasarathy, Medical image fusion: a survey of the state of the art,
Inf. Fusion 19 (1) (2014) 4–19, https://doi.org/10.1016/j.inffus.2013.12.002.
served. Nevertheless, this technique suffers from miss-registration, [3] PubMed, https://www.ncbi.nlm.nih.gov/pubmed/. (Accessed 10 February
including minimal detail preservation capacity. Deep learning tech 2020).
niques can make optimizing the image fusion process easier and maybe [4] J. Du, W. Li, K. Lu, B. Xiao, An overview of multi-modal medical image fusion,
Neurocomputing 215 (2016) 3–20, https://doi.org/10.1016/j.
customized to an application’s requirements. When there is a large neucom.2015.07.160.
amount of input data with several dimensions and variety, deep learning [5] F.E.Z.A. El-Gamal, M. Elmogy, A. Atwan, Current trends in medical image
approaches outperform other fusion techniques. However, these ap registration and fusion, Egypt. Inform. J. 17 (1) (2016) 99–124, https://doi.org/
10.1016/j.eij.2015.09.002.
proaches based on a dynamic process with many parameters, and
[6] B. Meher, S. Agrawal, R. Panda, A. Abraham, A survey on region based image
training a fusion model requires more effort and hardware than other fusion methods, Inf. Fusion 48 (2019) 119–132, https://doi.org/10.1016/j.
techniques. Another flaw of deep learning methods is that reliable re inffus.2018.07.010.
sults cannot be produce for smaller image datasets. [7] J.M. Dolly, A.K. Nisa, A survey on different multimodal medical image fusion
techniques and methods, in: Proc. 1st Int. Conf. Innov. Inf. Commun. Technol,
ICIICT 2019, 2019, https://doi.org/10.1109/ICIICT1.2019.8741445.
8.2. Future direction [8] B. Huang, F. Yang, M. Yin, X. Mo, C. Zhong, A review of multimodal medical
image fusion techniques, Comput. Math. Methods Med. (2020) 2020, https://doi.
org/10.1155/2020/8279342.
Even though researchers have suggested several image fusion algo [9] T. Tirupal, B.C. Mohan, S.S. Kumar, Multimodal medical image fusion techniques
rithms, these approaches even have certain flaws. Specific advanced – a review, Curr. Signal Transduct. Ther. 16 (2) (2020) 142–163, https://doi.org/
MMIF techniques and implementation of new image fusion metrics are 10.2174/1574362415666200226103116.
[10] H. Hermessi, O. Mourali, E. Zagrouba, Multimodal medical image fusion review:
still in need of the current research. Considering the mentioned short theoretical background and recent advances, Signal Process. 183 (2021), https://
comings, some possible future research aspects are as follows: doi.org/10.1016/j.sigpro.2021.108036.
[11] J. Sebastian, G.R.G. King, Fusion of multimodality medical images- A review, in:
Proc. - 1st Int. Conf. Smart Technol. Commun. Robot. STCR 2021, 2021, https://
• Investigate the effectiveness of image of fusion techniques, particu doi.org/10.1109/STCR51658.2021.9588882 no. October.
larly with advancements of machine learning and deep learning, [12] H. Zhang, H. Xu, X. Tian, J. Jiang, J. Ma, Image fusion meets deep learning: a
considering the absence of appropriate image representation meth survey and perspective, Inf. Fusion 76 (2021) 323–336, https://doi.org/10.1016/
j.inffus.2021.06.008. May.
odologies and generally recognized fusion assessment criteria.
[13] B. Singh, et al., Application of vibrational microspectroscopy to biology and
• The implementation of evolving methods to enhance image areas of medicine, Curr. Sci. 102 (2) (2012) 232–244.
interest before the fusion process is another area of concern. [14] J. Andreu-Perez, C.C.Y. Poon, R.D. Merrifield, S.T.C. Wong, G.Z. Yang, Big data
for health, IEEE J. Biomed. Heal. Inform. 19 (4) (2015) 1193–1208, https://doi.
• Develop innovative multi-scale decomposition algorithms for mul
org/10.1109/JBHI.2015.2450362.
timodality image fusion. [15] M.A. Azam, et al., Deep Learning Applied to White Light and Narrow Band
• Since most image fusion issues lack accurate ground truth, evalu Imaging Videolaryngoscopy : toward Real-Time Laryngeal Cancer Detection,
ating the quality of the fused output is highly challenging, we need to Laryngoscope, 2021, pp. 1–9, https://doi.org/10.1002/lary.29960.
[16] M.A. Haidekker, X-Ray Projection Imaging, 2013.
develop no-reference metrics with greater descriptive capabilities. [17] MITA. https://www.medicalimaging.org/about-mita/medical-imaging-primer/.
• Improve the image quality of resulting fusion output to manage the (Accessed 10 January 2020).
limitations of imaging modalities and excessive noise [18] V.I. Mikla, Medical Imaging Technology, 2013.
[19] F.E.Z.A. El-Gamal, M. Elmogy, A. Atwan, Current trends in medical image
• Identify solutions to the absence of adequate information in each registration and fusion, Egypt. Inform. J. 17 (1) (2016) 99–124, https://doi.org/
medical imaging modality, image noise, massive computing cost, and 10.1016/j.eij.2015.09.002.
the disparity in dimensions between images in multiple modalities. [20] 2013 John R. Giudicessi, BA.Michael J. Ackerman, Imaging techniques for Kaposi
Sarcoma (KS), Bone 23 (1) (2008) 1–7, https://doi.org/10.1038/jid.2014.371.
• A challenge to collect or build datasets that may deemed trustworthy [21] K.Y.C. Teo, et al., Introduction to optical coherence tomography angiography,
in various clinical aided-diagnosis such as eye, larynx, and other Swept-Source Opt. Coherence Tomogr. 1 (2018) 9–14, https://doi.org/10.1142/
bodily organs that have not been addressed in the literature to assess 9789813239579_0002.
[22] Microscopy. https://medical-dictionary.thefreedictionary.com/o. (Accessed 10
fusion methodologies is one of future research in this subject. Only
January 2020).
brain imaging data from Harvard dominates the rest of the field, [23] J. Cal-Gonzalez, et al., Hybrid imaging: instrumentation and data processing,
according to this current review. Front. Physiol. 6 (2018), https://doi.org/10.3389/fphy.2018.00047. MAY.
[24] M.A. Azam, K.B. Khan, M. Aqeel, A.R. Chishti, M.N. Abbasi, Analysis of the
MIDAS and OASIS biomedical databases for the application of multimodal image
Declaration of competing interest processing, Commun. Comput. Inf. Sci. 1198 (2020) 581–592, https://doi.org/
10.1007/978-981-15-5232-8_50.
The authors of this manuscript declare no conflict of interest. [25] TCIA. https://www.cancerimagingarchive.net/. (Accessed 5 January 2020).
[26] S.D. Ramlal, J. Sachdeva, C.K. Ahuja, N. Khandelwal, Multimodal medical image
fusion using non-subsampled shearlet transform and pulse coupled neural
Supporting Material network incorporated with morphological gradient, Signal, Image Video Process.
12 (8) (2018) 1479–1487, https://doi.org/10.1007/s11760-018-1303-z.
[27] E. Daniel, Optimum wavelet-based homomorphic medical image fusion using
(Source Link.1) https://github.com/RRuschel/Image-fusion hybrid genetic-Grey Wolf optimization algorithm, IEEE Sensor. J. 18 (16) (2018)
(MATLAB) 6804–6811, https://doi.org/10.1109/JSEN.2018.2822712.
(Source Link.2) https://github.com/zhiqinzhu123/Source-code [28] Y. Yang, Y. Que, S. Huang, P. Lin, Multimodal sensor medical image fusion based
on type-2 fuzzy logic in NSCT domain, IEEE Sensor. J. 16 (10) (2016) 3735–3745,
-of-medical-image-fusion-in-NSCT-domain (MATLAB) https://doi.org/10.1109/JSEN.2016.2533864.
(Source Link.3) https://github.com/uzeful/IFCNN (Python) [29] A. Torrado-Carvajal, et al., Multi-atlas and label fusion approach for patient-
(Source Link.4) https://github.com/yuliu316316/NSST-PAPC specific MRI based skull estimation, Magn. Reson. Med. 75 (4) (2016)
1797–1807, https://doi.org/10.1002/mrm.25737.
NN-Fusion (MATLAB)
19
[30] AANLIB. http://www.med.harvard.edu/AANLIB/home.html. (Accessed 5 [59] J. Bhardwaj, A. Nayak, Haar wavelet transform–based optimal Bayesian method
January 2020). for medical image fusion, Med. Biol. Eng. Comput. 58 (10) (2020) 2397–2411,
[31] ADNI. http://adni.loni.usc.edu/. (Accessed 5 January 2020). https://doi.org/10.1007/s11517-020-02209-6.
[32] M.A. Azam, K.B. Khan, M. Ahmad, M. Mazzara, Multimodal medical image [60] J. Bhardwaj, A. Nayak, Discrete wavelet transform and bird swarm optimized
registration and fusion for quality enhancement, Comput. Mater. Continua (CMC) Bayesian multimodal medical image fusion, Helix 10 (1) (2020), https://doi.org/
68 (1) (2021) 821–840, https://doi.org/10.32604/cmc.2021.016131. 10.29042/2020-10-1-07-12, 07–12.
[33] MIDAS. http://www.insight-journal.org/midas/. (Accessed 10 January 2020). [61] M. Yousuf, K.B. Khan, M.A. Azam, M. Aqeel, Brain tumor localization and
[34] M.R. Princess, V.S. Kumar, M.R. Begum, Comprehensive and comparative study segmentation based on pixel-based thresholding with morphological operation,
of different image fusion techniques, Int. J. Adv. Res. Electr. Electron. Instrum. Commun. Comput. Inf. Sci. 1198 (2020) 562–572, https://doi.org/10.1007/978-
Eng. (2014) 11800–11806, https://doi.org/10.15662/ijareeie.2014.0309015, 03, 981-15-5232-8_48. May.
no. 09. [62] A. Das, M. Bhattacharya, Evolutionary algorithm based automated medical image
[35] H.A. Mohammed, M.A. Hassan, The image registration techniques for medical fusion technique: comparative study with fuzzy fusion approach, in: 2009 World
imaging (MRI-CT), Am. J. Biomed. Eng. 6 (2) (2016) 53–58, https://doi.org/ Congr. Nat. Biol. Inspired Comput. NABIC 2009 - Proc., 2009, pp. 269–274,
10.5923/j.ajbe.20160602.02. https://doi.org/10.1109/NABIC.2009.5393715.
[36] H.M. Elhoseny, E.M. El-rabaie, O.S.F. Allah, F.E.A. El-samie, Medical image [63] H. Li, X. He, D. Tao, Y. Tang, R. Wang, Joint medical image fusion, denoising and
fusion: a literature review present solutions and future directions, Menoufia J. enhancement via discriminative low-rank sparse dictionaries learning, Pattern
Electron. Eng. Res. 26 (2017) 321–350, https://doi.org/10.21608/ Recogn. 79 (2018) 130–146, https://doi.org/10.1016/j.patcog.2018.02.005.
mjeer.2017.63510. September. [64] Y. Lecun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7553) (2015)
[37] S. Li, X. Kang, L. Fang, J. Hu, H. Yin, Pixel-level image fusion: a survey of the state 436–444, https://doi.org/10.1038/nature14539.
of the art, Inf. Fusion 33 (2017) 100–112, https://doi.org/10.1016/j. [65] T. Zhou, S. Ruan, S. Canu, A review: deep learning for medical image
inffus.2016.05.004. segmentation using multi-modality fusion, Array 3–4 (2019) 100004, https://doi.
[38] K. Parmar, R. Kher, A comparative analysis of multimodality medical image org/10.1016/j.array.2019.100004. July.
fusion methods, in: Proc. - 6th Asia Int. Conf. Math. Model. Comput. Simulation, [66] Z. Liu, et al., Automatic diagnosis of fungal keratitis using data augmentation and
AMS 2012, 2012, pp. 93–97, https://doi.org/10.1109/AMS.2012.46. image fusion with deep convolutional neural network, Comput. Methods Progr.
[39] F. Sadjadi, Comparative Image Fusion Analysais, 2006, p. 8, https://doi.org/ Biomed. 187 (2020) 105019, https://doi.org/10.1016/j.cmpb.2019.105019.
10.1109/cvpr.2005.436, 8. [67] K.S. Choi, J.S. Shin, J.J. Lee, Y.S. Kim, S.B. Kim, C.W. Kim, In vitro trans-
[40] S. Das, M.K. Kundu, A neuro-fuzzy approach for medical image fusion, IEEE differentiation of rat mesenchymal cells into insulin-producing cells by rat
Trans. Biomed. Eng. 60 (12) (2013) 3347–3353, https://doi.org/10.1109/ pancreatic extract, Biochem. Biophys. Res. Commun. 330 (4) (2005) 1299–1305,
TBME.2013.2282461. https://doi.org/10.1016/j.bbrc.2005.03.111.
[41] S. Zarif, I. Faye, R. Dayang, A comparative study of different image completion [68] M.D. Zeiler, D. Krishnan, G.W. Taylor, R. Fergus, Deconvolutional Networks for
techniques, in: 2014 Int. Conf. Comput. Inf. Sci. ICCOINS 2014 - A Conf. World Feature Learning, Cvpr, 2010, pp. 2528–2535, https://doi.org/10.1109/
Eng. Sci. Technol. Congr. ESTCON 2014 - Proc. 90, 2014, pp. 12–16, https://doi. CVPR.2010.5539957.
org/10.1109/ICCOINS.2014.6868411, 19. [69] Y.P. Wang, J.W. Dang, Q. Li, S. Li, Multimodal medical image fusion using fuzzy
[42] Q. Guihong, Z. Dali, Y. Pingfan, Medical image fusion by wavelet transform radial basis function neural networks, in: Proc. 2007 Int. Conf. Wavelet Anal.
modulus maxima, Opt Express 9 (4) (2001) 184, https://doi.org/10.1364/ Pattern Recognition, vol. 2, ICWAPR ’07, 2007, pp. 778–782, https://doi.org/
oe.9.000184. 10.1109/ICWAPR.2007.4420774.
[43] R. Singh, M. Vatsa, A. Noore, Multimodal medical image fusion using redundant [70] Z. Wang, Y. Ma, Medical image fusion using m-PCNN, Inf. Fusion 9 (2) (2008)
discrete wavelet transform, in: Proc. 7th Int. Conf. Adv. Pattern Recognition, 176–185, https://doi.org/10.1016/j.inffus.2007.04.003.
ICAPR 2009, 2009, pp. 232–235, https://doi.org/10.1109/ICAPR.2009.97. [71] J. Teng, S. Wang, J. Zhang, X. Wang, Neuro-fuzzy logic based fusion algorithm of
[44] E. Moustafa, For detection of hepatic lesions and acute intra-cerebral, Conf. Inf. medical images, in: Proc. - 2010 3rd Int. Congr. Image Signal Process, vol. 4, CISP
Commun. Technol. (2006), 00. 2010, 2010, pp. 1552–1556, https://doi.org/10.1109/CISP.2010.5646958.
[45] Y. Liu, PET/CT Medical Image Fusion Algorithm Based on Multiwavelet [72] S. Sivasangumani, P.S. Gomathi, B. Kalaavathi, Regional firing characteristic of
Transform. PCNN-based multimodal medical image fusion in NSCT domain, Int. J. Biomed.
[46] L. Yang, B.L. Guo, W. Ni, Multimodality medical image fusion based on multiscale Eng. Technol. 18 (3) (2015) 199–209, https://doi.org/10.1504/
geometric analysis of contourlet transform, Neurocomputing 72 (1–3) (2008) IJBET.2015.070575.
203–211, https://doi.org/10.1016/j.neucom.2008.02.025. [73] Y. Liu, X. Chen, J. Cheng, H. Peng, A medical image fusion method based on
[47] X.J. Wang, Y. Mu, A medical image fusion algorithm based on lifting wavelet convolutional neural networks, in: 20th Int. Conf. Inf. Fusion, Fusion 2017 - Proc.,
transform, in: Proc. - Int. Conf. Artif. Intell. Comput. Intell. AICI 2010 3, 2010, 2017, https://doi.org/10.23919/ICIF.2017.8009769.
pp. 474–476, https://doi.org/10.1109/AICI.2010.337, 8. [74] R. Hou, D. Zhou, R. Nie, D. Liu, X. Ruan, Brain CT and MRI medical image fusion
[48] S. V, B.R. Kumar, Directive contrast based multimodal medical image fusion in using convolutional neural networks and a dual-channel spiking cortical model,
NSCT with DWT domain, Int. J. Eng. Trends Technol. 9 (6) (2014) 288–294, Med. Biol. Eng. Comput. (2018), https://doi.org/10.1007/s11517-018-1935-8.
https://doi.org/10.14445/22315381/ijett-v9p257. [75] S. Daneshvar, H. Ghassemian, MRI and PET image fusion by combining IHS and
[49] A. Sahu, V. Bhateja, A. Krishn, Himanshi, Medical image fusion with Laplacian retina-inspired models, Inf. Fusion 11 (2) (2010) 114–123, https://doi.org/
pyramids, in: 2014 Int. Conf. Med. Imaging, M-Health Emerg. Commun. Syst. 10.1016/j.inffus.2009.05.003.
MedCom, 2014, pp. 448–453, https://doi.org/10.1109/MedCom.2014.7006050, [76] S. Das, M.K. Kundu, NSCT-based multimodal medical image fusion using pulse-
2014. coupled neural network and modified spatial frequency, Med. Biol. Eng. Comput.
[50] X.X. Xi, X.Q. Luo, Z.C. Zhang, Q.J. You, X. Wu, Multimodal medical volumetric 50 (10) (2012) 1105–1114, https://doi.org/10.1007/s11517-012-0943-3.
image fusion based on multi-feature in 3-D shearlet transform, in: 2017 Int. Smart [77] K. Sharmila, S. Rajkumar, V. Vijayarajan, Hybrid method for multimodality
Cities Conf. ISC2 1, 2017, https://doi.org/10.1109/ISC2.2017.8090797, 2017. medical image fusion using Discrete Wavelet Transform and Entropy concepts
[51] M. Yin, X. Liu, Y. Liu, X. Chen, Medical image fusion with parameter-adaptive with quantitative analysis, in: Int. Conf. Commun. Signal Process. ICCSP 2013 -
pulse coupled neural network in nonsubsampled shearlet transform domain, IEEE Proc., 2013, pp. 489–493, https://doi.org/10.1109/iccsp.2013.6577102.
Trans. Instrum. Meas. 68 (1) (2019) 49–64, https://doi.org/10.1109/ [78] C.T. Kavitha, C. Chellamuthu, Medical image fusion based on hybrid intelligence,
TIM.2018.2838778. Appl. Soft Comput. J. 20 (2014) 83–94, https://doi.org/10.1016/j.
[52] M. Arif, G. Wang, Fast curvelet transform through genetic algorithm for asoc.2013.10.034.
multimodal medical image fusion, Soft Comput. (2019), https://doi.org/ [79] S.D. Ramlal, J. Sachdeva, C.K. Ahuja, N. Khandelwal, An improved multimodal
10.1007/s00500-019-04011-5. medical image fusion scheme based on hybrid combination of nonsubsampled
[53] R. Stokking, K.J. Zuiderveld, M.A. Viergever, Integrated volume visualization of contourlet transform and stationary wavelet transform, Int. J. Imag. Syst.
functional image data and anatomical surfaces using normal fusion, Hum. Brain Technol. 29 (2) (2019) 146–160, https://doi.org/10.1002/ima.22310.
Mapp. 12 (4) (2001) 203–218, https://doi.org/10.1002/1097-0193(200104)12: [80] Q. Zhang, X. Maldague, An adaptive fusion approach for infrared and visible
4<203::AID-HBM1016>3.0.CO;2-X. images based on NSCT and compressed sensing, Infrared Phys. Technol. 74
[54] C. He, Q. Liu, H. Li, H. Wang, Procedia Engineering Multimodal medical image (2016) 11–20, https://doi.org/10.1016/j.infrared.2015.11.003.
fusion based on IHS and PCA, Procedia Eng. 7 (2010) 280–285, https://doi.org/ [81] Y. Liu, S. Liu, Z. Wang, A general framework for image fusion based on multi-
10.1016/j.proeng.2010.11.045. scale transform and sparse representation, Inf. Fusion 24 (2015) 147–164,
[55] R. Bashir, R. Junejo, N.N. Qadri, M. Fleury, M.Y. Qadri, SWT and PCA image https://doi.org/10.1016/j.inffus.2014.09.004.
fusion methods for multi-modal imagery, Multimed. Tool. Appl. 78 (2) (2019) [82] Q. Zhang, Y. Liu, R.S. Blum, J. Han, D. Tao, Sparse representation based multi-
1235–1263, https://doi.org/10.1007/s11042-018-6229-5. sensor image fusion for multi-focus and multi-modality images: a review, Inf.
[56] B. Meher, S. Agrawal, R. Panda, A. Abraham, A survey on region based image Fusion 40 (2018) 57–75, https://doi.org/10.1016/j.inffus.2017.05.006.
fusion methods, Inf. Fusion 48 (2019) 119–132, https://doi.org/10.1016/j. [83] S. Daneshvar, H. Ghassemian, Fusion of MRI and PET images using retina based
inffus.2018.07.010. July 2018. multi-resolution transforms, in: 2007 9th Int. Symp. Signal Process. its Appl.
[57] F. Tabib Mahmoudi, F. Samadzadegan, P. Reinartz, Object recognition based on ISSPA 2007, Proc., 2007, https://doi.org/10.1109/ISSPA.2007.4555524, 0–3.
the context aware decision-level fusion in multiviews imagery, IEEE J. Sel. Top. [84] Z. Fu, Y. Zhao, Y. Xu, L. Xu, J. Xu, Gradient structural similarity based gradient
Appl. Earth Obs. Rem. Sens. 8 (1) (2015) 12–22, https://doi.org/10.1109/ filtering for multi-modal image fusion, Inf. Fusion 53 (2020) 251–268, https://
JSTARS.2014.2362103. doi.org/10.1016/j.inffus.2019.06.025. April 2018.
[58] F. Shabanzade, M. Khateri, Z. Liu, MR and PET image fusion using nonparametric [85] S. Polinati, R. Dhuli, Multimodal medical image fusion using empirical wavelet
Bayesian joint dictionary learning, IEEE Sens. Lett. 3 (7) (2019) 2019–2022, decomposition and local energy maxima, Optik 205 (2020) 163947, https://doi.
https://doi.org/10.1109/LSENS.2019.2925072. org/10.1016/j.ijleo.2019.163947.
20
[86] Q. Hu, S. Hu, F. Zhang, Multi-modality medical image fusion based on separable [113] B. Rajalingam, R. Priya, R. Bhavani, Hybrid multimodal medical image fusion
dictionary learning and Gabor filtering, Signal Process. Image Commun. 83 using combination of transform techniques for disease analysis, Procedia Comput.
(2020) 115758, https://doi.org/10.1016/j.image.2019.115758. December 2019. Sci. 152 (2019) 150–157, https://doi.org/10.1016/j.procs.2019.05.037.
[87] Y. Liu, X. Chen, R.K. Ward, J. Wang, Image fusion with convolutional sparse [114] B. Lei, S. Chen, D. Ni, T. Wang, Discriminative learning for Alzheimer’s disease
representation, IEEE Signal Process. Lett. 23 (12) (2016) 1882–1886, https://doi. diagnosis via canonical correlation analysis and multimodal fusion, Front. Aging
org/10.1109/LSP.2016.2618776. Neurosci. 8 (2016) 1–17, https://doi.org/10.3389/fnagi.2016.00077. MAY.
[88] Y. Liu, X. Chen, R.K. Ward, Z.J. Wang, Medical image fusion via convolutional [115] O. Ben Ahmed, J. Benois-Pineau, M. Allard, G. Catheline, C. Ben Amar,
sparsity based morphological component analysis, IEEE Signal Process. Lett. 26 Recognition of Alzheimer’s disease and mild cognitive impairment with
(3) (2019) 485–489, https://doi.org/10.1109/LSP.2019.2895749. multimodal image-derived biomarkers and multiple Kernel learning,
[89] K. jian Xia, H. sheng Yin, J. qiang Wang, A novel improved deep convolutional Neurocomputing 220 (2017) 98–110, https://doi.org/10.1016/j.
neural network model for medical image fusion, Cluster Comput. 22 (2019) neucom.2016.08.041.
1515–1527, https://doi.org/10.1007/s10586-018-2026-1. [116] S.S. Chavan, A. Mahajan, S.N. Talbar, S. Desai, M. Thakur, A. D’cruz,
[90] J. Xia, Y. Lu, L. Tan, Research of multimodal medical image fusion based on Nonsubsampled rotated complex wavelet transform (NSRCxWT) for medical
parameter-adaptive pulse-coupled neural network and convolutional sparse image fusion related to clinical aspects in neurocysticercosis, Comput. Biol. Med.
representation, Comput. Math. Methods Med. (2020), https://doi.org/10.1155/ 81 (2017) 64–78, https://doi.org/10.1016/j.compbiomed.2016.12.006.
2020/3290136, 2020. December 2016.
[91] C. Panigrahy, A. Seal, N.K. Mahato, MRI and SPECT image fusion using a [117] M. Piccinelli, D.C. Cooke, E.V. Garcia, Multimodality image fusion for coronary
weighted parameter adaptive dual channel PCNN, IEEE Signal Process. Lett. 27 artery disease detection, Ann. Nucl. Cardiol. 4 (1) (2018) 74–78, https://doi.org/
(1070) (2020) 690–694, https://doi.org/10.1109/LSP.2020.2989054. 10.17996/anc.18-00065.
[92] L. Xu, Y. Si, S. Jiang, Y. Sun, H. Ebrahimian, Medical image fusion using a [118] E.S.K. B, C.S. Bindu, Emerging Technologies in Computer Engineering:
modified shark smell optimization algorithm and hybrid wavelet-homomorphic Microservices in Big Data Analytics, vol. 985, Springer Singapore, 2019.
filter, Biomed. Signal Process Control 59 (2020) 101885, https://doi.org/ [119] L. Singh, S. Singh, N. Aggarwal, Proceedings of 2nd International Conference on
10.1016/j.bspc.2020.101885. Communication, Computing and Networking, vol. 46, Springer Singapore, 2019.
[93] J. jing Zong, T. shuang Qiu, Medical image fusion based on sparse representation [120] A.D. Algarni, Automated medical diagnosis system based on multi-modality
of classified image patches, Biomed. Signal Process Control 34 (2017) 195–205, image fusion and deep learning, Wireless Pers. Commun. 111 (2) (2020)
https://doi.org/10.1016/j.bspc.2017.02.005. 1033–1058, https://doi.org/10.1007/s11277-019-06899-6.
[94] W. Jiang, et al., Medical images fusion by using weighted least squares filter and [121] X. Hao, et al., Multi-modal neuroimaging feature selection with consistent metric
sparse representation, Comput. Electr. Eng. 67 (2018) 252–266, https://doi.org/ constraint for diagnosis of Alzheimer’s disease, Med. Image Anal. 60 (2020)
10.1016/j.compeleceng.2018.03.037. March. 101625, https://doi.org/10.1016/j.media.2019.101625.
[95] Z. Zhu, H. Yin, Y. Chai, Y. Li, G. Qi, A novel multi-modality image fusion method [122] J. Jose, et al., An image quality enhancement scheme employing adolescent
based on image decomposition and sparse representation, Inf. Sci. 432 (2018) identity search algorithm in the NSST domain for multimodal medical image
516–529, https://doi.org/10.1016/j.ins.2017.09.010. fusion, Biomed. Signal Process Control 66 (2021), https://doi.org/10.1016/j.
[96] H.R. Shahdoosti, A. Mehrabi, Multimodal image fusion using sparse bspc.2021.102480. February.
representation classification in tetrolet domain, Digit. Signal Process. A Rev. J. 79 [123] J. Frp, J. Frp, J.D. Frp, E. Sdshu, G. Wkh, L. Ixvlrq, Comparison of medical image
(2018) 9–22, https://doi.org/10.1016/j.dsp.2018.04.002. fusion methods using image quaity metrics, in: 2018 Int. Conf. Commun. Comput.
[97] Y. Zhang, M. Yang, N. Li, Z. Yu, Analysis-synthesis dictionary pair learning and Internet Things 3, 2018, pp. 449–454.
patch saliency measure for image fusion, Signal Process. 167 (2020) 107327, [124] R. Singh, A. Khare, Multimodal medical image fusion using daubechies complex
https://doi.org/10.1016/j.sigpro.2019.107327. wavelet transform, in: 2013 IEEE Conf. Inf. Commun. Technol, ICT 2013, 2013,
[98] S. Maqsood, U. Javed, Multi-modal medical image fusion based on two-scale pp. 869–873, https://doi.org/10.1109/CICT.2013.6558217, no. Ict.
image decomposition and sparse representation, Biomed. Signal Process Control [125] W. Wang, F. Chang, A multi-focus image fusion method based on Laplacian
57 (2020) 101810, https://doi.org/10.1016/j.bspc.2019.101810. pyramid, J. Comput. 6 (12) (2011) 2559–2566, https://doi.org/10.4304/
[99] Y. Liu, X. Yang, R. Zhang, M.K. Albertini, T. Celik, G. Jeon, Entropy-based image jcp.6.12.2559-2566.
fusion with joint sparse representation and rolling guidance filter, Entropy 22 (1) [126] J. Sen Teh, A. Akhavan, A. Samsudin, High Quality Random Source Based on
(2020) 118, https://doi.org/10.3390/e22010118. Multi-Core CPUs and Chaotic Maps, 2006, pp. 1–2, https://doi.org/10.1049/el,
[100] N. Anantrasirichai, R. Zheng, I. Selesnick, A. Achim, Image fusion via sparse vol. 00, no. 00.
regularization with non-convex penalties, Pattern Recogn. Lett. 131 (2020) [127] S. Rajkumar, G. Malathi, A comparative analysis on image quality assessment for
355–360, https://doi.org/10.1016/j.patrec.2020.01.020. real time satellite images, Indian J. Sci. Technol. 9 (34) (2016), https://doi.org/
[101] H. Li, X. He, Z. Yu, J. Luo, Noise-robust image fusion with low-rank sparse 10.17485/ijst/2016/v9i34/96766.
decomposition guided by external patch prior, Inf. Sci. 523 (2020) 14–37, [128] V.P.S. Naidu, Discrete cosine transform-based image fusion, Defence Sci. J. 60 (1)
https://doi.org/10.1016/j.ins.2020.03.009. (2010) 48–54, https://doi.org/10.14429/dsj.60.105.
[102] S. Li, X. Kang, J. Hu, Image fusion with guided filtering, IEEE Trans. Image [129] Z. Wang, A.C. Bovik, A universal image quality index, IEEE Signal Process. Lett. 9
Process. 22 (7) (2013) 2864–2875, https://doi.org/10.1109/TIP.2013.2244222. (3) (2002) 81–84, https://doi.org/10.1109/97.995823.
[103] Z. Zhu, M. Zheng, G. Qi, D. Wang, Y. Xiang, A phase congruency and local [130] M. Gad, A. Zaki, Y.M. Sabry, 2017 , 34 th NATIONAL RADIO SCIENCE
Laplacian energy based multi-modality medical image fusion method in NSCT CONFERENCE Arab Academy for Science , Technology & Maritime Transport
domain, IEEE Access 7 (2019) 20811–20824, https://doi.org/10.1109/ Silicon photonic mid-infrared grating coupler based on silicon-on – insulator
ACCESS.2019.2898111. technology 2017 , 34 th NATIONAL RADIO SCIENCE CONFERENCE source
[104] Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao, L. Zhang, IFCNN: a general image fusion Gratin, in: Arab Acad. Sci. Technol. Marit. Transp. II., no. Nrsc, 2017,
framework based on convolutional neural network, Inf. Fusion 54 (2020) 99–118, pp. 400–406.
https://doi.org/10.1016/j.inffus.2019.07.011. August 2018. [131] L.F. Zoran, Quality evaluation of multiresolution remote sensing images fusion,
[105] S. Das, M.K. Kundu, Corrections to ‘A neuro-fuzzy approach for medical image UPB Sci. Bull. Ser. C Electr. Eng. 71 (3) (2009) 37–52.
fusion, IEEE Trans. Biomed. Eng. 62 (4) (2015) 1226, https://doi.org/10.1109/ [132] A. Kaur, L. Kaur, S. Gupta, Image recognition using coefficient of correlation and
TBME.2015.2405137. structural SIMilarity index in uncontrolled environment, Int. J. Comput. Appl. 59
[106] D. Mishra, B. Palkar, Image fusion techniques: a review, Int. J. Comput. Appl. 130 (5) (2012) 32–39, https://doi.org/10.5120/9546-3999.
(9) (2015) 7–13, https://doi.org/10.5120/ijca2015907084. [133] V. Radhika, K. Veera Swamy, S. Srininvas Kumar, Performance evaluation of
[107] S. Bhat, D. Koundal, Multi-focus image fusion techniques: a survey 54, Springer statistical measures for image fusion in spatial domain, in: 1st Int. Conf. Networks
Netherlands, 2021, 8. Soft Comput, ICNSC 2014, 2014, pp. 348–354, https://doi.org/10.1109/
[108] S. Masood, M. Sharif, M. Yasmin, M.A. Shahid, A. Rehman, Image fusion methods: CNSC.2014.6906716. Proc., no. Mi.
a survey, J. Eng. Sci. Technol. Rev. 10 (6) (2017) 186–194, https://doi.org/ [134] W. Pei, G. Wang, X. Yu, Performance evaluation of different references based
10.25103/jestr.106.24. image fusion quality metrics for quality assessment of remote sensing Image
[109] A.M. Sharma, A. Dogra, B. Goyal, R. Vig, S. Agrawal, From pyramids to state-of- fusion, in: Int. Geosci. Remote Sens. Symp., 2012, pp. 2280–2283, https://doi.
the-art: a study and comprehensive comparison of visible-infrared image fusion org/10.1109/IGARSS.2012.6351040.
techniques, IET Image Process. 14 (9) (2020) 1671–1689, https://doi.org/ [135] J. Liu, H. Wang, W. Qin, A new fusion image quality assessment based on edge
10.1049/iet-ipr.2019.0322. and structure similarity, in: 2011 IEEE Int. Conf. Cyber Technol. Autom. Control.
[110] P.K. Atrey, M.A. Hossain, A. El Saddik, M.S. Kankanhalli, Multimodal fusion for Intell. Syst. CYBER 2011, vol. 8, 2011, pp. 112–115, https://doi.org/10.1109/
multimedia analysis: A survey 16, 2010, 6. CYBER.2011.6011774.
[111] Y. Liu, X. Chen, R.K. Ward, Z.J. Wang, S. Member, Sparse representation, Comput. [136] G. Piella, H. Heijmans, A new quality metric for image fusion, in: IEEE Int. Conf.
Vis. 23 (12) (2021) 1185, https://doi.org/10.1007/978-3-030-63416-2_300100, Image Process. 3, 2003, pp. 173–176, https://doi.org/10.1109/
1185. icip.2003.1247209.
[112] N. Tawfik, H.A. Elnemr, M. Fakhr, M.I. Dessouky, F.E. Abd El-Samie, Survey study [137] J. Oliver, Objective image fusion performance measure, J. Chem. Inf. Model. 53
of multimodality medical image fusion methods, Multimed. Tool. Appl. 80 (4) (9) (2013) 1689–1699, https://doi.org/10.1017/CBO9781107415324.004.
(2021) 6369, https://doi.org/10.1007/s11042-020-08834-5, 6396.
21

A Review On Multimodal Medical Image Fusion Compendious Analysis

Uploaded by

Copyright:

Available Formats

A Review On Multimodal Medical Image Fusion Compendious Analysis

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A Review On Multimodal Medical Image Fusion Compendious Analysis

Uploaded by

Copyright:

Available Formats

Computers in Biology and Medicine 144 (2022) 105253

Contents lists available at ScienceDirect

Computers in Biology and Medicine

A review on multimodal medical image fusion: Compendious analysis of

Dolly JM et al. 2019 Classification: (Feature, Decision, and Pixel level). ✓ ✓

Fig. 2. Arrangement of sections encapsulated in this article.

Fig. 3. Biomedical image modalities w.r.t. electromagnetic spectrum.

2. Medical imaging modalities

In the medical field, each imaging modality has unique information

Section 2 contains a full discussion of medical modalities. Section 3

Table 2 of protons. Hence, more detailed images consist of many sequences of

2.4. Multimodal imaging

Fig. 5. Classification of medical imaging modalities.

Fig. 11. Overall multimodal medical fusion procedure.

Fig. 12. Medical image fusion step-by-step process.

Fig. 13. Classification of medical image fusion techniques.

QE (x, y, f) = QW (x, y, f).QW (x .y .f )x

MRI/CT with PET/SPECT CT with MRI

He et al. [54] Spatial domain 2010 2.9546 – – – – –

*Where MI and SSIM are reference-based metrics while SD is non-reference-based metrics.

You might also like