This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TMI.2015.2437894, IEEE Transactions on Medical Imaging
IEEE TMI-2015-0061 1

LRTV: MR Image Super-Resolution with Low-

Rank and Total Variation Regularizations
Feng Shi§, Jian Cheng§, Li Wang, Pew-Thian Yap, Senior Member, IEEE, and Dinggang Shen*, Senior
Member, IEEE

constraints. Image SNR is proportional to voxel size and the

Abstract—Image super-resolution (SR) aims to recover high- square root of the number of averages in the voxel. Reducing
resolution images from their low-resolution counterparts for the voxel size from 2×2×2 mm3 to 1×1×1 mm3 will require 64
improving image analysis and visualization. Interpolation averages for similar SNR [11]. This requires significantly
methods, widely used for this purpose, often result in images with
blurred edges and blocking effects. More advanced methods such
longer scanning time, which may not be practical clinically.
as total variation (TV) retain edge sharpness during image A possible alternative approach to this problem is image
recovery. However, these methods only utilize information from post-processing. For this, interpolation methods (nearest
local neighborhoods, neglecting useful information from remote neighbor, linear, and spine) are generally employed due to
voxels. In this paper, we propose a novel image SR method that their simplicity. However, as pointed out in [1], interpolation
integrates both local and global information for effective image methods generally blur the sharp edges, introduce blocking
recovery. This is achieved by, in addition to TV, low-rank
regularization that enables utilization of information throughout
artifacts in lines, and are unable to recover fine details. In view
the image. The optimization problem can be solved effectively via of this, we take a super-resolution (SR) approach for
alternating direction method of multipliers (ADMM). resolution enhancement of LR images [3]. Interpolation
Experiments on MR images of both adult and pediatric subjects methods are not considered as SR methods since they do not
demonstrate that the proposed method enhances the details in the consider the image degradation process (e.g., blurring, and
recovered high-resolution images, and outperforms methods such down-sampling). Multi-frame SR algorithms reconstruct a HR
as the nearest-neighbor interpolation, cubic interpolation,
iterative back projection (IBP), non-local means (NLM), and TV-
image from multiple LR images [12, 13]. These LR images
based up-sampling. are typically acquired repeatedly with slightly shifted field of
view (FOV). In this paper, we propose a single-frame SR
Index Terms—   Image enhancement, spatial resolution, image algorithm that requires the acquisition of only one LR image.
sampling, matrix completion, sparse learning A number of methods have been proposed for single-image
SR [3]. For example, iterative back-projection (IBP) was
proposed to estimate the HR image by back projecting the
I. INTRODUCTION difference between the LR image simulated based on the

H IGH-resolution (HR) medical images provide rich

structural details that are critical for accurate image post-
processing and pathological assessment of bodily organs [1-
estimated HR image via imaging blur and the input LR image
[14]. This process is repeated to minimize the energy of the
difference. Non-local means (NLM) is a method proposed to
10]. However, image resolution is limited by factors such as take advantage of image self-similarity [15]. Specifically, the
imaging hardware, signal to noise ratio (SNR), and time input LR image is first denoised and the similar patches are
used to reconstruct to a HR image. A correction step is then
Manuscript received January 22, 2015; accepted May 17, 2015. This work
applied to ensure that the down-sampled HR image is close to
was supported in part by National Institutes of Health grants MH100217, the denoised LR image. The reconstruction and correction
EB006733, EB008374, EB009634, AG041721, AG042599, MH088520, and steps are iterated in a multi-scale manner. In another work,
the National Research Foundation under Grant 2012-005741 from the Korean
NLM was employed to enhance the resolution of a single LR
government. §F. Shi and J. Cheng contributed equally to this work.
*
corresponding author. T2 image with the guidance from an HR T1 image [16].
F. Shi, L. Wang, and P.-T. Yap are with the Department of Radiology and Matrix completion algorithms have recently been shown to
Biomedical Research Imaging Center, University of North Carolina, Chapel be effective in estimating missing values in a matrix from a
Hill, NC 27599, USA (e-mails: fengshi, li_wang,
pewthian_yap@med.unc.edu). small sample of known entries [17-19]. For instance, it has
J. Cheng is with Department of Radiology and Biomedical Research been applied to the famous Netflix problem where one needs
Imaging Center, University of North Carolina, Chapel Hill, NC 27599, USA, to infer user preference for unrated movies based on only a
and also with Section on Tissue Biophysics and Biomimetics, PPITS, National
Institute of Child Health and Human Development, NIH, Bethesda, MD, USA small number of rated movies [20]. Matrix completion
(e-mail: jian.cheng.1983@gmail.com). methods assume that the recovered matrix has low rank and
D. Shen is with the Department of Radiology and Biomedical Research then uses this property as a constraint or regularization to
Imaging Center (BRIC), University of North Carolina at Chapel Hill, NC
27599 USA, and also with the Department of Brain and Cognitive
minimize the difference between the given incomplete matrix
Engineering, Korea University, Seoul 136-713, Korea and the estimated matrix. Candes et al. proved that, low-rank
(dinggang_shen@med.unc.edu)

0278-0062 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TMI.2015.2437894, IEEE Transactions on Medical Imaging
IEEE TMI-2015-0061 2

matrices can be perfectly recovered from a small number of

given entries under some conditions [18]. In image processing,
one advantage of matrix completion is that the remote
information from the whole image can be utilized in image
recovery. Besides, matrix completion has been widely applied
to image/video in-painting and decoding problems. However,
low-rank regularization is limited when the matrix to be
recovered contain rows or columns that are entirely missing.
As shown in Fig. 1, although voxels occluded by the words
“Low Rank” can be recovered very well, the voxels masked Fig. 1. Recovering the missing values using low-rank matrix completion [14].
by white horizontal and vertical lines cannot be recovered. In The red arrows mark the horizontal and vertical lines that the algorithm fails
to recover.
this case, each missing row or column will be simply filled by
regularization.
an arbitrary combination of other known rows or columns to
meet the low rank requirement. This precludes the application A. Super-Resolution Image Reconstruction Framework
of low-rank regularization to SR problem where missing As illustrated in Fig. 2, the acquired image is affected by
values for many rows and columns need to be recovered in the factors such as motion blur, field inhomogeneity, acquisition
process of recovering a HR image from a LR image. time, and noise.
The limitation of low-rank regularization in SR problem The observation model could be mathematically formulated
could be remedied by imposing the additional local spatial as:
consistency. Although local information may not be useful in
applications such as the Netflix problem, where different rows 𝑇 = 𝐷𝑆𝑋 + 𝑛 (1)
(e.g., users) can be considered independently, it is valuable in where T denotes the observed LR image, D is a down-
recovering images. One possible choice is total variation (TV) sampling operator, S is a blurring operator, X is the HR image
[21], defined as the integral of the absolute gradients of the that we want to recover, and n represents the observation
image. Recently, TV has been applied to avoid the ringing and noise. In the case of single-image SR, the HR image can be
smearing caused by interpolation [22]. estimated using this observation model by minimizing the
In this paper, we propose a novel low-rank total variation following cost function:
method, referred to as LRTV, for recovering a HR image from
!
a LR image. Our method 1) explicitly models the blurring 𝑋 = 𝑎𝑟𝑔 𝑚𝑖𝑛! 𝐷𝑆𝑋 − 𝑇 (2)
effects when an image is down-sampled, 2) combines both This is a data fidelity term for penalizing the difference
low-rank and TV regularizations for more effective image between the degraded HR image X and the observed LR image
recovery, and 3) works for 3D or 4D images by a tensor T. Since this is an ill-posed inverse problem, regularization
formulation [19]. A preliminary version of this work was terms are added to stabilize the solution. The cost function is
presented at a conference [23]. The work is significantly thus rewritten as:
extended in this article with more methodological details,
!
validations, and discussions. Experiments on MR images of 𝑋 = 𝑎𝑟𝑔 min! 𝐷𝑆𝑋 − 𝑇 + 𝜆ℜ 𝑋 (3)
both adults and pediatric subjects are conducted, and the
where ℜ 𝑋 is the regularization term often defined based on
proposed method is compared to various interpolation
prior knowledge. The parameter λ is used to balance the
methods, as well as the NLM and TV-based up-sampling
contributions of the fidelity term and the regularization term.
methods.

II. METHOD B. Low-Rank Total-Variation (LRTV) Method

We first describe how image degradation processes such as The proposed LRTV method is formulated as follows:
blurring and down-sampling effects are modeled. We then !
𝑋 = 𝑎𝑟𝑔 min! 𝐷𝑆𝑋 − 𝑇 + 𝜆!"#$ 𝑅𝑎𝑛𝑘(𝑋) +
describe the solution for the inverse problem of recovering the
𝜆!" 𝑇𝑉 𝑋 (4)
HR image from the LR image, using low-rank and TV
where the regularization can be separated into low-rank and

Fig. 2. Observation model in MR image acquisition process.

0278-0062 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TMI.2015.2437894, IEEE Transactions on Medical Imaging
IEEE TMI-2015-0061 3

total variation terms. 𝜆!"#$ and 𝜆!" are the respective tuning Algorithm 1. Low-Rank Total Variation (LRTV) for MR Image
parameters for those two terms. Super-Resolution
1) Low-Rank Regularization. The rank of a matrix is a Input: Low-resolution image T;
measure of nondegenerateness of the matrix, calculated by the Output: Reconstructed high-resolution image X;
a
Initialize: 𝑋 = 𝑢𝑝𝑠𝑎𝑚𝑝𝑙𝑒(𝑇) , 𝑀! = 0, 𝑌! = 0, i = 1,2,3
maximum number of linearly independent rows or columns in
Repeat
the matrix. The low-rank property implies that some rows or 1. Update X based on Eq. (9);
columns in the matrix can be linearly represented by other 2. Update M based on Eq. (11);
rows or columns, indicating redundant information in the 3. Update Y based on Eq. (12);
matrix. Low-rank prior can be used in matrix completion 4. Until difference in the cost function (Eq. (8)) is less than ε;
when only a subset of elements is known [18]. Since the rank End
a
of a matrix 𝑋 is a nonconvex function of 𝑋, a common The 𝑢𝑝𝑠𝑎𝑚𝑝𝑙𝑒(∙) operator is implemented by nearest-neighbor
interpolation.
approach is to approximate it using the trace norm 𝑋 !" ,
which leads to a convex optimization problem. Recently, Liu Based on ADMM in [24], the augmented Lagrangian of the
et al. extended the low-rank regularization to higher above cost function is written in below, where 𝑈! !
!!! are
dimensional images and further referred to as tensor Lagrangian parameters:
completion [19]. Basically, a N-dimensional image 𝑋 can be 𝑚𝑖𝑛!, ! ! ! 𝐷𝑆𝑋 − 𝑇 !
+
! !!! , !! !!!
seen as a high-order tensor. Since it is an NP-hard problem to
! !
compute the rank of a high-order tensor1, the rank is then 𝜆!"#$ !!! 𝛼! 𝑀! ! !" + 𝜆!" 𝑇𝑉 𝑋 + !!! 𝑈! 𝑋! −
!
approximated as the combination of trace norms of all 𝑀! ! + ! !!! ! 𝑋 − 𝑀!
!
(7)
matrices unfolded along each dimension:
We further set 𝑌! = (1/𝜌)𝑈! , and combine the last two
!
 𝑅𝑎𝑛𝑘 𝑋 = !!! 𝛼! 𝑋! !"
(5) linear and quadratic terms for simplicity:
!
where N is the number of image dimensionality. In this study, 𝑚𝑖𝑛!, ! ! ! 𝐷𝑆𝑋 − 𝑇 +
! !!! , !! !!!
we use 3D MR images and thus 𝑁 = 3. 𝛼! !
!!! are 𝜆!"#$ ! ! !
! !!! 𝛼! 𝑀!(!)
!"
+ 𝜆!" 𝑇𝑉 𝑋 + !!! ! 𝑋−
parameters satisfying 𝛼! ≥ 0 and !!! 𝛼! = 1. 𝑋(!) is the ! !
𝑀! + 𝑌! − 𝑌! (8)
unfolded X along the i-th dimension: 𝑢𝑛𝑓𝑜𝑙𝑑! 𝑋 = 𝑋(!) . For
example, a 3D image with size of 𝑈×𝑉×𝑊 can be unfolded According to ADMM [24], we break Eq. (8) into three sub-
into three 2D matrices, with sizes of 𝑈×(𝑉×𝑊), 𝑉×(𝑊×𝑈), problems below that could be solved by iteratively updating
and 𝑊×(𝑈×𝑉), respectively. 𝑋(!) is the trace norm of the the variables. In the below, k denotes the current iteration step.
!!
Subproblem 1: Update 𝑋 (!!!) by minimizing:
matrix X (!) .
! ! !
2) Total-Variation Regularization. Total variation was 𝑎𝑟𝑔 𝑚𝑖𝑛! 𝐷𝑆𝑋 − 𝑇 +𝜆!" 𝑇𝑉 𝑋 + !!! ! 𝑋−
proposed as a regularization approach to remove noise and (!) (!) !
handle proper edges in images [21]. It is defined as the 𝑀! + 𝑌! (9)
integral of the absolute gradients of an image: 𝑇𝑉 𝑋 = This subproblem can be solved by gradient descent, where
∇𝑋 𝑑𝑥𝑑𝑦𝑑𝑧. Minimizing TV will enforce local spatial the gradient of TV term is obtained from the associated Euler-
consistency in image recovery, remove noise, and preserve Lagrange equation [22].
edges. (!!!)
!
Subproblem 2: Update 𝑀! by minimizing:
!!!
C. LRTV Optimization
!
We use the alternating direction method of multipliers 𝑚𝑖𝑛 𝜆!"#$ 𝛼! 𝑀!(!)
(ADMM) algorithm to solve the cost function in Eq. (4). !
!! !!! !!! !"

ADMM is proven to be efficient for solving optimization ! 𝜌 (!!!) (!) !

+ 𝑋 − 𝑀! + 𝑌!
problems with multiple non-smooth terms in the cost function !!! 2
[24]. First, we introduce 𝑁 redundant variables 𝑀! ! !!! to
simulate X in each dimension i, by requiring that the unfolded (10)
𝑋 along the i-th dimension 𝑋(!) should be equal to the which can be solved using a close-form solution according to
unfolded 𝑀! along this dimension 𝑀!(!) . The new cost function [17]:
is as follows: (!!!) (!)
𝑀! = 𝑓𝑜𝑙𝑑! 𝑆𝑉𝑇!!"#$!! ! 𝑋(!) + 𝑌!(!) (11)
! !
𝑚𝑖𝑛!, ! ! 𝐷𝑆𝑋 − 𝑇 + 𝜆!"#$ !!! 𝛼! 𝑀!(!) +
! !!! !" where fold! (∙) is the inverse operator of unfold! (∙), i.e.,
𝜆!" 𝑇𝑉 𝑋 , subject to 𝑋(!) = 𝑀!(!) , 𝑖 = 1, … , 𝑁 (6)
fold! M! ! = M! . SVT ∙ is the Singular Value Thresholding
operator [17] using λ!"#$ α! ρ as the shrinkage parameter.
!
(!!!)
Subproblem 3: Update Y! by:
!!!
1
http://en.wikipedia.org/wiki/Tensor_rank_decomposition

0278-0062 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TMI.2015.2437894, IEEE Transactions on Medical Imaging
IEEE TMI-2015-0061 4

Fig. 4. Simulation of low-resolution image from high-resolution image.

Fig. 5. Parameter optimization based on ten images. SNR was used to evaluate
the reconstruction performance.

Fig. 3. Low-rank approximations of brain images. Top row shows the original brain images could be represented using their low-rank
image, singular-value plot, and zoomed singular-value plot of indices from
141 to 181. Bottom row shows the four reconstructed images and their approximations with a relatively high SNR.
differences with the original image by using top 30, 60, 90, and 120 singular
values, respectively. B. Experimental Settings
We applied our method to a set of down-sampled and
(!!!) (!) (!!!)
𝑌! = 𝑌! + 𝑋 (!!!) − 𝑀! (12) blurred 3D brain images and evaluated whether our method
can successfully recover the original high-resolution images.
III. EXPERIMENTAL RESULTS To do that, we use HR images as ground truth, and simulate
LR images as shown in Fig. 4. Blurring was implemented
A. Low-Rank Approximation for Brain Images
using a Gaussian kernel with a standard deviation of 1 voxel.
We first evaluated whether brain images can be sufficiently The blurred image was then down-sampled by averaging
characterized using their low-rank approximation. We selected every 8 voxels (to simulate the partial volume effect),
a representative 2D axial slice from the T1 MR phantom in resulting in half of the original resolution. The quality of
Brainweb2, which has a size of 181×181 with in-plane reconstruction of all methods from the input LR images was
resolution of 1 mm (Fig. 3). We then performed singular value evaluated by comparing with their corresponding original HR
decomposition (SVD) on this image and obtain 174 non-zero images.
singular values. As shown in Fig. 3, singular values decrease A number of comparison methods were also employed,
dramatically, with most values being close to zero. Next, we including nearest-neighbor interpolation (NN), spline
remove the small singular values and use the top 30, 60, 90, interpolation (Spline), IBP based up-sampling [14], NLM
and 120 singular values to reconstruct the image. Note that the based up-sampling [15], and TV based up-sampling [22]. The
number of used singular values equals to the rank of the estimated HR images from all methods were compared with
recovered image, implying that the recovered image is a low- the original HR image for accuracy of image recovery by
rank approximation of the original image. The reconstructed using SNR. Note that, for NLM, we used the implementation
images are shown in Fig. 3 for visual inspection and the made available by the authors3. TV-based up-sampling is
difference maps between original and reconstructed images realized by setting 𝜆!"#$ = 0 and 𝜌 = 0 in the proposed
are also provided. Signal-to-noise ratio (SNR) in decibels (dB) method and solving only the subproblem 1. Other methods
is used to evaluate the quality of reconstruction: 𝑆𝑁𝑅 = 20 ∗ were implemented by in-house tools.
𝑙𝑜𝑔10( 𝑓 𝑓 − 𝑔 ), where f is the original HR image and g Parameters were optimized based on a small dataset,
is the recovered HR image. consisting of 5 adult and 5 pediatric images as described in the
The results show that, by using the top 60 singular values, the Real Data Evaluation section. In particular, α! !!!! are weights
reconstructed image has high SNR (34.0db), although small to combine unfolded matrices along each spatial dimension in
edge information in the brain boundary is lost. When using the rank computation. All dimensions are assumed to be equally
top 90 or 120 singular values (out of 174), the resulting image important, i.e., 𝛼! = 𝛼! = 𝛼! = 1/3. The difference between
does not show visual differences with respect to the original iterations was measured by 𝑋 ! − 𝑋 !!! 𝑇 , and the
image. For the 3D Brainweb image with size 181×217×181, it
program was stopped when this difference is less than
has three ranks for its three unfolded matrices that are less
𝜀 = 1𝑒 − 5. Since TV is a major component of the proposed
than its longest image size 217. These ranks are relatively low
method, we first optimized TV by setting 𝜆 !" from a group of
in comparison to the total number of elements, suggesting
candidate values (Fig. 5). We thus chose 𝜆 !" = 0.01. Then,

2 3
http://www.bic.mni.mcgill.ca/brainweb/ https://sites.google.com/site/pierrickcoupe/

0278-0062 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TMI.2015.2437894, IEEE Transactions on Medical Imaging
IEEE TMI-2015-0061 5

Fig. 6. Illustration of upsampling results for simulated data. LR data with 2×2×2 mm3 resolution was upsampled to 1×1×1 mm3. A typical slice for each of
coronal, sagittal, and axial views is shown, and a zoom-up of frontal region in sagittal view is also provided.
we optimized the proposed method by employing the same
𝜆 !" while setting 𝜆!"#$ from a wide range of candidate values
(Fig. 5). We chose the rank regularization as 𝜆!"#$ = 0.01. As
for the input arguments, the blurring kernel (such as Gaussian
distribution with 1 voxel standard deviation) was used in IBP,
TV, and the proposed method. On the other hand, the default
parameters in NLM implementation were used [15]. Both NN
and Spline do not have any free parameters that require tuning.
We evaluate the performance of the proposed and
comparison methods by comparing the recovered HR images
with the original HR image. Besides SNR, we also employ Fig. 7. Results of upsampling on simulate data with different noise levels. (A)
shows the SNR result, and (B) shows the SSIM result.
another image quality measurement named Structural
Similarity Index (SSIM), which is considered to be correlated 5%, 7%, and 9% of the maximum intensity. The noise in the
with the quality perception of the human visual system (HVS) phantom images has Rayleigh statistics in the background and
[25]. SSIM has been used in many image SR studies [15, 26- Rician statistics in the signal regions. For our experiment, we
28]. SSIM is defined as: generated images with 2×2×2 mm3 resolution using the above-
𝑆𝑆𝐼𝑀 𝑓, 𝑔 = mentioned LR image simulation pipeline (Fig. 4), and
upsample it again to 1 mm isotropic resolution using the
2𝜇! 𝜇! + 𝑐! 2𝜎!" + 𝑐! 𝜇!! + 𝜇!! + 𝑐! 𝜎!! + 𝜎!! + 𝑐! ,
proposed and comparison methods, respectively.
where 𝜇! and 𝜇! are the mean values respectively in the Fig. 6 demonstrates the results when using noisy data at
original HR image f and recovered image g, 𝜎!! and 𝜎!! are the level of 3% as input image. A typical slice is shown for each
variances, σ!" is the covariance of two images, 𝑐! = 𝑘! 𝐿 ! of coronal, sagittal, and axial views, and the frontal region in
and 𝑐! = 𝑘! 𝐿 ! with 𝑘! = 0.01 and 𝑘! = 0.03, and L is the sagittal view is zoomed up for better visual inspection.
dynamic range of voxel values [25]. SSIM ranges from 0 to 1, Compared to the original HR image, results of NN, Spline,
and 1 means perfect recovery. IBP, and NLM appear blurry. TV provides better image
reconstruction while the proposed method shows more fine
C. Phantom Data Evaluation
details. Fig. 7 shows the SNR and SSIM measurements for all
We employed the T1 MR phantom from Brainweb to methods while changing the noise level from no noise to 9%
evaluate the recovery performance of the proposed and noise level. The proposed method outperforms all other
comparison methods in cases of no noise and with noise. The comparison methods in all noise levels for both SNR and
phantom has image size of 181×217×181 and spatial SSIM. Another observation is that, the SNR improvement of
resolution of 1×1×1 mm3. We downloaded images both the proposed method over other methods generally reduces
without noise and with noise at different levels of 1%, 3%, when noise level increases, while the SSIM improvement

0278-0062 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TMI.2015.2437894, IEEE Transactions on Medical Imaging
IEEE TMI-2015-0061 6

Fig. 8. Results for upsampling an adult image (upper panel) and a pediatric image (lower panel) with different methods. In each panel, the first row shows the
input image and the results by various methods, while the second row shows the close-up views of selected regions in the first row.

Fig. 10. Boxplot of SNR and SSIM results for recovering adult data in groups
of AD, MCI, and NC.

subjects from ADNI4, with 15 from Alzheimer's disease (AD),

15 from mild cognitive impairment (MCI), and 15 from
normal controls (NC). Their ages were 75±8 years old at MRI
scan. T1 MR images were acquired with 166 sagittal slices at
the resolution of 0.95×0.95×1.2 mm3. Second, we also
randomly selected 45 pediatric subjects from NDAR
(http://ndar.nih.gov/), with age of 11±3 years old at MRI scan.
Fig. 9. Boxplot of SNR and SSIM results for recovering adult data and
pediatric data using different methods. The proposed LRTV method
T1 MR images were acquired with 124 sagittal slices at the
significantly outperforms all other comparison methods (p<0.01 using two- resolution of 0.94×0.94×1.3 mm3.
sample t-tests). Fig. 8 shows the representative image SR results of an adult
scan (upper panel) and a pediatric scan (lower panel). From
maintains a similar level when the noise level increases.
left to right, the first row of each panel shows the input image,
D. Real Data Evaluation original HR image, the results of NN, Spline, IBP, NLM, TV,
We further evaluated the proposed method on two publicly and the proposed LRTV method. The close-up views of
available datasets. First, we randomly selected 45 adult
4
http://www.loni.ucla.edu/ADNI

0278-0062 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TMI.2015.2437894, IEEE Transactions on Medical Imaging
IEEE TMI-2015-0061 7

Fig. 11. Upsampling of an image from a typical AD subject. The resolution was thus increased from 0.95×0.95×1.2 mm3 to 0.475×0.475×0.6 mm3 . The first row
shows the reconstructed results and other rows show the close-up views of the regions selected from the first row.

selected regions are also shown for better visualization. It can

E. Computational Time
be observed that the results of NN and Spline interpolation
methods show severe blurring artifacts. The contrast is All programs were run in Linux environment on a standard
enhanced in the results of IBP, NLM, and TV up-sampling PC using a single thread of an Inter® Xeon® CPU (E5630 1.6
methods, while the proposed LRTV method best preserves GHz). Interpolation methods are computationally efficient as
edges and achieves the highest SNR values. it takes around 2 seconds for NN and 10 seconds for Spline for
Quantitative results on the images of 45 adults and 45 one 3D image. For the up-sampling methods, IBP takes 1 min,
pediatric subjects are shown in Fig. 9. Note that the original NLM takes 16 mins, TV takes 5 mins, and the proposed
images were used as ground truth and the input was LR image method takes about 30 mins.
simulated following the pipeline of Fig. 4. The proposed
method significantly outperforms all comparison methods IV. DISCUSSION
(p<0.01 using two-sample t-tests). Results on adult subjects We have presented a novel super-resolution method for
demonstrate less variance and higher accuracy than those on recovering high-resolution image from a single low-resolution
pediatric subjects, which may be because the image quality is image. For the first time, we show that combining low-rank
higher in the matured brain and also clearer gyri/sulci patterns and total-variation regularizations is a viable solution to the
appear in the adult images. No significant difference was SR problem. This combination brings together global and
found between the adult subjects of AD, MCI, and NC, as local information for effective recovery of the high-
shown in Fig. 10. dimensional image. Experimental results indicate that the
In addition, we applied our method directly to images with proposed method is able to remedy the partial volume effect
typical imaging resolution. Fig. 11 shows the results of a and recover the fine brain structure details from both adult and
typical AD subject for visual inspection, along with the close- pediatric images. Quantitative comparisons show that the
up views of selected representative regions. As it can be proposed method outperforms other popular methods such as
observed, the proposed method recovers fine details and also the interpolation methods based on NN and Spline, and the up-
preserves image edges. sampling methods based on IBP, NLM, and TV.

0278-0062 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TMI.2015.2437894, IEEE Transactions on Medical Imaging
IEEE TMI-2015-0061 8

The proposed method is computationally expensive [18] E. J. Candès and B. Recht, "Exact matrix completion via convex
optimization," Foundations of Computational mathematics, vol. 9,
compared with other methods. The major reason is that the pp. 717-772, 2009.
program spends most of the time solving TV, which is [19] J. Liu, P. Musialski, P. Wonka, and J. Ye, "Tensor Completion for
implemented using gradient decent [22] in MATLAB. In Estimating Missing Values in Visual Data," IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 35, pp. 208-220,
future work, we will implement a faster TV solver, such as
2013.
[29], and also accelerate the program using C++. On the other [20] Netflix, Netflix prize webpage, http://www.netflixprize.com/, 2007.
hand, we’d like to clarify that our main purpose is to show that [21] L. I. Rudin, S. Osher, and E. Fatemi, "Nonlinear total variation
the TV-based MR image reconstruction can be improved by based noise removal algorithms," Physica D: Nonlinear
Phenomena, vol. 60, pp. 259-268, 1992.
combining low-rank regularization. When a better TV solver [22] A. Marquina and S. J. Osher, "Image super-resolution by TV-
is available, the performance of the proposed method could regularization and Bregman iteration," Journal of Scientific
also be improved. Computing, vol. 37, pp. 367-382, 2008.
[23] F. Shi, J. Cheng, L. Wang, P.-T. Yap, and D. Shen, "Low-Rank
The proposed method is mainly developed for single-image Total Variation for Image Super-Resolution," in Medical Image
SR. In the future, we will extend the proposed method to use Computing and Computer-Assisted Intervention–MICCAI 2013,
multiple LR images [12, 13], training datasets [30], and also ed: Springer, 2013, pp. 155-162.
[24] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein,
for more applications such as 4D image recovery in functional "Distributed optimization and statistical learning via the alternating
MRI or diffusion MRI. The code for the proposed method will direction method of multipliers," Foundations and Trends in
be released at our website5. Machine Learning, vol. 3, pp. 1-122, 2011.
[25] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image
quality assessment: from error visibility to structural similarity,"
REFERENCES Image Processing, IEEE Transactions on, vol. 13, pp. 600-612,
[1] J. Van Ouwerkerk, "Image super-resolution survey," Image and 2004.
Vision Computing, vol. 24, pp. 1039-1052, 2006. [26] I. Begin and F. P. Ferrie, "Comparison of super-resolution
[2] H. Greenspan, "Super-resolution in medical imaging," The algorithms using image quality measures," in Computer and Robot
Computer Journal, vol. 52, pp. 43-63, 2009. Vision, 2006. The 3rd Canadian Conference on, 2006, pp. 72-72.
[3] S. C. Park, M. K. Park, and M. G. Kang, "Super-resolution image [27] S. Yang, Z. Liu, M. Wang, F. Sun, and L. Jiao, "Multitask
reconstruction: a technical overview," IEEE Signal Processing dictionary learning and sparse representation based single-image
Magazine, vol. 20, pp. 21-36, 2003. super-resolution reconstruction," Neurocomputing, vol. 74, pp.
[4] I. Bankman, Handbook of medical image processing and analysis: 3193-3203, 2011.
academic press, 2008. [28] Q. Yuan, L. Zhang, and H. Shen, "Multiframe super-resolution
[5] A. Buades, B. Coll, and J.-M. Morel, "A review of image employing a spatially weighted total variation model," Circuits and
denoising algorithms, with a new one," Multiscale Modeling & Systems for Video Technology, IEEE Transactions on, vol. 22, pp.
Simulation, vol. 4, pp. 490-530, 2005. 379-392, 2012.
[6] K. Doi, "Computer-aided diagnosis in medical imaging: historical [29] T. Goldstein and S. Osher, "The split Bregman method for L1-
review, current status and future potential," Computerized medical regularized problems," SIAM Journal on Imaging Sciences, vol. 2,
imaging and graphics, vol. 31, pp. 198-211, 2007. pp. 323-343, 2009.
[7] B. Zitova and J. Flusser, "Image registration methods: a survey," [30] J. Yang, J. Wright, T. S. Huang, and Y. Ma, "Image super-
Image and vision computing, vol. 21, pp. 977-1000, 2003. resolution via sparse representation," Image Processing, IEEE
[8] F. Shi, Y. Fan, S. Tang, J. H. Gilmore, W. Lin, and D. Shen, Transactions on, vol. 19, pp. 2861-2873, 2010.
"Neonatal brain image segmentation in longitudinal MRI studies,"
Neuroimage, vol. 49, pp. 391-400, Jan 1 2010.
[9] F. Shi, L. Wang, G. Wu, G. Li, J. H. Gilmore, W. Lin, et al.,
"Neonatal atlas construction using sparse representation," Human
brain mapping, 2014.
[10] P.-T. Yap, H. An, Y. Chen, and D. Shen, "Fiber-driven resolution
enhancement of diffusion-weighted images," NeuroImage, vol. 84,
pp. 939-950, 2014.
[11] B. Scherrer, A. Gholipour, and S. K. Warfield, "Super-resolution
reconstruction to increase the spatial resolution of diffusion
weighted images from orthogonal anisotropic acquisitions,"
Medical image analysis, vol. 16, pp. 1465-1476, 2012.
[12] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Advances and
challenges in super‐resolution," International Journal of Imaging
Systems and Technology, vol. 14, pp. 47-57, 2004.
[13] S. Borman and R. L. Stevenson, "Super-resolution from image
sequences-a review," in Circuits and Systems, 1998. Proceedings.
1998 Midwest Symposium on, 1998, pp. 374-378.
[14] M. Irani and S. Peleg, "Improving resolution by image
registration," CVGIP: Graphical models and image processing,
vol. 53, pp. 231-239, 1991.
[15] J. V. Manjón, P. Coupé, A. Buades, V. Fonov, D. Louis Collins,
and M. Robles, "Non-local MRI upsampling," Med Image Anal,
vol. 14, pp. 784-792, 2010.
[16] F. Rousseau, "A non-local approach for image super-resolution
using intermodality priors," Med Image Anal, vol. 14, p. 594, 2010.
[17] J.-F. Cai, E. J. Candès, and Z. Shen, "A singular value thresholding
algorithm for matrix completion," SIAM Journal on Optimization,
vol. 20, pp. 1956-1982, 2010.

5
http://www.med.unc.edu/bric/ideagroup/free-softwares

0278-0062 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.