AUTOMATED DIAGNOSIS OF

RETINOPATHY BY CONTENT-BASED
IMAGE RETRIEVAL
EDWARD CHAUM, MD, PHD,*†‡§ THOMAS P. KARNOWSKI, MS,¶
V. PRIYA GOVINDASAMY, MS,¶ MOHAMED ABDELRAHMAN, PHD,**
KENNETH W. TOBIN, PHD¶

Purpose: To describe a novel computer-based image analysis method that is being

developed to assist and automate the diagnosis of retinal disease.
Methods: Content-based image retrieval is the process of retrieving related images from
large database collections using their pictorial content. The content feature list becomes
the index for storage, search, and retrieval of related images from a library based upon
specific visual characteristics. Low-level analyses use feature description models and
higher-level analyses use perceptual organization and spatial relationships, including
clinical metadata, to extract semantic information.
Results: We defined, extracted, and tested a large number of region- and lesion-based
features from a dataset of 395 retinal images. Using a statistical hold-one-out method,
independent queries for each image were submitted to the system and a diagnostic
prediction was formulated. The diagnostic sensitivity for all stratified levels of age-related
macular degeneration ranged from 75% to 100%. Similarly, the sensitivity of detection and
accuracy for proliferative diabetic retinopathy ranged from 75% to 91.7% and for nonpro-
liferative diabetic retinopathy, ranged from 75% to 94.7%. The overall purity of the
diagnosis (specificity) for all disease states in the dataset was 91.3%.
Conclusions: The probabilistic nature of content-based image retrieval permits us to
make statistically relevant predictions regarding the presence, severity, and manifestations
of common retinal diseases from digital images in an automated and deterministic manner.
RETINA 28:1463–1477, 2008

E arly detection and treatment of prevalent eye dis-

eases, in particular diabetic retinopathy (DR),
glaucoma, and age-related macular degeneration
eras, laser-based imaging platforms such as optical
coherence tomography and scanning laser ophthal-
moscopy and computer-assisted diagnostic methods
(AMD), can have a significant impact on reducing the generate high quality digital data sets that have the
incidence of vision loss and blindness worldwide. potential to greatly enhance our ability to effectively
New high quality mydriatic and nonmydriatic cam- screen large, at-risk populations for specific disease

From the Departments of *Ophthalmology, †Anatomy and Neu- United States Army Medical and Material Command, Telemedi-
robiology, ‡Biomedical Engineering, and §Pediatrics, University cine and Advanced Technology Research Center (W81XWH-05-
of Tennessee Health Science Center, Memphis, Tennessee; ¶Image 1-0409), by an unrestricted UTHSC Departmental grant from Re-
Science and Machine Vision Group, Oak Ridge National Labora- search to Prevent Blindness, New York, NY, Fight for Sight, New
tory, Oak Ridge, Tennessee; and **Tennessee Technological Uni- York, NY, and by The Plough Foundation, Memphis, TN.
versity, Cookeville, Tennessee. Reprint requests: Edward Chaum, MD, PhD, Hamilton Eye
These studies were supported in part by grants from Oak Ridge Institute, 930 Madison Ave. Suite 731, Memphis, TN 38163;
National Laboratory, the National Eye Institute, (EY017065), the e-mail: echaum@utmem.edu

1463
1464 RETINA, THE JOURNAL OF RETINAL AND VITREOUS DISEASES ● 2008 ● VOLUME 28 ● NUMBER 10

states and significantly impact disease morbidity. methods used to locate the ON in an automated fashion
Novel computer-based image analysis and diagnostic include location regression, pixel classification, and
methods that can leverage this data have the potential graph search algorithms.14,15 Approaches to digital im-
to improve the sensitivity and specificity of remote age analysis of the retina have included techniques such
retinal diagnosis in large patient populations, and to as dynamic contours,16 model-based approaches.17 and
monitor therapeutic responses to treatment in a robust, vascular segmentation18 to name just a few.
objective, and deterministic manner. The literature is sparse regarding methods for lo-
Currently, networking technology applied to the calizing the macula region, primarily due to the lack
remote assessment of retinal disease consists primarily of visual structure in red-free and color fundus imag-
of “store and forward” digital retinal photography ery.19 Detection of the macula typically uses ancillary
with subsequent image analysis by certified techni- imaging to highlight pigmentation and manually as-
cians at commercial and academic reading centers. sisted methods.20 Although the ON head and the vas-
The reading center paradigm is an established and culature have well-defined boundaries and are readily
validated method, with high sensitivity and specificity segmented,7,21 the segmentation of retinal lesions is
for retinal disease detection and quantification in clin- more challenging and is being extensively addressed
ical trials.1–3 However, limited health care resources, today.10,22–24 Features of the macula that are more
access to imaging platforms and eye care specialists, specific to disease such as microaneurysms or drusen,
and the absence of real time diagnostic potential re- are of particular interest for the development of auto-
main major obstacles to achieving the goals of popu- mated screening for retinopathy.
lation-based disease detection and management using Content-based image retrieval (CBIR) is the pro-
the reading center method. The application of com- cess of retrieving related images from very large da-
puter-based image analysis to aid disease diagnosis tabase collections, based on their pictorial con-
has the potential to facilitate retinal disease detection tent.25–27 Pictorial content is typically defined by a set
and management by increasing throughput, reducing of intrinsic features extracted from an image that
costs, and assisting or potentially automating diagnos- describe the color, texture, shape, and regional struc-
tic capabilities if they can be effectively integrated ture of the image or of specific objects, i.e., attributes
into a telemedical network delivery system. that the human eye keys on to comprehend the image.
Current work in automated image analysis has dem- The pictorial content feature list becomes the index for
onstrated the ability to detect both anatomic features storage and retrieval, and facilitates searches of large
and retinal lesions using color and monochromatic historical image datasets based upon the specific vi-
retinal images.4 Two excellent overviews of work in sual characteristics of a query image. The procedures
this area have been published.5,6 In many approaches, for indexing are depicted in Figure 1. An image (Fig-
the first step in automated digital analysis of retinal ure 1A), is analyzed to locate anatomic structures and
disease is the systematic and reliable location of im- pathologic lesions as in Figure 1B. Statistical features
portant anatomic structures in the eye. Key elements are extracted from these structures as in Figure 1C.
include the ability to normalize images to accommo- These statistical features comprise a number matrix
date illumination and contrast from various fundus that describe shapes, texture, orientation, etc., of the
cameras used for data acquisition.7 A common ap- structures and lesions that have been found. A table of
proach is to detect and segment the vascular struc- these features is built (i.e., a database library) repre-
ture8 –10 to exploit the geometric relationship that ex- senting the image population (Figure 1D). Finally an
ists between the vasculature and the position of the optic indexing tree (Figure 1E), is generated that facilitates
nerve (ON) in the retina.11–13 Other recently reported rapid searching. For retrieval, Figure 2 demonstrates

Fig. 1. Representation of the basic elements of indexing of images in a CBIR environment. For indexing, an image (A) is analyzed to locate anatomic
structures and lesions (B). Statistical features (e.g., shape, texture) are extracted (C), and a database is built of the image population (D). Finally an
indexing tree (E), is generated to facilitate rapid searching.
 AUTOMATED RETINAL DIAGNOSIS BY CBIR ● CHAUM ET AL 1465

Fig. 2. For retrieval, a query

image is submitted to locate
anatomic structures and le-
sions followed by feature ex-
traction and analysis. The fea-
tures become an index for
searching the library to locate
similar images from the data-
base. The CBIR method per-
mits the use of both image
data and nonimage clinical
metadata.

how a digital query image is submitted and anatomic approximate nearest neighbors.29 Whereas an exhaus-
structures and lesions are found, followed by feature tive nearest-neighbor search of the n vectors (i.e.,
extraction and analysis. The features become an index patient records) in the database would be of O(n)
for searching in the database (CBIR “Library”) and computations, the approximate nearest neighbors ap-
are used to search the indexing tree generated in proach provides a more efficient search procedure
Figure 1 to locate a population of similar images from through the database of O(log(n)). This increased
the library. An additional advantage of the CBIR computational efficiency becomes significant when
method is the potential to use associated nonimage the number of records maintained in the system be-
data (e.g., clinical metadata such as disease history, comes large. For example, the approximate nearest
previous treatments, and visual acuity) through which neighbors tree method will show a 4,600-fold compu-
diagnostics capabilities begin to emerge. tational efficiency over brute force computation in a
Low-level image processing uses numerical de- library of 50,000 records, permitting real time analy-
scriptive object analysis to generate statistical fea- sis. The CBIR method provides a unique ability to
tures. These features are derived from the groups of describe “off-normal” occurrences in images. We
lesion pixels that have been segmented from the image have previously applied this to manufacturing envi-
using image processing and analysis. Features provide ronments25; however, the ability to identify and diag-
a representation of the “meaning” of objects that have nose retinal pathology based on deviation from normal
been located in the image that, in general, describe anatomy is inherent in the method.
attributes such as object texture, shape, and color In this article, we present an analytical method that
content. Once low-level analysis is complete, a high- automatically describes the anatomy of human retinal
level analysis is performed that maps the low-level images, localizes the critical regions of the macula and
objects and their statistical features to conform to the ON, and which detects and stratifies retinal diseases in
intended significance or meaning of these image ob- statistically significant and quantifiable terms. Our
jects and their features. This idea of meaning, or goal is to adequately describe key regions of the retina
semantics, attempts to map the statistical feature de- such that these statistical features can be used to index
scriptions of image populations to an image database images in a CBIR library that can be used for com-
index that captures the semantic intent of the user. puter-aided and potentially automated diagnosis of
Semantic mapping is performed using principle com- retinal disease.
ponent analysis and linear discriminant analysis28 to
reduce many features to a few succinct indices, al- Methods
though the indices are organized into a search tree.
There are many ways to implement search trees. For The anatomy of the retina is such that many of the
our CBIR research, we are using a kd-tree method of visually significant lesions that occur in DR, glau-
1466 RETINA, THE JOURNAL OF RETINAL AND VITREOUS DISEASES ● 2008 ● VOLUME 28 ● NUMBER 10

coma, and AMD can be detected by changes in the are shown. The retrievals are mapped in a figurative
ON and macula bordered by the temporal retinal vas- “feature space” according to their degree of similarity
cular arcades. Normalized digital retinal images can to the query image “q” (here shown in two vector
identify and record specific retinal features based upon dimensions Vy and Vx, upper right). The frequency
differences in location, contrast, and sharpness, of distribution of the various states “␻” in the population
detected lesions and groups of lesions (e.g., dark and of retrieved images is then determined (lower right),
bright lesion populations), and the absence of ex- providing a statistical method for determining the
pected features, to name a few. Our method for re- confidence limits.
trieving images on the basis of lesion populations (as An image exhibiting both dark (hemorrhages) and
one example) computes statistical values related to the light (exudates) lesions associated with nonprolifera-
number, size, orientation, etc., of lesions with regard tive diabetic retinopathy (NPDR) is used as a query in
to a macular-centered coordinate system. Other fea- Figure 4A. The top row images (Figure 4B) and
tures that are indirectly related to the lesion popu- (Figure 4C), show similar segmented examples re-
lation measure the characteristics of blood vessels turned from the library (all with similar states of
in the neighborhood of lesions, e.g., the average NPDR). In the bottom row, an image exhibiting bright
density or area of the vessel structure near or pass- lesions associated with AMD is used as a query in
ing through a lesion. These indirect measurements Figure 4A. The bottom row images (Figure 4B) and
of lesion properties help to place the lesion into the (Figure 4C), show the nearest examples returned (all
context of the retinal structure, therefore more fully diagnosed with moderate AMD). Note in these images
describing the lesions and their potential impact on that the demarcations in Figure 4B and 4C represent
vision. hand-labeled lesions from a set of data used to per-
A search through the database of retinal images is form testing and verification of the CBIR approach.
performed using the query-by-example method of When the image data were ground-truthed, different
submitting an uncharacterized digital fundus image to categories of lesions outlined by the retina specialist
the “query engine,” an algorithm which then retrieves are given different color designations (not shown in
a population of characterized data (in this case, retinal this image). This dataset represents an ophthalmolo-
images) that is similar to the unknown in terms of the gist-reviewed population that is particularly useful for
feature descriptions. For example, a query image with providing validation and quantification of CBIR per-
a few dark lesions in the central macula should locate formance. This dataset is further described in the
similar images with a few dark lesions in the central Results section.
macula in the library. This is demonstrated in the The numbers of features that can be used for char-
schematic in Figure 3, where a query image and the acterizing the structures of images are extensive.
first six of its corresponding most-similar retrievals Techniques used to quantify texture within images

Fig. 3. Schematic showing

the retrieval and analysis pro-
cess for a query image. An
example of the corresponding
most-similar retrievals from
the library is shown. The re-
trievals are mapped in a figu-
rative feature space according
to their degree of similarity to
the query image q (here
shown in just two vector di-
mensions Vy and Vx, upper
right). The frequency distri-
bution of the various states ␻
in the population of retrieved
images is then determined
(lower right), providing a sta-
tistical method for determin-
ing the confidence limits.
 AUTOMATED RETINAL DIAGNOSIS BY CBIR ● CHAUM ET AL 1467

Fig. 4. Examples of queries

in A and corresponding re-
trievals in B and C for a case
of nonproliferative diabetic
retinopathy with macular
edema (top row) and age-re-
lated macular degeneration
(AMD, bottom row). In each
row, an image of NPDR (or
AMD) is used as a query in
A. Images B and C show ex-
amples of segmented images
returned from the library
demonstrating similar patho-
logic findings.

calculate the relative brightness or color of selected vantage of a number of visual attributes of the vascu-
pairs of pixels including, degree of contrast, coarse- lar tree in a probabilistic framework.38 – 40 Figure 5
ness, directionality, periodicity, and randomness.30,31 presents an overview of our method for detection of
Shape features that are computed for CBIR databases the ON and localization of the macula. In Figure 5A,
include global features such as aspect ratio, circularity the original red-free fundus image undergoes analysis
and moment invariants, and local features such as sets to provide a segmentation of the vessels as shown in
of consecutive boundary segments and representations (Figure 5B). In our work, we leverage the method of
of object shape that can be compared using graph Zana and Klein.41 Vessel segmentation is by no means
matching techniques.32–35 Semantic-based image rec- a solved problem and other approaches are possi-
ognition uses color, texture, region, and spatial infor- ble.9 –12 Regardless of the actual algorithm employed,
mation to derive likely interpretations that generate the characteristics of the segmented vasculature such
text descriptors which can be input to a text retrieval as the density, orientation, and change in thickness,
system.36,37 In other words, feature analysis represents we are able to produce a spatial likelihood map of
an extensive assembly of measurements that can be the probable locations of the ON as in Figure 5C. The
used to describe imagery. Combining the diagnostic peak of this map defines the likely coordinate of the
experience of the ophthalmologist with the implemen- ON as in Figure 5D. Once the nerve coordinates have
tation of appropriate image features can result in a been detected, we make use of a parabolic model of
powerful predictive response in terms of specificity the vascular structure emanating from the ON coordi-
and sensitivity. nate to locate the horizontal raphe as in Figure 5E. We
use parabolic modeling because features of the normal
Digital Image Analysis macula and fovea such as carotenoid pigment are
Low-level analysis uses models that describe statis- often absent in diseased maculae. Finally, the center of
tical features related to the texture, shape, or structure the macula, our primary reference point in the descrip-
of regions of interest that are segmented from within tion of disease manifestation, is positioned two ON
the image. Therefore, low-level image processing to diameters from the temporal edge of the ON coordi-
achieve accurate segmentation of important structures nate as shown in Figure 5F.
is critical. In this regard, we must be able to accurately Once the ON and vessels have been segmented and
represent both anatomic structures of the retina to- the location of the macula defined, we focus on seg-
gether with the structures associated with the mani- menting the lesions in the regions within and imme-
festations of pathology. diately adjacent to the macula. There is a large body of
To address accurate segmentation of anatomic research on the segmentation of lesions and we are
structure, we have developed methods that take ad- investigating several approaches. Figure 6 shows an
1468 RETINA, THE JOURNAL OF RETINAL AND VITREOUS DISEASES ● 2008 ● VOLUME 28 ● NUMBER 10

of lesions due to variation in pigment, vascular pat-

terns, etc. The one on right is postprocessed and
reduces the number of false-positive lesions, but is
still somewhat imperfect at detecting small lesions.
The variability in retinal lesion populations seen
(bright, dark, sparse, feathery, “soft,” “hard,” etc.)
coupled with the natural variation of the surface of the
retina due to age and ethnicity make lesion segmen-
tation a particularly difficult challenge to accomplish
robustly. In Figure 6, the left-hand image represents
the initial output of the automated lesion detection
process and it contains a number of false positives that
arise from the inherent intensity fluctuations in the
RPE that exist across the image. This image is subse-
quently postprocessed to reduce the number of false
positives using an automated pattern recognition
method to separate candidate lesions from false posi-
tives as shown in the right-hand side of the figure.
For our research, we are focusing on the indepen-
dent segmentation and feature analysis of bright lesion
populations and dark lesion populations. Each popu-
lation is analyzed to extract features representing a
group as opposed to individual lesions. We analyze
lesion populations because, in general, a group of
bright (or dark) lesions are related to each other by
cause, i.e., the pathology of disease present. A pop-
Fig. 5. Overview of our method for localization of the optic nerve and ulation of bright lesions may also be inherently
macula. The fundus image (A) undergoes analysis to segment the
retinal vessels (B). We are able to produce a spatial likelihood map of
linked to dark lesions (e.g., exudates and microan-
the probable locations of the optic nerve (C) and the peak defines the eurysms), so understanding one population in the
coordinate of the optic nerve (D). We use of a parabolic model of the context of the other provides a powerful descriptive
vascular structure to locate the horizontal raphe’ (E) and then localize
the macula from the temporal edge of the optic nerve (F).
method.

Ground Truthing
example of an image in which bright and dark lesions
have been detected and segmented using methods For the purposes of this study, we are using a
based on the work of Rapantzikos et al23 and Ben hand-segmented and labeled population of images to
Sbeh.24 The image on the left shows oversegmentation validate our diagnostic approach. Ground-truth data

Fig. 6. Examples of prepro-

cessed (left) and postpro-
cessed (right) automated le-
sion detection isolating bright
and dark lesion populations
from an image with nonpro-
liferative diabetic retinopathy
with macular edema. The left-
hand image represents the ini-
tial output of the automated
lesion detection process con-
taining many false positives
that arise from the inherent
intensity fluctuations in the
RPE. The image is postpro-
cessed to reduce the number
of false positives using an au-
tomated pattern recognition
method to separate candidate
lesions from false positives.
 AUTOMATED RETINAL DIAGNOSIS BY CBIR ● CHAUM ET AL 1469

were used in lesion segmentation to help train a clas- the library therefore encapsulating semantic intent, or
sifier by providing examples of actual lesions and user meaning in association of image content to visi-
examples of “false positives.” For example, the com- ble pathology). We apply a novel directed indexing
puter may detect several “blobs” on an image of a method that uses principle component analysis and
retina which had no lesions on it. These blobs are linear discriminant analysis techniques to produce a
obvious examples of false positives. The algorithm is linear combination of features that optimally distin-
trained to distinguish between these false positives guish between the pathologic studies of interest.42
and actual lesions or true positives. This represents a “supervised learning” method that
Ground-truth data were used to characterize the uses training data. This method produces a projection
complete retina by identifying disease state of the of potentially hundreds of statistical features associ-
retinas in the training set so that the CBIR system can ated with lesions to a few discriminating indices op-
learn what characteristics (such as lesion population timized to represent the best meaning of image content
contrast, intensity, shape, etc.) distinguish one lesion (i.e., the meaning of lesion types and populations) for
type from another. The ground-truth data were used to the automated diagnosis application. All images in-
identify the lesion population on each retina. Numer- dexed into the system are described through the di-
ical descriptors of the lesion population were then rected indexing process to facilitate an accurate search
created from this ground-truth lesion population and and retrieval response from the system. Note, how-
used as training/testing data for the image retrieval ever, that the system does not perform classification of
algorithm. The ground-truth data were used in the image content into “disease state A” or “disease state
characterization of the lesion images by measuring B,” etc. Rather, it produces a probability that a par-
numerical descriptors on each ground-truth lesion ticular pathology is present, along with the probability
such as lesion area, intensity within the lesion, etc. that a particular stratification of that pathology exists.
These were then combined into a set of histograms This is a true CBIR method in which the human
that describe the total lesion population of the image. analyst ultimately makes the decision regarding diag-
This “feature vector,” or image description, consisted nostic correctness and appropriate follow-up.
of a total of 170 different values.
Although automating the segmentation of lesions is Estimating Pathology
key to achieving a fully automated method, our goal in
this article is to demonstrate the efficacy of the diag- We have developed our content-based diagnostics
nostic capabilities inherent in the CBIR method. For approach to support retrievals in a probabilistic frame-
the current research, we have fully automated the ON work that provides the user with measures of decision
detection and macula localization process, and we are confidence and controls. When performing retrieval
demonstrating the results of a fully automated lesion operations, we formulate the retrieved group of visu-
population feature extraction, analysis, and diagnostic ally similar images as a Bayesian posterior probabil-
assignment. Although the problem of automatic lesion ity, P(␻i/f) ⫾ ␴(␻i), where P represents the probability
detection is far from solved, one goal of our research that a pathology of class ␻i is present given the mea-
is to provide robust descriptions of the lesion popula- sured visual evidence of the statistical features, f. The
tions so that imperfect segmentations can still be used. class, ␻i, in this instance refers to the categories of
defined pathologies. The error estimate, ⫾␴(␻i), is
Discriminant Analysis determined by assuming that pathologies occur with a
frequency defined by a Poisson distribution. Recall
To achieve a succinct description of an image, we that a Poisson process is applied to phenomena of a
perform image analysis to locate groups of contiguous discrete nature whenever the probability of the phe-
pixels representing lesions, then extract the statistical nomena happening is constant over time, i.e., those
features that describe these lesions, and generate an that may occur 0, 1, 2 . . . times during a given period
index. The index is a series of numbers derived from of time, such as a patient with a given pathology
the statistical features to define an image in succinct walking into a doctor’s office.
terms (e.g., from megabytes of pixels to tens of num- Based on this Bayesian/Poisson formulation, and in
bers). For our application, to achieve high sensitivity particular when testing and validating the system, we
and specificity, we make use of our previously diag- have the ability to adjust classification performance of
nosed image archive which we refer to as a library of the system by specifying a confidence level. The con-
known pathology examples. The library contains ex- fidence level is based on the known characteristics of
amples of multiple pathology categories that represent the Poisson error process as being proportional to the
the semantic intent of the user (i.e., the user defines square root of the number of samples in each defined
1470 RETINA, THE JOURNAL OF RETINAL AND VITREOUS DISEASES ● 2008 ● VOLUME 28 ● NUMBER 10

class, or in our case the pathology. By adjusting the and supports the application of our algorithm using a
confidence level, the system will reject data that can- vascular model. We describe our results below with a
not be clearly diagnosed within a range of uncertainty. detailed description of our retinal features followed by
The result is that a query may produce a response diagnostic performance of the method estimated using
indicating that the pathology cannot be predicted be- this manually characterized dataset of 395 images as
cause of uncertainty in the data. These “unpredict- ground-truth for comparison.
able” records are set aside for manual review by a
trained reader or ophthalmologist. Although increas- Results
ing the confidence threshold will result in a higher
proportion of rejected data, the system performance Fundus Descriptors
on diagnosed data will increase. For this research, we
We tested and selected features that describe retinal
define system performance in terms of three com-
image features including region morphology based on
monly used medical statistics43 of sensitivity, speci-
invariant moments, shape descriptors, intensity de-
ficity, and accuracy as follows:
scriptors, textures derived from region variations, and
Sensitivity ⫽ images with disease classified as proximity features based on relationships within in the
dataset. We measured 170 independent features rep-
disease/all disease images resenting the number of lesions in a population (e.g.,
dark or bright lesions), including vascular density
Specificity ⫽ normal images classified as within the lesion population, population moments
normal/all normal images (means, variance, etc.) relative to the central macula,
sharpness and smoothness (texture) within and at le-
Accuracy ⫽ images classified correctly/all sion boundaries, and average morphologic properties
of lesions in the light and dark lesion groups.
classified images
We defined, extracted, and tested a large number
of region- and lesion-based features from the data
Imaging Methods generated. These features became the extensive de-
scription of the fundus image and were mapped to an
For this discussion, we will refer to a manually
index to capture the aspects of the fundus image that
characterized population of images. These images are
correspond to various disease states. We used mor-
from an anonymized dataset taken through dilated
phologic reconstruction concepts to segment drusen,
pupils using a Topcon TRC 50IA retina camera. Each
using a segmentation algorithm that is robust over
digital image encompasses a 50° field of the retina,
local changes in the background intensity. This im-
and is taken using a green filter in the flash path to
proved some aspects of the accuracy, including auto-
generate a “red-free” digital image. The image is
matic choice of parameters and preprocessing and
acquired using a 1.4 Megapixel CCD camera (Kodak
postprocessing thresholds. Figure 7 shows an example
Megaplus 1.4i) with 1,024 ⫻ 1,152 pixel resolution
of automated drusen detection by application of
that is saved at full fidelity without compression. The
this method.
images are from the ophthalmic clinical practice of
one of the authors (E.C.), a retina specialist at the
Diagnostic Performance
University of Tennessee Hamilton Eye Institute, and
were selected and characterized based upon determi- To demonstrate the expected performance of this
nation of the specific disease diagnosis present in the CBIR system for diagnosing both pathology and man-
eye. Images were selected from the patient database ifestations of disease, we use a statistical hold-one-
based upon their correlation with a standardized image out method.28 Because of the few manually analyzed
(Field 2) from the Early Treatment Diabetic Retinop- samples representing the variety of pathologic studies
athy Study (ETDRS) standard protocol for retinal tested in these results (395 images representing 14
imaging and lesion classification. For our purposes the disease states and normal fundus), we have used the
macula is centered in a 50° image. In this configura- features of the query data to generate the feature
tion, the ON occupies a location to the right or left of projection matrix during the indexing process. Once
the central macula, in the right and left eyes, respec- the indices have been generated, the query samples
tively. This standard image of the retina permits ex- are removed from the population during hold-one-out
amination of both the macula and ON in the same testing. This query is then submitted to the system and
image. In addition, we selected this standard field a prediction of the diagnosis is formulated by estimat-
because it shows the vasculature that feeds the macula ing a Bayesian posterior probability of pathology us-
 AUTOMATED RETINAL DIAGNOSIS BY CBIR ● CHAUM ET AL 1471

Fig. 7. Example of a pro-

cessed fundus image showing
demarcated vasculature, lo-
calized optic nerve and mac-
ula, segmented macular le-
sions, and oriented macular
coordinate system.

ing a neighborhood of similar returned images as intraretinal microvascular abnormalities, have not yet
shown in Figure 3. This process is repeated for all been developed.
samples in the dataset and the accuracy is then deter- Instead, images of DR were graded by a retina
mined. It should be noted for these results that the specialist (E.C.) and stratified into different levels
lesion populations in the 395 images have been seg- according to the International Council of Ophthalmol-
mented and the pathologies characterized manually as ogy, International Clinical Diabetic Retinopathy Dis-
a means of providing ground-truth comparative data. ease Severity Scale (ICDRS, http://www.icoph.org/
For this work, the prior distribution of pathologies in standards/dmedetail.html). These were: no retinopathy
the dataset is a function not only of the category of (“normal”), minimal or mild retinopathy without clin-
pathology, but of the degree to which these pathologies ically significant macular edema (CSME) (“1”), min-
are present in the patient. By structuring the data in imal or mild retinopathy with CSME (“2”), moderate
this manner, it becomes possible for the CBIR system to severe retinopathy without CSME (“3”), and mod-
to provide information not only on the presence or erate to severe retinopathy with CSME (“4”). Prolif-
absence of pathology, but also on the associated erative DR was stratified into four levels; PDR with-
severity. In total, there were 14 pathologies or degrees out high-risk characteristics (by DRS criteria) (1),
of pathology listed plus examples of a normal fundus. PDR without high-risk characteristics with CSME (2),
These represent AMD, NPDR, proliferative diabetic PDR with high-risk characteristics (3), and PDR with
retinopathy (PDR), retinal vein occlusions and a gen- high-risk characteristics with CSME (4).
eral category of “Other” which contains numerous Figure 8 shows an example of a posterior pathology
disease states and retinal abnormalities including ret- estimate made for a patient diagnosed with Grade 1
inal artery and retinal vein occlusions, juxtafoveal NPDR. The digital fundus image is inset showing the
telangiectasis, retinitis pigmentosa, and choroidal le- manual demarcation of lesions associated with the
sions, to name a few. disease. Note that based on the population of similar
The stratified grades (levels) of disease for AMD, images returned by the system, the cluster of large
NPDR, and PDR, correspond to currently used or hemorrhages in the upper right of the macula suggests
common strategies for disease quantification in clini- that there is also a probability greater than zero that the
cal use. Grades 1 through 4 for AMD correspond to lesion is consistent with a small retinal vein occlusion.
the disease stratification used in the AREDS clinical Also note that the error bars represent the uncertainty in
trials. The optimal stratification system grading NPDR the estimates of either probability and that the probability
for comparative purposes would be that used for the of retinal vein occlusion pathology is not significant
ETDRS studies. However, we are unable to use the relative to its own associated error bar. Because of the
ETDRS grading system for several reasons: 1) we use nature of Poisson statistics, as the number of samples of
only single field (macula) images, the ETDRS uses each type of pathology increases in the data system (i.e.,
seven standard fields including four peripheral fields; as the size of the CBIR library grows), the relative error
2) lesions are not counted per se using the CBIR in each category will constrict.
method, the ETDRS quantifies severity levels by type, Not only is the method capable of predicting dis-
location, and number of fields in which they are ease pathology, but it can also infer the specific man-
present; and 3) algorithms to recognize certain fea- ifestations of disease states. For example, in our data
tures in used the ETDRS grading system, such as population, there are eight manifestations associ-
1472 RETINA, THE JOURNAL OF RETINAL AND VITREOUS DISEASES ● 2008 ● VOLUME 28 ● NUMBER 10

Fig. 8. Posterior probability

of pathologic study for an ex-
ample query image (inset) in-
dicating the presence of min-
imal or mild retinopathy
without clinically significant
macular edema (Level 1
NPDR). Raw and segmented
images are shown.

ated with specific disease pathologies; hemorrhages/ in the population. In these tables, the vertical axis
microaneurysms, exudates, cotton wool spots, hard represents manually labeled pathologies whereas the
drusen, soft drusen, basal laminar drusen, retinal pig- horizontal axis represents classifications assigned by
ment epithelial detachments, and choroidal neovascu- the CBIR system. The diagonal of the matrix repre-
larization. Our ability to identify these manifestations sents the number of correct classifications. The col-
based upon a population of visually similar data umn labeled “Unknown” shows the number of data
strengthens our ability to assign a diagnostic interpre- that were rejected for each pathology class at the
tation of the pathology present. specified confidence level. The right-most column
To represent the sensitivity and specificity of the shows the accuracy for each defined class (row) of
system over all of the data using the hold-one-out data in the table. The data in the Unknown column
process, we produce a confusion matrix representing was excluded from the calculation of accuracy, i.e.,
the response of the system to all pathology classes for accuracy is only reported for data that was classified
a specified confidence level as shown in Table 1 and (either correctly or incorrectly) by the CBIR system
Table 2. The data in these tables were generated for a using our probabilistic method. The purity along the
confidence level of ␴ ⫽ 1.0, where ␴ represents the bottom row is similar to accuracy, but instead of
SD of the variability in the data. A value of ␴ ⫽ 1.0 showing the fraction of each manually labeled dis-
refers to ⫾1 SD encompassing 67% of the variability eases that were correctly categorized, the purity shows
 AUTOMATED RETINAL DIAGNOSIS BY CBIR ● CHAUM ET AL 1473

Table 1. Confusion Matrix Detailing Classification Performance Over Four Primary Disease Categories of AMD, PDR,
NPDR, and “Other,” Along With the Normal (Norm) Category. The Vertical Axis Represents Human-Labeled Classes
Whereas the Horizontal Axis Represents Estimates Made by the CBIR System. For This Example, Confidence Is Set
to ␴ ⫽ 1.0

Norm AMD PDR NPDR Other Unknown Performance (%)

Norm 40 0 0 0 5 5 88.9
AMD 1 61 0 1 6 12 88.4
PDR 0 0 14 1 0 7 93.3
NPDR 0 0 0 46 5 7 90.2
Other 6 4 0 1 154 17 93.3
Purity (%) 85.1 93.9 100.0 93.9 90.6 12.2 91.3

what fraction of the classes categorized by the CBIR when considered as one agglomerated disease cate-
system were correct. Note that the purity and accuracy gory. The overall accuracy of the diagnosis for all
for each class are not necessarily equal. disease states in the grouped (Table 1) and stratified
The diagnostic predictions formulated and the over- disease (Table 2) dataset was 91.3%. It is this
all performance of the method was determined for the ability of a CBIR systems to not only identify but
general disease categories of AMD, PDR, and NPDR also to stratify disease states that makes it useful as
in Table 1. Note that a general exclusion category of an automated diagnostic method.
Other (diseases) is also listed along with a population
of normal retina images (i.e., no visible pathology). Discussion
Although this result is indicative of the ability of the
system to categorize major diseases, Table 2 details The use of digital photographic analysis to extract
performance over a stratification of the major disease features of DR has been reported in numerous stud-
states. The diagnostic performance for all stratified ies.44 – 46 Simple image extraction algorithms have
levels of AMD ranged from 75–100% when treated as demonstrated the ability to detect features of DR such
four independent, stratified disease states, and 88.4% as exudates and microaneurysms using color and mono-
when taken as one agglomerated disease category. chromatic retinal images. These small pilot studies have
The performance for PDR ranged from 75–91.7% shown that up to 75% to 85% sensitivity and specificity
when considered as four independent disease states, for certain features of DR can be achieved.
and 93.3% when taken as one agglomerated disease Retinal images have previously been classified us-
category, and for NPDR ranged from 75–94.7% for ing various perceptual metrics including: topographic
four independent, stratified disease states, and 90.2% analysis,47,48 image segmentation,8,49 geometric mod-

Table 2. Confusion Matrix Detailing Classification Performance as a Function of the Stratification of a Variety of
Disease States. This Stratified Data Represents the Ability of a CBIR System to Infer Useful Diagnostic Information
from an Image Archive. The Vertical Axis Represents Human-Labeled Classes Whereas the Horizontal Axis
Represents Estimates Made by the CBIR System. For This Example, Confidence is Set to ␴ ⫽ 1.0
AMD AMD AMD AMD PDR PDR PDR PDR NPDR NPDR NPDR NPDR Performance
Norm (1) (2) (3) (4) (1) (2) (3) (4) (1) (2) (3) (4) RVO Other Unknown (%)

Norm 40 0 0 0 0 0 0 0 0 0 0 0 0 0 4 6 90.9
AMD (1) 1 0 0 0 0 0 0 0 0 0 0 0 0 0 3 8 0.0
AMD (2) 0 0 3 0 0 0 0 0 0 0 0 0 0 0 1 4 75.0
AMD (3) 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 9 100.0
AMD (4) 0 0 0 0 25 0 0 0 0 0 0 0 0 0 6 4 80.7
PDR (1) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 ND
PDR (2) 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 5 100.0
PDR (3) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 ND
PDR (4) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 ND
NPDR (1) 0 0 0 0 0 0 0 0 0 11 0 0 0 0 1 4 91.7
NPDR (2) 0 0 0 0 0 0 0 0 0 0 9 0 0 0 3 7 75.0
NPDR (3) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 ND
NPDR (4) 0 0 0 0 0 0 0 0 0 0 0 0 18 0 1 4 94.7
RVO 0 0 0 0 0 0 0 0 0 0 0 0 0 14 1 1 93.3
Other 7 0 0 0 0 0 0 0 0 0 0 0 0 0 152 7 95.6
Purity 83.3 ND 100.0 100.0 100.0 ND 100.0 ND ND 100.0 100.0 ND 100.0 100.0 88.4 18.8 91.3
1474 RETINA, THE JOURNAL OF RETINAL AND VITREOUS DISEASES ● 2008 ● VOLUME 28 ● NUMBER 10

eling and filtering, and neural networks,50 to name a Our results using a small library (comprising 395
few. These techniques are based on modeling or ex- retinal images) demonstrate the potential to achieve a
tracting each class of object to be recognized, identi- relatively high level of automated disease detection
fying regions that contain examples of the object, and and stratification using CBIR methods. Within each
then using algorithms to confirm or rule out the fea- disease category; however, accuracy is variable de-
ture. In addition to these methods, there are a number pending on the severity of disease present (Table 2).
of reported studies detailing approaches for detecting For example, although overall accuracy for all
and classifying the fovea and blood vessels,51 classi- AREDS levels of AMD was 88.4%, the diagnostic
fication of blood vessels into arteries and veins,52 and specificity was best for higher levels of disease Grades
probabilistic retinal vessel segmentation.53 Many ret- 3 and 4 (100% and 80.7%, respectively) and diagnos-
inal disease screening methods referenced are note- tic specificity for Grade 1 disease was poor (0/12). In
worthy for the use of digital photography using lim- 75% of these cases (8 of 12), the inability to assign a
ited 1- or 2-field retinal photographs, and for attempts diagnosis occurred because the confidence limits for
to develop automated detection algorithms.4,54,55 Im- two different levels of AMD disease overlapped
age extraction algorithms have been used by others in [e.g., AMD (1) and AMD (2)], and therefore the
automated screening of digital color and monochro- disease state was determined to be unknown. Sim-
matic retinal images for features of NPDR such as ilarly, 18.8% of images in all disease categories
hard exudates, hemorrhages, and microaneurysms.56 could not be definitively stratified and were as-
Although our research currently focuses on the diag- signed to the unknown category.
nosis of previously unseen patients in a screening By varying the confidence level through a range of
environment, longitudinal changes associated with values, we generate a set of confusion matrices from
DR in fundus image sequences collected from a single which we can produce a series of receiver-operator
patient over time are also possible.57 characteristic curves56 detailing overall performance
In our previous work, we have developed an auto- as a function of the number of data rejected, i.e., in
mated method for localizing the ON using a combined terms of specificity, sensitivity, and accuracy. A set of
principle component analysis/linear discriminant anal- receiver-operator characteristic curves for the 395 im-
ysis and feature-based likelihood ratio approach. This age database is shown in Figure 9. Note the positive
method demonstrated automatic localization of the response of the system to the increased rejection of
nerve within one ON radius in 92% of 345 images data. Recall that the fraction of data rejected increases
tested and within 1⁄2 of an ON radius in 89% of the as the confidence (stringency) specified by the user is
images.38 Similarly, preliminary work demonstrated increased. A higher confidence level (i.e., increasing
92% performance for automatically locating the cen- ␴) results in a larger fraction of the data being re-
tral macula within one ON diameter of the fovea using jected, due essentially to confusion in the classifica-
red-free fundus images in 345 patients with 19 differ- tion process. This characteristic of a probabilistic
ent retinal diseases.37 CBIR approach is advantageous. Although the system

Fig. 9. Performance curves

for the CBIR system indicating
accuracy, sensitivity, and spec-
ificity as a function of data re-
jected. A higher confidence
level (i.e., increasing ␴) results
in a larger fraction of the data
being rejected. As the CBIR
image library grows, the strin-
gency of the retrieval process
can be increased to improve the
diagnostic sensitivity and spec-
ificity of the method.
 AUTOMATED RETINAL DIAGNOSIS BY CBIR ● CHAUM ET AL 1475

can make predictions regarding the presence, degree, associated manifestations of the disease. We anticipate
and manifestations of pathologic study, the ability to that incorporating clinical metadata will improve our
select the stringency within the retrieval process for a assessment of disease risk in parallel with increased
similar population of images for human analysis and sensitivity and specificity. With the advent of electronic
decision-making is a key feature. As the CBIR image medical records in modern clinical practice, there is the
library grows and larger numbers of images with potential to leverage clinical and image electronic med-
stratified levels of diseases are added, the stringency ical records databases by linking relevant clinical meta-
of the retrieval process can be increased to improve data to processed retinal images thereby enhancing the
the diagnostic specificity of the method. power of the image analysis and management algorithms
Thus, it is clear that the use of computerized image inherent in CBIR. We have developed our approach to
extraction algorithms that rely upon statistical meth- support retrievals in a probabilistic framework that pro-
ods and image-processing algorithms can identify ret- vides the user with confidence metrics to control system
inal lesions. However, the objectives of automated sensitivity, specificity, and accuracy.
analysis and diagnosis have not been adequately met
to date. The detection of focal lesions in the retina Summary
such as dot hemorrhages in NPDR may not have
relevance to the presence of vision threatening disease There is increasing interest in the use of CBIR
in the absence of exudates and fluid. Focal hemor- techniques to aid in the development of diagnostic
rhages, microaneurysms, and exudates outside the ma- algorithms by identifying similar, well defined cases
jor vascular arcades are a common finding in DR but related to a query case. Most research in picture ar-
have little relevance to vision. The extent and prox- chiving and communication is still directed toward
imity of lesions to central macula defines to a certain providing basic functionality and access; however,
degree, their potential visual impact. Conversely, the clinical CBIR systems are now in use.58,59 Ophthal-
presence of intraretinal edema and fine exudates are mology practice, in particular Retina, generates large
difficult to detect on standard digital photographic im- amounts of digitized patient data documenting retinal
ages, and their location relative to the fovea has previ- and ON diseases in the form of retinal photographs,
ously been ignored by lesion segmentation methods. fluorescein angiography and OCT images, topography
The optimal analytical method would not only maps, ultrasound, and other digital images for diag-
screen for vision-threatening lesions in the region of nosis and monitoring purposes. CBIR methods have
the macula and ON, but would also detect and quan- the potential to not only assist physicians in arriving at
tify the nature, location, and extent of retinal patho- diagnoses based upon similar pathologic findings in
logic study, determine whether it was visually signif- referential datasets, but also to enhance the diagnostic
icant by its location relative to the fovea, and generate process through development of disease-directed au-
a diagnosis that includes clinically relevant metadata tomated algorithms.
in the analysis. Metadata includes any number of The number of electronic images that are poten-
physical or medical characteristics of the patient (i.e., tially available to researchers and clinicians is enor-
excluding the fundus image) that can be used to focus mous, because of the advent of inexpensive and pow-
the data mining process of CBIR on other clinically erful computing and storage devices. Both the context
relevant cases in an archive of previously diagnosed (metadata) and the content (intrinsic properties of the
patients. Metadata includes age, gender, ethnicity, and digital image) of these datasets represent a rich source
disease history. Patient data in the archive would also of clinical information and data mining that is only
contain metadata related to the diagnosis of existing marginally accessed for medical care today. Impor-
disease and the description of the presence and abun- tantly, the CBIR method has the potential to accu-
dance of the manifestations of any disease present. rately describe and index human retinal images using
Pure image analysis methods exclude important both the image content and the “patient context” (the
clinical metadata for example: the type and duration of clinical metadata). The method can be used to enhance
diabetes, presence of comorbid disease such as hyper- human analysis and image throughput in reading cen-
tension, historical hemoglobin A1C values, and a pre- ters and in clinical practice. In this article, we have
vious history of laser treatment, to name a few, which demonstrated, using a manually segmented lesion
have direct relevance to the presence of retinal lesions population and a statistical hold-one-out validation,
and the severity of the underlying disease. Using a that CBIR has the potential to provide a reliable and
Bayesian probabilistic framework to make use of the potentially automated diagnostic method for detecting
metadata from an image population assists us in arriving and quantifying important blinding retinal diseases,
at a prediction of the severity of disease together with the including DR and AMD, and may eventually permit
1476 RETINA, THE JOURNAL OF RETINAL AND VITREOUS DISEASES ● 2008 ● VOLUME 28 ● NUMBER 10

diagnostic image analysis in an automated and deter- boundary using dynamic contours. Image Vis Comput 1999;
ministic manner. 17:169 –174.
17. Li H, Chutatape O. Automated feature extraction in color
Key words: content-based image retrieval, comput- retinal images by a model based approach. IEEE Trans
er-aided diagnosis, retina, retinopathy, diabetic reti- Biomed Eng 2004;51:246 –254.
nopathy, age-related macular degeneration, image 18. Niemeijer M, Abramoff MD, van Ginneken B. Segmentation
analysis. of the optic disc, macula and vascular arch in fundus photo-
graphs. IEEE Trans Med Imaging 2007;26:116 –127.
19. Pinz A, Bernogger S, Datlinger P, Kruger A. Mapping the
References human retina. IEEE Trans Med Imaging 1998;17:606 – 619.
20. Solouma NH, Youssef AM, Badr YA, Kadah YM. A new
real-time retinal tracking system for image-guided laser treat-
1. The Diabetic Retinopathy Study Research Group. Preliminary
ment. IEEE Trans Biomed Eng 2002;49:1059 –1067.
report on effects of photocoagulation therapy. Am J Ophthal-
21. Trucco E, Kamat PJ. Locating the optic disk in retinal images
mol 1976;81:383–396.
via plausible detection and constraint satisfaction. In: IEEE
2. The Early Treatment Diabetic Retinopathy Study Research
International Conference on Image Processing. Singapore:
Group. Photocoagulation for diabetic macular edema: early
2004:55–158.
treatment diabetic retinopathy study report number 1. Arch
22. Niemeijer M, van Ginneken B, Staal J, et al. Automatic de-
Ophthalmol 1985;103:1796 –1806.
tection of red lesions in digital color fundus photographs. IEEE
3. Age-Related Eye Disease Study Research Group. Risk factors
Trans Med Imaging 2005;24:584 –592.
associated with age-related macular degeneration. A case-
23. Rapantzikos K, Zervakis M, Balas K. Detection and segmen-
control study in the age-related eye disease study: age-related
tation of drusen deposits on human retina: potential in the
eye disease study report number 3. Ophthalmol 2000;107:
diagnosis of age-related macular degeneration. Med Image
2224 –2232.
Anal 2003;7:95–108.
4. Larsen M, Godt J, Grunkin M. Automated detection of diabetic
24. Ben Sbeh Z, Cohen LD, Mimoun G, Coscas G. A new ap-
retinopathy in a fundus photographic screening population.
Invest Ophthal Vis Sci 2003;44:767–771. proach of geodesic reconstruction for drusen segmentation in
5. Teng T, Lefley M, Claremont D. Progress towards automated eye fundus images. IEEE Trans Med Imaging 2001;20:1321–
diabetic ocular screening: a review of image analysis and 1333.
intelligent systems for diabetic retinopathy. Med Biol Eng 25. Tobin KW, Karnowski TP, Arrowood LF, et al. Content-based
Comput 2002;40:2–13. image retrieval for semiconductor process characterization.
6. Patton N, Aslam TM, MacGillivary T, et al. Retinal image EURASIP J Appl Signal Process 2002;7:704 –713.
analysis: concepts, applications and potential. Prog Retin Eye 26. Karnowski TP, Tobin KW, Ferrell RK, Lakhani F. Using an
Res 2006;25:99 –127. image retrieval system for image data management. Design,
7. Foracchia M, Grisan E, Ruggeri A. Luminosity and contrast process integration, and diagnostics in IC manufacturing. Proc
normalization in retinal images. Med Image Anal 2005;9:179 – SPIE 2002;4692:120.
190. 27. Santini S. Exploratory Image Databases, Content-Based Re-
8. Martinez-Perez ME, Hughes AD, Stanton AV, et al. Segmen- trieval. Duluth, MN: Academic Press; 2001.
tation of retinal blood vessels based on the second directional 28. Duda RO, Hart PE, Stork DG. Pattern Classification. 2nd ed.
derivative and region growing. In: IEEE International Confer- New York, NY: John Wiley & Sons, Inc; 2001.
ence on Image Processing. Kobe, Japan 1999:173–176. 29. Arya S, Mount DM, Netanyahu NS, et al. An Optimal algo-
9. Lin T, Zheng Y. Adaptive image enhancement for retinal rithm for approximate nearest neighbor searching in fixed
blood vessel segmentation. Electronic Lett 2002;38:1090 – dimensions. Journal ACM 1998;45:891–923.
1091. 30. Gong Y, Chuan CH, Xiaoyi G. Image indexing and retrieval
10. Staal J, Abramoff MD, Niemeijer M, et al. Ridge-based vessel based on color histogram. Multimedia Tools Appl 1996;2:133.
segmentation in color images of the retina. IEEE Trans Med 31. Rubner Y, Tomasi C. Perceptual Metrics for Image Database
Imaging 2004;23:501–509. Navigation. Boston, MA: Kluwer Academic Publishers; 2001.
11. Hoover A, Goldbaum M. Locating the optic nerve in a retinal 32. Vicario E, ed. Image Description and Retrieval. New York,
image using the fuzzy convergence of the blood vessels. IEEE NY: Plenum Press; 1998.
Trans Med Imaging 2003;22:951–958. 33. Mehtre BM, Kankanhalli MS, Lee WF. Shape Measures for
12. Foracchia M, Grisan E, Ruggeri A. Detection of the optic disc content based image retrieval: a comparison. Inform Process
in retinal images by means of a geometrical model of vessel Manag 1997;33:319.
structure. IEEE Trans Med Imaging 2004;23:1189 –1195. 34. Gary JE, Mehrotra R. Similar shape retrieval using a structural
13. Fleming AD, Goatman KA, Philip S, et al. Automatic detec- feature index. Information Syst 1993;18:525–545.
tion of retinal anatomy to assist diabetic retinopathy screening. 35. Bradley A. The use of the area under the ROC curve in the
Phys Med Biol 2007;52:331–345. evaluation of machine learning algorithms. Pattern Recog
14. Abramoff MD, Niemeijer M. The automatic detection of the 1997;30:1145–1159.
optic disc location in retinal images using optic disc location 36. Bingham PR, Price JR, Tobin KW, Karnowski TP. Semicon-
regression. Presented at: 28th IEEE EMBS Annual Confer- ductor sidewall shape estimation. SPIE Journal of Electronic
ence. New York, NY 2006. Imaging, 2004;13:474 – 485.
15. Abramoff MD, Alward WL, Greenlee EC, et al. Automated 37. Manjunath BS, Ma WY. Texture features for browsing and
segmentation of the optic disc from stereo color photographs retrieval of image data. IEEE Trans Pattern Anal Mach Intell
using physiologically plausible features. Invest Ophthalmol 1996;18:837– 842.
Vis Sci 2007;48:1665–1673. 38. Tobin KW, Chaum E, Govindasamy VP, Karnowski TP, Sezer
16. Morris DT, Donnison C. Identifying the neuroretinal rim O. Characterization of the optic disk in retinal imagery using
 AUTOMATED RETINAL DIAGNOSIS BY CBIR ● CHAUM ET AL 1477

a probabilistic approach. In: SPIE International Symposium on with a confocal scanning laser: new optic disk analysis without
Medical Imaging. San Diego, CA: Proceedings of SPIE; 2006. any observer input. Surv Ophthalmol 1999;44:33– 40.
39. Karnowski TP, Govindasamy VP, Tobin KW, Chaum E. Lo- 49. Qing ZH. Segmentation of blood vessels in retinal images
cating the optic nerve in retinal images: comparing model- using 2-D entropies of gray level-gradient co-occurrence ma-
based and Bayesian decision methods. Conf Proc IEEE Eng trix. In: IEEE International Conference on Acoustics, Speech,
Med Biol Soc 2006;1:4436 – 4439. and Signal Processing. Montreal, Quebec 2004:509 –512.
40. Tobin KW, Chaum E, Govindasamy VP, Karnowski TP. De- 50. Morgan-Davies J, Taylor NK, Armbrecht AM, Aspinall P,
tection of anatomic structures in human retinal imagery. IEEE Dhillon B. Progressive assessment of age related macular
Trans Med Imaging 2007;26:1729 –1739. degeneration using an artificial neural network approach. Br J
41. Zana F, Klein J. Segmentation of vessel-like patterns using Ophthalmol 2001;85:246 –247.
mathematical morphology and curvature evaluation. IEEE 51. Estabridis K, Defigueiredo R. Fovea and vessel detection via
Trans Image Process 2001;10:1010 –1019. multi-resolution parameter transform. In: Medical Imaging.
42. Tobin K, Bhaduri B, Bright E, et al., Automated feature San Diego, CA: Proceedings of SPIE; 2007.
generation in large-scale geospatial libraries for content-based 52. Kondermann C, Kondermann D, Yan M. Blood vessel classi-
indexing. Photogrammetric Eng Remote Sens 2006;72:531– fication into arteries and veins in retinal images. In: Medical
540. Imaging. San Diego, CA: Proceedings of SPIE; 2007;6512:
43. Muller H, Michoux N, Bandon D, Geissbuhler A. A review of 651247.
content-based image retrieval systems in medical applica- 53. Wu C-H, Agam G. Probabilistic retinal vessel segmentation.
tions— clinical benefits and future directions. Int J Med Inf In: Medical Imaging. San Diego, CA: Proceedings of SPIE;
2004;73:1–23. 2007.
44. Stellingwerf C, Hardus P, Hooymans J. Two-field photography 54. Lee SC, Lee ET, Kingsley RM, et al. Comparison of diagnosis
can identify patients with vision-threatening diabetic retinop- of early retinal lesions of diabetic retinopathy between a com-
athy: a screening approach in the primary care setting. Diabe- puter system and human experts. Arch Ophthalmol 2001;119:
tes Care 2001;24:2086 –2090. 509 –515.
45. Lin DY, Blumenkranz MS, Brothers RJ, Grosvenor DM. The 55. Sinthanayothin C, Boyce JF, Williamson TH, et al. Automated
sensitivity and specificity of single-field nonmydriatic mono- detection of diabetic retinopathy on digital fundus images.
chromatic digital fundus photography with remote image in- Diabet Med 2002;19:105–112.
terpretation for diabetic retinopathy screening: a comparison 56. Ke Huang K, Yan M, Aviyente S. Edge directed inference for
with ophthalmoscopy and standardized mydriatic color pho- microaneurysms detection in digital fundus images. In: Med-
tography. Am J Ophthalmol 2002;134:204 –213. ical Imaging. San Diego, CA: Proceedings of SPIE; 2007.
46. Lim JI, Labree L, Nichols T. Comparison of nonmydriatic 57. Narasimha-Iyer H, Can A, Roysam B, et al. Robust detection
digitized video fundus images with standard 35-mm slides to and classification of longitudinal changes in color retinal fun-
screen for specific lesions of age-related macular degeneration. dus images for monitoring diabetic retinopathy. IEEE Trans
Retina 2002;22:59 – 64. Biomed Eng 2006;53:1084 –1098.
47. Dichtl A, Jonas JB, Mardin CY. Comparison between tomo- 58. Antani S, Long LR, Thoma GR. Content-based image retrieval
graphic scanning evaluation and photographic measurement of for large biomedical image archives. Medinfo 2004;11:829 – 833.
the neuroretinal rim. Am J Ophthalmol 1996;121:494 –501. 59. Lehmann TM, Guld MO, Deselaers T, et al. Automatic cate-
48. Iester M, DeFerrari R, Zanini M. Topographic analysis to gorization of medical images for content-based retrieval and
discriminate glaucomatous from normal optic nerve heads data mining. Comput Med Imaging Graph 2005;29:143–155.