IMAGE SEGMENTATION BY SPATIALLY ADAPTIVE COLOR AND TEXTURE FEATURES

Junqing Chen, Thrasyvoulos N. Pappas Aleksandra Mojsilovic, Bernice E. Rogowitz

Electrical and Computer Engineering Dept. IBM T. J. Watson Research Center

Northwestern University, Evanston, IL 60208 Hawthorn, NY 10532

ABSTRACT dian type of operation which, as we pointed out in [12], responds

We present an image segmentation algorithm that is based on to texture within uniform regions and suppresses textures associ-
spatially adaptive color and texture features. The proposed al- ated with transitions between regions. Finally, we use an elaborate
gorithm is based on a previously proposed algorithm but intro- border refinement procedure [13], which extends the idea of the
duces a number of new elements. We use a new set of texture adaptive clustering algorithm [7] to color texture, and results in
features based on a steerable filter decomposition. The steerable accurate border locations.
filters combined with a new spatial texture segmentation scheme The focus of this work is in the domain of photographic im-
provide a finer and more robust segmentation into texture classes. ages. The range of topics is essentially unlimited (people, nature,
The proposed algorithm includes an elaborate border estimation buildings, textures, indoor scenes, etc.). An important assumption
procedure, which extends the idea of Pappas’ adaptive clustering is that the images are of relatively low resolution (e.g., 200 × 200)
segmentation algorithm to color texture. The performance of the and occasionally degraded or compressed. The image segmenta-
proposed algorithm is demonstrated in the domain of photographic tion results can be used to derive region-wide color and texture
images, including low resolution compressed images. features. These can be combined with other segment informa-
tion, such as location, boundary shape, and size, in order to ex-
1. INTRODUCTION tract semantic information. A key to the success of the proposed
The field of Content-Based Image Retrieval (CBIR) has made sig- approach is the recognition of the fact that it is not necessary to ob-
nificant advances during the past decade [1, 2]. Many CBIR sys- tain a complete understanding of a given image: In many cases, the
tems rely on scene segmentation. However, image segmentation identification of a few key segments (such as “sky,” “mountains,”
remains one of the most challenging problems. While significant “people,” etc.) may be enough to classify the image in a given
progress has been made in texture segmentation (e.g., [3–6]) and category [14]. In addition, regions that are classified as complex
color segmentation (e.g., [7–9]) separately, the combined spatial or “none of the above,” can also play a significant role in scene
texture and color segmentation problem remains quite challeng- analysis.
ing [10, 11]. In Section 2, we review the color feature extraction. Our new
In [12], we presented an image segmentation algorithm that is approach for spatial texture feature extraction is presented in Sec-
based on spatially adaptive color and spatial texture features. The tion 3. Section 4 discusses the proposed algorithm for combining
perceptual aspects of this algorithm were further developed in [13], the texture and color features to obtain an overall segmentation.
including the use of a steerable filter decomposition instead of the
discrete wavelet transform. As we saw in [12], the resolution of 2. COLOR FEATURE EXTRACTION
the spatial texture segmentation is limited because it is defined on In this section, we review the color feature extraction algorithm
a finite neighborhood, while color segmentation can provide accu- that was used in [12]. The main idea is to use spatially adap-
rate and precise edge localization. In this paper, we improve and tive dominant colors as features that incorporate knowledge of hu-
refine the algorithm presented in [12, 13]. The main structure of man perception [12]. The color feature representation consists of
the algorithm remains the same, i.e., the color and spatial texture a limited number of locally adapted dominant colors and the cor-
features are first developed independently, and then combined to responding percentage of occurrence of each color within a certain
obtain an overall segmentation. While the color features also re- neighborhood:
main the same, we use a new set of spatial texture features based
fc (x, y, Nx,y ) = {(ci , pi ), i = 1, . . . , M, pi ∈ [0, 1]} (1)
on the steerable filter decomposition. The steerable filters com-
bined with a new texture segmentation scheme provide a finer and where each of the dominant colors, ci , is a three dimensional vec-
more robust segmentation into different texture classes (smooth, tor in Lab space, and pi are the corresponding percentages. Nx,y
horizontal, vertical, + 45 o , −45 o , and complex). A key to the represents the neighborhood around the pixel at location (x,y). M
proposed method is the use of the “max” operator to account for is the total number of colors in the neighborhood Nx,y .
the fact that there is significant overlap between the filters that cor- The spatially adaptive dominant colors are obtained by the
respond to the different orientations. This avoids misclassification adaptive clustering algorithm (ACA) proposed in [7] and extended
problems associated with the previously proposed texture extrac- to color in [8]. The ACA is an iterative algorithm that uses spatial
tion technique [13]. The “max” operation is followed by a me- constraints in the form of Markov random fields (MRF). The al-
This material is based upon work supported by the National Science
gorithm starts with global estimates (obtained using the K-means
Foundation (NSF) under Grant No.CCR-0209006. Any opinions, findings algorithm) and slowly adapts to the local characteristics of each
and conclusions or recomendations expressed in this material are those of region. We found that a good choice for the number of (locally
the authors and do not necessarily reflect the views of the NSF. adapted) dominant colors is M = 4.

0-7803-7750-8/03/$17.00 ©2003 IEEE. ICIP 2003

 to obtaining good segmentations from low-resolution compressed
images (e.g., 200 × 200 pixels).
Fig. 2 shows a circular cross-section from the steerable filter
responses. The x-coordinate denotes spatial orientation in degrees.
Thus, the filter with peak at 0o represents the horizontal subband
s0 , the filter with peak at 90o represents the vertical subband s2 ,
and so on. Note that there is a large overlap between neighbor-
ing filters. In [13], we account such overlap by comparing the 1st
and 2nd maximum among the four subband coefficients. However,
Fig. 1. One-Level Steerable Filter Decomposition this method could misclassify, as complex, textures with orienta-
tions that fall between the main orientations of the steerable filters.
Note that the ACA was developed for images of objects with That’s because for such textures the responses of the two filters
smooth surfaces and no texture. In textured regions, the ACA over- are close. The complex category should instead be reserved for
segments the image, but the segments do correspond to actual tex- textures with many different orientations. Note that using sharper
ture details. Thus, some other mechanism is needed to consolidate orientation filters will narrow the range of misclassified orienta-
these small segments into regions. Such a mechanism is provided tions, but will not entirely eliminate the problem. As we will see
by the “local histograms” we describe next and the texture classes below, we solve this problem by introducing a “max” operator, and
we present in the next section. using the local histogram of orientations to determine the texture
Color Features: The ACA provides a segmentation of the image orientation.
into classes. In the example of Fig. 5(b), each pixel is painted with The first step in the texture classification is to identify and lo-
the average color of the pixels in its neighborhood that belong to cate the smooth regions in the image. We use s0 (x, y), s1 (x, y),
the same class [7]. Assuming that the dominant colors are slowly s2 (x, y), s3 (x, y) to represent the steerable subband coefficients
varying, we can assume that they are approximately constant in at location (x, y) that correspond to the horizontal (0o ), diagonal
the immediate vicinity of a pixel. We can then count the number with positive slope (+45o ), vertical (90o ), and diagonal with nega-
of pixels in each class within a given window, and average their tive slope (−45o ) directions, respectively. For each image location
color values to obtain a feature vector that consists of a few (up (x, y), we find the maximum of the four coefficients, denoted by
to four) dominant colors and the associated percentages. Thus, smax (x, y). The subband index si (x, y) that corresponds to that
the color feature vector at each pixel is essentially a crude “local maximum is also stored for use in the next step. Then, a median
histogram” of the image. operation is performed on smax (x, y). Recall that the values in
Color Metric: To measure the perceptual similarity of two color smax (x, y) come from four different subbands; thus, the cross-
feature vectors, we use the “Optimal Color Composition Distance subband median can only help in determining whether a pixel be-
(OCCD)” proposed by Mojsilovic et al. [15]. The OCCD was longs to a smooth region, not which texture class it belongs to.
designed to provide the optimal mapping between the dominant Finally, a two-level K-means algorithm, segments the image into
colors of two images, and thus obtain a better measure of their smooth and non-smooth regions.
similarity. Since we are using OCCD to compare local histograms
A cluster validation step is necessary at this point. If the clus-
that contain only four bins, its implementation can be simplified
ters are too close, then the image may contain only smooth or non-
considerably [13].
smooth regions, depending on the actual value of the cluster center.
We have also experimented with alternative ways to obtain
3. SPATIAL TEXTURE FEATURE EXTRACTION
smooth vs. non-smooth classification. For example, we tried an
As in [12], the spatial texture feature extraction is independent approach similar to the one described in [12], whereby a median is
from that of color. Thus, we use the gray-scale component of the applied to each subband followed by a 2-level K-means. A pixel is
image to obtain the spatial texture features, based on which we then classified as smooth if all subbands are classified in the lower
obtain an intermediate segmentation, which is then used together class. This leads to similar results yet involves much more com-
with the color features described in the previous section to produce putation. Another approach is to apply a median to each subband,
the final image segmentation. followed by K-means applied to the vector of the four subband
As indicated in [13], the steerable filter decomposition [16] coefficients. We found that the proposed algorithm has the best
provides a finer frequency decomposition that more closely cor- performance in terms of accuracy and robustness.
responds to human visual processing. We use a one-level steer- The next stage is to further classify the pixels in the non-
able filter decomposition with four orientations which provide four smooth regions. As we discussed above, there is significant over-
texture classes: horizontal, vertical and two diagonal directions lap between neighboring directional filters. Thus, even in a texture
(+45o and −45o ), as shown in Fig. 1. of a single orientation (e.g., horizontal), the responses of the two
As noted in our earlier work [12], many textures are not di- neighboring filters will still be significant. Thus, the maximum
rectional; thus, it is necessary to include a complex or “none of of the four coefficients is the one that carries significant informa-
above” category. Even though such a category does not provide tion about the texture orientation. Based on this observation, we
much information about a given region, it nevertheless plays an use the index si (x, y) of the subband with the maximum value in
important role in the overall image classification. Note that even order to determine the texture orientation for each pixel location.
with the addition of the diagonal texture categories, the texture de- We then consider a window, and find the percentage of indices
scription is still quite crude. However, unlike the texture synthesis for each orientation. Only non-smooth pixels within the window
problem that requires a very precise model in order to accurately are considered. If the maximum of the percentages is higher than a
synthesize a wide range of textures, a crude model can be quite ad- given threshold (e.g., 36%) and the difference between 1st and 2nd
equate for segmentation. Such a simple model is actually the key maximum is significant (e.g., greater than 15%), we conclude that
 (a) Original image (b) Max index (c) Texture Classes
Fig. 3. Texture Map Extraction

Fig. 4. Illustration of border refinement.

any remaining small regions neighboring non-smooth texture re-
Fig. 2. Steerable Filter Frequency Response gions are relabeled as complex so that they can be considered in
the next step.
there is a dominant orientation in the window and the pixel is clas-
Next, we consider the non-smooth texture regions. First, we
sified accordingly. Otherwise, there is no dominant orientation,
use a region growing approach to obtain an initial “crude” seg-
and the pixel is classified as complex. Thus, our texture classifica-
tion is based on the local histogram of the indices corresponding to mentation that is based on the grayscale texture classification and
maximum subband values. This is essentially a “median” type of the color features presented in Section 2. Since both of these fea-
tures are slowly varying, we use a multi-grid approach. We start
operation, which is necessary, as we saw in [12], for boosting the
from pixels located on a coarse grid. We set the window size for
response to texture within uniform regions and to suppress the re-
the color feature equal to twice the grid spacing, i.e., there is a
sponse due to textures associated with transitions between regions.
An example is given in Fig. 3. Fig. 3 (a) shows the grey-level com- 50% window overlap. A pair of pixels belong to the same region if
the color features are similar in the OCCD sense. The threshold is
ponent of the original color image. Fig. 3(b) shows the matrix si of
higher for pixels that belong to the same texture class (i.e., easier
indices indicating the maxima. The smooth regions are shown in
to merge), and lower for pixels in different texture classes. In ad-
black, while the other 4 orientations are shown in shades of gray.
Fig. 3 (c) shows the resulting texture classes, where black denotes dition, we use MRF-type spatial constraints (as in [13]). That is, a
pixel is more likely to belong to a region if many of its neighbors
smooth, white denotes complex, and light grey denotes horizontal
belong to the same region. The symmetric MRF constraint makes
textures. The window used in this example was 23 × 23.
it necessary to iterate a few times for a given grid spacing. The
The window size for median operation should be reasonably
grid spacing is then reduced, and the procedure repeated until the
big to obtain an accurate estimate of the histogram and to suppress
grid spacing is equal to one pixel.
texture edges. On the other hand, a very big window will result
in texture classes too crude to be useful. Our experiments indi- Finally, the crude segmentation is refined using an adaptive
cate that a window size in the range of 17 × 17 to 29 × 29 works algorithm similar in nature to the ACA [7]. Fig. 4 illustrates the
well. Since the texture classes are obtained through window oper- idea. The dotted line in Fig. 4 represents the real boundary and the
ations, we know the texture boundaries are not accurate. Thus, we solid line denotes the boundary given by our algorithm in current
have to rely on the color texture features to obtain a more accurate iteration or an initial segmentation obtained in the previous step.
segmentation. Given a segmentation, we use two windows to update the clas-
4. FINAL SEGMENTATION sification of each pixel. The larger window provides a localized
estimate of texture characteristics of each region that overlaps the
We now discuss the combination of texture and color features to window. For each texture, within the segmentation boundaries, we
obtain the final segmentation. First, we consider the smooth tex- find the average color and the corresponding percentage for each
ture regions. As in [12], we rely on the color segmentation which of the dominant colors. The smaller window provides an estimate
provides regions of different uniform colors. Recall that the ACA of the pixel texture. This consists of the dominant colors corre-
provides slowly varying color. To avoid oversegmentation, we find sponding percentages within the smaller window ignoring the cur-
all the connected segments that belong to different color classes, rent boundary. Then the texture of the pixel is compared with the
and then merge neighboring segments if the average color differ- textures of the different regions using the OCCD criterion. The
ence across the common border is below a given threshold. Finally, procedure is repeated for each pixel in a raster scan. As in [7],
 (a) Original Color Image (b) Color Segmentation (ACA) (c) Texture Segmentation

(d) Crude Segmentation (e) Final Segmentation (f) Final Segmentation (on original image)
Fig. 5. Color and Texture Image Segmentation (a,b,d,e,f shown in color)
an MRF-type constraint is necessary to insure region smoothness. [6] T. Randen and J. H. Husoy, “Texture segmentation using
A few iterations are necessary for convergence. An iteration has filters with optimized energy separation,” IEEE Tr on Im. Pr.,
converged when the number of pixels that change class is below a vol. 8, pp. 571–582, Apr. 1999.
given threshold. The overall procedure must then be iterated for a [7] T.N. Pappas, “An adaptive clustering algorithm for image
series of window pairs starting from 35/5 and ending with 11/3. segmentation,” IEEE Tr. SP, vol. 40, pp. 901–914, Apr. 1992.
One of the important details in the above procedure is that each [8] M.M. Chang, M.I. Sezan, and A.M. Tekalp, “Adaptive
of the candidate regions in the larger window must be large enough Bayesian segmentation of color images,” JEI, vol. 3, pp.
in order to obtain a reliable estimate of its texture attributes. Oth- 404–414, Oct. 1994.
erwise, the region is not a valid candidate. A reasonable choice [9] D. Comaniciu and P. Meer, “Robust analysis of feature
for the threshold for deciding whether a region should be a valid spaces: Color image segmentation,” CVPR, June 1997, pp.
candidate is to use the product of the two window sizes divided by 750–755.
2. The crude and final segmentation results are shown in Figs. 5(e) [10] S. Belongie, et al., “Color- and texture-based image segmen-
and (f). tation using EM and its application to content based image
retrieval,” ICCV, 1998, pp. 675–682.
The use of both color and texture information to estimate re- [11] Y. Deng and B.S. Manjunath, “Unsupervised segmenta-
gion boundaries finds further justification in psychophysical ex- tion of color-texture regions in images and video,” IEEE
periments [17], which showed that the perceived edge location is a Tr. PAMI, vol. 23, pp. 800–810, Aug. 2001.
combination of the position signaled by texture and by other cues [12] J. Chen, T. N. Pappas, A. Mojsilovic, and B. Rogowitz,
(motion, luminance, color etc). “Adaptive image segmentation based on color and texture,”
Proc. ICIP 2002, Rochester, New York, Sept. 2002.
5. REFERENCES [13] J. Chen, T. N. Pappas, A. Mojsilovic, and B. Rogowitz, “Per-
[1] Y. Rui, T.S. Huang, and S.-F. Chang, “Image retrieval: ceptual color and texture features for segmentation,” Human
Current techniques, promising directions and open issues,” Vision and Electronic Imaging VIII, Jan. 2003, Proc. SPIE
J. Vis. Comm. Im. Repr., vol. 10, pp. 39–62, Mar. 1999. Vol. 5007.
[14] A. Mojsilovic and B. Rogowitz, “Capturing image semantics
[2] W.M. Smeulders, et al., “Content-based image retrieval at
with low-level descriptors,” Proc. ICIP, Oct. 2001, pp. 18–
the end of the early years,” IEEE Tr. PAMI, vol. 22, pp. 1349–
21.
1379, Dec. 2000.
[15] A. Mojsilović, J. Hu, and E. Soljanin, “Extraction of percep-
[3] A. Kundu and J.-L. Chen, “Texture classification using QMF tually important colors and similarity measurement for image
bank-based subband decomposition,” CVGIP, GMIP, vol. 54, matching, retrieval, and analysis,” IEEE Tr. Im. Pr., vol. 11,
pp. 369–384, Sept. 1992. no. 11, pp. 1238–1248, Nov. 2002.
[4] T. Chang and C.-C.J. Kuo, “Texture analysis and clas- [16] E.P. Simoncelli and W.T. Freeman, “The steerable pyramid:
sification with tree-structured wavelet transform,” IEEE A flexible architecture for multi-scale derivative computa-
Tr. Im. Pr., vol. 2, pp. 429–441, Oct. 1993. tion,” Proc. ICIP, vol. III, Oct. 1995, pp. 444–447.
[5] M. Unser, “Texture classification and segmentation using [17] J. Rivest and P. Cavanagh, “Localizing contours defined by
wavelet frames,” IEEE Tr. Im. Pr., vol. 4, pp. 1549–1560, more than one attibute,” Vision Research, vol. 36, pp. 53–66,
Nov. 1995. 1996.