Proceedings of the International Conference on Theory and Applications of

Mathematics and Informatics - ICTAMI 2004, Thessaloniki, Greece

AUTOMATIC TRAFFIC SIGN RECOGNITION

IN DIGITAL IMAGES
by
John Hatzidimos

Abstract. The objective of this project is the development of an algorithm for the automatic
recognition of traffic signs in digital images. The program An.Si. was created (from the Greek
words Anagnorisi Simaton which means Sign Recognition). Up to now, many algorithms for
the traffic sign detection and classification have been introduced. Extensive research is being
made by major car manufacturing companies in collaboration with Universities and other
institutes on real-time and automatic recognition of traffic signs, so that it can be a part of the
so called “Driver Support Systems” ([7]).
Two major problems exist in the whole detection process. Road signs are frequently
occluded partially by other vehicles. Many objects are present in traffic scenes which make the
sign detection hard (pedestrians, other vehicles, buildings and billboards may confuse the
detection system by patterns similar to that of road signs). Colour information from traffic
scene images is affected by varying illumination caused by weather conditions, time (day-
night) and shadowing (buildings) ([7]).
The proposed method detects the location of the sign in the image, based on its geometrical
characteristics and recognises it using colour information. Partial occlusion is dealt by the use
of the Hough Transform and suggestions are made for future improvements so that the
robustness of the algorithm in light condition changes can be increased.

1. Introduction

The whole process is part of a larger project concerning the Mobile

Mapping. Mobile Mapping is the automated “mapping”-registration of features
of interest from images acquired from a moving registration system, mounted
either on an airplane or a vehicle. Research is being made, mostly in land-
based Mobile Mapping Systems ([1], Tsioumas 2003).
Apart from traffic signs, other features of interest are traffic lights, road
centerlines and edges and building facades. For building facades, the
registration system doesn’t necessarily have to be movable. For the rest of the
features, effort is being made for real time recognition and mapping, so they
can be a part of the so called “Driver Support Systems”. The DSS is in a way
the predecessor of the car autopilot.

174
 John Hatzidimos - Automatic traffic sign recognition
in digital images

Traffic sign recognition is part of the general case of Pattern Recognition.

Major problem in pattern recognition is the difficulty of constructing
characteristic patterns (templates). This is because of the large variety of the
features being searched in the images, such as people faces, cars, etc.. On the
contrary, traffic signs a) are made with vivid and specific colors so as to attract
the driver’s attention and to be distinguished from the environment b) are of
specific geometrical shapes (triangle, rectangle, circle-ellipse) and c) for each
sign there is a specific template. It is therefore rather easy to develop an
algorithm in such a way that the computer has “a priori knowledge” of the
objects being searched in the image.
The developed algorithm is divided in two basic phases each one composed
of a certain number of steps. In the first phase the detection of the location of
the sign center of gravity (which is used as a location reference point) in the
image coordinate system is carried out, based on its geometric characteristics.
The second phase is the sign recognition with the matching between the search
image and the template images, already stored in a database.
The programming language used is IDL (Interactive Development
Language), of Research Systems Inc., version 5.4. It was selected because of
its simple syntax (similar to Visual Basic), but mainly because it is focused in
graphic and image processing. Also, it is very effective in matrix manipulation,
which is very important, considering the fact that images are processed in
matrix form.

2. The algorithm

The basic steps of the algorithm in each phase are:

a. First phase: detection of the location of the sign in the image

1. ROI segmentation with image thresholding

The first step of the algorithm is the region of interest segmentation,
using sign color information. A “region of interest” is an area of the
image that may contain a traffic sign. A new black and white image is
constructed in which all the pixels that satisfy certain thresholds of the
sign color are black and the background is white (binary image – an
image with only two possible pixel values). The color thresholds are
measured in test images.

175
 John Hatzidimos - Automatic traffic sign recognition
in digital images

First image Image after colour thresholding

(binary image)

2. Thinning and Edge Detection

After image thresholding, a thinning algorithm is implemented. The
goal of this process is the reduction of the edge thickness in the binary
image. In result, the edges after the implementation have a thickness of
one pixel. This process makes it easier for the algorithm to detect lines
in a later step and also increases its speed (less pixels of interest). In the
case of blue traffic sings which don’t have a clear outline, an edge
detection filter is implemented (Roberts filter) for the removal of their
interior pixels.

3. Region identifying and region clustering

Region identifying is the calculation of the down left and upper right
apex coordinates of the rectangles which include the regions. This way
the search is constrained only in the regions of interest. After that, the
regions are clustered according to their total number of pixels and their
176
 John Hatzidimos - Automatic traffic sign recognition
in digital images

center distances. Furthermore, regions with total number of pixels less

than a certain value are eliminated and not examined in the later steps.
4. Line Detection
In each roi, line detection is carried out so as to check the kind of shape
in the roi (triangle, rectangle, circle – ellipse). In the case of the triangle
and the rectangle, the line detection process also calculates the
necessary data for the calculation of the sign’s centre of gravity
coordinates. The Hough Transform is used for the detection, a unique
and effective way for the analysis of shapes and movement in images
that include noise, missing or surplus data. It is a curve detection
technique, that can be applied when the object location is unknown but
its shape can be described as a parametric curve ([5], Adamos, and
Faig).
5. Shape Check
The angles between the lines can be now calculated, using the line
direction coefficients deprived from the previous check. If there are
three angles between [50,70] or [-70,50] degrees then the shape is
accepted as a triangle. In the other case, the algorithm continues with
the ellipse detection step.
6. Hough transform for ellipse detection.
For the ellipse detection the algorithm uses a different approach of the
Hough Transform, the RHT (Randomized Hough Transform) ([4],
Inverso, May 2002). The general idea of the transform is the same. The
term “Random” is used because the pixels of interest in the current roi
are being tested in random groups of three. After the detection of an
ellipse, the algorithm runs a check whether this ellipse is acceptable or
not, based on its eccentricity and the number of points that belong to the
ellipse. In the case of an accepted ellipse, detection of the location of
the sign is complete because during the ellipse detection process, the
center coordinates have been calculated. From this point on, the ellipse
is considered a circle with a radius equal to the large half-axis. If there
isn’t an accepted ellipse, the algorithm loops back to the line detection
step and examines the next roi.
7. Calculation of the apexes and the center of gravity (triangle and
rectangle sign

177
 John Hatzidimos - Automatic traffic sign recognition
in digital images

From the line equations, the coordinates of the center of gravity and of
the apexes are calculated. The coordinates of the apexes are needed in
the later step of the template transformation. At this point the detection
of the sign location is complete.

b. Second phase: recognition of the sign.

The sign recognition is carried out with the use of the cross-correlation
matching between the region of interest extracted from the previous steps and
specific template images.
For the matching process, the two images must have the same coordinate
system; therefore the templates are transformed to the search image coordinate
system.

1. Affine transformation (triangle, rectangle).

The two coordinate systems are different by two different components
of parallel translation in each axis, in two rotation angles and in two
scale coefficients. The common points used for the calculation of the
transformation parameters are the apexes of the detected shape. Their
coordinates in the template system are already known and their
respective ones in the search system have been calculated in the
location detection phase. All the template images are transformed and
the new images constructed have the same dimensions with the search
image. The template is located in the same place as the detected sign.

178
 John Hatzidimos - Automatic traffic sign recognition
in digital images

2. Similarity transformation (circle)

In the case of a circle sign, there is only one common point, the center,
therefore the affine transformation cannot be used. A similarity
transformation is used instead. It is considered that the two systems are
different by one scale coefficient (same by x and y) and by two
different components of parallel translation in each axis. The scale
parameter is calculated from the ratio of the sign radius in the two
images. In the same way as in the affine transformation process, new
images are constructed for each template.
3. Search area definition (bounding box)
The cross correlation coefficient is calculated for the entire roi. The
search image pixels which are included in the roi but do not belong to
the sign, affect the correlation process and may lead in wrong results.
Their colour is changed to that of the background, so their influence is
eliminated. In the case of the circle sign, it is easy to determine whether
a pixel belongs to the sign or not. All the pixels with a distance to the
circle center greater than the radius are whitened (set to the background
colour). In the case of triangles and rectangles those pixels are found
using a “Point Inclusion In Polygon Test” ([6], WR Franklin). In the
new image, all the pixels with a distance to the circle center greater than
the radius are whitened (set to the background color). The pixels
outside the region are not of any interest because the calculations are
only made for the pixels inside the region, so they are left black. The
sign pixels maintain their colour.

4. Cross correlation matching

For every roi pixel, the cross correlation coefficient ([3]) between the
template and the search image is calculated, for each colour channel
179
 John Hatzidimos - Automatic traffic sign recognition
in digital images

(red, green and blue). The final coefficient is the mean of the three
RGB coefficients. The template with the largest coefficient corresponds
to the sign searched.

C. SEPARATE RECTANGLE CASE

Rectangle signs are detected with a slight altered and separate process
because of two basic differences:
a) During the Hough Transform, only (approximately) vertical and horizontal
lines are searched
b) The correspondence of the common points in the affine transformation step
is more complicated to program than that of the triangle shape.

3. Results

Below are some digital images, which the algorithm was tested:

The algorithm successfully detected and identified the sign.

180
 John Hatzidimos - Automatic traffic sign recognition
in digital images

The algorithm successfully detected and identified the sign.

The algorithm successfully detected and identified both of the signs. However,
it accepted the region at the left edge of the image as a sign, because of its
similar blue colour and its rectangle shape.

The algorithm successfully detected the sign location, but the greater cross-
correlation coefficient was that of the sign . That’s due to the shadowing

of the sign in the search image, which made it’s interior colour more similar to
light blue than the white of the correct sign:

181
 John Hatzidimos - Automatic traffic sign recognition
in digital images

4. Conclusions – future directions

Implementation of the algorithm in test images showed that it is very

effective in the sign location phase. There is a slight weakness in the second
phase, in cases of color similarity between signs and other areas of the image.
It is sensitive in light condition changes during the image acquisition, because
of the effect they have in the color thresholds used in the regions of interest
segmentation step. The use of proper thresholds is very important as it affects
in a great deal the success of the sign location detection and it’s final
recognition.
Based in the experience acquired from the tests, the aspects which should be
further researched and be improved in the future are:
• Recognition of signs of more complex shape (eg. Stop sign-octagon, right of
way-diamond). The shape of such signs could be detected using another
variation of the Hough Transform, the Generalized Hough Transform. The
GHT can detect shapes in an image, which doesn’t necessarily have to be
described by an analytical mathematical equation such as the triangle or the
circle.
• Recognition of two (or more) signs in the same region of interest.
It is possible in some cases that there two or more signs in the same region of
interest.

Such a problem can be dealt with changes in the algorithm’s structure

• Increase of the speed of the algorithm by improving the source code and
again, by possible changes in its structure.

182
 John Hatzidimos - Automatic traffic sign recognition
in digital images

• Increase of the robustness of the algorithm in light condition changes.

The RGB representation is far from the human concept of colour. Furthermore,
image processing of the RGB model has several disadvantages:
1. all three components (R, G, and B) depend on intensity,
2. colours, which for a human might be perceptually close, do nat
have to be close to each other (Euclidean distance) in RGB
space.
3. smoothly shaded surfaces might correspond to several clusters
in a colour space.
These factors indicate that colour thresholding, a step very essential to the
whole process, is greatly affected by light condition changes. One possible
solution to the problem is the contiguous replacement of the threshold values
during the acquiring of the images, by photographing a metal plate constructed
with the same colour as the signs in the current light conditions and
measurement of the new threshold. Alternatively, the HSI colour system could
be used instead of the RGB. The HSI system (Hue, Saturation, Intensity) is
much less sensitive in light condition changes than the RGB ([2],
Gadzamowicz, April 1999.)
• Merging of the rectangle and triangle-ellipse detection process
• Use of “Neural Networks”.
“Neural Networks” is a method for pattern recognition which has recently
started being researched and is associated with the Artificial Intelligence. It
simulates the human nervous system and it uses its ability to “learn” in order to
find “recognisable” shapes and objects in images. Programming of “Neural
Networks” is much more difficult, but also a more effective and faster
recognition process.

References:

[1] Introduction to Mobile Mapping Systems, Tsioumas Michael, Post

Graduate Thesis, May 2003
[2] Automated Processing of Georeferenced Color Stereo Images for Road
Inventory, Krzysztof Gajdamowicz Dissertation, April 1999.

183
 John Hatzidimos - Automatic traffic sign recognition
in digital images

[3] Detection and sub-pixel location of photogrammetric targets in digital

images, Photogrammetria, Elsevier Science Publishers B.V., Amsterdam.
[4] Ellipse Detection Using Randomized Hough Transform, Samuel Inverso,
May 2002.
[5] Hough Transform in Digital Photogrammetry, C. Adamos and W. Faig.
Department of Surveying Engineering, Univ. of New Brunswick, Canada
Comission III.
[6] PN-POLY, Point inclusion in polygon test, WR Franklin (WRF),
www.ecse.rpi.edu/Homepages/wrf/
[7] Road Sign Recognition Survey, http://euler.fd.cvut.cz/research/rs2/

Author:
John Hatzidimos - Aristotle University of Thessaloniki, Greece, E-mail
address: johnhatz111@hotmail.com

184