FAST GEOREFERENCING IMAGES THROUTH GENERALIZED PHOTOGR AMMETR IC ALGORITHMS

Chao-Hong Liu

Department of Computer Science and Information Engineering, National Dong Hwa University, 1, Sec. 2, Da Hsueh Rd., Shou-Feng, Hualien, Taiwan - chliu@gnu.org
Working Group I/6

KEY WORDS: Photogrammetry, Algorithms, Georeferencing, Image, Three-dimensional, Geometry, Triangulation

ABSTRACT: Traditional photogrammeric workstations for georeferencing images are usually time- and human resource-consuming, input dat a source specific, and lack of flexibility to use in a generalized triangulation problem. T hese characteristics have made them inadequate to be used in case of emergency. The approach presented in this paper started after Chi-Chi Earthquake of Richter Magnitude 7.3 struck central Taiwan on September 11th, 1999, in response to the situation that we did not have a proper tool to deal as soon and easy as possible with the problem that the Earth surface had been changed, and acceptable data of what it was is still not available even a few months later. In responding the need of a proper tool to deal with, generalized photogrammetic algorithms are developed totally based on geometry to reduce the computing overheads needed to calculate space resection and relative orientation. A hexagonal approach to pattern matching is also provided in this paper to reduce the intensive time spent in the process of feature extraction.

1. INTRODUCTION The approach presented in this paper is a direct response to the need in the survey of surface rupture of Chelungpu Fault invoked by Chi-Chi Earthquake of Richter Magnitude 7.3 struck central Taiwan on September 11th, 1999 (Liu et al., 2000). The situation encountered is that the Earth surface had been changed, and acceptable data of what it was is still not available even a few months later. This resulted in the use of the older and incorrect maps for almost half a year by the in situ disaster relief, geological surveying, civil engineering, and reconstruction teams. It is a critical problem since the terrain had been drastically changed in some areas, and the decision maker had no idea where the urgent resources should be placed. Although several techniques have been tried to derive the displacement data of surface rupture after the quake, the use of aerial photos through photogrammetry seems to be the last and the only one applicable technique. This paper presents how to fast georeference images taken by any source, such as aerial photos and surveyors field photos, through a generalized photogrammetric algorithm to ease the situation by providing acceptable geographic information of a vast area in few days. The design principles of the algorithms are 1) independent of specific input data sources, 2) relevant to geometry, 3) built-in with three-dimensional architecture, and 4) no default automation will be executed or ignored if ambiguous conditions are encountered, in which a black box like proprietary software

generally could not provide. To make sure each calculation is explicitly defined by primitive mathematical equations from the bases, a concise fundamental spatial library and a photogrammetric package for the algorithm, both written in Java and available as Free Software under GNU General Public License version 2, are developed. The algorithms could be regarded as generalized triangulation methods since they are designed and programmed totally based on geometry, and are used to derive the displacement data of surface rupture of Chelungpu Fault. In order to make them more powerful and efficient in practical uses, the development of the software will be gradually migrated into ANSI C/C++, will the Java version still be usable and under maintenance, however. A hexagonal pattern recognition algorithm is also presented in this paper. The preliminary version of this algorithm was first written in ArcView/Avenue and AML, and as the Avenue scripting language is no longer under maintenance, these codes are planned to re-program in Java and GRASS in their next versions.

2. GENERALIZED ALGORITHMS Since the major applications of photogrammetry lie on aerial usage, most workstations are designed and well developed for this specific application. This unfortunately makes them inadequate to be used in a generalized photogrammetric problem, and forces MIT City Scanning Project to develop their own

algorithms and imaging system of photogrammetry (Ho, 1995; Schenkel and Teller, 1998). Although the system of MIT City Scanning Project is used in a specific application, the concept of hemisphere used in their algorithm can be regarded as an attempt to have the images be transformed into a better geometric form that can be easily handled. This idea is pretty much the same as to calibrate and rectify the images of photographs. This is a good approach, but the algorithm is designed to handle images of the geometric shape of hemisphere, and this has limited its further applications. There are several approaches toward a more generalized photogrammetry , and they can usually be regarded as multiplecamera photogrammetry (El-Hakim et al., 1998; Mostafa et al., 2001; Zhang et al., 2000). In this paper, the author presents another approach through the development of general photogrammetric algorithms, and the objective is to have the system to accept different input data sources in a single photogrammetric process. This means that even the photographs are taken by different cameras with totally different interior orientations, the algorithm will still be working normally. The design principles of the algorithms are: 1. Independent of specific input data sources 2. Relevant to geometry 3. Built-in with three-dimensional architecture 4. No default automation will be executed or ignored if ambiguous conditions are encountered Here below shows the detailed geometries and descriptions of the algorithms. 2.1 Geometry of space resection The design of one of the two geometric algorithms is used to solve space resection problem: suppose that we have three point s g0, g1, g2 of known geographic coordinates (Fig. 1), their corresponding image points p0, p1, p2 in a photograph, and the focal length are known values . How to re-construct the position and orientation of the camera at the time that it took the photograph, in terms of a positional vector of the coordinate system of the three ground points? With the problem stated above, and according to the geometry shown in Fig. 1, the following notations are defined: f0 c0 o0 p0, p1, p2 g0, g1, g2, g3 gx q1, q2 r1, r2 s 1, s2 Focal length of photograph 0 Focus of photograph 0 FC of photograph 0 Image points of g0, g1, g2 Control points (with known coordinates ) A randomly selected point of reference Line q1-q2 is a parallel with p1-p2 Nearest points of g0 on lines c0-p1 and c0-p2 Two proposed values of g1, g2
r1

c0 p0 f0 o0 Photograph 0 p1 p2 m c0 s1 d r1 s1 d m b a gx n s2 e r2 q1 g1 z x a n c s2 e r2 q2 g2 b y g0 Locations and Lengths. g3

Figure 1:

T he geometry of proposed space resection method. It should be noted that gx is not necessary g0, and in most cases, gx is not g0. It was drawn as the same point to show the geometry more clear.

A pinhole perspective projection model is assumed in this algorithm, and re-constructing the position and orientation of the photograph 0 is stated in the following: By providing a value of d, the algorithm calculates its corresponding two values of e, as shown in the lie-flat lower left drawing, and derives their c values by the geometry of triangle c0-s 1-s 2 or the distance between s 1 and s2. The orientation of photograph 0 is derived when m :n:c is of the same proportion with x:y:z . Spatial coordinates of g0, g1, g2, their corresponding image points p0, p1, p2, the interior orientation of photograph 0, and g3 are known values, where g3 is used to select which of the two candidates is the right one. It should be noted that the position and orientation, combined represented as a positional vector, will, in most cases, have two possible candidates since the procedure is generalized based on geometry. And additional information, i.e. g3, is needed to determine which the actual one is. The ratio of the three sides of triangle p0-p1-p2 is proportional to its counterpart of triangle g0-q1-q2, therefore these two triangles are parallel in 3D space. The lower left drawing of Fig. 1 is a lieflat version of part of the geometry. It shows the basic idea we used to find two possible candidate geometries of m :n:c. 2.2 Relative orientation It is sufficient if we take every photograph ideally with enough control points when we do photogrammetry only by space resection. In practice, however, most of the photographs do not even have a single control point, and this is why we have to develop another algorithm, with similar geometry of Fig. 1, to determine how adjacent photographs are related.

vector to c0, photograph 0s focus. vector to c1 s4

c1 f1 o1 Photograph 1

3.1 Pattern matching and feature extraction Pattern matching techniques are mostly used to compress images by replacing several duplicated, or almost the same, patterns in one or a sequence of images , and their applications in this field are proved to be very effective. These techniques can also be used to extract a feature by recognizing the pattern and its geographical relationships with other features of the same image space. And the procedure is often called a feature extraction. Lay et al. (2004) have shown a feature extraction method through the use of basic grid arithmetic, and this technique is also used in the proposed algorithm.
1 I 2 3 II 4 5 III 6 I (a) II (b) III

s3 g0

Line 0

Line 1

s3 v3

Line 2 s1 g1 s2 g2

t3

Figure 2:

The geometry of relative orientation.

Additional notations used in Fig. 2: f1 c1 o1 s 3, t3 v3 Focal length of photograph 1 Focus of photograph 1 FC of photograph 1 Nearest points on Line 1, Line 2 to each other A vector from t3 to s 3

Figure 3:

The geometry of relative orientation of this algorithm is shown in Fig. 2. To determine how two adjacent photographs of a common sight are relatively located and oriented, in terms of the coordinate system provided by photograph 0, the relative location and orientation of photograph 1, represented as a pointvector defined by point c1 and vector c1-o1, is derived by the geometry shown in Fig. 1 through an invariant g0 and two variants s 1 and s 2 proposed initially by photograph 0. The correcting vectors v3 and v4 (not shown here) are used to guess closer coordinates of s 1 and s 2 to g1 and g2, respectively. Ideally, the norms of v3 and v4 should be zeros, but we usually set a threshold, such as two- to five- pixel width length, to proceed.

Pyramidal process and m apping of a regular raster space into a hexagonal space and an n-row regular space is t ransited into a 2n-row hexagonal space. The transition is used as the first-step of pyramidal process in the algorithm. (a) Consecutive pyramidal processes of the algorithm. The first two steps are shown in this diagram. (b) Diagram of the mapping algorithm.

Although the processes used in pattern matching and feature extraction are practically the same, they have a major difference in their basic ideas: pattern matching is intended to locate where there are patterns of as many as areas in images are, to their most extent and under acceptable errors, the same; whereas feature extraction used in photogrammtry application is intended to find where there are similar features, and most importantly, where they are going to differentiate? These minor differences are just as important as matched features are in photogrammetry. 3.2 Hexagonal space

3. PATTERN RECOGNITION One of the two major topics to enable fast georeferencing images is pattern recognition of images. Although the author presents in this paper first with two generalized geometric photogrammetric algorithms since they are of the very fundamental enablers of photogrammetry, the most resourceconsuming task lies in pattern recognition. Just as Celikoyan et al. (1999) pointed out that it takes a lot of time to match the continuous non-geometric items. It is of the most key techniques since we must first identify the common features of two different images of a common sight of interest, and the p reviously developed algorithms can then be used to calculate the relative orientations of a group of images. Only when this technique is developed can we finally design an eligible automatic digital photogrammetric system.

Hexagonal spaces is adopted by the algorithm, and the reasons are listed below: Distances from any hexagonal grid to its six adjacent neighbours are equal. For small angles of rotation, images of hexagonal space have shown to have a better representation over regular square space (Tirunelveli et al., 2002). Mapping from a regular raster space into a hexagonal space can easily be done by simple grid arithmetic (Fig. 3). Hexagonal spaces provide six equally scaled profiles for use of pattern matching (Fig. 4).

In Fig. 3, the g rid of bold line denotes that the value of a hexagonal grid is directly assigned as the value of the grid value of regular space where its centre resides, and this is adopted by the current algorithm; Grid of dotted line denotes that the value of a hexagonal grid is assigned by several values of the nearest grids of regular space. Noted that grid values of odd-number columns of hexagonal space are derived from even-number row of regular space, and grid values of even-number columns of hexagonal space are derived from odd-number row of regular space. Of the same scale of x and y directions, there is no one-to-one and pixel-to-pixel mapping between these two raster spaces exists. The proposed mapping algorithm is not designed to preserve all information kept in the original regular raster space.
5 55 45 44 4 (100 2) 64 6(110 2) 66 26 22 (a) (c) 2(0 10 2) 32 65 80 75 85 95 0(000 2) 3(0112) (b) 33 2 5(1 01 2) 1(0012) 13 6 3 51 11 4 1

process of calibration. Intuitively, an imperfect lens with distortion can be calibrated through a matrix of distortion values , and this characteristic would be a major advantage for a nonmetric camera to be used in photogrammetry. The author has also shown how to fast georeference images through the use of grid arithmetic and hexagonal spaces. The known abilities of grid arithmetic are used to implement pattern recognition functionality, and the nature of hexagonal spaces ensures the behaviours of each grid have equal relationships with the grids six adjacent neighbours.

5. CONCLUSION The author presents in this paper newly designed generalized algorithms for photogrammetry , totally based on the characteristics of perspective geometry and an approach toward pattern recognition by the use of grid arithmetic and hexagonal spaces. These two branches of techniques are both intended to improve the efficiencies of generalized photogrammetric tasks by designing new algorithms to reduce the overheads needed to complete the calculations. The latest information of related source codes of the algorithms presented in this paper can be download from the authors working page: http://savannah.gnu.org/users/chstoneliu/.

Figure 4:

The use of hexagonal space in pattern matching. (a) Numerating of neighbours of a grid in question in a hexagonal raster space. (b) The six lines of profile provided by the nature of hexagons. (c) An example profile used in pattern matching, where the left-most hexagon is the grid in question.

ACKNOWLEDGEMENTS The author would like to thank Syue-Ping Ma for her useful idea in simplifying the geometry of space resection. The general support on grid arithmetic of Dr Jinn-Guey Lay is also gratefully acknowledged.

4. RESULTS AND DISCUSSION As the author described above, to make sure each calculation is explicitly defined by primitive mathematical equations from the bases, a concise fundamental spatial library is developed, and it plays a role of primitives in implementing the algorithms shown in Fig. 1 and Fig. 2. This photogrammetric package is already used to derive the displacement data of the surface rupture of a test area in northern segment of Chelungpu Fault, and the results are prepared for further publication. The architecture of using a spatial library was designed in the beginning of developing the photogrammetric algorithms. As the algorithms are developed, this library seems to be unnecessary although it is in essence a concise one. The geometry is in such a simple form that the author has implemented part of the algorithms as an ASIC (Application Specific Integrated Circuits) in Verilog. It should be an easy task for a single experienced programmer to implement these algorithms, too. Since the photogrammetric algorithms are developed totally based on geometry, there would be some influences on the

REFERENCES Celikoyan, T. M. and Altan M. O., 1999. Automatic matching of non-geometric items in digital photogrammetry. Third Turkish-German Joint Geodetic Days, 1-4 June 1999, Istanbul, Turkey, pp. 423430. El-Hakim, S. F., Brenner, C., and Roth G., 1998. A multisensor approach to creating accurate virtual environments. ISPRS Journal of Photogrammetry & Remote Sensing 53 (1998), pp. 379391. Ho, W. S., 1995. Aerial photogrammetry (in Chinese). Great China Books Ltd., Taiwan. Lay, J.-G., Liu, C.-H., Lee, J.-F., and Wang, W.-L., 2004. Morphometric Analysis of Landslides (in Chinese). Submitted to Bulletin of Geography for further publication, Taiwan. Liu, C.-H., Lee, J.-F., and Huang, C.-C., 2000. Threedimensional displacement in northern segment of Chi-Chi Earthquake surface rupture. International Workshop on Annual

Commemoration of Chi-Chi Earthquake, September 18-20 2000, Taipei, Taiwan. Mostafa, M. M. R. and Schwarz, K.-P., 2001. Digital image georeferencing from a multiple camera system by GPS/INS. ISPRS Journal of Photogrammetry & Remote Sensing 56 (2001), p p . 112. Schenkel, T. and Teller, S., 1998. MIT City Scanning Project. September 1998. http://city.lcs.mit.edu/city.html (accessed 12 Nov. 2003) Tirunelveli, G., Gordon, R., and Pistorius, S., 2002. Comparison of square-pixel and hexagonal-pixel resolution in image processing. Proceedings of the 2002 IEEE Canadian Conference On Electrical & Computer Engineering, Banff, Canada. Zhang, Z., Shan, Y., and Loop C., 2000. Object modelling from a casual collection of images. 4th Asian Conference on Computer Vision, Taipei, Taiwan, January 2000, pp. 905910.