Aerial Data Acquisition for Disaster Management: Exploiting the Potential of Integrated Sensor Orientation

Khoshelham et al., Aerial data acquisition for disaster management Aerial data acquisition for disaster management: exploiting the potential of integrated sensor orientation Kourosh Khoshelham Optical and Laser Remote Sensing Delft University of Technology The Netherlands k.khoshelham@tudelft.nl Ben Gorte Optical and Laser Remote Sensing Delft University of Technology The Netherlands b.g.h.gorte@tudelft.nl ABSTRACT Aerial photogrammetry is an efficient technique for data acquisition over large areas. The georeferencing of aerial images often relies on the availability of ground control information. The requirement for the field measurement of ground control points is a limiting factor in the application of aerial photogrammetry as a rapid response data acquisition technique for crisis management. In recent years, the integration of GPS and INS has provided the opportunity of georeferencing aerial images without ground control, by directly measuring the position and attitude of the camera at the time of photography. In this paper, we investigate the accuracy potential of an integrated orientation approach for the georeferencing of aerial images without control points. Experimental results demonstrate that improved accuracy in image space can be achieved by the integrated orientation approach when a larger number of tie points is introduced in the adjustment process. Also, it is shown that variations in the initial weights for the tie points and GPS/INS measurements result in variations of the errors in both image and object space. The dependence of accuracy on the initial weights suggests that variance component estimation should be seen as a necessary process in the integrated orientation approach. Keywords Photogrammetry, sensor orientation, bundle adjustment, aerial triangulation, direct georeferencing. INTRODUCTION Aerial photogrammetry is an efficient technique for data acquisition over large areas. The georeferencing of aerial images relies on the availability of ground control information. The requirement for the field measurement of ground control points is a limiting factor in the application of aerial photogrammetry as a rapid response data acquisition technique for crisis management. In the event of natural disasters such as earthquake, bush fire, and flood, the collection of ground truth information is extremely difficult and, in most cases, impractical. In recent years, the integration of GPS (Global Positioning System) and INS (Inertial navigation System) has provided the opportunity of georeferencing aerial images without ground control, by directly measuring the position and attitude of the camera at the time of photography (Cramer and Stallmann, 2001; Schwarz et al., 1993; Skaloud, 2002; Yastikli and Jacobsen, 2005). Previous research has shown that direct orientation using GPS/INS yields georeferencing accuracies that are relatively lower in image and object space when compared to the conventional photogrammetry (Heipke et al., 2002). Although in disaster management and crisis response, accuracy in object space is often not of the utmost importance, the difficulty of stereo compilation of images due to the presence of Y-parallax (error in image space) is a major drawback of direct georeferencing approach. In practice, improving the georeferencing accuracy is essential for making visual 3D measurements in aerial images. In a previous work, we have shown that an integrated orientation approach (Ip, 2005; Jacobsen, 2004), in which image measurements in the overlapping areas (tie points) contribute to the refinement of the exterior orientation parameters measured by GPS/INS, results in an improvement of the accuracy in image space (Khoshelham et al., 2007b). The extent of the improvement, however, is dependent on a number of factors, including the number of tie points and the a priori weights assigned to them and to other Proceedings of the Joint ISCRAM-CHINA and GI4DM Conference Harbin, China, August 2008 543 Khoshelham et al., Aerial data acquisition for disaster management observations within the adjustment process. The objective of this paper is to investigate the role of tie points as well as weight assignment in improving the accuracy of integrated orientation. The paper is structured in five sections. In section 2, the principle of direct georeferencing using an integrated GPS/INS is described. In section 3, direct and integrated orientation approaches are discussed. Section 4 presents detailed results of the experimental analysis of the two orientation approaches. Conclusions appear in Section 5. GEOREFERENCING WITHOUT CONTROL POINTS The georeferencing of images is a basic requirement in aerial photogrammetry. The conventional approach to the georeferencing of aerial images is aerial triangulation. Aerial triangulation relies on the availability of a sufficient number of ground control points evenly distributed in the block of images. When aerial photogrammetry is used as a rapid response technique for acquiring data over areas hit by natural disasters, such as earthquake, flood and bushfire, establishing the ground control points is practically not feasible. Using an integrated GPS/INS onboard the aircraft, it is possible to georeference the aerial images without control points. Since this method is based on direct measurement of the camera position and attitude with respect to the ground coordinate system, it is referred to as direct georeferencing. The mathematical model for the direct georeferencing of aerial images using the GPS/INS measurements is based on the 3D conformal transformation: riL = rCL + λi ⋅ R CL ⋅ riC (1.) where riL and riC are vectors containing the coordinates of a point i in the local (map) coordinate system and camera coordinate system respectively, rCL is a vector containing the coordinates of the camera perspective centre in the local coordinate system, RCL is the rotation matrix from camera to the local coordinate system and λ is a scale factor. Since the integrated GPS/INS system measures the position and attitude of the centre of the INS (and not directly the camera perspective centre), rCL and RCL are expressed as: rCL = rBL + RCL ⋅ aCB (2.) R CL = R LB ⋅ R CB (3.) where rBL and RBL denote respectively the measured position and attitude of the centre of the INS with respect to the local coordinate system, and aBC and RCB are the unknown distance and rotation between the INS and the camera. Figure 1 illustrates the components of the above equations. The unknown lever arm distance between the GPS/INS and the camera perspective centre (aBC), and the three misalignment angles (in RCB) that model the relative orientation of the INS with respect to the camera comprise the main calibration parameters, which are determined during a calibration procedure. Calibration procedure The calibration of GPS/INS is usually carried out through one or more calibration flights over a test field with accurately measured control points. The exterior orientation parameters are computed using a conventional bundle adjustment aerial triangulation, and compared with the GPS/INS direct measurements. In the simplest case, the calibration model consists of three shifts and three misalignment angles that model the discrepancies between the computed and measured values. This model is adequate if the discrepancies remain constant over the duration of the flight. Otherwise, a more general calibration model in the form of a polynomial function of time is adopted. Proceedings of the Joint ISCRAM-CHINA and GI4DM Conference Harbin, China, August 2008 544 Khoshelham et al., Aerial data acquisition for disaster management IMU body frame (B) YB ZL a CB ZB Camera rBL Frame (C) ZC XB G XC λi ⋅ R CL ⋅ riC YL riL Local Frame (L) Figure 1: Components of the mathematical model of direct georeferencing. XL Relation between navigation and photogrammetric angles Attitude parameters measured by GPS/INS are navigation angles, roll, pitch and yaw, which define the relative orientation of the INS with respect to the navigation coordinate system. In photogrammetry, however, attitude parameters are omega (ω), phi (φ), and kappa (κ), which determine the relative orientation of the camera with respect to a local coordinate system. To perform the calibration procedure, a conversion of the navigation angles to the photogrammetric angles, or vice versa, is required. Figure 2 depicts the coordinate systems that are used in navigation and photogrammetry (the definition of the camera coordinate system is conventional). The conversion of navigation angles to photogrammetric angles can be expressed as: R CL = R LN ⋅ R BN ⋅ R CB (4.) where RCL is a rotation matrix that contains photogrammetric angles, ω, φ and κ, and brings the camera axes parallel to the local coordinate system; RNL denotes the rotation from the navigation to the local system; RCB denotes the rotation from the camera to the IMU body, and RBN is the rotation from the body to the navigation system and contains navigation angles, roll, pitch and yaw. As illustrated in Figure 2, the rotation from the camera to the body coordinate system, RCB, is simply a 180o-rotation around the camera x axis: 1 0 0  R = 0 − 1 0  0 0 − 1 B C (5.) The rotation from the navigation to the local coordinate system, however, is not constant during the flight, because the origin of the navigation system is at the centre of the IMU, which moves with the aircraft. In other words, while the local system remains fixed, the navigation system slightly rotates with the movement of the aircraft to keep one axis towards north. Thus, in order to express the rotation from the navigation system at a given location to the local system, an initial navigation system, No, at a hypothetical location above the local coordinate system is assumed (Figure 2). We have: Proceedings of the Joint ISCRAM-CHINA and GI4DM Conference Harbin, China, August 2008 545 Khoshelham et al., Aerial data acquisition for disaster management R LN = R LN o ⋅ R NN o (6.) where the rotation matrix from the initial navigation system to the local system, RNL, can now be obtained by a 180o-rotation around the E axis followed by a -90o-rotation around the Z axis. That is: 0 1 0  R LN o = 1 0 0  0 0 − 1 (7.) o Finally, the rotation from the navigation system at an arbitrary location to the initial navigation system, R NN , is carried out by making use of a geocentric coordinate system, G, as an intermediate system: (8.) R NN = R GN ⋅ R GN o o where the rotation from the navigation coordinate system at geographic coordinates (Λ, Φ) to the geocentric coordinate system is:  − sin Φ cos Λ R GN ( Λ , Φ ) =  − sin Φ sin Λ  cos Φ − sin Λ cos Λ 0 − cos Φ cos Λ  − cos Φ sin Λ  − sin Φ  (9.) and R GN (Λ o , Φ o ) = R G o T (Λ o , Φ o ) . N o Figure 2: Coordinate systems used in navigation and photogrammetry. Proceedings of the Joint ISCRAM-CHINA and GI4DM Conference Harbin, China, August 2008 546 Khoshelham et al., Aerial data acquisition for disaster management SOME REMARKS ON DIRECT AND INTEGRATED ORIENTATION APPROACHES The georeferencing of the image data in direct orientation approach is based on a least-squares forward intersection model. The image coordinates of a point on the ground serve as observations and the object-space coordinates are computed in such a way that the sum of the squared residuals of the observations is minimized. The exterior orientation parameters are treated as constants in the forward intersection model, and no corrections are applied to the GPS/INS measurements. Therefore, possible errors in the measured values of the exterior orientation parameters cannot be corrected for, and such errors would reflect on errors in object space and large residuals in image space. In the direct orientation approach, the estimated standard deviation of the computed object-space coordinates depends only on the a priori standard deviation of the observations and the number of images used for forward intersection. The integrated orientation approach is based on a simultaneous adjustment of a number of tie points together with the exterior orientation parameters. GPS/INS measurements of the exterior orientation parameters are introduced in the adjustment model as constraints or additional observations, and corrections are estimated for these measured values. Comparing to the direct orientation approach, where the exterior orientation parameters are treated as constants, the integrated orientation adjustment model is more flexible since it allows for the adjustment of the exterior orientation parameters as well as the tie points. Also, the integration of a number of tie points distributed across the block in the adjustment model results in smaller residuals in image space. That is, a better accuracy in image space can be expected in the integrated orientation approach. The accuracy of the integrated orientation approach depends, among other factors, on the number of tie points in the adjustment model and also on the initial weights assigned to the observations. The initial weights are inversely related to the a priori standard deviation of the observations. While an estimate of the a priori standard deviation of the image measurements is often available, the a priori standard deviations of the exterior orientation parameters measured by GPS/INS are usually unknown. Therefore, the assigned weights to the exterior orientation parameters in the adjustment model are approximate values. The assignment of suitable weights to the exterior orientation parameters is a determinant factor in the accuracy of integrated orientation. If large weights are assigned to the exterior orientation parameters and small weights to the image coordinates, then the estimated correction to the exterior orientation parameters will be minor, and large image residuals are expected similar to the direct orientation approach. On the other hand, if the image coordinates are assigned larger weights as compared to the exterior orientation parameters, then large corrections will be estimated for the GPS/INS measurements, which might result in larger errors in the object space. EXPERIMENTS Experiments were conducted to compare the performance of the direct and integrated orientation approaches, and to investigate the role of tie points and weight assignment in the integrated orientation approach. A dataset of the EuroSDR (formerly OEEPE) test on integrated sensor orientation (Nilsen, 2002) acquired by Applanix integrated GPS/INS system was used for the experiments. The dataset included a block of 181 aerial images obtained at the scale of 1:5000 together with GPS/INS measurements of the exterior orientation parameters for each image. In addition, image measurements of 2294 tie points as well as object-space coordinates of 13 control points and 18 check points evenly distributed in the block were available. The GPS/INS measurements were calibrated using the data of a calibration flight. For the accuracy evaluation in object space the mean and the RMS error over the check points were used. In the image space, the reference standard deviation (Mikhail and Ackermann, 1976) of the image residuals of the tie points, σο, was used as an indicator of the accuracy. In the computations of the integrated orientation approach, we distinguished between several schemes of tie point selection. Figure 3 demonstrates various tie point selection schemes where tie points in approximate von Gruber areas are preferred. The location of the tie points in the models has been shown in previous research (Khoshelham et al., 2007a) to not have an influence on the result of the integrated orientation approach. Therefore, here only the regular distribution schemes, as shown in Figure 3, are chosen for the experiments. Table 1 summarizes the results of the direct orientation approach as well as integrated orientation with various numbers of tie points. The first row of Table 1 shows the accuracies obtained by the bundle adjustment aerial triangulation using 13 control points. The results of the direct and integrated orientation approaches, which were obtained without using control points, can be compared with the results of the aerial triangulation as the standard approach. As can be seen, the direct orientation approach results in a noticeable 7.8cm mean error in Z, whereas systematic shifts in the integrated orientation approach (with various tie point distributions) remain within a small range comparable to the results of the bundle adjustment. Also, the mean error Proceedings of the Joint ISCRAM-CHINA and GI4DM Conference Harbin, China, August 2008 547 Khoshelham et al., Aerial data acquisition for disaster management values do not show a correlation with the number of tie points in the integrated orientation approach. A close examination of the RMSE measures shows similar results for the direct and integrated orientation approaches, and that an increase in the number of tie points within the integrated orientation approach does not lead to an improvement in the object space accuracy. Table 1 includes also σο values as an indication of errors in the image space. As can be seen, the image-space accuracy obtained by the direct orientation approach is much worse than that obtained by the standard aerial triangulation. The image-space accuracy of the integrated orientation approach, however, improves as the number of tie points increases. Figure 4 demonstrates the improvement of the accuracy in image space as a result of increasing the number of tie points in the integrated orientation approach. A B E C D F Figure 3. Various tie point selection schemes: A. four points per model; B. two points per model; C. one point per model; D. one point in every second model; E. one point in every third model; F. one point in every fifth model; G. one point in every tenth model. To investigate the effect of the initial weights, integrated orientation was performed with various values for the a priori standard deviation of the observations. The tie point selection scheme C, one point per model, was adopted for the integrated orientation, and the number of tie points was kept constant during this experiment. Table 2 summarizes the results of integrated orientation with various a priori standard deviations assigned to the observations. In Table 2 object-space errors are represented by the L2 norm of the RMSE values in X, Y, and Z directions (RMSE_P). The a priori standard deviations used in the previous experiment with the tie point distributions are presented in the first row of Table 2. These values were selected based on initial estimates of the accuracy of tie points (sigmaTie = 20 ), the accuracy of GPS/INS position measurements (sigmaEOp = 2cm), and the accuracy of GPS/INS attitude measurements (sigmaEOa = 10 arc seconds). As can be seen, the accuracies in image and object space are more influenced by variations in the a priori standard deviation of the tie points and also the GPS/INS position measurements, and less influenced by variations in the a priori standard deviation of the GPS/INS attitude measurements. Proceedings of the Joint ISCRAM-CHINA and GI4DM Conference Harbin, China, August 2008 548 Khoshelham et al., Aerial data acquisition for disaster management Method Scheme Nr. of tie points Mean X (cm) Mean Y (cm) Mean Z (cm) RMSE X (cm) RMSE Y (cm) RMSE Z (cm) σο ( m) Bundle AT - 2294 0.0 0.1 1.6 3.3 3.5 10.5 4.1 Direct Orientation - 0 -1.3 -1.9 7.8 6.7 7.7 14.7 36.2 17 1.6 -0.1 -0.2 7.6 7.5 12.2 26.2 33 1.6 -0.1 -0.2 7.6 7.5 12.2 23.2 58 1.6 0.0 -0.6 7.6 7.6 12.2 20.1 87 1.6 -0.1 -0.6 7.8 7.6 12.2 18.5 172 1.8 -0.1 -0.1 7.7 7.4 11.9 15.0 296 2.2 -0.2 -0.2 7.7 7.4 11.7 12.6 516 2.3 -0.5 -0.4 8.4 7.0 11.9 13.1 G (one point in every tenth model) F (one point in every fifth model) E (one point in every third model) Integrated Orientation D (one point in every second model) C (one point per model) B (two points per model) A (four points per model) Error in image space ( m) Table 1: Error measures of direct orientation and integrated orientation with various tie point distributions. 40 35 30 25 20 15 10 5 4 point per model 2 point per model 1 point per model 1 point every 2nd model 1 point every 3th model 1 point every 5th model 1 point every 10th model Direct Orientation 0 Distribution scheme Figure 4: Image-space errors of the integrated orientation approach with various tie point distributions. Figure 5 shows the errors in image and object space obtained by assigning various a priori standard deviations to the GPS/INS position measurements (sigmaEOp), while the a priori standard deviation of the tie points (sigmaTie) and of the GPS/INS attitude measurements (sigmaEOa) remain 20 and 10 arc seconds respectively. As can be seen, the accuracy in image space improves as a result of decreasing the initial weight of the GPS/INS position measurements, whereas the object-space errors increase for smaller weights (corresponding to 10cm and 15cm for Proceedings of the Joint ISCRAM-CHINA and GI4DM Conference Harbin, China, August 2008 549 Khoshelham et al., Aerial data acquisition for disaster management sigmaEOp). In Figure 6 errors are shown for various sigmaEOa, while sigmaTie and sigmaEOp are fixed at 20 and 7cm respectively. The two plots in this figure show an insignificant effect of the initial weight of the GPS/INS attitude measurements on the accuracy in image space; although the errors in object space show a noticeable increase at sigmaEOa=20sec. Figure 7 demonstrates the effect of the a priori standard deviation of the tie points on the accuracy in image and object space. An examination of the errors in image space shows an improvement as a result of reducing the a priori standard deviation of the tie points. In the object space, however, errors increase dramatically for smaller sigmaTie values (5 ). sigmaTie ( ) sigma0 ( ) RMSE_P (cm) 2 15 16 3 14.4 15.7 4 13.8 15.6 13.2 15.4 7 12 15.2 10 10.6 15.5 15 9 17.1 2 12.2 15.4 5 12.2 15.3 7 12.1 15.2 10 12 15.2 12 12 15.4 15 12 16 20 12.2 19.2 5 6 26.1 10 8.9 17.1 10.8 15.5 12 15.2 25 12.9 15.2 30 13.4 15.5 20 20 15 20 sigmaEOp (cm) 5 7 7 sigmaEOa (sec) 10 10 Table 2: Error measures of integrated orientation with various a priori standard deviations assigned to observations CONCLUSIONS An experimental investigation of the accuracy of direct and integrated sensor orientation approaches was presented in this paper. It was shown that the accuracy of the integrated orientation approach is dependent on the number of tie points and also on the a priori standard deviations assigned to the observations. Experimental results indicated that image-space error, which is an indication of Y-parallax for stereo compilation, reduces when a larger number of tie points are introduced in the integrated orientation approach. Also, experiments with various a priori standard deviation values, which determine the initial weights assigned to the observations, showed that a variation in the initial weights results in variations of the errors in both image and object space. This finding suggests that variance component estimation should be seen as a necessary process in the integrated orientation approach in order to fully exploit the potential of GPS/INS measurements for aerial data acquisition over disaster hit areas. Proceedings of the Joint ISCRAM-CHINA and GI4DM Conference Harbin, China, August 2008 550 Khoshelham et el., Aerial data acquisition for disaster management sigma0 ( ) RMSE_P (cm) 20 18 16 14 12 10 8 6 4 2 0 2 3 4 5 7 10 15 sigm aEOp (cm ) sigmaTie = 20 ; sigmaEOa = 10sec Figure 5: Effect of a priori standard deviation assigned to the GPS/INS position measurements on the accuracy in image and object space. sigma0 ( ) RMSE_P (cm) 20 18 16 14 12 10 8 6 4 2 0 2 5 7 10 12 15 20 sigm aEOa (sec) sigmaTie = 20 ; sigmaEOp = 7cm Figure 6: Effect of a priori standard deviation assigned to the GPS/INS attitude measurements on the accuracy in image and object space. sigma0 ( ) RMSE_P (cm) 26 24 22 20 18 16 14 12 10 8 6 4 2 0 5 10 15 20 25 30 sigm aTie ( ) sigmaEOp = 7cm; sigmaEOa = 10sec Figure 7: Effect of a priori standard deviation assigned to the tie point measurements on the accuracy in image and object space. Proceedings of the Joint ISCRAM-CHINA and GI4DM Conference Harbin, China, August 2008 551 Khoshelham et el., Aerial data acquisition for disaster management In conclusion the presented results show the advantage of the integrated orientation approach over the direct orientation approach in photogrammetric data acquisition for disaster management. This is because the image-space errors are a determinant factor in photogrammetric map compilation and making measurements in stereo view, and integrated orientation can provide accuracies in image space that are sufficiently high for this purpose. With the assignment of suitable initial weights and integrating only one tie point per model accuracies in the range of 9 in the image space and 17cm in the object space can be achieved. The presented results demonstrate the potential of aerial photogrammetry as a suitable data acquisition technique for disaster management. ACKNOWLEDGMENTS The authors would like to thank the Applanix Company and Prof. Karsten Jacobsen for providing the data. REFERENCES 1. Cramer, M. and Stallmann, D., 2001. On the use of GPS/inertial exterior orientation parameters in airborne photogrammetry, ISPRS Workshop "High Resolution Mapping from Space 2001", Hannover, Germany, pp. 32-44. 2. Heipke, C., Jacobsen, K. and Wegmann, H., 2002. Analysis of the results of the OEEPE test "Integrated Sensor Orientation", Proceedings of the OEEPE Workshop: Integrated Sensor Orientation, Institute for Photogrammetry and Geoinformation, University of Hannover, Hannover, Germany, pp. 31-49. 3. Ip, A.W.L., 2005. Analysis of integrated sensor orientation for aerial mapping. MSc Thesis, University of Calgary, Calgary, 181 pp. 4. Jacobsen, K., 2004. Direct/integrated sensor orientation - pros and cons, XX ISPRS Congress, Commission III, Istanbul, Turkey, pp. 829-835. 5. Khoshelham, K., Saadatseresht, M. and Gorte, B.G., 2007a. Influence of tie point distribution on integrated sensor orientation. In: N. El-Sheimy and A. Vettore (Editors), 5th International Symposium on Mobile Mapping Technology (MMT'07), Padua, Italy, pp. 179-184. 6. Khoshelham, K., Saadatseresht, M. and Milan, A., 2007b. Calibration of integrated GPS/INS system and direct sensor orientation, Geomatic '86 annual conference, National Cartographic Centre, Tehran, Iran. 7. Mikhail, E.M. and Ackermann, F., 1976. Obsevations and least squares. IEP, New York, 497 pp. 8. Nilsen, B., 2002. Test field Fredrikstad and data acquisition for the OEEPE test " Integrated Sensor Orientation", Proceedings of the OEEPE Workshop: Integrated Sensor Orientation, Institute for Photogrammetry and Geoinformation, University of Hannover, Hannover, Germany, pp. 19-30. 9. Schwarz, K., Chapman, M., Cannon, M.E. and Gong, P., 1993. An integrated INS/GPS approach to the georeferencing of remotely sensed data. Photogrammetric Engineering and Remote Sensing, 59(11): 16671674. 10. Skaloud, J., 2002. Direct georeferencing in aerial photogrammetric mapping. Photogrammetric Engineering and Remote Sensing, 68(3): 207-210. 11. Yastikli, N. and Jacobsen, K., 2005. Direct sensor orientation for large scale mapping - potential, problems, solutions. Photogrammetric Record, 20(111): 274-284. Proceedings of the Joint ISCRAM-CHINA and GI4DM Conference Harbin, China, August 2008 552