MicMac – a free, open-source solution for photogrammetry

Abstract

The publication familiarizes the reader with MicMac - a free, open-source photogrammetric software for 3D reconstruction. A brief history of the tool, its organisation and unique features vis-à-vis other software tools are in the highlight. The essential algorithmic aspects of the structure from motion and image dense matching problems are discussed from the implementation and the user’s viewpoints.

Background

Photogrammetry is the art, science, and technology of obtaining geometric information on the 3-dimensional shape and orientation of objects from images and other imaging sensors. It is a cheap measurement methodology as it can be executed by practically any digital camera of a decent make. It is instantaneous as it captures the observed phenomena at once and in a split second, and highly automated, therefore accessible to non-expert users. Thanks to the field of computer vision, photogrammetry rejuvenated and today places among other competitive remote sensing techniques (e.g. Light Detection and Ranging LiDAR) [1]. The several milestones leading to this progress are the automated interest points detection [2], the Structure from Motion (SfM) algorithms capable of reconstructing scenes from sets of unordered image collections [3, 4], and the dense image matching techniques delivering surface models of resolution equal to the pixel size on the ground [5, 6].

All this contributes to an ever growing visibility of the photogrammetric tools across various fields of science and engineering, the growing market interest, and subsequently a multitude of photogrammetric/computer vision libraries and software solutions, be it commercial or free/open-source. MicMac – together with Bundler, PMVS, VisualSfM, openMVG, OpenCV and others – belongs to the open-source solutions¹. This publication aims at familiarizing the reader with the philosophy behind MicMac, some of its crucial algorithmic aspects, the software architecture and the pool of available tools.

Historical notes

MicMac has been developed at the National Institute of Geographic and Forestry Information (IGN) and the National School of Geographic Sciences (ENSG), since 2003. Initially, the software tools were developed having in mind exclusively the IGN’s cartographic production. The independent tools were interfaced in 2005 via an XML framework, allowing the user to freely parametrize the calculations at all processing stages. In 2007, IGN began to freely distribute MicMac under the CECILL-B license that is a version of the L-GPL license adapted to the french law.

Until 2008, the dense image matching of already oriented images was possible only with the IGN’s internal image file format. In the same year the Apero tool was added to the sofware kernel, offering from now on the possibility to estimate camera exterior and interior orientations, with no restriction on the image format. In 2010, the XML interface was replaced by a simplified command line. This evolution contributed to an improved accessibility, diffusion and subsequently a better visibility of the software in the scientific communities, and the general public.

Since 2010 MicMac has been undergoing a significant evolution due to its implication in many french and european projects. Besides the contributions to the source code, new software distributions under Windows, Mac OSX, as well as the GPU processing for certain tasks became available.

MicMac vis-à-vis other tools

From the architecture point of view, the unique feature of MicMac with respect to its alternatives is that the user can operate on different software levels, cf. Fig. 1. The least experienced user can steer the processing by a simple command line where little or no parameters must be set. An experienced user can operate from the same command line by adapting the parameters typically filled with default values. An expert in photogrammetry can access any parameter at any processing step via a set of XML files (cf. Fig. 1). On top of that, developers and scientists can use MicMac as a library, implement their own algorithms, and share them for the benefit of photogrammetric community. This said, MicMac was designed primarily as a tool for professionals and consequently is far from a model design in the sense of HCI².

Fig. 1

The simplified architecture of the core MicMac modules. a the low-level and high-level modules dependencies, b the processing workflow. Marked in red are the outcome products, in green are the direct georeferencing inputs

MicMac, as a photogrammetric tool, stresses the aspects of accuracy, reliability, and provides tools typically unavailable in existing software alternatives, for instance:

• access to intermediary results in open data formats allowing to interact with the processing chain at any desired stage,

• qualitative evaluation of the results via quality indicators (e.g. bundle block adjustment residuals, covariance matrices, parameter’s sensitivity, correlation images),

• a wide range of camera calibration models (e.g. the models adapted to consumer grade cameras, large-frame aerial cameras, cameras with very long focal lengths, fish-eye and spherical cameras),

• two-dimensional dense image matching for deformation studies,

• processing of frame camera and pushbroom sensor images,

• processing of scanned analogue images,

• architecture adapted to big datasets.

Software organisation

MicMac is organised in a number of modules, accessible through a common command mm3d. Execution with a unique command has a twofold advantage: (a) it allows to factorize certain developments while keeping the binary files compact, and from the user point of view, (b) it is easy to remember and gives instantaneous access to all modules (cf. Fig. 2). Invoking a selected module is followed by a number of parameters. The obligatory parameters must be specified at first, and since they do not carry names, their order of occurrence is important. In the example shown in Fig. 2, the obligatory parameter would be the format specification, while the change coordinate file, ChSys, serves as an option.

Fig. 2

Each module is executed with some obligatory and optional parameters. Adding -help at the end will print out the expected input arguments

Implementation

The algorithmic aspects

The photogrammetric workflow encompasses the passage from images, through the estimation of the orientation parameters, finalizing with a 3D surface model. In other words, it is a passage from a 2D representation of the world captured by a camera, through inference of the position and rotation of that camera at the moment of the image taking, towards a 3D restitution of the lost dimension. As long as the quality of the final output depends on the skillful engineering of many small processing steps, estimation of the camera orientation parameters and the matching algorithms constitute the heart of the pipeline. Respectively, the coming sections concentrate on these two aspects, report on the adopted methods and give a global look on what is possible in MicMac.

Structure from motion

Recovery of the structure (i.e. 3D coordinates of the observed scene) and the model. It is well known that this model (i.e. the transformation from 3D to 2D space and vice-versa also referred to as collinearity equation) is not linear, hence requires linearization. Moreover, there exists no direct algorithm able to compute orientation parameters globally consistent with a number of images (generally n>3). To overcome this gap, bootstrap solutions were proposed. Using direct algorithms for a single image, a pair or triplet of images, the global orientation is deduced sequentially [3] or hierarchically [7] starting from a seed image pair. The so-obtained parameters serve as input to a system of equations composed of the linearized collinearity equations, where their optimal values – in the stochastic sense – are found iteratively. The observations (e.g. tie points) are redundant therefore the solution is deduced with the least squares method by minimizing an objective function. The typical function is the squared differences between the nominal observations and those predicted from the estimated model, possibly subject to constraints. The latter stage is also known as the bundle block adjustment (BBA).

MicMac implementation.

MicMac solves the BBA with the Levenberg-Marquardt (L-M) method. The L - M is in essence the Gauss-Newton method enriched with a damping factor to handle rank-deficient (or nearly so) Jacobian matrices [8]. A range of different observations equations, constraints and unknowns (i.e. parameters) are allowed in the adjustment routine, among them:

• tie points,

• lines,

• perspective centers derived from direct georeferencing (i.e. GNSS),

• ground control points (GCP),

• the lever-arm vector separating the GNSS antenna and the perspective center,

• rigid relative orientation between cameras.

On top of that, a broad spectrum of camera calibration models are available. Both argumentation are supported, the physical and the phenomenological. The former models any imaging error attributed to a phenomenon of a physical nature, while the latter models solely the effects of the lens imperfections, without interrogating its causes. The total distortion is defined as a composition of predefined elementary distortions, such as radial, decentric or a generic polynomial. Typically, the major distortion are modeled by the physical model, while the generic polynomial models remove the less significant remaining systematism.

The critical element of the BBA’s mathematical model is the observation weighting, known as the stochastic model. MicMac specifies three different weighting strategies. The first strategy corresponds to the weighting in the classical model known in photogrammetry – the Gauss-Markov – where the observations are weighted by their true standard deviations known a priori. The second strategy controls the influence of a particular category of observations so as to avoid solutions that are driven by a single category only for the sheer reason of its abundant observations. The last strategy addresses the robustness. The weight is a function of the observation’s residual in the BBA, therefore giving more credibility to observations that are close to the estimated model, and contrary, limiting the influence of observations with high residuals.

User’s viewpoint.

There are two principal modules that handle the camera orientation recovery with the simplified interface – Tapas, Campari – and both of the modules call the parent orientation tool Apero (cf. Fig. 1). Tapas calculates the purely relative orientation of images, using observed tie points as the only input. Since at this point there is no a priori on positions and rotations of the cameras, Tapas also entails the initialization step where it progressively reconstructs the cameras with the help of the direct orientation algorithms, and intertwines it with the BBA routine.

Unlike Tapas, Campari is a BBA module adapted to handle heterogeneous observations. Besides the tie points, it works with GCPs, the GNNS data and can impose various constraints on the solution. This module is typically executed once a good relative orientation is established, and the cameras are moved two a coordinate system (CS) consistent with that of auxiliary observations, e.g. the GCPs. The latter is performed with any Bascule tool, and it is a spatial similarity transformation (cf. Fig. 1). Both Campari and all variations of the Bascule can be regarded as georeferencing tools.

Since recently, MicMac provides the user with the internal accuracy estimates of the BBA parameters (their standard deviations and correlations) derived from covariance matrices.

Multi-view stereo image matching (MVSM)

Given several overlapping images with known orientation parameters, MVSM is the process of reconstructing a complete 3D model of the scene by finding corresponding pixels (i.e. their disparities or depths, respectively) between image pairs, and triangulating them in space. The generic algorithm is defined as an energy minimization problem that searches for a disparity map minimizing the energy. The minimization can be solved with local, semi-global and global algorithms. Today the golden standard method for producing dense 3D models from images is the semi-global matching [6].

Image matching breaks down into three processing stages: computation of the pixel matching cost, cost aggregation and disparity calculation. The matching cost is a measure of dissimilarity, i.e. describes the unlikelihood that two pixels belong to a unique point in 3D. As the matching can be ambiguous, and the minimum cost insignificant, an additional a priori is imposed on the energy function that penalizes disparity changes at neighboring pixels (the case of local and semi-global algorithms). This aggregation takes place within a window, a 1D path (locally) or multiple paths (semi-globally). In the latter case, the cost for a given disparity is an accumulated cost along many paths that end in the pixel in question [9].

MicMac implementation

Image dense matching in MicMac is possible with a version of semi-global and global algorithms. The objective of matching is to find a mapping $\mathcal {F}_{px} : \tau \otimes \varepsilon _{px}$. Depending on the restitution geometry, the τ can be the 3D euclidean space or the image space of the master image, hence it is the space where the surface model is produced. The ε _px can be either the depth of the euclidean $\mathcal {Z}$ or the image disparity (cf. Eq. 1 and Fig. 3).

Fig. 3

Three restitution geometries in MicMac. a ground, euclidean space, b image geometry discretised along the ray, no resampling c image space resampled to epipolar geometry. For a range of the potential Z-coordinates/depths/disparities the similarity measure is computed and further passed to the energy minimization solver

The motivation to distinguish between these two restitution geometries – the ground and the image – is twofold. On the one hand, a Digital Surface Model (DSM) produced from aerial images is normally defined in some reference CS, therefore it is more intuitive to perform the reconstruction directly in the target CS (i.e. ground geometry), where the disparity space ε _px is the $\mathcal {Z}$ coordinate (cf. Fig. 3 a). On the other hand, it is known that matching in image space with a master and a set of slaves images is more reliable, especially in close-range photogrammetry applications and for small datasets. The disparity space ε _px in the image geometry is then either the depth along the ray or the respective disparity in image space (i.e. the image geometry; cf. Fig. 3 b and c). It is up to the user which geometry to employ.

MicMac offers a very flexible formulation of the matching cost function as far as its terms are concerned, the optimization algorithm and the dimension of the matching are regarded (cf. Eq. 1).

$$ {{\mathcal{E}}}({{\mathcal{F}}_{px}}) = \iint_{{\mathcal{T}}} {{\mathcal{A}}}(x,y,{{\mathcal{F}}_{px}}(x,y)) + \| \nabla({{\mathcal{F}}_{px}}) \|^{reg} $$

(1)

$$ \| \nabla({{\mathcal{F}}_{px}}) \|^{reg} = \alpha_{1}*|\nabla\left({{\mathcal{F}}_{px}}^{1}\right)|+\alpha_{2}*|\nabla\left({{\mathcal{F}}_{px}}^{2}\right)| $$

(2)

where ${{\mathcal {A}}}(x,y,{{\mathcal {F}}_{px}}(x,y))$ is the measure of similarity between pixels with ${{\mathcal {A}}}=0$ when they are identical; $\| \nabla ({{\mathcal {F}}_{px}}) \|^{reg}$ a norm on the gradient, which is used as a regularity criteria (it penalizes the variation of ${{\mathcal {F}}_{px}}$); α ₁ is the regularization on first component of disparity and α ₂ the regularization on the second component (equivalent of matching in the direction transverse to the epipolar line).

The $\| \nabla ({{\mathcal {F}}_{px}}) \|^{reg}$ controls the a priori on the disparity space in a way that:

• if the desired model is smooth, a convex function F is adequate (it’s better to climb up a given jump by regular steps),

• if the desired model has many discontinuities, a concave function F is adequate (it’s better to climb up a given jump in a single step),

• when there is no strong prior, the default choice is to have F linear,

• if a priori knowledge on scene slope exists, it can impose an allowable maximum scene slope,

• for 2D matching, non-isotropic smoothing factors can be set.

The actual similarity measure ${{\mathcal {A}}}(x,y,{{\mathcal {F}}_{px}}(x,y))$ is calculated from the the normalized cross correlation coefficient (1−C o r), defined as a function of multiple images, rather than a single pair. The coefficient can privilege an image out of a set of images (e.g. a master image), or consider the set as “symmetrical”. Varying cross correlation window sizes and weighted windows are also possible.

To find the disparity map that minimizes the energy, MicMac by default employs a multi-directional variant of the dynamic programming algorithm. The results along independent directions are aggregated by taking a mean of all results or simply the best result. Optionally, a global minimization by the Min-Cut/Max-Flow algorithm [10] can be enforced.

In order to limit the disparity search-space, speed up the calculation and to reduce the noise, MicMac adopts a multi-resolution matching approach where coarse resolution disparity maps serve as predictors of the fine resolution output.

User’s viewpoint

DSM creation in MicMac takes place via several tools that are semi- or fully-automated. In both cases the parent tool to handle the low-level dense matching is the MicMac, which an expert user can access via an XML file.

Malt, the semi-automated tool, operates in three predefined modes, i.e. Ortho, UrbanMNE and GeomImage. The modes are adapted to producing orthophotography, DSMs in ground and image geometries. Accordingly, Ortho and UrbanMNE would be selected for flat terrains or planar surfaces (i.e. 2.5D; cf. Figs. 4, 5, 6) and GeomImage for true 3D surfaces (cf. Fig. 7).

Fig. 4

Bas Relief stone in Louvre: the image and the 3D model rendered in shading and depth map

Fig. 5

Under induced force the concrete beam deformed, while a still camera measured the displacements in during the breaking phase. Upper left: the full view image, upper right: a close-up of the concrete structure, bottom left: displacement in the x-coordinate (max amplitude $\frac {1}{4}$ pixel), bottom right: displacement in the y-coordinate (max amplitude $\frac {1}{4}$ pixel)

Fig. 6

Munich dataset, acquired with a DMC camera at GSD of 10 cm. Left: the depth map, right: the y-parallax generated with MMTestOrient. The amplitude of transverse parallax is ± 0.1 pixel

Fig. 7

Indoor architecture: Chapelle imperiale Ajaccio, with 100 fish-eye images; left: retrieved camera poses, right: 3D model

The inconvenience of the Malt GeomImage is that for a complete 3D restitution of a surface, one must indicate several master and their relevant slave images. As long as for simple object geometries it is feasible, it becomes inefficient for more complex and extensive objects. The fully automated tools overcome this impediment by making this choice for the user, while prioritising completeness and precision of the output. The two automated tools are C3DC and PIMs which furnish a point cloud and a set of depth maps, respectively. To proceed with the orthophotomap generation, the PIMs2MNT and subsequently the Tawny shall be executed (an exemplar orthophoto in Fig. 8). The latter collects the per-image orthophotos and merges it to a single mosaic. The individual orthophotos can be created either by PIMs2MNT tool or Malt; cf. Fig. 1.

Fig. 8

Forteresse de Salses, photo acquired from a drone, in collaboration with Map-CNRS/ Left: hypsometry and shading maps overlapped, right: an orthophotography

Besides the classical 1D matching (along an image ray, an image row or the Z-coordinate) for reconstruction of object geometry, MicMac also implements a 2D matching strategy. This strategy is useful in 2D deformation studies between rectified images (cf. Fig. 5; also MM2DPosSism), as an orientation quality estimation tool to asses the remaining y-parallax (MMTestOrient; cf. Fig. 6), or in cases when orientation parameters are unknown or known with poor precision (executed with XML file). Because the expected disparities pertinent to geometry are of higher frequencies than those along the y-parallax, the regularization of the energy function in the two direction is managed separately; cf. Eq. (2).

Lastly, in the event where all camera optical centers are located on a 3D line, parallel to the surveyed object, and the reconstruction is defined in object space, the matching becomes ambiguous along the epipolar lines. To avoid the ambiguities and the resultant noise, MicMac calculates the matching in ortho-cylindrical coordinates (see RepLocBascule).

Interactive tools

Although MicMac by default is based on simplified command lines, for some processes – i.e. image measurements, visualization of the results – a graphical user interface (GUI) is available. Besides this, for multiple commands there exists a GUI that replaces the standard command line. To run it, the user should simply type the name of the command preceded by the letter ’v’. For example, in the GrShade command, the GUI is called by typing vGrShade (cf. Fig 9).

Fig. 9

Graphical interface to launch commands

All results produced by MicMac are exclusively in open data formats (tif, xml, ply) and as such it opens up the possibility to interact with the processing chain at any desirable stage. For instance, the user can import interior and exterior image orientation data and proceed with the dense matching and orthophoto generation. Or vice versa, the orientation parameters or epipolar images can be generated to proceed with an alternative dense image matcher. The reader is advised to refer to the software documentation [11] for more details on the conversion procedures.

Image measurement tools

In the following, several tools for image measurements and the use of 2D/3D masks are discussed.

Many of the interactive tools are available in a version requiring the installation of the QT library. For example, for the SaisieMasq, the corresponding QT tool is the SaisieMasqQT. In either case, the masking works by creating a binary image mask from a 2D polygon, or a 3D cylinder from a 3D mask (cf. Fig. 10). Masks can be used to, e.g., remove tie points (cf. HomolFilterMasq), or to limit the image matching to a selected part.

Fig. 10

Left: drawing of a 2D mask on an image; right: and a 3D mask in a point cloud

The SaisieAppuisInitQT/SaisieAppuisPred icQT tools allow for GCPs’ image measurements (cf. Fig 11). Provided that the camera orientation parameters are known with a reasonable accuracy and expressed in a CS coherent with the GPCs, the latter tool serves as a point position prediction tool. Measurements necessary to perform a spatial similarity transformation, i.e. the origin, the axes directions and distances, can be collected within another georeferencing tool – the SaisieBascQT. These tools normally precede the georeferencing operations with GCPBascule and/or Campari.

Fig. 11

Interface for measuring GCPs in images

Visualization tools

To visualize the intermediary and final results MicMac offers a range of different tools.

The SEL tool visualizes the tie points between a stereo pair. The AperiCloud tool converts the result of the BBA, that is the poses and 3D tie points, to the point cloud ply format.

The Vino tool is a simple image viewer adapted to visualization of very large images (i.e. in size of a few gigabytes), irrespective of their type (i.e. integer, float or >8Bit). Moreover, it lets the user to modify the histogram of the image or generate an image crop.

Other tools allow a more intuitive visualization of depth maps. For instance, the to8bits tool converts 32-bit or 16-bit images to 8-bit images, while the GrShade tool computes shading from a depth map image (cf. Fig 4). Lastly, the Nuage2Ply transforms a depth map into a point cloud ply format (cf. Fig 7).

Discussion

MicMac is an open-source project for photogrammetry. As a research implementation platform, it serves the IGN’s internal needs as well as aims at bringing the state-of-the-art 3D image processing algorithms to a wider audience, be it universities, public institutions or enterprises. Like any other open-source project, MicMac authorizes the volunteers to contribute or change the source code according to their needs. For more information please refer to:

• http://micmac.ensg.eu, as the software’s reference webpage;

• http://forum-micmac.forumprod.com/, for technical support;

• https://github.com/micmacIGN/, for the source code.

The high priority on-going developments and short-term perspectives are those concerning effective processing of big datasets. The developments pertain to (a) the tie points extraction, (b) the SfM, as well as (c) the storage of 3D data. Regarding the tie points, very precise tie point detectors and detectors invariant to affine distortions are being developed. Within the SfM, global, structure-less methods [12–14] are under development. As a further perspective, adequate methods for storing very large 3D point clouds will be conceived. Lastly, as there has been numerous demands to create a GUI, computer programmers willing to contribute are strongly encouraged to contact the team of developers. A GUI in form of a stand-alone application or a GUI integrated with other open-source software tools, e.g. QGIS, are possible.