GIWIS - Micosoft Windows (TM) binary setup.exe for writer-identification application

1. Introduction 1.1 Description The GIWIS system is an application for the Microsoft Windows™ operating system which allows to compare the image of an unknown handwritten document to documents in a collection of handwritten image samples from known writers. The goal is to find the 'nearest neighbor' in terms of the similarity of its handwritten shapes with the unknown or questioned document. The GIWIS algorithms use handwritten shape identifiers that have a solid basis in handwriting research (graphonomics) and may be related to intrinsic properties of the writing process (slant, curvature) on the one hand, and character (letter) attributes on the other hand. The system is not an oracle which generates a true writer identity. Instead, the found nearest neighbor should be considered as a mere hypothesis which is stated with a given state of certainty. The similarity of two samples of handwriting is computed on the basis of the differences in image characteristics ('features') between two documents. The result of this computation is a number, i.e., a single distance value for a pair of handwritten document samples. The lower the distance, the better a match. If several samples of a single writer's handwriting are compared, different distance values will be found among them, within that writer, with a statistical distribution that reflects the within-writer variability. It is a basic assumption of GIWIS, that a new sample of handwriting from that writer, written under similar writing conditions, will fall within the cloud of known samples. Normally, a natural writing attitude is assumed by GIWIS, and naturally written alternative documents in the reference data set can be found. However, if a writer produces several samples in a comparable forged style, the forged samples will also be mutual neighbors. Similarly, for arbitrary pairs of writers of different identity, distance values can be computed between their handwritten samples. The image features that are computed from the ink trace are chosen in such a way that the average distance of same-writer document pairs tends to be smaller than the average distance between two document from two different writers. After having computed the distance values of a questioned document to all known documents in the database -which is mainly a directory with handwritten image files-, the candidates are ranked in order of increasing distance. At the top of the list will be the document from a writer which, according to the calculation, is the best match, i.e., the nearest neighbor of the query document (questioned document). This is the primary function of GIWIS, similar to the concept of a 'hit list' in an internet search engine. However, it is well possible that the actual, correct document by the same writer is not at the top, but at a later position in the ranking. It is important to realize that even in the case where the target writer is not in the collection, a nearest neighbor will be found. How to know whether the 'top-1 hit' is indeed the correct writer? We consider the answer to this question as a secondary function of GIWIS. We will return to this topic at a later stage. Concretely, the major function of GIWIS is the sorting of handwritten documents in terms of similarity to a query document. GIWIS does not require to learn or compute a 'model' for a writer. Instead, instances of concrete documents are compared: "the sample is the model", in principle. 1.2 GIWIS System Concept The GIWIS program is the result of ten man years of scientific research. The software is entirely written by researchers and programmers of the institute for Artificial Intelligence and Cognitive Engineering of the University of Groningen, The Netherlands. The underlying theory has been detailed in a number of scientific publications. Summarizing, the system aims at capturing three aspects of handwriting biometrics: 1) curvature and slant, capturing distributions of writing directions; 2) allography, capturing shape elements within the handwritten characters (letter shapes); 3) ink density or thickness variation, indirectly capturing some aspects of 'pressure' and pen wielding. Furthermore, the system is able to compute a number of additional shape features for handwriting which are much less powerful in writer identification but which were historically available before the start of the GIWIS development and which are useful in performance comparisons and evaluations. In particular, GIWIS implements the HINGE feature by Marius Bulacu (Schomaker & Bulacu, 2004; Bulacu & Schomaker, 2007) to capture the distribution of angle combinations along the ink trace. Second, the FRAGLET feature by Lambert Schomaker (Schomaker & Bulacu, 2004) breaks up the ink trace in little fragments which are of sufficient shape complexity to be informative concerning the shape elements a writer uses in the allographs. Third, the QUILL and QUILLHINGE feature by Axel Brink (Brink, Smit, Bulacu & Schomaker, 2011) is directed at representing the variations in ink width in different directions. The reader is referred to the cited publications for more details. Although the features that GIWIS computes are not directly based on a motor model of handwriting, the system is deeply rooted in the knowledge on movement control, collected between 1984 and 2000 by the 'Nijmegen group on handwriting research'. For example, the HINGE feature can be considered a descendant of the writer identification methods in on-line handwriting by Maarse et al, (1988). The user interface of GIWIS is newly designed. The writer and document-specific data fields for the data set are inspired by the results of the Wanda project (Franke, Guyon, Schomaker & Vuurpijl, 2004.). 1.3 Usage scenario The typical usage scenario entails that the user will digitize (scan) a handwritten document at 300dpi. Using a general image-processing tool, all non-handwritten material needs to be removed (whitened) in the image. Then the image is entered into the GIWIS system together with metadata such as unique writer identificaton code, a code for the document and other known facts related to the document and the writer. These items are based on an earlier study, the Wanda project (Franke,Schomaker,Vuurpijl,Guyon, 2004), on the basis of the needs of forensic handwriting examiners. However, it should be stressed that the system has no claims of completeness in this respect. Additionally, it is important to know that GIWIS is simultaneously designed for paleographic researchers and other applications for writer/author identification. After entering an image into the system, the user can perform a perspective transform, if necessary, in case the handwriting was not acquired on a flat-bad scanner but photographed from a non-orthonormal angle. A number of basic image processing steps make the image suitable for shape feature computation. However, the user must visually inspect the results of the image normalizations. In case GIWIS is unable to separate the paper and ink colors sufficiently, external image-processing tools from a different provider need to be used for additional preprocessing. An example constitutes the presence of the lineations on a paper notepad or table borders on a form. Such non-handwritten patterns may disturb the computation of the shape features which where designed to handle handwritten ink, only. The next step concerns the computation of the shape features. One of the reasons GIWIS is called 'intelligent' because it allows for the computation of single features but also feature combinations. The reason behind this is that any feature scheme -in any system- will have implicitly attached to it the exemplary type of data it is intended to represent properly. If these assumptions are violated, which is difficult to predict, a match using multiple features will therefore be more robust than a match based on a single method. For a given data collection, the optimal feature set for writer identification within that set can be computed by GIWIS. This may take quite some time on a realistic reference data set, but in the end the user will know what is the best feature combination to use [cf. Section on Performance, later in this document]. After selecting and computing the shape features, the inter-writer distances will be computed. The last step is the presentation of a sorted 'hit list' of matching documents, with the nearest neighbor to the questioned document at the top of the list. On the basis of the probability distribution of distances between document (writer) pairs in the image collection, a rough estimate can be made of the log-likelihood ratio for deciding Same-Writer or Different-Writer, in a verification scenario where the user not only wants to find 'the needle in the hay stack' but also needs to decide on the likelihood that the shape features come from the same source, i.e., the same writer. It should be stressed that this is not the strongest aspect of GIWIS. Its primary purpose is to find the best match. The reliability of the computed likelihood ratios is directly determined not by the software, but by the size and quality of the reference document collection. This also implies that the reference set (the hay stack) should be representative for all possible questioned documents. For example: if the reference document collection contains mainly right-handed students from a single schooling system writing a page of text in pencil, whereas the questioned document is from a left hander, is ball-point based and contains just four words from an unknown writer generation (cohort), the chances for useful matching are minimal. It is well possible that an arbitrary sample from a major writing style (same cohort) in the set will act as a bad attractor for that document. Carelessness and negligence of the above caveat will lead to meaningless results. Furthermore, with badly cleaned images, it is possible to obtain a 'hit' on the basis of an irrelevant shape feature such as lineation or even paper structure (texture) when the background has not been cleaned. 1.4 Design considerations It is known that in many application domains, users state to need systems which explain their results in an understandable, preferably verbal report. This is beyond the current state of the art. We have experimented with explainable methods (Brink, Schomaker & Bulacu, 2007), but concluded that this approach entails a price in terms of writer-identification performance, so that road has been abandoned. Some computer-based writer-identification systems focus on the manual segmentation of individual letters which will then individually be matched between questioned and known documents. Such a manual effort costs a lot of human labor. The concept of GIWIS is: scan, cleanup and match. Rather than making statements on individual letters, a distance value and a log-likelihood ratio estimate will be provided if necessary. We know that some users object to a 'black box' approach. It should be noted that the GIWIS system is not a black box: All of its algorithms have been published. Adopting new tools costs time. In other domains of expertise, devices are introduced which need an incubation period before users obtain an insight in the meaning of the numbers produced by the algorithm. As an example, in medical diagnostics, many devices have been introduced, generating numbers which were arbitrary at first, but which could be interpreted as more material for comparison and calibration was collected. In the ideal case, all GIWIS users would annually exchange new data to increase the reliability of the statistical estimates. In the future, new uses can be foreseen, with calibrated probabilities for, e.g., 'a writer appearing at rank 10 in the hit list when there are four thousand reference documents in the collection'. We predict that the GIWIS distance values and probability numbers will become quickly familiar for a regular user. For comparability of results, GIWIS is a fixed-binary program. 1.5 Development history The actual development of GIWIS started in a Dutch NWO project, Trigraph. PhD student Axel Brink developed a graphical user interface in the programming language Python, for the algorithms and tools developed by Lambert Schomaker and Marius Bulacu running under Linux. Furthermore, Axel Brink developed new features (QUILL, QUILLHINGE), specifically aimed at historical writer identification. This prototype version of GIWIS was enthusiastically used by researchers in paleography (Smit, 2010). However, it appeared that the ink-width variation detection by the QUILL family of features is also powerful in increasing writer-identification performance in contemporary handwriting. On the basis of this prototype, scientific programmer Aswin van Woudenberg wrote an integral C++ reimplementation for the Windows platform. The GIWIS software uses source code, exclusively developed by researchers at ALICE. For some image-processing steps, modules of the netpbm package have been used as inspiration. It should be noted that the distribution of GIWIS is not commercial, but can be considered as 'freeware with obligations for the user'. When reporting on GIWIS, the user should cite the relevant paper [ref]. In addition, we, the developers, would like to develop a reference base of documents. However, given the sensitive nature of many writer-identification problems, we would also be helped much by a regular reporting of GIWIS performance. A standard form will be developed. The statistical 'Same-writer' and 'Different-writer' distributions for pairwise distances are a completely anonymous, image-less representation of your data set: The underlying handwriting can never be reproduced from a single number (distance values) that describes the dissimilarity of a pair of handwritten samples. In the future, we will ask users to kindly provide us with these statistics, such that the reliability of pairwise verification will improve over the years. 1.6 Caveats The identification and verification of writers is a process with a large number of pitfalls. Some of the risks are similar to all biometric methods: face recognition, fingerprint and to human-by-human identification. Other risks are typical for handwriting biometrics. The following list of caveats itemizes a number of these: 1. Amount of handwriting. We found that samples need to consist of at least 100 characters (letters) of Western Handwriting (Brink, Bulacu & Schomaker, 2008). 2. Style of handwriting. The main and tested handwriting style for GIWIS is Western (Latin) handwriting. However, GIWIS and/or its algorithm has been used successfully on other script types such as Arabic, Persian and others. Performance evaluations for other scripts than Latin are ongoing. For these scripts, nothing is known about the minimum amount of necessary text, as of yet. The FRAGLET feature assumes that spatial minima in the ink trace are good points for automatic segmentation (fragmentation). The RUNLENGTH feature assumes a horizontal spacing of words. 3. Image problems. Residual speckles, lineation traces, paper texture, concentrations of book-worm holes, calligraphy, ornate capitals differing from the body of text are just but a small subset of possible problems. GIWIS assumes that its input consists of a handwritten trace under natural conditions (habitual pen grip, stable writing surface) against a white background. 4. Unbalanced collection yielding outliers as 'matches'. Example: the database mainly consists of upper case samples. The questioned document is in connected-cursive, lower case. There is one sample of the connected-cursive style in the database. Under these conditions the user should not be surprised of the GIWIS system response, most probably finding the cursive writer as the most likely nearest neighbor. This problem comes in a number of variations on a theme: e.g, (a) the questioned document is a small snippet of text, while the reference database consists of full-page letters (or vice versa); (b) The data base consists of many ball-point samples, of high quality, originally in color at 600dpi, whereas the questioned document is in pencil, originally black and white (bitonal) because it was faxed at 200dpi; etc. This pitfall can be avoided by constructing balanced datasets, with a comparable image quality and amount of text for all samples. Some degree of scale invariance is achieved by GIWIS but it is better to strive for comparable sizes (ink-trace width typically 7-9 pixels, similar character height across samples). 5. If the writing style is apertly different, and several human experts would agree on this, it is better to enter a writer code for that (forged or special) sub style, e.g., "Writer5437.Variant-B" than entering one single writer code for all possible styles, upper/lower case variants, etc., for this writer. There is no magic in GIWIS: If the major style looks different to you, GIWIS will usually find large distance values for a pair of documents, too, and will not find a match with a document from the same writer if that document is written in an entirely different major style. 6. Context dependency of handwritten samples. This is a variant of problem 3: It is easier to match two samples from the same document, e.g., two paragraphs by a single writer on the same page, than matching different samples from a single writer but from widely different origins and production contexts, spanning several years. We have done informal tests with samples from a handful of individuals providing samples spanning several decades. Results give an indication that GIWIS is well able to find matches, but the fact remains true that the easiest pairwise comparison is from two regions of interest from one document, i.e., from one page of text produced a particular moment in time with one, single writing implement, other pairwise comparisons becoming more difficult as the writing contexts differ. 7. Dependency on writing implement. Similarly, if one sample is written with a very thin ink trace by fountain pen, and a target sample is written with a thick felt pen, the chances of finding a match are limited. This can be counteracted by applying a thinning image operator on the 'felt pen' sample with an external image processing tool such as Photoshop™, before matching. Also, not all shape features are similarly sensitive to the sample differences. In the latter case, the FRAGLET feature may detect correspondences in letter shapes, due to the overall shape fit of letters, whereas a micro feature such as the QUILLHINGE or HINGE may suffer from the unbalanced ink thickness condition for two document samples. Conversely, a highly broken ink trace yields only a few reliable full-letter samples for the FRAGLET feature, such that the other methods (e.g., HINGE) are needed to help out and regain robust estimation. 8. Preferred feature combination. The best choice for feature combinations is, to our knowledge, to combine: FRAGLETS, HINGE and QUILLHINGE in a search. If these results are not satisfactory, exploration with other feature combinations such as BRUSH or RUNLENGHT in addition to at least one stronger feature is advisable. 9. Prior probability of finding a writer in the dataset. If there is an unbalance in the number of samples per writer, e.g., one fraudulent subject occurs 50 times, whereas the other writers occur only twice in the data set, this will yield a bias for questioned samples from the fraudulent subject. For serious tests, the number of documents per writer should be balanced: Removing a number of samples from that writer until the distribution is even, and repeating the search for a number of such random permutations. The RANDOM feature will give an insight on the a priori probability for a writer to obtain a spurious (chance) hit, given the data set. The GIWIS user interface allows for disabling of document instances. 10. Image processing pitfall. It goes without saying that a proper image preprocessing is a must. Please look at the Firemaker data set for visual examples of what GIWIS considers as 'proper data'. The scaling function can be used to make samples of a comparable ink width and line height. For paleographic identification of the hand, care must be taken that the image contains clean handwriting only. Otherwise, GIWIS will quickly turn into a book-identification system rather than a hand-identification system, inadvertently zooming in on paper texture and/or lineation as the essential source of information. 11. Performance estimates. The Firemaker data set is 'academic', very clean, with a good, single scanner, from a population of mainly young Dutch writers. Moreover, our research has revolved around this collection for several years. We do have shown that the results remain stable, going to sets of 900 and more international writers. However, also these latter additional sets are 'academic', collected under controlled conditions, with comparable content and page layout. Currently, (2011) experiments with more practical real-life data sets are ongoing. In any case, it is important to note that the reported percentages in the scientific articles and in the tables of this report are obtained under ideal conditions, similar to the fuel-usage ratings that a car company would report. 12. Verification, error rates and likelihood ratio. GIWIS was designed for identification and contains a number of preliminary functions for verification of handwritten samples. To this purpose, false-reject/false acceptance curves are computed and shown in relation to the measured distance for a given document pair. The reliability of the probability estimates and likelihood ratios is limited. The user is responsible for a dataset which is large enough to yield smooth and reliable distributions for Same-writer and Different-writer distance values, with single-peaked 'mountains' and limited overlap. Hundreds to thousands of samples are needed and experience needs to be collected over several years to be able to relate GIWIS-based decisions to the correct decisions. This can be done on the basis of confessions, auxiliary evidence, etc. This aspect of bookkeeping Bayesian statistics concerning GIWIS as a tool in an operational environment is beyond the scope of this image-processing and pattern-recognition tool. 13. Blind spots. For all the aspects of handwriting style that GIWIS takes into account, there are evidently some variables that are ignored or missed. As an example, writing size variation is considered a nuisance rather than a shape feature. Even with a habitual pen grip, the range of character heights that can be produced by a writer is quite large. In case there is an absolute reference, such as the line separation of lineated paper, and sufficient comparison data from many writers are available, a user may consider to take this variable into account in the total evaluation process, in addition to the GIWIS results. Similarly, word and line spacing or other pen-tip placement subtleties are ignored by GIWIS, which zooms in on detailed shape. 14. Human expertise. TheGIWIS tool is not in any way a replacement for human expertise, neither in forensic, nor in paleographic applications. The final verdict should be based on additional detailed visual analysis, for which an understandable and convincing narrative can be extracted, supported by other, extraneous sources of information (microscopy, ink or paper composition analysis etc.).