Visual appearance is described as a cue with which we discriminate images. It has been conjectured that appearance similarity emerges from similarity between features of image surfaces. However, the design of effective appearance features and their efficient representations is an open problem. In this dissertation, appearance features are developed by decomposing image brightness surfaces differentially in space, and in scale. Image representations constructed from multi-scale differential features are compared to determine appearance similarity. The first part of this thesis explores image structure in scale and space. Multi-scale differential features are generated by filtering images with Gaussian derivatives at multiple scales (GMDFs). This provides a robust local characterization of the brightness surface; filtered outputs can be transformed to seek rotation, illumination, view and scale tolerance. Differential features are also shown to be descriptive; both local and global representations of images can be composed from them. The second part of this thesis begins by illustrating local and global representations including feature-templates, -graphs, -ensembles and -distributions. It continues by developing one algorithm, CO-1, in detail. In this algorithm, two robust differential-features, the orientation of the local gradient and the shape-index, are selected for constructing representations. GMDF distributions of the first type are used to represent images and euclidean distance measure is used to determine similarity between representations. The first application of CO-1 is to image retrieval, a task central to developing search and organization tools for digital multimedia collections. CO-1 is applied to example-based browsing of image collections and trademark: retrieval, where appearance similarity can be important for adjudicating relevance. The second application of this work is to image-based and view-based object recognition. Results are demonstrated for face recognition using several standard collections. The central contribution of this work in the words of a reviewer is “… in the simplicity and elegance of the approach of using low-level multi-scale differential image structure.” We posit that this thesis highlights the utility of exploring differential image structure to synthesize features effective in a wide range of appearance-based retrieval and recognition tasks
