In this paper we propose a novel approach to select a summary set of images from a large image collection by improved Random Sample Consensus (RANSAC) and Affinity Propagation (AP) clustering. It can automatically select a small set of representatives to highlight all the significant visual properties of a given image collection. The proposed framework mainly composes four stages. First, the scale-invariant feature of each image is extracted by Scale Invariant Feature Transform (SIFT). Second, keypoints of two images are matched and ranked based on nearest neighbor ratio. The representative dataset of RANSAC is established by a minimal number of optimal matches. Third, the target homographic matrix is fitted based on the representative dataset. Mismatches are filtered out via the homographic matrix. Finally, summarization is automatically formulated as an optimization framework by AP clustering. We conduct experiments on a set of Paris which is consisting of 1000 images downloaded from Flickr. The results show that the proposed approach significantly outperforms other methods.