The explosive growth and wide-spread accessibility of community-contributed multimedia contents on the Internet have led to a surging research activity in social image search. However, the existing tag-based search methods frequently return irrelevant or redundant results. To quickly target user's intention in the result returned by an ambiguous query, we first put forward that the top-ranked search results should meet some criteria, i.e., relevance, typicality and diversity. With the three criteria, a novel ranking scheme for social image search is proposed which incorporates both semantic similarity and visual similarity. The ranking list with relevance, typicality and diversity is returned by optimizing a measure named Average Diverse Precision. The typicality score of samples is estimated via the probability density in the space of visual features. The diversity among the top-ranked list is achieved by fusing both semantic and visual similarities of images. A comprehensive approach for calculating visual similarity is considered by fusing the similarity values according to different features. To further benefit ranking performance, a data-driven method is implemented to refine the tags of social image. Comprehensive experiments demonstrate the effectiveness of the approach proposed in this paper.