Relevance feedback has been proposed as an important technique to boost the retrieval performance in content-based image retrieval (CBIR). However, since there exists a semantic gap between low-level features and high-level semantic concepts in CBIR, typical relevance feedback techniques need to perform a lot of rounds of feedback for achieving satisfactory results. These procedures are time-consuming and may make the users bored in the retrieval tasks. For a long-term study purpose in CBIR, we notice that the users' feedback logs can be available and employed for helping the retrieval tasks in CBIR systems. In this paper, we propose a novel scheme to study the log-based relevance feedback (LRF) technique for improving retrieval performance and reducing the semantic gap in CBIR. In order to effectively incorporate the users' feedback logs, we propose a modified support vector machine (SVM) technique called soft label support vector machine (SLSVM) to construct the LRF algorithm in CBIR. We conduct extensive experiments to evaluate the performance of our proposed algorithm. Compared with the typical approach using query expansion (QEX) technique, we demonstrate that our proposed scheme can significantly improve the retrieval performance of semantic image retrieval from detailed experiments.