Detecting and predicting searcher success is essential for automatically evaluating and improving Web search engine performance. In the past, Web searcher behavior data, such as result clickthrough, dwell time, and query reformulation sequences, have been successfully used for a variety of tasks, including prediction of success in a search session. However, the effectiveness of the previous approaches has been limited, as they tend to ignore how searchers actually view and interact with the visited pages. We show that fine-grained interactions, such as mouse cursor movements and scrolling, provide additional clues for better predicting success of a search session as a whole. To this end, we identify patterns of examination and interaction behavior that correspond to search success, and design a new Fine-grained Session Behavior (FSB) model to capture these patterns. Our experimental results show that FSB is significantly more effective than the state-of-the-art approaches that do not use these additional interaction data.