Two-stage salient object detection based on prior distribution learning and saliency consistency optimization

Abstract

Although two-stage methods have favorably improved the accuracy and robustness of saliency detection, obtaining a saliency map with clear foreground boundaries and fine structure is still challenging. In this article, we proposed a novel and effective two-stage method to intelligently detect salient objects, which constitutes the coarse saliency map construction and the fine saliency map generation. Firstly, we develop the prior distribution learning algorithm (PDL) to explore the mapping relationship between input superpixel and its corresponding superpixels in various prior maps. The PDL can calculate the corresponding weights according to the contribution of various priors to each region in the image. Therefore, it can provide more reliable pseudo-labels for training subsequent learning models. Secondly, through learning the implicit representation between reliable samples and multiple priors, the learning model can accurately predict the salient values of those regions that are difficult to judge the saliency, so as to obtain an instructive coarse saliency map. Thirdly, in order to optimize the details of the coarse saliency map, we propose a framework called saliency consistency optimization, which can get clear foreground boundaries and effectively suppress the background noise. We compare the proposed algorithm with other state-of-the-art methods on four datasets. Experimental results adequately demonstrate the effectiveness of our approach over other comparison methods, especially two-stage-based methods.

Appendix

In the multi-prior distribution learning model, five prior knowledge are utilized to extract saliency cues, i.e., background prior, objectness prior, color distribution prior, global contrast prior and center prior. Here, we will give more details about these five prior knowledge.

Background prior is the universal prior knowledge in saliency detection. It usually assumes image boundary superpixels as initial background seeds. Then, each superpixel’s saliency value is determined by its feature contrast with background seeds. Thus, we construct a background-based map ${\text{BG}} = \left[ {{\text{bg}}_{1} ,{\text{bg}}_{2} , \ldots ,{\text{bg}}_{N} } \right]^{T}$ as follows:

$$ {\text{bg}}_{i} = \frac{1}{n}\sum\limits_{{j = 1}}^{n} {\exp } \left( {\frac{{\left\| {d_{i} ,d_{j}^{b} } \right\|}}{\sigma }} \right) $$

(13)

where ${d}_{i}$ and ${d}_{j}^{b}$ represent the deep features of $i$-th superpixel and $j$-th boundary superpixel, respectively. $n$ is the total number of boundary superpixels, and the parameter $\sigma$ is set to 0.1. Higher ${\text{bg}}_{i}$ indicates that superpixel $m_{i }$ has higher contrast to background seeds and is more likely to be the salient object. Otherwise, superpixel ${ } m_{i }$ tends to be background regions.

Objectness prior is proposed in [50]. It firstly constructs a large number of windows, each of which is a part of the image. Then several rules are utilized to compute the likelihood of each window containing salient objects. Objectness prior is widely applied to numerous saliency detection methods due to its effectiveness. We define the objectness map to be ${\text{OB}} = \left[ {{\text{ob}}_{1} ,{\text{ob}}_{2} , \ldots ,{\text{ob}}_{N} } \right]^{T}$.

Color distribution prior is proposed in our previous works [12]. It firstly divides image pixels into eight regions according to color cues. Secondly, those regions that jointly have compact spatial structures and are near the image weight center are more likely to be salient objects. Finally, according to [12], we obtain a color distribution map ${\text{CD}} = \left[ {{\text{cd}}_{1} ,{\text{cd}}_{2} , \ldots ,{\text{cd}}_{N} } \right]^{T}$ without any modifications.

Global contrast prior is based on the observation that the human eye tends to detect regions which have higher contrast to other regions. Thus, we construct the global contrast-based map ${\text{GC}} = \left[ {{\text{gc}}_{1} ,{\text{gc}}_{2} , \ldots ,{\text{gc}}_{N} } \right]^{T}$ as follows:

$$ {\text{gc}}_{i} = \frac{1}{N}\sum\limits_{{j = 1,j \ne i}}^{N} {{\text{exp}}} \left( {\frac{{\left\| {d_{i} ,d_{j} } \right\|}}{\sigma }} \right) $$

(14)

where $d_{i}$ and $d_{j}$ are the deep features of superpixel $m_{i }$ and $m_{j} ,$ respectively,$\,{ }N$ is the total number of superpixels and parameter $\sigma$ is set to 0.1. The global contrast value of each superpixel is determined by the mean contrast between it and other superpixels.

Center prior is also effective prior knowledge, which assumes that image center regions are more likely to be salient objects. To this end, we construct the center-based map ${\text{CB}} = \left[ {{\text{cb}}_{1} ,{\text{cb}}_{2} , \ldots ,{\text{cb}}_{N} } \right]^{T}$ as follows:

$$ {\text{cb}}_{i} = \exp \left( { - \frac{{\left\| {{\mathbf{p}}_{i} ,{\mathbf{p}}^{{\mathbf{c}}} } \right\|}}{\sigma }} \right) $$

(15)

where ${\mathbf{p}}_{i}$ is the position coordinate of superpixel $m_{i }$ and ${\mathbf{p}}^{{{\mathbf{c}} }}$ is the image center position, parameter $\sigma$ is set to 0.1.

In multi-prior distribution learning algorithm, five prior maps are represented as ${\text{pm}}_{1}$,${\text{pm}}_{2}$,…, ${\text{pm}}_{5}$, i.e., ${\text{pm}}_{1}$ for ${\text{ BG}}$,${\text{pm}}_{2}$ for ${\text{ OB}}$,${\text{pm}}_{3}$ for ${\text{ SD}}$,${\text{pm}}_{4}$ for ${\text{ GC}}$ and ${\text{pm}}_{5}$ for ${\text{ CB}}$.