Matrix Separation and Poisson Multi-Bernoulli Mixture Filtering for Extended Multi-Target Tracking with Infrared Images

Abstract

Multi-target tracking using infrared images is receiving more and more attention. There are many state-of-the-art methods, and the deep learning network and low-rank and sparse matrix separation are two kinds of methods with high accuracy. However, the former suffers from heavy training samples, and the latter requires high-dimensional processing, meaning its computing cost is huge. In this work, a united detection and tracking method with matrix separation and PMBM filtering is proposed. In the detection process, a low-rank and sparse matrix separation algorithm with a differentiable form based on a single image is constructed. In the filtering process, the multi-target state is modeled as a PMBM distribution, which is conjugate in the Bayesian framework. The two processes interact mutually in that the detection provides measurements, and the filtering offers prior information for the next detection to improve accuracy. The computational complexity is given by a theoretical analysis, which shows a significant reduction. The numerical analysis, carried out on a practical dataset, verifies an enhancement in the BSF and SCRG metrics and ROC curves.

1. Introduction

Driven by the demands of applications such as video surveillance and space target monitoring, multi-target tracking technology based on infrared images has made great progress [1,2,3]. In these applications, one often encounters challenges such as the coexistence of the target of interest and other interfering targets or complex image backgrounds. Existing target tracking and detection algorithms mainly include two categories: Track-Before-Detect (TBD) [4] and Detect-Before-Track (DBT) algorithms [5]. Particle filtering is the main technology in the TBD category [6]. Since sampling of the probability density function is required, the computational complexity of the method is high, and the performance will be seriously degraded when the background is complex. The corresponding technologies in the DBT group include the transform domain [7], deep learning [8], image separation [9,10] and other technologies. The current research mainly focuses on the latter two categories because of their superior performance. However, since deep learning methods require a large number of labeled training samples [8] and the existing technology based on low-rank and sparse matrix separation requires joint spatial–temporal processing of multiple frames [11,12,13,14,15,16], the computational complexity is high. In addition, due to the improvement in current imaging sensors, targets often show “extended” characteristics; i.e., they occupy several pixels in the image rather than “point” targets [17]. Reasonable exploitation of this feature will further improve the accuracy of target tracking methods.

The main innovations of this work are as follows:

(1)

Low-rank and sparse matrix separation and the multi-target filtering method based on random finite sets (RFSs) are integrated into our framework. It effectively utilizes the physical characteristics of the target and background in the image and avoids the pure data-driven properties of the deep learning method; at the same time, the efficiency is deeply improved by filtering compared to traditional spatial–temporal, multi-frame, high-dimensional processing.

(2)

The target obtained in the detection process of low-rank and sparse separation naturally has position and scale information, which is modeled as measurement by RFSs; meanwhile, the filtering process also uses RFSs to describe state parameters. The whole tracking process is realized within a conjugate Bayesian framework.

(3)

In the filtering process, the continuity of the target’s motion is introduced through the state equation so that any false alarm that may occur in the detection process can be effectively suppressed; in addition, the result obtained by filtering can be used as prior information for detection in the next frame. Thus, deep fusion is achieved, and this makes the entire framework more accurate.

2. Differential Low-Rank and Sparse Matrix Separation

In target detection for infrared images, the different physical properties of the target and background, namely sparsity and low rank, respectively, are often used to construct a separation model, as shown in (1) [18]:

$${\min }_{L,O\in {ℝ}^{m\times n}}\left\{\frac{1}{2}\Vert Y-L-O{\Vert }_{\mathrm{F}}^2+{\lambda }_{\ast }\Vert L{\Vert }_{\ast }+{\lambda }_1\Vert O{\Vert }_1\right\}$$

(1)

where $Y$ is a “patch image” matrix whose columns are formed by a series of sliding windows of size $n_w\times n_w$ from top left to bottom right of the original image (see Figure 1). Here, $L$ and $O$ are the low-rank and sparse matrices, which represent background and targets, respectively. $\Vert \cdot {\Vert }_{\ast }$ is the nuclear norm of the matrix, which reflects the low-rank constraint, while $\Vert \cdot {\Vert }_1$ is the ${\mathcal{l}}_1$-norm that describes the sparsity. The parameters ${\lambda }_{\ast }$ and ${\lambda }_1$ adjust the weights of the low-rank and sparse terms. Without losing mathematical equivalence, the matrix $L$ and its nuclear norm in (1) can be further decomposed into

$$\Vert L{\Vert }_{\ast }={\min }_{A,B}\frac{1}{2}\left\{\Vert A{\Vert }_{\mathrm{F}}^2+\Vert B{\Vert }_{\mathrm{F}}^2\right\}\hspace{1em} \mathrm{s}.\mathrm{t}. \hspace{1em}AB=L$$

(2)

where $L=US{V}^{\mathrm{T}}$, and $\Vert \cdot {\Vert }_{\mathrm{F}}^2$ is the Frobenius norm of the matrix. Minimization is achieved when $A=U{\Sigma }^{\frac{1}{2}}$ and $B={\Sigma }^{\frac{1}{2}}V$.

Inspired by (2), the low-rank matrix is decomposed into two parts: the dictionary, $D_0$, and coefficient, $S$. Thus, (1) becomes

$${\min }_{D_0,S,O}\left\{\frac{1}{2}{‖Y-D_{0}S-O‖}_{\mathrm{F}}^2+\frac{{\lambda }_{\ast }}{2}\left({‖D_0‖}_{\mathrm{F}}^2+\Vert S{\Vert }_{\mathrm{F}}^2\right)+\lambda \Vert O{\Vert }_1\right\}$$

(3)

where $L=D_{0}S$, and the dimensions of each matrix are $D_0\in {ℝ}^{m\times q},\hspace{0.33em}S\in {ℝ}^{q\times n}$ and $O\in {ℝ}^{m\times n}$, respectively. A unified fast algorithm can be used to solve (3), and the specific process is shown in Algorithm 1. In this algorithm, the formula $\left(Z^{k+1}-Z^k\right)$ is multiplied by the matrix $H$, which comes from the differentiability of the objective function in (3), which greatly improves the computational efficiency. At the same time, $p_t\left(B^k\right)$ is an operation carried out on each component of the vector, and its $i$-th component is ${\tau }_{t_i}\left(B^k{ }_i\right)=\max \{0,B^k{ }_i-t_i\}$, which is only a threshold operation. Reference [19] proves that if a solution obtained by the algorithm in Algorithm 1 satisfies ${‖Y-D_{0}S-O‖}_2\le {\lambda }_{\ast }$, then it is global optimal [19].

Algorithm 1 Algorithm for solving the differential form of low-rank and sparse matrix separation.

Input: Data $Y$, dictionary $D_0$, parameters $\lambda ,\hspace{0.33em}{\lambda }_{\ast }$ and Step size controller $\alpha$ Output: Low rank matrix $L$, sparse matrix $O$ Define: $H=I-\frac{1}{\alpha }\left(\begin{array}{cc}{D}_0^{\mathrm{T}}{D}_0+{\lambda }_{\ast }I& D_0^{\mathrm{T}}\\ D_0& \left(1+{\lambda }_{\ast }\right)I\end{array}\right)$$, W=\frac{1}{\alpha }\left(\begin{array}{c}{D}_0^{\mathrm{T}}\\ I\end{array}\right)$$, t=\frac{\lambda }{\alpha }\left(\begin{array}{l}0\hfill \\ 1\hfill \end{array}\right)$ Initialize: $Z^0=0,B^0=WY$ For k = 1, 2, …. (until convergence) $$Z^{k+1}={\pi }_t\left(B^k\right)$$ (4) $$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{B}^{k+1}=B^k+H\left(Z^{k+1}-Z^k\right)$$ (5) End where $Z^{k+1}=\left(S;\hspace{0.33em}O\right)$$, L=D_{0}S$

3. PMBM for Extended Multi-Target Tracking

In recent years, due to the development of infrared sensors, imaging resolution has become higher and higher, so small targets will also occupy several pixels in an image, showing the characteristics of “extended” targets. In order to make better use of this information, in extended target filtering, the state of the target is added with the shape (scale) information on the basis of the original kinematic (such as position and velocity) information only.

3.1. PMBM for State Modeling

Extended target tracking requires appropriate mathematical tools to model the kinematic and shape information of the target. In this paper, the Bernoulli RFS in (6) is first introduced to describe a single extended target.

$$p(X)=\left\{\begin{array}{l}1-r\hspace{1em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}X=\varnothing \\ r\hspace{0.33em}p\left(x\right)\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}X=\left\{x\right\}\\ 0\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\left|X\right|\ge 2\end{array}\right.,$$

(6)

In (6), the target state vector takes an empty set, which means that the target “disappears” with a probability of $1-r$. The state $X$ “exists” as a single element set, $\left\{x\right\}$, with a probability of $r$. The function $p\left(x\right)$ is a probability density function describing the position (through the expectation) and the extension (through the standard deviation) of the target. Generally, $p\left(x\right)$ can be chosen to be a Gaussian distribution.

Furthermore, in order to represent multiple targets, an index set, $\mathbb{I}$, is introduced. In this case, the number of targets in $X$ is equal to the cardinality of $\mathbb{I}$, thus obtaining multi-Bernoulli (MB) RFSs.

$$p(X)=\left\{\begin{array}{c}{\displaystyle {\sum }_{X={\cup }_{i\in \mathbb{I}}\hspace{0.33em}{X}^i}{\displaystyle {\prod }_{i\in \mathbb{I}}{p}^i\left(X^i\right)}}\hspace{1em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\left|X\right|\le \left|\hspace{0.33em}\mathbb{I}\hspace{0.33em}\right|\hfill \\ 0\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\left|X\right|\ge \left|\hspace{0.33em}\mathbb{I}\hspace{0.33em}\right|\end{array}\right.$$

(7)

where $\forall i,j\in \mathbb{I},\hspace{0.33em}{X}^i\cap X^j=\varnothing$ and $X={\cup }_{i\in \mathbb{I}}{X}^i$ represent the union of multiple targets. From (6) and (7), one can conclude that the key parameters of the MB model are ${\left\{\left(r^i,p{\left(\cdot \right)}^i\right)\right\}}_{i\in {\mathbb{I}}^{ }}$, which represent the survival probability of the $i$-th target and the probability density representing the “extended” state, respectively.

Data association is one of the key steps in multi-target tracking, and its purpose is to establish a connection between the measurement data and target status. The mathematical representation of data association is also considered in state modeling. Let ${\mathcal{W}}^j,\hspace{0.33em}j\in \mathbb{J}$ be the corresponding weights of several MB RFSs; i.e., for a fixed $j$, there is an MB RFS ${\left\{\left(r^{j,i},p{\left(\cdot \right)}^{j,i}\right)\right\}}_{i\in {\mathbb{I}}^j}$, and then their combination forms a multi-Bernoulli mixture (MBM) RFS. The key parameters of the MBM RFS can be listed as ${\left\{\left({\mathcal{W}}^j,{\left\{\left(r^{j,i},p{\left(\cdot \right)}^{j,i}\right)\right\}}_{i\in {\mathbb{I}}^j}\right)\right\}}_{j\in \mathbb{J}}$. The weight ${\mathcal{W}}^j$ represents the probability that a certain measurement belongs to the target states.

When multiple targets exist, some target could be missed in some time steps. In (8), the undetected targets, $X^u$, are described through the Poisson point process (PPP):

$$p^u\left(X^u\right)=e^{-〈D^u\left(x\right),\hspace{0.33em}1〉}{\displaystyle {\prod }_{x\in X^u}{D}^u\left(x\right)}$$

(8)

where $D^u\left(x\right)={\mu }^u{p}^u\left(x\right)$, ${\mu }^u$ are constant numbers, and $p^u\left(x\right)$ is the probability density of undetected targets. $〈D^u\left(x\right),\hspace{0.33em}1〉={\displaystyle {\int }_{ }^{ }{D}^u\left(x\right)\mathrm{d}x}={\mu }^u$ is the inner product.

By combining (7) and (8), the state model for extended multi-target tracking, the Poisson multi-Bernoulli mixture (PMBM), can be obtained:

$$p(X)={\displaystyle {\sum }_{\hspace{0.33em}{X}^u+X^d=X}{p}^u(X^u)}{\displaystyle {\sum }_{\hspace{0.33em}j\in \mathbb{J}}{\mathcal{W}}^j{p}^j\left(X^d\right)}$$

(9)

where $X^d$ and $X^u$ ($X^u\cap X^d=\varnothing$) are the detected and undetected targets, respectively, and

$$p^j\left(X^d\right)={\displaystyle {\sum }_{X^d={\cup }_{i\in {\mathbb{I}}^j}{X}^i}{\displaystyle {\prod }_{i\in {\mathbb{I}}^j}{p}^{j,i}\left(X^i\right)}}$$

(10)

Once the state model was obtained, we also used the probability density function $f_{k+1,k}\left(x_{k+1}|x_k\right)$ to represent state transition. In addition, $p_{\mathrm{S}}\left(x_k\right)$ was employed to represent the probability that the target still “exists” from time $k$ to time $k+1$, and a PPP model with density $D_{k+1}^b\left(x\right)$ was introduced to represent the emergence of new target.

3.2. Measurement Modeling

The measurement refers to the “targe part” in the infrared image obtained through the detection process. Let $p_{\mathrm{D}}\left(x\right)$ be the probability that the target can be detected, and after detection, the model of the measurement is given as

$${\mathcal{l}}_Z\left(x\right)=p_{\mathrm{D}}(x)p(Z|x)=p_{\mathrm{D}}\left(x\right)e^{-\gamma \left(x\right)}{\displaystyle {\prod }_{z\in Z}\gamma (x)\varphi (z|x)}$$

(11)

which is a PPP model with an intensity of $\gamma (x)\varphi (z|x)$. In (11), $\gamma (x)$ is the Poisson intensity and $\varphi (z|x)$ is the spatial distribution of the measurement. On the other hand, the probability that the target is not measured is

$$q_{\mathrm{D}}(x)=1-p_{\mathrm{D}}(x)+p_{\mathrm{D}}(x)e^{-\gamma (x)}$$

(12)

Assuming the measurement dataset is $Z={\left\{z^m\right\}}_{m\in \mathbb{M}}$ and $\mathbb{M}\cup {\mathbb{I}}^j$ is the union of the measurement index set $\mathbb{M}$ and the target’s state index set ${\mathbb{I}}^j$, the process of data association is basically a partition of $\mathbb{M}\cup {\mathbb{I}}^j$, and all possible results of the partition are set to ${\mathcal{A}}^j$. For a simple interpretation, if $\mathbb{M}=\left(m_1,\hspace{0.33em}{m}_2,\hspace{0.33em}{m}_3\right)$ and ${\mathbb{I}}^j=\left(i_1^j,\hspace{0.33em}{i}_2^j\right)$, then the process of data association is carried out to assign these three measurement values to two targets, where each target can have multiple measurements or none. Thus, one result may be $\left\{m_1,\hspace{0.33em}{m}_2,\hspace{0.33em}{i}_1^j\right\},\hspace{0.33em}\left\{m_3\right\},\hspace{0.33em}\left\{i_2^j\right\}$, where the measurements $m_1$ and $m_2$ are associated with the target $i_1^j$, the target $i_2^j$ is not measured, and the measurement $m_3$ is not associated with any detected targets. The set $\left\{m_3\right\}$ indicates that it may be a measurement of a new target or clutter. In multi-target tracking, clutter is also modeled as a PPP with an intensity of $\kappa \left(z\right)=\lambda c\left(z\right)$.

3.3. Conjugate Bayesian Filtering

The process of filtering in multi-target tracking refers to obtaining the target state at time $k$ by combining the state transition information with the target state at time $k-1$ and the measurement at time $k$. Since the states at time $k-1$ and time $k$ are the prior and posterior distributions, respectively, it is a Bayesian filtering process.

As discussed above, the target state is modeled as a PMBM distribution. It has been proven that in Bayesian filtering, if the prior distribution is a PMBM, then the posterior distribution also satisfies a PMBM. Therefore, the filtering process only needs to update the parameters of the model, eliminating the calculation of the density function itself, thus significantly improving efficiency.

The filtering process can be divided into two processes: prediction and update. The prediction means using the state at time k − 1 to “predict” a value at time k through the state transfer function $f_{k+1,k}\left(x_{k+1}|x_k\right)$. Since the distribution in a PMBM is fixed (as Gaussian and Poisson), the parameters

$$D^u(x),\hspace{0.33em}\hspace{0.33em}{\left\{\left({\mathcal{W}}^j,{\left\{\left(r^{j,i},p^{j,i}(x)\right)\right\}}_{i\in {\mathbb{I}}^j}\right)\right\}}_{j\in \mathbb{J}}$$

(13)

are enough to describe the target state at time $k-1$. Furthermore, the predicted state also satisfies the PMBM, and the corresponding parameters are [20,21]

$$D_{+}^u(x),{\left\{\left({\mathcal{W}}_{+}^j,{\left\{\left(r_{+}^{j,i},p_{+}^{j,i}(x)\right)\right\}}_{i\in {\mathbb{I}}_{ }^j}\right)\right\}}_{j\in {\mathbb{J}}_{ }}$$

(14)

where

$$\begin{array}{c}{D}_{+}^u(x)=D^b(x)+〈D^u(x),\hspace{0.33em}{p}_{\mathrm{S}}(x)f_{k+1,k}\left(x_{k+1}|x_k\right)〉\\ r_{+}^{j,i}=〈p^{j,i}(x),p_{\mathrm{S}}(x)〉r^{j,i}\\ p_{+}^{j,i}(x)=\frac{〈p^{j,i}(x), \hspace{0.17em}{p}_{\mathrm{S}}(x)f_{k+1,k}\left(x_{k+1}|x_k\right)〉}{〈p^{j,i}(x),p_{\mathrm{S}}(x)〉}\end{array}$$

(15)

Since the process of prediction is independent of the measurement, ${\mathcal{W}}_{+}^j$ equals ${\mathcal{W}}^j$.

The update process refers to the use of the predicted state value in (14), combined with the measurement $Z={\left\{z^m\right\}}_{m\in \mathbb{M}}$, to obtain the final target state at time $k$. It has been shown that in the PPP measurement model, the updated state also satisfies the PMBM [20,21]:

$$\begin{array}{c}p(X|Z)={\displaystyle \sum _{X^u+X^d=X}}{p}^u\left(X^u\right){\displaystyle \sum _{j\in {\mathbb{J}}_{ }}}{\displaystyle \sum _{A\in {\mathcal{A}}^j}}{\mathcal{W}}_A^j{p}_A^j\left(X^d\right)\\ p^u\left(X^u\right)=e^{-〈D^u(x),1〉}{\displaystyle \prod _{x\in X^u}}{D}^u(x)\\ p_A^j\left(X^d\right)={\displaystyle \sum _{X={\cup }_{C\in A}{X}^c}} {\displaystyle \prod _{C\in A}}{p}_C^j\left(X^C\right)\end{array}$$

(16)

where

$${\mathcal{W}}_A^j=\frac{{\mathcal{W}}_{+}^j{\displaystyle \prod _{C\in A}}{\mathcal{L}}_C}{{\displaystyle \sum _{j^{\prime }\in \mathbb{J}}}{\displaystyle \sum _{A^{\prime }\in {\mathcal{A}}^{j^{\prime }}}}{\mathcal{W}}_{+}^{j^{\prime }}{\displaystyle {\displaystyle \prod _{C^{\prime }\in A^{\prime }}}}{\mathcal{L}}_{C^{\prime }}}$$

(17)

and the PPP intensity is $D^u(x)=q_{\mathrm{D}}(x)D_{+}^u(x)$.

4. The United Framework and Complexity Analysis

4.1. Detection and Filtering Framework

The infrared image small-target tracking method in this paper deeply integrates (single-frame-image) low-rank and sparse matrix separation and the extended multi-target filtering algorithm. The whole framework is shown in Figure 1. Firstly, the original image traverses the sliding windows (small red rectangle) and is rearranged as the input patch image, $Y$, for low-rank and sparse matrix separation. Then, through the algorithm in Section 2, the “target pixels” in the image are separated, which have specific positions and shapes. Thirdly, taking the “target pixels” as the measurement from extended multi-target filtering, a state estimation of the multiple targets can be obtained by using the state prediction and update process given in Section 3. Finally, the target state is used as prior information in the low-rank and sparse separation in the next frame, and a new round of detection and filtering is carried out. It is worth noting that this work combines multiple frames of images through the Bayesian filtering process to improve the accuracy of detection and tracking, rather than using multiple frames of images through high-dimensional processing like in the existing spatial–temporal method, so the computational complexity is significantly reduced.

4.2. Computational Complexity

Let’s recall the dimensions of the matrices mentioned in Algorithm 1 first, i.e., $D_0\in {ℝ}^{m\times q},\hspace{0.33em}S\in {ℝ}^{q\times n}$ and $O\in {ℝ}^{m\times n}$. The computational complexity mainly comes from the matrix multiplication in Algorithm 1, i.e., the multiplication of matrices $H$ and $Z^{k+1}-Z^k$ in (5). From their construction, it can be seen that their dimensions are $q\left(m+q\right)\times q\left(m+q\right)$ and $q\left(m+q\right)\times n$, respectively. Thus, the complexity is $O\left[{\left(q\left(m+q\right)\right)}^2\times n\right]$, which is needed in each iteration. A threshold operation is also required during the iteration in (4). Since this operation is performed on the elements, the time complexity is negligible compared to the above matrix multiplication. In the process of filtering the PMBM model, since the “sparse” part is obtained from low-rank and sparse decomposition as a measurement, its size is very small compared to the original matrix. At the same time, the dimensions of the state vectors of the filtering process are similar to those of the measurement; therefore, the complexity of the filtering process is negligible compared with low-rank and sparse matrix decomposition. In summary, the computational complexity of the entire algorithm is of the order of $O\left[K\left({\left(q\left(m+q\right)\right)}^2\times n\right)\right]$, where $K$ is the number of iterations in the low-rank and sparse decomposition process. This complexity allows the algorithm to satisfy the needs of real-time applications.

5. Experiments

In this section, an infrared image dataset from an unmanned aerial vehicle (UAV) is used to verify the performance of the algorithm proposed in this paper. This dataset has the following characteristics: (1) Due to the huge difference in distance between the sensor and the UAV, the scale of the target in the images varies greatly. At a closer distance, the UAV appears as an extended target, and at a longer distance, it appears as a point target. Therefore, it matches the features of the extended target filtering algorithm discussed in this paper. (2) The dataset contains multiple flight processes, and each flight process forms an image sequence. These sequences include both single UAV flights and multiple UAVs flying simultaneously. Meanwhile, there are both sky backgrounds and complex ground backgrounds, which enhances the difficulty of detection and tracking.

After comprehensive analysis, three representative state-of-the-art methods were selected for comparison. These three methods are four-dimensional spatial–temporal tensor decomposition with a block term decomposition-based norm and multidirectional derivative-based priors (4DST-BTMD) [14]; a weighted adaptive Schatten p-norm and spatial–temporal tensor transpose variability (WASpN-STTTV) model [13]; and a Spatial Temporary Tensor Modeling With Saliency Filter Regularization (STTM-SFR) algorithm [10]. It has been shown that these algorithms outperform multiple frame sequential detection methods such as spatial temporal difference measurement (STLDM) [22] and edge and corner awareness-based spatial temporal tensors (ECASTTs) [23]. In addition, the advantages of PMBM filtering compared with other RFS filters have been verified in [24].

There are some control parameters in the process of low-rank and sparse matrix decomposition, and their values are chosen as follows. In the construction of the patch image matrix, $Y$, the size of the sliding window is 16*16 pixels. The balance parameters, ${\lambda }_{\ast }$ and $\lambda$, in (3) are chosen according to the recommendations in [13,14] with fine-tuning for the infrared dataset. Specifically, we set ${\lambda }_1=c/\sqrt{\min \left(n_1,n_2\right)\times n_3}$, where $n_1$ and $n_2$ are the 2D patch sizes, and $n_3$ is the number of patches from the original image. The value of the constant $c$ is initially set to 2.2 and adaptively adjusted in real time according to PMBM filtering based on the result of the previous timepoint. The value of ${\lambda }_{\ast }$ is set to $0.05\cdot {\lambda }_1$ to balance the measurements and clusters in the filtering. As for PMBM tracking, the detection probability is 0.95, and the false alarm probability is $8\times {10}^{-4}$.

The metrics used for the performance comparison were also obtained from studies of the above methods, and they include the Background Suppression Factor (BSF) and the Signal-to-Clutter Ratio Gain (SCRG) [13]. The BSF is a commonly used indicator for infrared image target detection, defined as

$$\mathrm{BSF}={\sigma }_{\mathrm{in}}/{\sigma }_{\mathrm{out}}$$

(18)

where ${\sigma }_{\mathrm{in}}$ and $\hspace{0.33em}{\sigma }_{\mathrm{out}}$ are the standard variances of the input and the detected image, respectively. The SCRG is defined as follows:

$$\mathrm{SCRG}=20\cdot \log 10\left({\mathrm{SCR}}_{\mathrm{out}}/{\mathrm{SCR}}_{\mathrm{in}}\right)$$

(19)

$$\mathrm{SCR}=\left(I_{\max }-I_{\mathrm{mean}}\right)/\sigma$$

(20)

where $I_{\max },\hspace{0.33em}{I}_{\mathrm{mean}}$, and $\sigma$ are the maximum, minimum, and standard variance of the image, respectively. The receiver operating characteristic (ROC) curve [13] is also used to compare the performance of the different algorithms, where the x-axis is $P_f$ and the y-axis is $P_d$.

$$P_d=\frac{\mathrm{number} \mathrm{of} \mathrm{detected} \mathrm{true} \mathrm{pixels}}{\mathrm{groundtruth} \mathrm{target} \mathrm{pixels}}$$

(21)

$$P_f=\frac{\mathrm{false} \mathrm{alarm} \mathrm{pixels} }{\mathrm{total} \mathrm{number} \mathrm{of} \mathrm{pixels}}$$

(22)

In the dataset [11], six image sequences (each containing about 300 image frames with 256 × 256 pixels) were selected for performance verification. Sequences 1 and 2 are multiple extended targets in the sky background (see the two frames in Figure 2a,b, respectively). The trajectories between the targets overlapped, which makes the tracking process difficult. Sequences 4 to 6 are images including complex backgrounds (see the four frames in Figure 2c,g–i, respectively). Their backgrounds contain objects with different characteristics, which can cause confusion and make the detection process difficult. The corresponding results of low-rank and sparse matrix separation are listed below the original image frame, respectively (see Figure 2d–f,j–l). One can see that the targets can be detected effectively.

For a quantitative comparison, the mean of the BSF and SCRG metrics are calculated over the six image sequences, respectively, and listed in

Table 1. A quantitative comparison of BSF and SCRG metrics among four methods on the six selected image sequences.

		STTM-SFR	WASpN-STTTV	4DST-BTMD	The Proposed
BSF	Seq. 1	43.54	45.33	45.74	46.07
	Seq. 2	42.90	44.33	45.06	45.46
	Seq. 3	27.62	28.44	28.23	28.53
	Seq. 4	35.11	36.61	37.62	37.35
	Seq. 5	36.54	37.60	38.52	38.94
	Seq. 6	34.05	35.11	35.53	36.06
SCRG	Seq. 1	37.01	38.71	39.27	40.29
	Seq. 2	36.69	37.99	38.58	39.14
	Seq. 3	29.92	30.71	30.45	31.28
	Seq. 4	37.70	38.03	38.94	38.62
	Seq. 5	38.72	39.04	39.39	39.51
	Seq. 6	36.37	36.52	36.97	37.23

. Higher values mean better suppression of the background and better accuracy in object detection. The results show that the proposed algorithm outperforms the state-of-the-art algorithms, which are STTM-SFR, WAS_pN-STTTV, and 4DST-BTMD, except in one situation (see Seq. 4). It is worth mentioning that performance was enhanced not only by the low-rank and sparse separation algorithm itself but also by the use of accurate prior information in the filtering process.

Finally, the ROC curve is used to compare the values of $P_f$ and $P_d$. In Figure 3, the six panels (a) to (d) are plotted from image sequence 1 to 6, respectively. The horizontal axis in the figure represents the $P_f$ value after logarithmic transformation (log10), and the vertical axis represents the $P_d$ value. There are four curves in each panel, representing the four methods we employed. The red curve in each panel represents the proposed method, and it is almost always above the other curves.

6. Conclusions

A unified framework for small-target tracking in infrared images is proposed in this work. The framework deeply combines a fast low-rank and sparse matrix separation algorithm and a PMBM based on the Bayesian multi-target filtering process. The low-rank and sparse matrix separation completes the process of target detection based on a signal image, and it constitutes a fast algorithm benefiting from its differentiable form with global convergence. On the other hand, extended multi-target filtering is carried out, which matches the characteristics of the target in high-resolution infrared images. In the process of filtering, the state of multiple targets is modeled by the PMBM distribution. Due to the conjugate feature of the PMBM in the Bayesian framework, we only need to update the parameters rather than the whole distribution, so the computational complexity is significantly reduced. Unlike simultaneous spatial–temporal processing based on multiple frames in the existing literature, the enhancement in accuracy comes from the sequential use of detection and filtering, where the former provides the measurement for the latter and the latter provides precise prior information for the former. A theoretical complexity analysis is given in this work, and it is found to be lower than in existing 3D or 4D low-rank and sparse decomposition in multiple frames, which explains the efficiency of the whole framework. Finally, verification on a practical infrared dataset was carried out, and corresponding quantitative metrics (BCF and SCRG) and ROC curves proved the accuracy of the proposed method in cluster suppression and target detection. One of the limitations of the united framework comes from the parameters, which need fine-tuning based on the theoretical criteria.

Although the proposed framework has led to some improvements in terms of computational efficiency and accuracy, there is still some room for improvement. For example, the current combination is feature-level. Future work will consider a direct integration of matrix decomposition and PMBM filtering; that is, we will integrate the matrix decomposition process into the state modeling, transition, and update of the PMBM and develop a framework that advances from the current feature-level fusion to signal-level fusion.