A Method of Fusing Probability-Form Knowledge into Object Detection in Remote Sensing Images

Abstract

In recent years, dramatic progress in object detection in remote sensing images has been made due to the rapid development of convolutional neural networks (CNNs). However, most existing methods solely pay attention to training a suitable network model to extract more powerful features in order to solve the problem of false detections and missed detections caused by background complexity, various scales, and the appearance of the object. To open up new paths, we consider embedding knowledge into geospatial object detection. As a result, we put forward a method of digitizing knowledge and embedding knowledge into detection. Specifically, we first analyze the training set and then transform the probability into a knowledge factor according to an analysis using an improved version of the method used in existing work. With a knowledge matrix consisting of knowledge factors, the Knowledge Inference Module (KIM) optimizes the classification in which the residual structure is introduced to avoid performance degradation. Extensive experiments are conducted on two public remote sensing image data sets, namely DOTA and DIOR. The experimental results prove that the proposed method is able to reduce some false detections and missed detections and obtains a higher mean average precision (mAP) performance than the baseline method.

1. Introduction

Object detection classifies and locates geospatial objects in aerial images and is a fundamental task in remote sensing. Recently, there have been great advances in object detection in aerial images (ODAI) [1,2] with the advent of deep convolutional neural networks. Because of this, applications in some fields, such as (UAVs) and wireless sensor networks (WSNs), have also benefited dramatically [3,4]. Although a large number of excellent methods [5,6,7,8,9] have been invented in the past few years, updating the previous state-of-the-art methods, there are still problems, such as missing detections and false detections caused by densely distributed objects, varied object sizes, and occluded objects. To illustrate, Figure 1a is a visualization of false harbor and helicopter detection, where a plane is detected as the harbor, and a piece of land is regarded as a helicopter. In Figure 1b, a piece of agricultural land is detected as a soccer ball field. In Figure 1c, a shaded basketball court is not detected. In the right of Figure 1d, two bridges are not detected, which damages the detection performance. The traditional object detection paradigm only takes the extracted feature into consideration. However, this is the bird eye view in aerial images. Therefore, detection algorithm inference only uses the roof information from aerial images. Moreover, due to its complex background features, such as the shape similarity and appearance similarity between objects and the background, the wrong labels are predicted. Because of the ambiguous appearance, those algorithms with an advantage in terms of extracting features do not have a significant effect. This is because when the instances are ambiguous, the extracted features are not that powerful, which leads to missed detection and false detection.

However, the human visual recognition system informs us how to tackle this problem, because humans take their surroundings, as well as objects into account when conducting recognition tasks [10], which might be beneficial for geospatial object detection. When humans see one thing, it is common for them to think of another thing related to the seen one. Normally, for instance, when a harbor comes into view, there might be ship parking along the harbor. In addition, the ship and the harbor appear with water, which provides the link between the water and the object. Relationships between objects and relationships between a scene and objects can be assembled into a knowledge pool, which is then used in the object detection task, the process of which is shown in Figure 2. With the utilization of such a knowledge pool, the human visual recognition system is superior to the artificial intelligent algorithm in terms of its detection performance. The visual recognition method mentioned above benefits from the utilization of knowledge, which is worth adopting in the object detection algorithm.

Obviously, it is by converting pictures into data that object detection algorithms process converted data and can detect objects. Thus, in order to improve the detection method added to work, such as the human visual system, the transformation of knowledge into data is essential. Refs. [11,12] were our sources of inspiration. Li et al. [11] presented statistics regarding the probability of objects appearing in each scene and assembled these data into a probability matrix with the aim of improving scene classification. Consequently, thinking in opposition, it is profitable to use the probability of an object appearing in the relevant scene to guide detection. In [12], Xu et al. utilized a class relation matrix to boost the object detection performance. Specifically, they integrated a class relation matrix with a high-dimensional feature to obtain an enhanced feature, which is not that intuitively explainable due to the high-dimensional space. As a result, to deal with false detections, such as a harbor being detected on land, in this work, we explored the co-relations between the water area and objects by analyzing a data set. Moreover, in order to utilize the implicit relation as knowledge, we analyzed the conditional co-occurrence probabilities of different categories, which is the expression of the implicit relation.

In this paper, an approach to utilizing knowledge is proposed. Our overall contributions are as follows:

• We extract two kinds of knowledge in the form of probabilities, namely the correlations between classes and the correlations between the water area and classes, by analyzing the DOTA [13] and DIOR [2] training sets. Then, we transform the extracted knowledge into a knowledge factor using a novel equation improved from [14].
• We propose a method, namely the Knowledge Inference Module (KIM), of integrating knowledge into object detection in remote sensing images. Through an evaluation of two public aerial data sets, our method obtains a higher mAP than the baseline model, Oriented R-CNN [15] with fewer false and missed detections.

2. Related Work

2.1. Object Detection in Remote Sensing Images

It is obvious that remote sensing images are very different from natural scene images due to their large aspect ratio, arbitrary orientation, variation in appearance and scale, densely distributed objects, etc., leading to difficulties in directly transferring object detection in natural scene images to object detection in aerial images. Therefore, references [9,16,17] proposed different ways to improve the detection performance. Han et al. [9] used the Feature Alignment Module (FAM) to generate anchors, encode orientation information, and obtain orientation-sensitive features using the Oriented Detection Module (ODM). Yang et al. [17] $R^{3}Det$ refined features by re-encoding the positional information of the bounding box to the corresponding feature points through pixel-wise feature interpolation. Ming et al. [16] constructed critical features through the Polarization Refinement Module (PAM) and used the Rotation Anchor Refinement Module (R-ARM) to finally obtain a powerful semantic representation. Yang et al. [7] used a circular smooth label (CSL) to solve the problem of having discontinuous boundaries due to angular periodicity or corner ordering. Ding et al. [6] proposed the RoI-Transformer, which contains Rotated RoI Warping (RRoI Warping), which extracts rotation-invariant features, and the Rotated RoI Learner (RRoI Learner), which acquires objects’ orientation information, to address detection misalignment. The aforementioned [6] achieved great success in boosting the detection performance. However, in this method, the amount of computation increases, the detection speed decreases, and the GPU memory is burdened. Consequently, some methods [15,18] with reduced computation have been proposed with the condition of maintaining accuracy. Xie et al. [15] built the Oriented-RCNN including the Oriented RPN to generate high-quality anchors and the Oriented R-CNN head, which has a faster detection speed than the RoI-Transformer [6]. Han et al. [18] proposed a novel backbone, the Rotation-equivariant ResNet (ReResNet), which has a reduction in parameters of over 60% compared to ResNet [19].

2.2. Utilization of Existing Knowledge

Works that integrate knowledge into object detection can be divided into two kinds. One involves learning relationships between objects and scenes or relationships between categories during the training process; another makes use of existing knowledge to improve the detection performance.

2.2.1. Utilization of Knowledge Learned in Training

Chen et al. [20] directly added a global image feature to an ROI feature, which is a simple and effective method. Liu et al. [21] designed a Structure Inference Network (SIN) with two parallel branches, one for global information extraction and another for global information extraction. Though this method is more sophisticated and useful, the GRU cell in each branch requires more GPU memory, making the process time-consuming. Siris et al. [22] applied an Attention [23] mechanism to combine global information and local information and produced a great performance in terms of salient object detection. Li et al. [24] obtained local features and contextual features from the Region of Interest (RoI), then integrated them into a Local-Contextual joint feature for geospatial object detection. Zhang et al. [25] proposed CAD-net, which can learn correlations between objects and scenes from global features and object features. Refs. [26,27,28] designed a module to enhance scene or global information in order to capture contextual information.

2.2.2. Utilization of Existing Knowledge

In [11,29], the prior scene-class graph was adopted to infer the relationship between a scene and an object through the Bayesian criterion. The adjacency matrix learned from the visual feature was adopted in the relation-reasoning module in [30,31]. Shu et al. [32] introduced the Graph Attention Network (GAT) and Graph Convolutional Network (GCN) to learn hidden knowledge from the obtained co-occurrence matrix and scene–object matrix. Fang [14] proposed a probability-based knowledge graph and graph-based knowledge graph with a cost function containing a knowledge graph to carry out knowledge-aware detection. In [33], an explicit knowledge module and an implicit knowledge module, containing an explicit knowledge graph and an implicit knowledge graph, respectively, were introduced, to enhance the RoI features.

3. Establishment of the Knowledge Matrix

3.1. Knowledge Matrix Establishment

In this section, the procedures used to create knowledge matrices, the category conditional co-occurrence knowledge matrix and the water area knowledge matrix from the data set are illustrated.

3.1.1. Conditional Co-Occurrence Knowledge Matrix

We first count the probability $n\left(l\right)$ that every category appears in images using Equation (1)

$$n\left(l\right)=\frac{N_{img}\left(l\right)}{N_{allimg}}.$$

(1)

where $N_{img}\left(l\right)$ denotes the number of images in which category l occur, and $N_{allimg}$ denotes the number of images in the trainset. An analysis of the DOTA trainset is shown in

Table 1. The number of images in which the category occurs $N_{img}\left(l\right)$ and the probabilities of the class occurrence $n\left(l\right)$.

Object Categories	${\mathit{N}}_{\mathit{img}}\left(\mathit{l}\right)$	$\mathit{n}\left(\mathit{l}\right)$
Plane	197	0.1396
Baseball diamond	122	0.0864
Bridge	210	0.1488
Ground track field	177	0.1254
Small vehicle	486	0.3444
Large vehicle	380	0.2693
Ship	326	0.2310
Tennis court	302	0.2140
Basketball court	111	0.0786
Storage tank	161	0.1141
Soccer ball field	136	0.0963
Roundabout	170	0.1204
Harbor	339	0.2402
Swimming pool	144	0.1020
Helicopter	30	0.0212

, and an analysis of DIOR is shown in Appendix A.

Then, we determine the probability of conditional co-occurrence. In detail, we first count $N_{img}\left(l\right|l^{\prime })$, the number of images in which class l and class $l^{\prime }$ appear together. Then, we use $N_{img}\left(l\right)$ diving $N_{img}\left(l\right|l^{\prime })$.

Fang et al. [14] proposed Equation (2)

$$S_{l,l^{\prime }}=max(log\frac{n\left(l\right|l^{\prime })N_{allimg}}{n\left(l\right)n\left(l^{\prime }\right)},0)$$

(2)

where $n\left(l\right)$ denotes the probability of occurrence for category l, and $N_{allimg}$ denotes the number of all images to transform the frequencies into a knowledge matrix. We applied this formula to the DOTA and DIOR data sets.

However, this knowledge matrix has some disadvantages. On one hand, the numerical dimension, which is over 10, is too large to be suitable for optimizing predicted class scores. The oversized numerical dimension has an excessive influence on the predicted class scores, causing the knowledge matrix to become the decisive element, whereas our original intention was to make use of the knowledge matrix to improve predictions. On the other hand, when two categories do not co-occur, the corresponding position in the knowledge matrix will be set as 0, which is a rigid way to deal with the aforementioned situation. Thus, we propose a novel processing approach that introduces a zero factor to replace 0. Additionally, simply setting zero could exert a negative impact on the generalization performance. This is because the none-conditional-co-occurrence of two categories in the data set only denotes that the two categories have little correlation, but it does not mean that the two categories never co-occur in the real world.

In order to address aforementioned problems, we modify Equation (2) and propose Equation (3).

$$S_{l,l^{\prime }}=\left\{\begin{array}{ccc}\hfill \frac{log\frac{n(l,l^{\prime })}{n\left(l\right)n\left(l^{\prime }\right)}(n\left(l\right)+n\left(l^{\prime }\right))}{N_{img}\left(l\right)+N_{img}\left(l^{\prime }\right)}& \hfill & \hfill n(l,l^{\prime })\ne 0\\ \hfill \frac{log\frac{ϵ}{n\left(l\right)n\left(l^{\prime }\right)}(n\left(l\right)+n\left(l^{\prime }\right))}{N_{img}\left(l\right)+N_{img}\left(l^{\prime }\right)}& \hfill & \hfill n(l,l^{\prime })=0\end{array}\right.$$

(3)

The modifications are as follows:

• We abandon the $max\left(\right)$ function, which means that the range of $S_{l,l^{\prime }}$ extends to the negative axis, which is suitable for situations where category l and category $l^{\prime }$ do not co-occur or barely co-occur;
• We regard the log part $log\frac{n(l,l^{\prime })}{n\left(l\right)n\left(l^{\prime }\right)}$ as a conditional co-occurrence factor and abandon $N_{allimg}$ and add $\frac{(n\left(l\right)+n\left(l^{\prime }\right))}{N_{img}\left(l\right)+N_{img}\left(l^{\prime }\right)}$ as a scale factor into the equation in order to the scale knowledge factor into a proper numerical dimension and the make knowledge factor adaptive to the probability of category occurrence;
• We split our novel equation into two branches, where the upper one is for situations where l and $l^{\prime }$ appear together in the training set and the lower one computes the knowledge factor when l and $l^{\prime }$ do not co-occur;
• In the lower branch, we replace $n(l,l^{\prime })$ with the zero factor $ϵ$ in Equation (4), where $N_{object}$ denotes the number of instances belonging to category l, making the equation more elegant when $n(l,l^{\prime })$ equals 0.

  $$ϵ=\frac{1}{N_{object}\left(l\right)N_{object}\left(l^{\prime }\right)}$$

  (4)

3.1.2. Water Area Knowledge Matrix

In this section, we first count the number of images containing water areas and the number not containing water areas and compute the occurrence probability of water appearing and not appearing as 0.5102 and 0.4898 in DOTA, respectively. Then, we determine the probability of a class appearing with water area or not using Equations (5) and (6)

$$n\left(l\right|w)=\frac{N_{img}\left(l\right|w)}{N_{img}\left(l\right)}$$

(5)

$$n\left(l\right|\overline{w})=\frac{N_{img}\left(l\right|\overline{w})}{N_{img}\left(l\right)}$$

(6)

where $n\left(l\right|\overline{w})$ and $N_{img}\left(l\right|\overline{w})$ denote the probability of class l in a water area and the number of images in which class l does not occur in a water area. Those with a water area are denoted by $n\left(l\right|w)$ and $N_{img}\left(l\right|w)$. Probabilities of classes of DOTA appearing in a water area and not in a water area are demonstrated in

Table 2. Probabilities of classes appearing with water area and not with water area. Column $n\left(l\right|w)$ denotes the probability of category l appearing with water area; $n\left(l\right|\overline{w})$ is the probability of category l not appearing with water area.

Object Categories	$\mathit{n}\left(\mathit{l}\right|\mathit{w})$	$\mathit{n}\left(\mathit{l}\right|\overline{\mathit{w}})$
plane	0.1748	0.8252
baseball-diamond	0.5543	0.4457
bridge	0.9755	0.0245
ground-track-field	0.6462	0.3538
small-vehicle	0.2837	0.7163
large-vehicle	0.1584	0.8416
ship	0.9994	0.0006
tennis-court	0.2945	0.7055
basketball-court	0.2544	0.7456
storage-tank	0.9402	0.0598
soccer-ball-field	0.5215	0.4785
roundabout	0.6166	0.3834
harbor	1	0
swimming-pool	1	0
helicopter	0.0015	0.9985

. The probabilities of the classes of DIOR appearing in a water area and not in a water area are demonstrated in

Table A2. Probabilities of classes of DIOR appearing with water area and not appearing with water area. Column $n\left(l\right|w)$ denotes the probability of category l appearing with water area; $n\left(l\right|\overline{w})$ is the probability that category l appears with no water area.

Object Categories	$\mathit{n}\left(\mathit{l}\right|\mathit{w})$	$\mathit{n}\left(\mathit{l}\right|\overline{\mathit{w}})$
airplane	0.0412	0.9588
airport	0.4482	0.5518
baseball field	0.0714	0.9286
basketball court	0.0981	0.9019
bridge	0.9525	0.0475
chimney	0.1228	0.8772
dam	1	0
expressway service area	0.2719	0.7281
expressway toll station	0.1335	0.8665
golf field	0.9114	0.0886
ground track field	0.1293	0.8707
harbor	1	0
overpass	0.1179	0.8821
ship	0.9998	0.0002
stadium	0.1126	0.8874
storage tank	0.3126	0.6874
tennis court	0.1366	0.8634
train station	0.2398	0.7602
vehicle	0.2140	0.7860
windmill	0.0489	0.9511

.

Similar to the conditional co-occurrence knowledge matrix, we also partly modify Equation (2) and propose Equation (7)

$$S_{l,*}=\left\{\begin{array}{ccc}\hfill \frac{log\frac{n\left(l\right|*)}{n\left(l\right)n(*)}n\left(l\right)}{N_{img}\left(l\right)}& \hfill & \hfill n\left(l\right|*)\ne 0\\ \hfill \frac{log\frac{ϵ}{n\left(l\right)n(*)}n\left(l\right)}{N_{img}\left(l\right)}& \hfill & \hfill n\left(l\right|*)=0\end{array}\right.$$

(7)

where $ϵ$, taken as the zero factor, equals $\frac{1}{N_{object}\left(l\right)}$, and n(*) is the probability of a water area appearing, where * is w or $\overline{w}$ meaning that there is a water area and that there is no water area, respectively. Equation (7) was also applied to DOTA and DIOR.

4. Methods

In this paper, we propose a knowledge-inferencing module that uses a residual structure to optimize the predicted class scores, which helps the detector to perform better. Our method can be applied to any two-stage object detection framework. In this work, we chose the Oriented R-CNN [15] as the framework and baseline model. The overall structure of the framework filled with the knowledge-aware bounding box head is presented in Figure 3.

4.1. Feature Extraction

Resnet-50 [19] and FPN [34] are adopted into the Feature Extraction module to extract multi-scale features with which size-various objects can more easily be detected. Figure 4 shows the overall structure of the Feature Extraction module. In the bottom–up part, the input image is first input into a series of convolutional layers to generate low-semantic feature maps and high-semantic feature maps {$C_1$, $C_2$, $C_3$, $C_4$, $C_5$}. Moreover, the top–down part obtains more powerful features by fusing features. To be specific, $M_4$ is equal to the sum of the result of the upsampling of $M_5$ and the result of the $1\times 1$ convolution of $C_4$. In this way, $M_3$ and $M_2$ are generated. $P_2$, $P_3$, $P_4$, and $P_5$ are obtained by feeding $M_2$, $M_3$, $M_4$, and $M_5$ into a 3 × 3 convolutional layer, and $P_6$ is obtained by maxpooling $P_5$. Finally, this module outputs the fused multi-scale features {$P_2$, $P_3$, $P_4$, $P_5$, and $P_6$}.

4.2. Oriented RPN

The oriented RPN takes {$P_2$, $P_3$, $P_4$, $P_5$, and $P_6$} as the input and output region proposals with location information and an objectness score. Figure 5 shows the structure of the Oriented RPN.

We set three horizontal anchors with aspect ratios of {1:2, 1:1, and 2:1} for every location in the features of all scales. The anchors correspond to {$P_2$, $P_3$, $P_4$, $P_5$, and $P_6$} and have pixel areas of {${32}^2$, ${64}^2$, ${128}^2$, ${256}^2$, and ${512}^2$}, which are represented by a 4-dimensional vector $a=(a_x,a_y,a_w,a_h)$. $a_x$ and $a_y$ denote the horizontal and vertical locations of the anchor center; $a_w$ and $a_h$ correspond to the width and height of the anchor. The upper branch of Figure 5, i.e., the regression branch, outputs the offset $\delta =({\delta }_x,{\delta }_y,{\delta }_w,{\delta }_h,{\delta }_{\alpha },{\delta }_{\beta })$ of proposals related to anchors. We decode the offset using Equation (8) to obtain oriented proposals $(x,y,w,h,\mathsf{\Delta }\alpha ,\mathsf{\Delta }\beta )$, where $(x,y)$ denotes the location of the proposed center coordinate, w and h correspond to the width and height of the external rectangle box of the proposal, $\mathsf{\Delta }\alpha$ and $\mathsf{\Delta }\beta$ denote the offsets of the proposal box vertex oriented to the midpoints of the top and right sides of the corresponding external rectangle. The lower branch of Figure 5, i.e., the classification branch, outputs objectness scores.

$$\left\{\begin{array}{c}\mathsf{\Delta }\alpha ={\delta }_{\alpha }·w,\mathsf{\Delta }\beta ={\delta }_{\beta }·h\hfill \\ w=a_w·e^{{\delta }_w},h=a_h·e^{{\delta }_h}\hfill \\ x={\delta }_x·a_w+a_x,y={\delta }_y·a_h+a_y\hfill \end{array}\right.$$

(8)

To represent the oriented object in elegant manner, the midpoint offset representation is introduced, the schematic of which is illustrated in Figure 6. In detail, the black horizontal box, i.e., the external rectangle of the blue one, is obtained from the anchor, where $a_w$ and $a_h$ are the width and height of the anchor, and the blue oriented box is the predicted oriented proposal box. The black dots and the light green dots are the midpoint of the external rectangle edges and the vertices of the oriented box, respectively. The predicted oriented proposal box can be represented as O$=(x,y,w,h,\mathsf{\Delta }\alpha ,\mathsf{\Delta }\beta )$, which can be computed by Equation (8). Furthermore, the vertices of predicted oriented proposal are denoted by a set of coordinates $v_1,v_2,v_3,v_4$. Similarly, $\mathsf{\Delta }\beta$ is the distance between $v_2$ and the midpoint $(x,y-\frac{h}{2})$ of the top side, and because of the symmetry, the distance between $v_3$ and the midpoint $(x,y+\frac{h}{2})$ of the bottom side equals $-\mathsf{\Delta }\alpha$. It is noticing that $\mathsf{\Delta }\alpha$ is distance between $v_1$ and the midpoint $(x,y-\frac{h}{2})$ of top side, and because of the symmetry distance between $v_3$ and the midpoint $(x,y+\frac{h}{2})$ of bottom side equals to $-\mathsf{\Delta }\alpha$. Similarly, $\mathsf{\Delta }\beta$ is the distance between $v_2$ and the midpoint $(x+\frac{w}{2},y)$ of the right side, and the distance between $v_4$ and the midpoint $(x,y+\frac{h}{2})$ of the left side equals $-\mathsf{\Delta }\beta$. As a result, the vertices of the oriented proposal $\{v_1,v_2,v_3,v_4\}$ can be computed using Equation (9).

$$\left\{\begin{array}{c}{v}_1=(x,y-\frac{h}{2})+(\mathsf{\Delta }\alpha ,0)\hfill \\ v_2=(x+\frac{w}{2},y)+(0,\mathsf{\Delta }\beta )\hfill \\ v_3=(x,y+\frac{h}{2})+(-\mathsf{\Delta }\alpha ,0)\hfill \\ v_4=(x-\frac{w}{2},y)+(0,-\mathsf{\Delta }\alpha )\hfill \end{array}\right.$$

(9)

In the training process, we assign positive and negative samples using the following rules:

• An anchor that has an Intersection-over-Union(IoU) over 0.7 with any ground-truth box is regarded as a positive sample;
• An anchor that has an IoU over 0.3 with a ground-truth box and the IoU is the highest;
• An anchor that has an IoU lower than 0.3 is regarded as a negative sample;
• Anchors that do not belong to the above cases are discarded during the training process.

4.3. Oriented RCNN Head with the Knowledge Inference Module

In this section, we apply our proposed Knowledge Inference module to the Oriented RCNN head in order to reduce missed and wrong detections by improving the predicted class scores. Specifically, the proposed module is applied on two kinds of knowledge: conditional co-occurrence knowledge and water area knowledge. Thus, the proposed module has two similar inferencing modes. The details can be seen in the middle part of Figure 3.

The oriented RCNN head first takes {$P_2$, $P_3$, $P_4$, $P_5$, and $P_6$} and the oriented proposals from the oriented RPN as the input. In detail, feature vectors are obtained by rotating the RoI alignment to extract rotated RoI features according to the oriented proposals and transform them into fixed-length vectors. This is followed by two fully-connected layers. Then, we use two fully-connected sibling layers outputting classification scores and location predictions. For each image, we generate 512 predictions. Thus, the classification scores are denoted by the tensor of shape $[512,K+1]$, where $K+1$ denotes the number of classes plus the background, and the location predictions are denoted by the tensor of shape $[512,5]$. With the aim of optimizing the predicted classification scores with the knowledge matrix, we propose the Knowledge Inference Module, the structure of which is illustrated in Figure 7. To be specific, Figure 7a shows the structure of the Knowledge Inference module applied to class conditional co-occurrence knowledge. The class scores $[512,16]$ are first fed into the main-class-seeking-module to compute the major class in the image outputting the index of the main class. Then, the conditional co-occurrence matrix is sliced in terms of the main class index. The sliced matrix denotes the relationship between the main class and other classes, which is represented by the tensor of shape $[1,16]$. Therefore, the $\mathsf{\Delta }$ class score is a tensor with knowledge integrated. It is obtained by dot-multiplying class scores $[512,16]$ and transposing the sliced matrix $[16,1]$. However, our initial idea is to use knowledge to guide detection, so we apply a residual structure [19] into our proposed module, avoiding degradation brought about by using $\mathsf{\Delta }$ class scores only. Enhanced class scores are the result of $\mathsf{\Delta }$ class scores plus class scores. Additionally, the Knowledge Inference module on water area knowledge is shown in Figure 7b and is similar to that of the category conditional co-occurrence matrix. The difference is that the Main Class Seeking module and the main class index are replaced by water information showing whether there is a water area in the image.

The structure of the Main-Class-Seeking module is illustrated in Figure 8. We first slice the tensor class scores $[512,16]$ into 512 small tensors $[1,16]$ and encode them into values of 1 to 512. Then, the sliced class scores are fed into the argmax() function outputting the classes with the highest classification scores. The function max check() is used to count the number of each category in the 512 predictions. This is performed sequentially, where the class with largest number is the main class.

4.4. Loss Function

To train the Oriented RPN and Oriented RCNN head, we introduce Cross-Entropy Loss $L_{cls}$ for the classification task and Smooth L1 Loss $L_{reg}$ for the regression task. The whole loss function L is defined as follows:

$$L(p_i,t_i)=\frac{1}{N}\sum _i^N{L}_{cls}(p_i,p_i^{*})+\frac{1}{N}\sum _i^N{p}_i^{*}{L}_{reg}(t_i,t_i^{*})$$

(10)

$$L_{cls}(p_i,p_i^{*})=-[p_i^{*}log\left(p_i\right)+(1-p_i^{*})log(1-p_i)]$$

(11)

$$L_{reg}(t_i,t_i^{*})=\left\{\begin{array}{c}0.5{(t_i-t_i^{*})}^{2}if|t_i-t_i^{*}|<1\hfill \\ |t_i-t_i^{*}|-0.5otherwise\hfill \end{array}\right.$$

(12)

where N and i are, respectively, the number and index of the predicted anchors in a image, $p_i$ is the probability of predicted anchors, $p_i^{*}$ denotes the ground-truth label that belongs to $\{0,1\}$, i.e., negative and positive, and $t_i$ and $t_i^{*}$ are the predicted box and the ground-truth box.

5. Experiments

In this section, we introduce the two geospatial object data sets used in this work. Then, evaluation metrics and implementation details are illustrated.

5.1. Data Sets

To evaluate the proposed method, we conduct experiments on two public aerial image data sets, i.e., DOTA and DIOR.

DOTA is the most popular large-scale data set for geospatial object detection, containing 2806 images and 188,282 instances with arbitrary-oriented objects. Moreover, there are 15 classes in the data set: bridge, harbor, ship, plane, helicopter, small vehicle, large vehicle, baseball diamond, ground track field, tennis court, basketball court, soccer ball field, roundabout, swimming pool, and storage tank. The image width ranges from 800 to 4000 pixels. In this work, the training set was used for training, and the validation set was used for evaluation.

DIOR is another data set that is widely used for geospatial object detection. It contains 23,463 optimal remote sensing images and 192,472 object instances annotated by a horizontal bounding box. There are 20 object classes in total, namely, airplane, airport, baseball field, basketball court, bridge, chimney, dam, expressway service area, expressway toll station, harbor, golf course, ground track field, overpass, ship, stadium, storage tank, tennis court, train station, vehicle, and windmill. The image size of the DIOR data set is 800 × 800 pixels. We trained the network on the training set and evaluated the method on the validation set.

5.2. Evaluation Metrics

To evaluate the performance of the proposed method, we utilized four popular evaluation metrics, i.e., precision, recall, average precision, and mean average precision, the calculation formulas of which are shown as follows:

$$Precision=\frac{TP}{TP+FP}$$

(13)

$$Recall=\frac{TP}{TP+FN}$$

(14)

$TN$, $FN$, and $FP$ denote the number of true positives, the number of false negatives, and the number of false positives, respectively. $Precision$ measures the number of correctly identified positive detections of the total number of positive detections and $Recall$ measures the fraction of correctly identified positive detections of all positive samples.

$AP$ is computed by calculating the average value of precision from recall = 0 to recall = 1.

$$AP={\int }_0^{1}P\left(R\right)dR$$

(15)

$mAP$ is used to describe the multi-class object detection performance.

$$mAP=\frac{1}{N_{class}}\sum _{j=1}^{N_{class}}{\int }_0^1{P}_j\left(R_j\right)d{R}_j$$

(16)

where $N_{class}$ is the number of data set classes, j denotes the index of the class, and $P_j$ and $R_j$ are the precision rate and recall rate of the j-th class.

5.3. Implementation Details

The experiments were conducted on a single CPU, Intel Xeon CPU E5-2650 V4 at 2.20 GHz with a single GPU, NVIDIA Tesla P40 24 GB. The operating system was Ubuntu 18.04. The MMrotate [35] repository provided the training strategy. The size of the training image was 1024 × 1024, and the original DOTA images were split. All images in DIOR were 800 × 800 in size. Thus, there was no need to split the DIOR images. The objects in DIOR were annotated in the horizontal direction by the left-top vertex $(x_1,y_1)$ and right-bottom vertex $(x_1,y_1)$. Thus, we converted the annotations into a form suitable for the Oriented RCNN: $(x_1,y_1,x_2,y_1,x_2,y_2,x_1,y_2)$, corresponding to the vertices of oriented ground truth box in clockwise order. As for the hyperparameters, the optimizer was the stochastic gradient descent (SGD) with a learning rate of 0.005, a momentum of 0.9, and a weight decay of 0.0001. The batch size was 1, and the number of training epochs was 12.

6. Results

In this section, the results of the experiments on DOTA and DIOR are displayed.

6.1. DOTA Results

We applied the knowledge inference module to two kinds of knowledge: class co-occurrence knowledge and water area knowledge. The results show that our method achieved increases in mAP of 1.0% and 0.6%, respectively. Table 3 reports the comparison between the baseline model Oriented RCNN and our proposed method, in which the proposed method basically maintains the performance for both kinds of knowledge and improves the accuracy of several classes, for example 8.7% for the term helicopter with conditional co-occurrence knowledge, and 2.9% for the soccer ball field with water area knowledge.

Table 3. Comparison between the baseline model Oriented RCNN and our proposed method on DOTA data set.

METHOD	PL	BD	BR	GTF	SV	LV	SH	TC	BC	ST	SBF	RA	HA	SP	HC	mAP
$R^3$Det [17]	88.8	67.4	44.1	69.0	62.9	71.7	78.7	89.9	47.3	61.2	47.4	59.3	59.2	51.7	24.3	61.5
CSL [7]	88.1	72.2	39.8	63.8	64.3	71.9	78.5	89.6	52.4	61.0	50.5	66.0	56.6	50.1	27.5	62.2
$S^2$A-net [9]	89.1	72.0	45.6	64.8	65.0	74.8	79.5	90.1	60.2	67.3	49.3	62.2	60.6	53.4	37.6	64.8
FR-O [5]	89.3	76.0	49.3	74.7	68.1	75.5	87.1	90.7	64.2	62.3	57.0	65.8	66.6	59.6	38.2	68.3
RoI Trans [6]	89.9	76.5	48.1	73.1	68.7	78.2	88.7	90.8	73.6	62.7	62.0	63.4	73.7	57.2	47.9	70.3
Baseline [15]	89.8	75.7	50.2	77.3	69.4	84.8	89.3	90.8	69.2	62.6	63.1	65.0	75.3	57.5	45.3	71.0
Water area	89.6	75.6	50.3	76.4	68.4	84.3	89.4	90.7	72.9	62.6	66.0	67.2	75.6	56.5	48.8	71.6
Co-occurrence	89.6	76.0	50.7	77.0	68.3	84.4	89.3	90.7	73.6	62.4	63.8	66.8	75.1	57.6	54.0	72.0

The baseline is the Oriented RCNN [15], the Water area denotes the Knowledge Inference module on water area knowledge, and Co-occurrence is the Knowledge Inference module applied on conditional co-occurrence knowledge, where: PL: plane, BD: baseball diamond, BR: bridge, GFT: ground field track, SV: small vehicle, LV: large vehicle, SH: ship, TC: tennis court, BC: basketball court, ST: storage tank, SBF: soccer ball field, RA: roundabout, HA: harbor, SP: swimming pool, and HC: helicopter.

Table 3. Comparison between the baseline model Oriented RCNN and our proposed method on DOTA data set.

METHOD	PL	BD	BR	GTF	SV	LV	SH	TC	BC	ST	SBF	RA	HA	SP	HC	mAP
$R^3$Det [17]	88.8	67.4	44.1	69.0	62.9	71.7	78.7	89.9	47.3	61.2	47.4	59.3	59.2	51.7	24.3	61.5
CSL [7]	88.1	72.2	39.8	63.8	64.3	71.9	78.5	89.6	52.4	61.0	50.5	66.0	56.6	50.1	27.5	62.2
$S^2$A-net [9]	89.1	72.0	45.6	64.8	65.0	74.8	79.5	90.1	60.2	67.3	49.3	62.2	60.6	53.4	37.6	64.8
FR-O [5]	89.3	76.0	49.3	74.7	68.1	75.5	87.1	90.7	64.2	62.3	57.0	65.8	66.6	59.6	38.2	68.3
RoI Trans [6]	89.9	76.5	48.1	73.1	68.7	78.2	88.7	90.8	73.6	62.7	62.0	63.4	73.7	57.2	47.9	70.3
Baseline [15]	89.8	75.7	50.2	77.3	69.4	84.8	89.3	90.8	69.2	62.6	63.1	65.0	75.3	57.5	45.3	71.0
Water area	89.6	75.6	50.3	76.4	68.4	84.3	89.4	90.7	72.9	62.6	66.0	67.2	75.6	56.5	48.8	71.6
Co-occurrence	89.6	76.0	50.7	77.0	68.3	84.4	89.3	90.7	73.6	62.4	63.8	66.8	75.1	57.6	54.0	72.0

The baseline is the Oriented RCNN [15], the Water area denotes the Knowledge Inference module on water area knowledge, and Co-occurrence is the Knowledge Inference module applied on conditional co-occurrence knowledge, where: PL: plane, BD: baseball diamond, BR: bridge, GFT: ground field track, SV: small vehicle, LV: large vehicle, SH: ship, TC: tennis court, BC: basketball court, ST: storage tank, SBF: soccer ball field, RA: roundabout, HA: harbor, SP: swimming pool, and HC: helicopter.

Moreover, to a certain degree, some missed detections and wrong detections were improved. Figure 9 and Figure 10 display the reductions in missed detection and wrong detection using conditional co-occurrence knowledge and water area knowledge, respectively. In each subfigure, the left half is the result of the baseline model and the right half is the result of the proposed method. We use yellow circles to draw missed detections and red circles to draw false detections. Additionally, the first three subfigures shown in Figure 9 and Figure 10 display missed detections, and the second three subfigures in Figure 9 and Figure 10 display the false detections. The positive detections are shown in Figure 11.

Improvement occurs due to the utilization of knowledge. On the one hand, knowledge is used to optimize the predicted class scores. Thus, the performance of the classification is promoted; on the other hand, the class predictions optimized by knowledge can help the network iterate better during backpropagation. As a result, the more powerful features can be extracted by the network.

In terms of the inferencing speed, we compared the baseline and Knowledge Inference module for two kinds of knowledge, as shown in

Table 4. Comparison of the inferencing speed and accuracy between the baseline and proposed methods for two kinds of knowledge in the DOTA data set.

METHOD	FPS	mAP
Baseline	11.8	71.0
Water area	11.7	71.6
Conditional co-occurrence	11.5	72.0

. With the Knowledge Inference module, there was no significant drop in speed.

6.2. DIOR Results

The experiments conducted on the DIOR data set achieved a better performance. The ap values and mAP values are shown in

Table 5. Comparison between the baseline model Oriented RCNN and our proposed method on DIOR data set.

METHOD	APL	APT	BF	BC	BR	CM	DA	ESA	EST	GF
$R^3$Det [17]	89.6	6.40	89.5	71.2	14.4	81.7	8.90	26.5	48.1	31.8
CSL [7]	90.9	2.60	89.4	71.5	7.10	81.8	9.30	31.4	41.5	58.1
$S^2$A-Net [9]	90.8	14.0	89.9	72.7	17.6	81.7	9.50	32.9	50.1	50.9
RoI Trans [6]	90.8	12.1	90.8	79.8	22.9	81.8	8.20	51.3	54.1	60.7
FR-O [5]	90.9	13.1	90.7	79.9	22.0	81.8	10.4	49.8	53.1	58.5
Baseline	90.9	17.0	90.7	80.6	33.5	81.8	19.2	59.8	53.1	56.7
Water area	90.9	21.2	90.7	80.3	34.2	81.8	20.7	60.0	52.9	55.1
Co-occurrence	90.9	23.3	90.8	80.9	38.0	81.8	20.5	62.2	53.7	61.3
GTF	HA	OPS	SP	STD	ST	TC	TS	VEH	WD	mAP
65.6	8.20	33.4	69.2	51.9	72.9	81.1	21.4	54.2	44.7	48.5
63.2	17.5	26.6	69.3	53.8	72.8	81.6	18.4	47.2	46.3	49.0
70.9	16.3	43.6	80.1	52.5	75.7	81.7	23.9	59.0	45.6	53.0
77.2	30.6	40.5	89.9	88.2	79.5	81.8	20.6	67.9	55.1	59.2
75.4	35.5	41.6	89.1	85.4	79.3	81.8	32.7	66.4	55.5	59.6
76.9	26.1	54.5	89.9	88.8	79.6	81.8	30.2	68.3	55.1	61.7
76.7	26.8	54.6	89.8	88.3	79.5	81.8	35.5	68.7	55.6	62.2
81.3	32.1	56.6	90.0	88.3	79.7	90.1	44.3	69.2	56.5	64.6

The baseline is the Oriented RCNN, the Water area denotes the Knowledge Inference module’s knowledge on a water area, and conditional co-occurrence is the Knowledge Inference module for category conditional co-occurrence knowledge. APL: airplane, APT: airport, BF: baseball filed, BC: basketball court, BR: bridge, CM: chimney, DA: dam, ESA: expressway service area, ETS: expressway-toll-station, GF: golf field, GTF: ground track filed, HA: harbor, OPS: overpass, SP: ship, STM: stadium, ST: storage tank, TC: tennis court, TS: trainstation, VEH: vehicle, WD: windmill.

. As can be seen, both kinds of knowledge had beneficial impacts on the detection performance, and the mAP values increased by 0.5% and 3.9%, respectively. Similarly, missed detections and wrong detections were effectively eliminated.

For the DIOR data set, we also visualized the impacts of two kinds of knowledge on the baseline, as shown in Figure 12 and Figure 13. As for the visualization of the DOTA data set, the yellow circle and red circle denote missed detections and false detections, respectively. The positive detections are shown in Figure 14.

A comparison of the inferencing speed and accuracy between the baseline and proposed methods for two kinds of knowledge is shown in

Table 6. Comparison of the inferencing speed and accuracy between the baseline and proposed methods for two kinds of knowledge in the DIOR data set.

METHOD	FPS	mAP
Baseline	9.3	61.7
Water area	9.1	62.2
Conditional co-occurrence	9.1	64.6

. As can be seen, the Knowledge Inference module improved the detection accuracy with a negligible negative influence on the inferencing speed.

7. Conclusions

In this paper, in order to utilize knowledge to reduce false detections and missed detections caused by variation in the object appearance, varied object sizes, and complicated backgrounds, a series of steps were taken. We first established a knowledge matrix between the classes and a knowledge matrix between water areas and classes by analyzing the training set and proposed a novel equation, which can effectively avoid generalization degradation, to transform the relationship into form applicable for inferencing. Then, we proposed a method, the Knowledge Inference module, for integrating knowledge into object detection. The experiments were conducted on two public remote sensing data sets: DOTA and DIOR. The experimental results show that, compared to the baseline model, the proposed method achieved higher mAP values with fewer false detections and missed detections at an almost equal inferencing speed.