STAR: A First-Ever Dataset and A Large-Scale Benchmark for Scene Graph Generation in Large-Size Satellite Imagery

Visual Phrases [29] is an early dataset for visual relationship detection, which is a task similar to SGG, and it has 8 object categories and 13 relationship categories. RW-SGD [13] may be the first dataset for SGG, is constructed by gathering 5,000 images from YFCC100m [30] and Microsoft COCO datasets [31], containing 93,832 object instances and 112,707 relationship instances. VRD [32] is constructed for the task of visual relationship detection, which has 100 object categories from 5,000 images and contains 37,993 relationship instances. Visual Genome (VG) [33] is a large-scale scene graph dataset, which consists of 108,077 images with tens of thousands of object and relationship categories. Later, VG variants [34, 35] are gradually introduced by researchers to address its key shortcomings. Specifically, VG-150 [34] retains only the most common 150 object categories and 50 predicate categories to reflect a more realistic scenario. VrR-VG [35] argues that many predicates in VG-150 could be easily estimated by statistics, and then re-filters the original VG categories. However, there are still cases of redundancy and ambiguity of predicates in VrR-VG. Similar shortcomings are also found in the popular datasets GQA [36], Open Images V4 and V6 [37]. In addition, the co-occurrence of multiple relationships between subject-object pairs is common in the real world, but most previous datasets treat edge prediction as single-label categorization.

Compared to the natural imagery field, the development of SGG in SAI field has been relatively slow. SGG datasets in SAI [19, 20, 38, 39] with detailed annotation and considering the characteristics of SAI are rare. GRTRD [19] may be the first SGG dataset in SAI, which focuses on 12 kinds of objects and contains 19,904 object instances and 18,602 relationship instances. RSSGD [20] is a SAI-oriented SGG dataset constructed based on the remote sensing caption dataset RSICD [40], which has about 39 object categories and 16 relationship categories. Unfortunately, the image size of GRTRD dataset is 600×600 pixels, and the image size of RSSGD dataset is only 224×224 pixels. Since the large-size VHR SAI scenarios are cropped into small image blocks, the objects and relationships contained in each image are very sparse, and many object pairs containing high-value relationships are corrupted, which makes it difficult to support high-level cognitive understanding of SAI. In addition, the objects in the above dataset are annotated by HBB with too much background disturbance, which does not provide accurate information for recognizing relationships between regular man-made objects with distinct oriented boundaries in large-size VHR SAI. To meet the need for cognitive understanding of SAI, a large-scale SGG dataset named STAR for large-size VHR SAI is constructed, which contains multiple complex scenarios and meaningful relationships. In Table I, we give the statistics of popular OBD datasets and SGG datasets in SAI. Compared to small-scale datasets for the SGG task in SAI, such as RSSGD dataset and GRTRD dataset, our STAR outperforms them by at least an order of magnitude in both the number of objects and relationships on all/single images. For OBD datasets in SAI [23, 24, 22] lacking relationship annotations, our STAR dataset provides effective improvements in both object category diversity and scene complexity. The STAR dataset, a large-scale and comprehensive SGG dataset, supports both the OBD task and the SGG task based on the HBB-based and OBB-based detectors in large-size VHR SAI, serving as an invaluable resource for cognitive understanding of SAI.

Existing SGG methods for natural imagery have been dominated by the two-stage process consisting of OBD and relationship prediction between objects. Given the need for compatibility between OBD models and relationship prediction models within the SGG framework, the majority of existing SGG models, in alignment with the literature [41, 42], adopt Faster R-CNN as the standard for the OBD task. To generate high-quality scene graphs from images, a series of works have been explored in several directions, such as task-specific [34, 43, 44, 45, 46] and dataset-specific [41, 47, 48]. Notably, aggregating contextual information between objects has been shown to be effective for SGG. Xu et al. [34] first combined contextual information to refine the features of objects and relationships by using GRU to pass messages between graphs. Based on the discovery that there are some inherent relationship patterns in subgraphs, Motifs [43] constructed a new global context computing mechanism. A series of context-based studies [49, 50, 51] have consecutively explored the messaging mechanisms of objects and relationships. Most recently, HetSGG [46] attempted to capture the relationships between objects in more detail by proposing a heterogeneous graph and proved its effectiveness. However, the above methods of exhaustive enumeration based on all pairs tend to result in excessive computation, which makes it difficult to run under common computational resources (e.g., 24 GB of memory in a single GPU) for large-size VHR SAI with a large number of objects and more object pairs.

Limited by a few semantic understanding datasets in SAI, there are only a few approaches related to SGG in SAI. Shi and Zou [52] proposed a framework for SAI captioning by exploring whether machines can automatically generate human-like linguistic descriptions of SAI, and this work provides an initial exploration of high-level understanding of SAI. Chen et al. [19] proposed a geospatial relationship triplet representation dataset (GRTRD) based on the characteristics of high-resolution SAI and then adopted the ”object-relationship” message-passing mechanism to realize the prediction of geographic objects and geospatial relationships. Li et al. [20] proposed a multi-scale semantic fusion network (MSFN) for SGG in SAI based on their dataset RSSGD, which integrates global information through a graph convolution network to improve the accuracy of SGG. However, the above methods are mainly designed for small-size SAI with HBB and do not take into account the characteristics of large-size VHR SAI, such as the extreme aspect ratio of the objects, the great scale variation among the objects, and the random orientation of the objects. Furthermore, due to the limitation of the dataset, these methods do not have the ability to predict relationships between objects separated at long-range in large-size VHR SAI.

To meet the needs of practical applications, images in the STAR dataset are collected from Google Earth, with a spatial resolution ranging from 0.15m to 1m. Considering the value of cognitive understanding of large-size VHR SAI scenarios in practical applications closely related to human activities, such as transportation, energy, and life, we collected scenarios from more than 1,200 airports, ports, wind power stations, nuclear power stations, thermal power stations, construction sites, sports, service areas, toll stations, traffic bridges, and dams from all over the world. Unlike objects in natural imagery, objects in SAI are oriented in a variety of directions. Therefore, HBB-based annotation cannot provide accurate spatial information for oriented objects. To wrap objects more accurately and develop more suitable algorithms for oriented objects in SAI, all object instances in the STAR dataset are annotated with OBB. Samples of annotated instances in the STAR dataset can be seen in Fig. 2.

The construction of the SGG dataset for large-size VHR SAI mainly contains two tasks: object annotation and relationship annotation. Referring to the category system of mainstream OBD datasets in SAI, and combining with the actual requirements of the SGG task and some downstream tasks such as visual question answer and image caption, we carry out detailed interpretation and value analysis of the object categories appearing in the above scenarios, and finally choose 48 objects categories as shown in Fig. 3(a). Specifically, we select to annotate ”smoke” and ”vapor” because they can be used as a basis for judging the working status of power plants. In addition, the collection and annotation of objects like ”containment vessel”, ”lattice tower”, ”substation” and ”gravity dam” are expected to play an important role in SAI-based electricity research [18].

For the relationship description, unlike the “human-eye view” perspective of natural imagery, the “birds-eye view” perspective in SAI makes the semantic relationships between objects absolute, so we discard the orientation words such as “east”, “south”, “west”, and “north”. Moreover, we have learned a lesson from the ambiguous labeling of SGG datasets in natural imagery and removed relationship descriptions whose semantics are obviously ambiguous. After careful screening, the semantic relationships are defined into 8 major categories: “distance warning”, “spatial topology”, “functional description”, “circuit layout”, “movement status”, “emission status”, “construction status” and “parking status”, which contain 58 subcategories as in Fig. 3(b).

Fig. 4 gives the interaction mapping between objects and relationships. There are eight colors, representing the eight major categories of relationships consistent with Fig. 3(b). It can be seen that almost every relationship category is associated with multiple object categories, which reflects the complex interactions between objects in large-size VHR SAI.

STAR dataset has a standardized dataset construction pipeline: pre-annotation and rules refinement stage, large-scale detailed annotation stage, and quality-checking stage. In the pre-annotation stage, 6 experts in the SAI field are first invited to formulate annotation rules, and then the main authors pre-annotate the images with 11 types of scenarios, which are examined and evaluated by these experts, thereby forming a detailed object and relationship annotation document. During the large-scale annotation stage, the main authors provide comprehensive training to 9 professionals with rich interpretation experience in SAI, and each image is annotated by 2 professionals and checked by 1 professional at the same time. During the quality-checking stage, cross-sampling and revisions were performed by main authors.

Specifically, we use RoLabelImg4 to manually generate fine-grained oriented labeled boxes for the objects. Each object is represented by four clockwise-aligned angular coordinates $\left(x_{1},y_{1},x_{2},y_{2},x_{3},y_{3},x_{4},y_{4}\right)$ and category labels, and each triplet is represented by $<$subject, relationship, object$>$. For a more meaningful annotation, all pre-annotated pairs (both long-range and short-range) are selected based on the interaction graph in Fig. 4 during the relationship annotation process.

Compared with the existing OBD dataset in SAI, the STAR dataset has unique properties on task diversity, category diversity, and scale diversity. Sparse high-value pairs, high intra-class variation and inter-class similarity are also important characteristics of the STAR dataset.

Task diversity. STAR supports both the OBD and SGG tasks in large-size VHR SAI. After processing the annotated images, it can further satisfy the various task requirements like OBD based on HBB/OBB in large-size VHR SAI, SGG based on HBB/OBB in large-size VHR SAI.

Category diversity. STAR has various scene categories, various object categories, and various relationship categories. Specifically, it contains 11 categories of VHR SAI scenarios over the world, 48 categories of important objects, and 58 categories of high-value relationships, which will bring new opportunities for cognitive understanding of SAI.

Scale diversity. In the STAR dataset, there are many typical objects with extreme aspect ratios (e.g., runways, bridges, gravity dams), and there are large scale variations between objects of the same category (e.g., airplanes, ships) and different categories (e.g., “boats and docks”, “cars and car parkings”) that contain rich knowledge, which will bring great challenges to the OBD and SGG tasks.

Sparse high-value pairs. There are an enormous number of objects in a large-size SAI. If relationship prediction is performed for all object pairs, more than $N(N-1)$ triplets will appear for $N$ objects in the case of one-to-many relationships (e.g., in the STAR dataset where the number of objects in one image is up to 6,800, the possible triplets will be more than 40,000,000), which will cause memory overflow under common computational resources. In fact, not all object pairs are of interest to us, and only a few of them contain high-value relationships. Therefore, it will be a new challenge to select high-value pairs from the numerous pairs for the SGG task in large-size VHR SAI.

High intra-class variation and inter-class similarity. Intra-class variation of relationships is mainly due to the differential appearance and diverse subject-object pairs. The inter-class similarity of relationships arises from similar appearance representations, but manifests itself differently for different relationship categories. As shown in Fig. 5, for the object pairs of the same type, the relationship labels between them are non-unique (e.g., $<$ship, away from, dock$>$, $<$ship, approach, dock$>$), and for the object pairs of different types, their relationship labels may be the same (e.g., $<$ship, away from, dock$>$, $<$truck, away from, toll gate$>$).

The statistical results of object and relationship annotations from the STAR dataset are shown in Fig. 3(a) and Fig. 3(b), respectively. It can be seen that there is an obvious long-tail effect for both object and relationship categories in the STAR dataset, which is also consistent with reality. The highest-frequency objects are common small objects such as “cars”, “airplanes”, “tanks” and “boats”. The highest-frequency relationships are mainly related to the parking states, such as ”parked alongside with”, “parking in the different apron with”, “parallelly parked on” and “parking in the same apron with”.

In the sparse graph formulated based on the proposed PPG network, the semantic representations of both objects and relationships are influenced by neighboring objects and relationships. However, the interplay between object-object, object-relationship, and relationship-relationship interactions may exhibit substantial diversity in different contexts. Attention graph network [65], which leads the model to focus on reliable context information by assigning the corresponding weights to different regions of an image, has shown its effectiveness for SGG in natural imagery [50, 46]. Nonetheless, due to the complexity of large-size VHR SAI, conventional SGG models often exhibit deficiencies in relationship context inference capabilities when processing SAI. Moreover, although the integration of global context information assists in enhanced relationship recognition, more fine-grained discriminative features are primarily extracted from the triplets themselves. An overabundance of global context information may induce significant confusion. In our work, a novel progressive bi-context augmentation (PBA) component is proposed. As shown in Fig. 7, PBA mainly consists of entity-relationship progressive global context-aware messaging and global-local feature fusion. The progressive global context-aware messaging module for entity-relationship adaptively learns global information from neighboring entities and relationships using a progressive strategy. The global-local feature fusion module fuses the collected global and local features to obtain augmented features. After several iters of nodes and edges in the sparse graph, the dual optimization and updating of entity and relationship representations are accomplished.

Progressive Global Context-aware Messaging. Considering that it is easy to confuse when introducing too much unnecessary context information, we adopt a progressive training strategy to solve this problem. In this strategy, all entities and relationships first collect reliable information from their direct neighboring nodes and edges via the attention graph network, and then learn their indirectly related contexts with the help of neighboring nodes and edges, so that each entity and relationship can learn more reliable contextual information without introducing extra noise.

Sparse scene graphs constructed based on the proposed PPG network can provide multiple types of messaging indexes for entities and relationships. For $i+1$ iters, different types of messaging (i.e., entity-entity, relationship-subject, and relationship-object) are performed on entities to learn context-aware information of entities:

where $\sigma(\cdot)$ denotes the sigmoid activation function [66]. $\alpha^{ee},\alpha^{rs}$ and $\alpha^{ro}$ represent the importance of messaging from entity to entity, relationship to subject, and relationship to object, respectively. By learning to adjust $\alpha$ [65], each iteration leads to a change in attention and affects subsequent iters. $w^{ee},w^{rs}$ and $w^{ro}$ denote the weight matrix of the directed messaging from entity to entity, relationship to subject, and relationship to object, respectively. ${F}_{i}^{e}$ and ${F}_{i}^{r}$ are entity and relationship representation, respectively.

Similarly, different types of messaging (i.e., relationship-relationship, subject-relationship, object-relationship) on relationships are used to learn the global contextual information of relationships:

Global-Local Feature Fusion. Introducing reliable global information helps to better recognize relationships, but more fine-grained discriminative features mainly come from local. Therefore, we adopt a global-local feature fusion strategy to capture reliable global information and maintain the local more discriminative features. Specifically, the context augmented feature of entity ${F}_{i+1}^{e}$ can be obtained by fusing the context-aware feature of entity $\widehat{F}_{i+1}^{e}$ from global and the entity feature ${F}_{0}^{e}$ from local. Similarly, the context augmentation feature of relationship ${F}_{i+1}^{r}$ can be obtained. After $L$ iters, the predicted relationship representations $Rel$ with context-aware augmentation are updated. Note that the predicted relationship representations in Sec. 4.3.1 are optimized and updated simultaneously with the prototype representations in Sec. 4.3.2.

Prototype learning allows SGG models to focus on all categories equally in training, and is effective in alleviating the category imbalance problem of datasets [45]. However, previous linear classifier-based methods often lack the ability to model data at different semantic levels and suffer from a lag in representational capabilities (e.g., $<$ship, away from, dock$>$ and $<$ship, approach, dock$>$ correspond to different relationship labels, but it is clear that $<$ship, away from, dock$>$ and $<$ship, approach, dock$>$ are closer than $<$truck, away from, gas station$>$ in the visual representation). We propose a prototype-guided relationship learning method that utilizes category labels as prompts to help generate learnable relationship prototypes, and prototype representations updated dynamically by instance-level constraints and prototype-level constraints to help the SGG model enhance the discriminability between prototypes.

Prototype Construction. To construct more discriminative prototype representations, word embeddings (Glove) of different category labels $[label_{1},label_{2},...,label_{C}]$ are utilized to construct initial class-specific semantic prototypes $\mathbf{T}=[t_{1},t_{2},...,t_{C}]$. After obtaining the initial prototype representations, they are fed into a learnable prototype encoder $Enc_{p}$, during which the prototype representations $Pro=Enc_{p}(\mathbf{T})$ are allowed to be updated and matched with the predicted relationship representations $Rel$ in the same semantic space by $r=MAP(Rel)$ and $p=MAP(Rro)$, where $MAP(\cdot)$ is two-layer MLPs.

Instance-wise Matching Loss. To enable the model to learn more fine-grained category variation, the matching constraints between samples and prototypes are injected into the learning objective during instance-level matching to achieve positive samples closer to the prototypes and negative samples further away from the prototypes. Specifically, we obtain instance-level loss $\mathcal{L}_{IC}$ and $\mathcal{L}_{ID}$ based on cosine distance and euclidean distance, respectively:

where $q^{-}=\left\|r^{Neg}-\overline{p}_{i}\right\|_{2}^{2},q^{+}=\left\|r^{true}-\overline{p}_{i}\right\|_{2}^{2}$. $r^{true}$ denotes positive samples with label annotations, and $r^{Neg}$ is the negative samples sampled in order of distance with the exclusion of positive samples.

Prototypical Matching Loss. The purpose of prototype contrast learning is to make samples of the same category tightly clustered around the corresponding prototype to form tight clusters, with clusters corresponding to different prototypes pulling away from each other. To achieve this goal, we utilize the semantic structure captured by relationship prototypes to realize prototype contrast learning loss $\mathcal{L}_{PC}$ and loss $\mathcal{L}_{PD}$.

where $\mathcal{N}_{select}^{k}(D_{M})$ denotes the selection of the top $k$ smallest values in the distance matrix $D_{M}$.

The overall training loss $\mathcal{L}$ of our RPCM is as follows, without any bias on the weight of each term:

In the inference stage, the prototype type with the highest cosine similarity $s_{i}$ is used for relationship prediction:

where $\overline{r}$ and $\overline{p}$ are both regularization operations.

Table V gives the quantitative comparison of the proposed PPG network with other pair pruning methods on the STAR dataset when the sorted top 10,000 pairs are selected. It can be seen that the ABBS method based on statistics has only a weak improvement over the randomized method in all metrics, which shows that the semantic and spatial combination types of the object pairs in large-size VHR SAI are so complex that it is difficult to obtain combination patterns of objects by statistics. The score matrix, as depicted in the first row of Fig. 10, becomes sparse after the PPG network learning, effectively retaining the annotated pairs. This highlights the robust learning capacity of the proposed PPG network when dealing with pairs containing rich knowledge. The local visualization shows that the PPG network can retain the annotated pairs with high scores, as well as some pairs that are non-annotated but may contain high-value relationships, this is because the annotation of triplets in the real annotation is hardly exhaustive, so achieving the same sparsity as ground-truth is not necessary.

In Table VI, we compare the results on PredCls for different numbers of PBA iters L (L is set to 1, 2, 3, 4, 5). We find that the number of iters has little effect on our model. Notably, the HMR@K keeps improving with the increase in iters when iters are less than 4. This suggests that proper contextual information can effectively improve the model performance, but excessive introduction of contextual information may cause disturbance. In this paper, the number of iters for PBA is set to 4.

To explore the intrinsic impact of context introduction on the model, Fig. 11 gives examples of visualizations obtained from different PBA iters. It can be seen that lattice towers that are far apart (e.g., ”lattice tower11 and lattice tower28”, ”lattice tower15 and lattice tower26”, and ”lattice tower16 and lattice tower26”) have incorrect relationship predictions due to insufficient contextual information when L is set to 1. As L increases, the introductions of more contextual information cause the wrong relationships to be gradually corrected. When L is set to 4, the relationships between lattice towers at different distances are predicted with better results. It is worth noting that the previous incorrect relationship prediction recurred when L is 5, which may be caused by excessive contextual interference. Overall, the introductions of proper contexts help relationship inference and excessive context brings disruption, consistent with the quantitative metrics in Table VI.

We perform component ablation by gradually removing them from the RPCM on the PredCls task, as shown in Table VII. Not surprisingly, the experimental results show that all three components, object context augmentation, relationship context augmentation, and prototype matching, are important for the performance of SGG. Specifically, prototype matching enables it to capture the discriminative features of different relationship categories in the semantic space, which in turn effectively distinguishes different categories with strong similarities. The object augmentation and relationship augmentation modules assist the model in learning contextual reasoning constraints for relationship prediction, as shown by the visualization in Fig. 9.