BERT-PAGG: a Chinese relationship extraction model fusing PAGG and entity location information

Input layer

The input of the BERT-PAGG model mainly consists of sentences $S=\left\{c_1,c_2,\dots ,c_n\right\}$ and entity positions. Firstly, the input sequence $\left\{c_1,c_2,\dots ,c_n\right\}$ is encoded by the BERT pre-training model to obtain $\left\{e^{c_1},e^{c_2},\dots ,e^{cn}\right\}$ , where each word c_i in the sentence corresponds to the encoded e^c_i .

Vector representation layer

The model proposed in this article is also built on the BERT pre-training model, which can obtain richer information from the text and provide better text representation. In the relational extraction task, how to obtain entity information efficiently is the key to improving the performance of the relational extraction model. Wu & He (2019) proposed the R-BERT relational extraction model for the relational classification task, and proved that the tagging process for entities in the relational extraction model is effective. Special tags, such as “#” and “$”, are added to the head and tail entities in the input text, and such tags enable the BERT pre-training model to capture entity information better.

For the text in the dataset, special markers are added to the head and tail entities according to their positions, respectively, in the form shown in Table 1.

Wu & He (2019) conducted ablation experiments for the method of adding markers to the head and tail entities, and the experimental results showed that the effect of adding markers to the head and tail entities separately is better than that of adding markers to the head or tail entities, and also better than the effect of not adding markers. Therefore, in this article, we add “#” and “$” marks on both sides of the head and tail entities of the input text, which enables the Attention mechanism of the Transformer structure in the BERT pre-training model to better capture the information of the two entities and the relationship information between them, including the relationship information between the two entities, the information between the two entities and other words, and the semantic information between the two entities.

We input the above text sequence with “#” and “$” marks and entity positions into BERT to obtain the corresponding entity vector representation, where the vector representation of the first word of the head entity and the vector representation of the first word of the tail entity are spliced to represent the vector representation of the head and tail entities as shown in Eq. (1). (1)${\displaystyle Entity=\left[h_{head};h_{end}\right].}$

Segmented convolutional neural network

To highlight the importance of the first and last entities and their location information, we use a segmental convolutional neural network (PCNN) to extract sentence local features. Segmented convolutional neural network PCNN is a widely used model for relation extraction. Considering that the relative positions of words and entities also have an impact on the effect of relation extraction in the relation extraction task, the position information is also added to the encoding layer. Since the traditional single max-pooling can only get one value per convolutional kernel, which cannot capture the structural information features between two entities, PCNN introduces a segmented max-pooling mechanism, which cuts the text into three parts according to the head entity and the tail entity, extracts the local features separately and then splices the three vectors obtained by segmented max-pooling to obtain the final feature vector, This mechanism can extract the sentence feature information more effectively. The structure of the PCNN network is shown in Fig. 2.

The PCNN relationship extraction model differs from the traditional relationship extraction model by using only the word vector representation as the input of the model. Considering the importance of the positional relationship between words and entities, PCNN distinguishes entities from other words and also splices the positional features of words and head and tail entities with word embedding in the vector representation layer as the input of the final model, enabling the model to obtain rich positional information. The calculation is shown in Eqs. (2)–(4). (2)${\displaystyle B=BERT\left(S\right)}$ (3)${\displaystyle Position=Embedding\left(Index\right)}$ (4)${\displaystyle P=PCNN\left(B,Position\right)}$

where B is the embedding representation of the sentence S after BERT encoding, index is the position of the word relative to the head and tail entities, the corresponding position embedding representation is obtained by Embedding, and finally the word embedding B and position embedding Position are stitched together and input to PCNN to get the output P.

Through the studies and analysis of the distance between the word and the entity in the sentence, it was found that the smaller the distance between the word and the entity, the greater the influence on the final relationship extraction results, so the relative distance between the word and the head and tail entity was used to represent the location feature in PCNN. As shown in Table 2, the head entity in the sentence is “旅行车” and the tail entity is “崇山峻岭”, and the location features of other characters in the sentence are expressed as the relative distance from the current word to the head and tail entities, which are encoded by the location representation matrix $D\in R^{d^p\times \left|v^p\right|}$ , where d^p is is the dimension of the position vector, and v^p denotes a fixed-size position table.

The word vector representation and the position vector representation are concatenated to form the final vector representation as the input of the model, for the input text $S=\left\{c_1,c_2,\dots ,c_n\right\}$ , the word vector representation $\left\{e^{c_1},e^{c_2},\dots ,e^{cn}\right\}$ is obtained by BERT, and the position vector representations and are obtained by position features. The final stitching yields SP = [S; HEAD; TAIL] , where $SP\in R^d\left(d\in d^w+2\times d^p\right)$ is taken as the input to the model. (5)${\displaystyle PCN{N}_{\text{outputs}}=tanh\left(P_{1:n}\right).}$

Self-attention mechanism

The PCNN has the advantages of capturing local features, small computation and simple structure, and is the mainstream model in relation extraction. The contextual representation of sentences can be obtained by using PCNN, but long-term dependency features of sentences cannot be obtained by using PCNN alone, even by stacking multiple convolutional neural networks to achieve better performance. In this article, the self-attention mechanism is employed to capture global dependency features that complement the contextual representation obtained by PCNN.

Instead of using the PCNN in tandem with the self-attention mechanism in the previous way, a parallel approach is adopted in this article. For the sentence input, the contextual representation of the sentence is obtained by PCNN, the global dependent features are captured by the self-attention mechanism, and finally the global dependent features and the contextual representation are put into the gating mechanism to adaptively fuse the two features and get the final output. the PCNN has been introduced in the previous section, and the output calculation formula of the self-attention mechanism module is given here as Eqs. (6)–(8) shows. (6)${\displaystyle Att=W^2\sigma \left(W^{1}SP+b^1\right)+b^2}$ (7)${\displaystyle P^{\left(Att\right)}=softmax\left(Att\right)}$ (8)${\displaystyle A=\sum P^{\left(Att\right)}\odot SP}$

where W¹ and W² are the weight matrices, b¹ and b2 are the biases, and the final output A is obtained.

Gating mechanism

To introduce both local and global dependent features of a sentence into the relational extraction task, we need to fuse the two features. Traditional features fusion approaches (e.g., feature stitching (Bell et al., 2015), feature summation (Kong et al., 2016), etc.) do not consider whether the participating features are useful for the final task, so the fused features may contain a lot of unwanted information. Considering that the gating mechanism based on Gated Tanh Units (GTU) and Gated Linear Units (GLU) will measure the usefulness of each feature vector in the feature map during fusion and select the information that is useful for the final task for aggregation, therefore, this article uses the gating mechanism to adaptively fuse the entity features extracted by the PCNN network and the global dependent features extracted by the attention mechanism are fed to the classification layer as a representation of the whole text for classification, and the gating mechanism is calculated as shown in Eqs. (9)–10. (9)${\displaystyle g=sigmod\left(W^{g1}tanh\left(W^{g2}A+b^{g2}\right)+b^{g1}\right.}$ (10)${\displaystyle outputs=g\cdot PCN{N}_{outputs}}$

where W^g1 and W^g2 are the weight matrices, b^g1 and b^g2 are the biases, and finally the outputs is obtained through the gating mechanism.

Entity-specific graph convolutional neural networks

Graph convolutional neural network

We feed the adjacency matrix A constructed above into the graph convolutional network, which in this article is stacked in multiple layers, and then update the node representation in the next layer based on the hidden representation of the node’s neighborhood, with the graph convolutional formula shown in Eq. (12). (12)${\displaystyle h_i^l=relu\frac{A_i{h}_i^{l-1}{W}^l}{D+1}+B^l}$ (13)${\displaystyle D_{i,j}=\left\{\begin{array}{c}1,A_{i,j}\ne 0\hfill \\ 0,A_{i,j}=0\hfill \end{array}\right.}$

where relu is the nonlinear activation function, W^l is the linear transformation weight, B^l is the linear transformation bias term, $D={\sum }_{j=1}^n{D}_{i,j}$ is the degree of A_i , and $h_i^{l-1}$ is the hidden representation of the GCN transformed from the previous layer.

Classification layer

Formally, we first fuse the local features extracted by PCNN and the global features focused on by the self-attention mechanism using the gating mechanism, and subsequently splice the fused features with the dependent features extracted by GCN, which in turn yields the final representation that is fed into the classification layer to produce the probability distribution p, as shown in Eqs. (19)–(20). (19)${\displaystyle r=\left[R_{gate};R_{gcn}\right]}$ (20)${\displaystyle p=fc\left(relu\left(W_{p}r+b_p\right)\right)}$

where ’;’ denotes the splicing of vectors, fc is the fully connected function, W_p and b_p are trainable parameters.

Datasets and setup

This article evaluates the BERT-PAGG model on two public datasets: the SanWen dataset and the FinRE dataset, in which the SanWen dataset collects a corpus of 837 Chinese prose works, including 695 training sets, 58 validation sets and 84 test sets. The dataset includes 10 types of relations including “Unknown”. While the FinRE dataset collected 2,647 Sina financial news, and extracted 13,486 relationship instances as the training set, 3,727 relationship instances as the test set, and 1,489 relationship instances as the validation set, which contains 44 types of relationships.

In this article, a 768-dimensional word vector is generated using the BERT model with Adam as the optimizer and the learning rate set to 5e−5. To prevent overfitting, the dropout strategy is applied to the input and hidden layers in the training model and the dropout rate is set to 0.1. The hyperparameter settings of the BERT-PAGG model are shown in Table 3.

Baselines

To evaluate the performance of the BERT-PAGG model for Chinese relationship extraction task, this article compares it with the following classical models.

• Att-BLSTM: Zhou et al. (2016) proposed a BLSTM model with word-level attention mechanism.

• BLSTM: A bidirectional LSTM model for relation extraction proposed by Zhang & Wang (2015).

• PCNN: A piecewise convolutional neural network model with multi-instance learning proposed by Le et al. (2018).

• PCNN+Att: An improved PCNN model with selective attention mechanism proposed by Lee, gyu Seo & Choi (2019).

• Lattice LSTM: A model for Chinese named entity recognition proposed by Zhang & Yang (2018), which explicitly utilizes word and word order information.

• MG Lattice: A model for Chinese relation extraction proposed by Li et al. (2019), which incorporates word-level information into character sequences and introduces external knowledge.

• MGRSA: Zhong (2021) proposed a Chinese entity relationship extraction model based on a multi-level threshold recursion mechanism and self-attention mechanism.

Main results

As shown in Table 4, we report the performance of the BERT-PAGG model and the baseline model on two datasets, where the BERT-PAGG model is marked with “#” and “$” on both sides of the head and tail entities in the sample, respectively.

As can be seen from the above table, our proposed BERT-PAGG shows effectiveness and superiority in the relational extraction task compared to others, achieving a state-of-the-art Macro-F1 score. First, piecewise convolutional neural networks and self-attention mechanism can capture local features and global features well, while the addition of graph neural networks allows inter-word dependency features to be used effectively. Thus, we can observe that BERT-PAGG achieves the highest Macro-F1 value compared to the model using only sequence information.

Ablation study

To investigate the effect of different components, we performed ablation experiments, and the results are shown in Table 5, where rows 1-4 are removing the specified components separately, in row 5, we use the lightweight Bert model, i.e., Albert, to replace Bert to verify the effectiveness of BERT. From the Table 5, we can draw the following conclusions.

1. The attention mechanism and the PCNN module can improve the feature extraction, thus improving the performance of the Chinese relationship extraction model.

2. The introduction of gating mechanism can better integrate the features extracted by the attention mechanism and PCNN module, and the gating mechanism can filter out the key features more wisely. Removing the gating mechanism reduces the BERT-PAGG model by at least 3% in the Macro-F1 metric, which indicates that the representation of text by PCNN and self-attention can be better obtained by the gating mechanism.

3. Without using inter-word dependency information, i.e., removing the GCN module, the Macro-F1 of M₃ is lower than that of M₅ , this shows that the GCN module can effectively extract the dependency information between words and apply it to the relation extraction task. However, it is worth noting that the drop in Macro-F1 score is not significant after removing the GCN module, which may be due to the poor accuracy of the syntactic analyzer in the Chinese context, which in turn leads to errors in the construction of the adjacency matrix.

4. To verify the effectiveness of BERT for the Chinese relationship extraction task, M₄ uses Albert with fewer parameters to replace BERT. Compared with M₅, M₄has a significant decline in Macro-F1, which indicates that BERT pre-training model can obtain more information from text and provide better text representation.