6.1 Lessons Learned
Lesson #1: Incorporating post embeddings from an external approach does not boost the performance of neural network models. Xu et al. [
54] demonstrated that appending the distributed post representation learned by Post2Vec to the manually crafted feature vector can increase the performance of traditional machine learning algorithms, such as SVM [
55] and Random Forest [
5], in a set of Stack Overflow-related tasks. However, these benefits are not observed for the SOTA techniques that are based on deep neural networks. This is potentially caused by the design of neural networks that automatically extract feature vectors and continuously optimize the representations. It indicates that deep neural networks may lose the effectiveness of external embeddings while optimizing the parameters of the feature extractor.
Lesson #2: Models with broader background knowledge derive better results than those with specific knowledge.
Intuitively, BERTOverflow is expected to produce the desired Stack Overflow post representation, as it is specifically designed for Stack Overflow data. A major difference between BERTOverflow and others is the vocabulary. As BERTOverflow is pre-trained from scratch with the Stack Overflow data, its vocabulary should be more suitable for modeling Stack Overflow posts than general domain models.
Surprisingly, our experiment results show that other transformer-based language models outperform BERTOverflow by a substantial margin across all three tasks. It gives an extremely poor performance in the tag recommendation task. By inspecting the prediction results of BERTOverflow in the tag prediction task, we notice that the top-5 predictions made by BERTOverflow are always the most frequent tags (‘python’, ‘java’, ‘c#’, ‘java-script’, and ‘android’) from the dataset. We observe that seBERT has performance similar to BERTOverflow in the tag recommendation task.
We hypothesize that the poor performance of BERTOverflow and seBERT is because these models lack a sufficient amount of pre-training to perform well. Since seBERT and BERTOverflow are trained from scratch, they require much more pre-training effort than continued pre-training with existing models. To prove this concept, we perform additional pre-training on BERTOverflow with the same dataset as SOBERT. The further pre-training was done with the same hyper-parameters as SOBERT, and it took 23 hours on four 16-GB NVIDIA V100 GPUs to complete. We denote this new model as BERTOverflow
\(_{\text{NEW}}\) .
6To independently evaluate the effect of domain-specific vocabulary, we train additional two transformer-based models from scratch. Both models contain 12 layers of encoder modules, but one with the vocabulary of BERTOverflow and the other with the vocabulary of RoBERTa. The pre-training process is the same as for BERTOverflow
\(_{\text{NEW}}\) . We refer to these two models as BERTOverflow
\(_{\text{vocab}}\) , and RoBERT
\(_{\text{vocab}}\) . From Table
8 (presented later), we observe the significant performance improvements of BERTOverflow
\(_{\text{NEW}}\) compared to BERTOverflow. We also observe that the performance of BERTOverflow
\(_{\text{vocab}}\) and RoBERTa
\(_{\text{vocab}}\) are quite similar to each other. Overall, our experiments have shown that domain-specific vocabulary has a negligible effect on all three tasks. The more important factor tends to be the amount of pre-training. Moreover, pre-training from scratch is commonly considered an expensive process. Initializing new representation models based on the checkpoint of a popular model reduces the risk, and the tag recommendation task is a good indicator to demonstrate the generalizability and the sufficiency of pre-training of transformer-based representation models.
For a more comprehensive insight, we analyze the lengths of tokenized text for different representation models on the datasets used in this article. Tables 5 through 7 show the statistics for the lengths of different tokenizers. Given that representation models like CodeBERT, GraphCodeBERT, Longformer, GPT2, and CodeGen share the same tokenizer as RoBERTa, we omit these representation models from this analysis. We notice that PLBart’s tokenizer can generate the shortest tokenizations across all datasets. Interestingly, even though BERTOverflow and seBERT are being developed with SE-specific vocabularies, the lengths of their tokenized text are almost the same as that of RoBERTas. For example, BERTOverflow’s average tokenization length in the tag recommendation task is 250, whereas RoBERTa’s is slightly higher at 254.7.
Lesson #3: Despite considering a longer input length, Longformer does not produce better representations for posts.
Conventional transformer-based models like CodeBERT and RoBERTa cannot handle long sequences due to the quadratic complexity of the self-attention mechanism [
48] and accept a maximum of 512 tokens as the input. From Tables 5 through 7, we can observe that the ratios of data that are longer than 512 tokens are approximately 9%, 0%, and 94% in tag recommendation, API recommendation, and relatedness prediction, respectively.
In Table
6, we can see that the dataset of API recommendation has a short length, where the longest tokenized text has a length of 53. This is because only the title of a post is considered in this task. As Longformer is implemented with a simplified attention mechanism (introduced in Section 2), which only gives its advantage in handling long text, this explains why CodeBERT and RoBERTa outperform Longformer in API recommendation.
From Tables
5 and
7, we can see that both datasets contain data samples that are longer than the 512 limit. Especially in related prediction (see Table
7), the average length of each KU is more than 1,800 tokens. The lengthy text is because each KU consists of question posts and a set of corresponding answer posts. Surprisingly, Longformer fails to perform better than the other model that belongs to the general domain (i.e., RoBERTa), as well as models from the SE domain, even though it takes much longer input in this task.
We further compare the performance of Longformer by varying the input size considering the first 512 and 1,024 tokens. The additional experimental results are shown in Table
8. These additional settings do not differ in performance. This indicates that diversifying the input size does not affect Longformer’s performance on post representation. A potential interpretation would be the important features for representing Stack Overflow posts lie in the first part of each post (e.g., Title serves as a succinct summary of the post). It is not worth trying Longformer unless one strictly needs the entire content of Stack Overflow posts.
Lesson #4: We advocate that future studies related to Stack Overflow consider SOBERT as the underlying baseline.
Our experiment results demonstrate that further pre-training based on in-domain data leads to better Stack Overflow post representation. By initializing SOBERT with the CodeBERT checkpoint and performing further pre-training on Stack Overflow data, we have noticed that SOBERT consistently outperforms the original CodeBERT and produces new SOTA performance for all three tasks.
In Table
9, we present three examples of the prediction results of CodeBERT and SOBERT for the tag recommendation task. We observe that CodeBERT is making wrong predictions like “.net” and “c#” when the question is about “haskell,” whereas SOBERT is capable of making the correct predictions. CodeBERT may lack knowledge of PLs like Haskell and Lua since it is pre-trained on artifacts from Python, Java, JavaScript, PHP, Ruby, and Go. Taking the Stack Overflow post with ID 13202867 as another example, the question is about Flexslider, a jQuery slider plugin. In the given example, SOBERT could successfully make connections to tags like ‘jQuery’ and ‘css’ while CodeBERT struggles to give meaningful predictions.
Overall, by continuing the pre-training process on Stack Overflow data, SOBERT outperforms CodeBERT in three popular Stack Overflow-related tasks. We advocate future studies to consider SOBERT as their underlying baseline. To facilitate the usage of the SOBERT proposed in this work, we plan to release it to HuggingFace
7 so that it can be used by simply calling the interface.