Measurement of Embedding Choices on Cryptographic API Completion Tasks

Token-level embeddings, such as word2vec [31, 32, 33], FastText [8, 25], GloVe [39], assign one numeric vector to represent a token. In our work, we follow the skip-gram [31, 32] algorithm to train token-level embeddings for API methods and constants. Specifically, a three-layer linear neural network is used to automatically learn the embeddings of all tokens in an embedding sequence corpus. The token (API method or constant) to be embedded is the input, and the tokens before and after it within a sliding window are used as the labels to train the neural network. During the embedding process, the entire embedding corpus is scanned and all the tokens and their neighbors are used for training. After that, the latent vector at the hidden layer is kept as the embedding of the input token. In this way, a token’s embedding vector is determined by the statistics of its neighboring tokens in a large corpus.

Sequence-level embedding assigns a vector for every occurrence of a token. In other words, a token is represented with different vectors when it appears in different sequences. To generate this contextualized vector, not only the token itself but also other tokens in a given sequence are used. There is a neural network-based language model to take a sequence as input and output the embedding vectors of every token in this sequence. For example, the GPT family [41], BERT [17], RoBERTa [28], are sequence-level embeddings generated from Transformer neural networks. The sequence-level embedding ELMo [40] is generated from a BiLSTM neural network. This neural network is pretrained with carefully crafted tasks for producing the sequence-level embedding. Therefore, it is also referred to as a pretrained language model. The sequence-level embedding of a token is dynamically generated by the pretrained language model.

To apply the sequence-level embeddings for downstream tasks, a common way is to use the pretrained model that produces sequence-level embedding as the initial states. An extra application layer is added after the embedding layer, and the entire model is fine-tuned with extra data for a specific downstream task.

We evaluate different embeddings in cryptographic API completion. API completion refers to a task that suggests one or more next API methods given a preceding sequence of API elements (i.e., API methods and constants). We define two types of cryptographic API completion tasks, i.e., next API completion and next API sequence completion. The former aims to predict one API method in the next line while the latter produces a sequence of API methods to invoke sequentially.

We examine the impacts of using program analysis to guide the embedding. There could be unlimited program analysis strategies to extract different program sequences. Specifically, we compare three types of program sequences: (i) bytecode, (ii) program slices, and (iii) API dependence paths. The bytecode is from Android apps without program analysis. The program slices are obtained by conducting interprocedural backward slicing on bytecode. Moreover, the API dependence paths are extracted from API dependence graphs we construct on program slices with the dataflow dependence between the API calls. We select these three because they embody the increasing levels of program analysis guidance.

Bytecode sequences. We extract the API sequences directly from the Android bytecode. For each method implementation, we extract the API methods and constants used in it into one sequence. There is no ordering between sequences collected from different method implementations. Based on our observation, the order of the API methods and constants in these sequences is close to their order in the source code. We cover the bytecode option because it reflects the effect of embedding without program analysis guidance.

Program slices. We apply a program analysis strategy, interprocedural backward slicing, to obtain program slices. The slicing starts from the variables used with a cryptographic API invocation. By backwardly tracing the data flows reaching these variables, all the code statements influencing the API invocation are kept while irrelevant code statements are excluded. When reaching the entry point of the current method, we jump to its callers to continue the backward tracing until the tracked data facts are empty or there is no caller found. In this way, the influencing code context beyond a local method is also collected. When meeting a self-defined method (i.e., a method that is written by the developer and is not provided by Java libraries) call, we replace it with its implementation code if available. An example of program slices is shown in Figure 1(b). A major difference between program slices and bytecode is that the irrelevant predecessors are removed by program analysis. API dependence paths. With program analysis, the code semantic information, such as program dependencies, can be extracted. We perform the API dependence graph construction and extract the API dependence paths for embedding. The API dependence graphs are built through dataflow analysis. We add the data dependence edges between API calls on slices. An example of our API dependence graph is shown in Figure 1(c). It uses an API or a constant as a node. Two nodes having the data dependence (def-use) relationship are connected directly. The API dependence paths are covered in our measurement as a representative of the state-of-the-art code semantic-based approaches. [7, 22].

Experimental setup of program analysis preprocessing. We implement an interprocedural, context- and field-sensitive dataflow analysis to achieve our backward slicing and API dependence graph construction. The analysis is implemented with the Java program analysis framework Soot [50]. Soot takes the Android bytecode as input and transforms it into an intermediate representation (IR) Jimple. The program analysis (i.e., slicing or API dependence graph construction) is performed on Jimple IR. We use Soot 2.5.0, Java 8, and Android SDK 26.1.1.

We perform the token-level embedding training to produce vectors for tokens in an embedding vocabulary, as illustrated in Figure 2(b).

Cryptographic code identification. All the embeddings are produced from the cryptographic code corpus we extract from decompiled Android APKs. We refer to the code implemented with cryptographic API calls as cryptographic code. To identify cryptographic code from an Android App, we first search all the cryptographic API callsites within the codebase. All the method signatures within the Java package java.security and javax.crypto (see Table 2) are included in our search list. Then, we start from these cryptographic API callsites to find other standard API calls happening before a cryptographic API callsite as its context. However, there might be different accurate levels and scopes of the context according to preprocessing. In bytecode sequences, we can only extract all the previous API calls within the same method of a cryptographic callsites as its context. When program analysis technique is applied, we are able to generate more meaningful API call context based on its program dependency. In our experiments, a cryptographic API callsite and its program-wide dependency code is extracted as an inter-procedural (cross-method) program slice. The entire slice are regarded as cryptographic code and all the previous API calls within this slice will be gathered as the context of a cryptograhpic API call.

Embedding vocabulary. The embedding vocabulary is collected during the cryptographic code identification. The vocabulary initially includes the standard JAVA cryptographic APIs. Then, we scan the App and perform interprocedural backward slicing from the detected cryptographic API callsites as entry points. In this way, the vocabulary expands with all the encountered API calls and constants during backward program slicing. When an API call is encountered, we first check whether it is a self-defined method.² If it is, then the analysis jumps into the implementation of this method according to the interprocedural slicing algorithm. Otherwise, the API method will be collected as an element in our vocabulary. For the collected API methods, we further filter those that appear less than five times. For constants, we manually identified 104 reserved string constants used as the arguments of cryptographic APIs. Other constants that appear more than 100 times in the slices are also kept in the embedding vocabulary. Finally, we have a vocabulary of 4,543 tokens (3,739 APIs and 804 constants). The API methods include the standard APIs from Java and Android platforms, as well as some third-party APIs that cannot be inlined because of recursion or phantom methods (whose bodies are inaccessible during the analysis). Table 2 shows the library distribution of these API methods.

We train the skip-gram embedding model [32] to obtain the word2vec-like embedding. With different program analysis preprocessing, three types of token-level embeddings—byte2vec, slice2vec, and dep2vec—are produced.

Experimental setup for token-level embeddings. We follow the convention of the natural language embedding word2vec to set hyperparameters. The embedding vector length is 300. The sliding window size for neighbors is 5. We also applied subsampling and negative sampling to randomly select 100 false labels to update in each batch. Based on our preliminary experiments, we train embeddings with a mini-batch size of 1,024. The embedding terminates after 10 epochs, because we did not observe significant improvement by longer epochs and smaller batch size. Our embedding model is implemented using Tensorflow 1.15. Training runs on the Microsoft AzureML GPU clusters, which support distributed training with multiple workers. We use a cluster with 8 worker nodes. The VM size for each node is the (default) standard NC6.

We obtain sequence-level embeddings by applying the method of training the well-known natural language embedding BERT [17] on program sequences, as shown in Figure 2(c).

byteBERT vs. sliceBERT vs. depBERT. On bytecode, program slices, and API dependence paths, we obtained three different versions of BERT embeddings for API elements: byteBERT, sliceBERT, and depBERT. These BERT-like API embeddings are produced by pretrained Transformer neural networks. We apply the masked language modeling (MLM) task to pretrain them. MLM is a task that reconstructs language sequences with masked tokens. It predicts the missing tokens for a given sequence with random masks. The masked tokens in the input sequence are either replaced by a special token [MASK] or an arbitrary random token in the vocabulary or kept in original sequences. We set the probabilities of the three situations as 80%, 10%, and 10% and follow the convention in NLP. The masked tokens are randomly selected with a probability of 30%, and one sequence is limited to having two masked tokens at most. These are similar to the setting of the MLM for training BERT [17]. We discard the next sentence prediction (NSP) of BERT, as there is no corresponding concept of the “next sentence” between two code sequences. Three types of sequence-level embeddings are trained with identical hyperparameters. Same with the LSTM training with token-level embedding, the neural network is trained for 10 epochs with a batch size of 1,024. When training the Transformer model, the input tokens are represented as our token-level embedding. To apply these sequence-level embeddings in cryptographic API completion, the pretrained neural networks are fine-tuned by the given task-specific data.

Training parameters. Due to the resource constraint, we cannot thoroughly try every possible parameter combination (grid search). Instead, we select the optimal parameters with some preliminary experiments. We considered different numbers of epochs (up to 20), batch sizes (512, 1,024), and learning rates (0.1, 0.01, 0.001). Choosing different learning rates improves the accuracy by no more than 0.02. After training for 10 epochs, the accuracy increment is less than 0.001 for each extra epoch. Therefore, we chose the final set of parameters (10 epochs, 0.001 learning rate, and 1,024 batch size) to achieve a balance between computational resources and model performance.

For the pretraining of our byteBERT, sliceBERT, and depBERT models, we choose the ratio of masking strategies (80%, 10%, and 10%) following the original BERT paper. We choose the mask ratio of 30% because some of the sequences are short and we want each sample having one or two masked tokens. As those parameters are optimized by the authors of BERT, they are also the most commonly used settings in the NLP field.

Differential evolution (DE) is not a common practice in choosing hyperparameters for deep learning models, while it is more frequently used for tuning kernel parameters for SVM. Applying DE on top of LSTM models and evaluating its impact on model performance can form an interesting research topic by itself. We leave this extension as our future research direction.

We conduct experiments on Android apps collected from the Google Play store. We choose the Android platform because of its widespread use and popularity among users. We collect apps from various categories to ensure the dataset reflects a diverse usage of Java cryptographic API in practice. According to the way we split data for training and testing, we have a basic dataset and an advanced cross-app data setup. Table 3 gives an overview.

The impact of applying token-level embedding (RQ1) is measured by comparing it with one-hot encoding on bytecode, slices, and dependence paths, respectively. The accuracies of the API completion tasks are shown in Tables 5 and 6, and a comparison is shown in Figure 3. We visualize the accuracy differences brought by various design choices, namely, applying token-level embedding, program analysis preprocessing (i.e., program slicing and API dependence graph build), or increasing the model sizes in Figure 8 in the Appendix. Experimental setup for cryptographic API completion tasks. We train LSTM-based models for the task. For next API completion task, we train the LSTM-based sequence model to accept a sequence of API methods or constants ( \(t_1, t_2, \dots , t_{n-1}\) ) and output the next API \(t_n\) . For next API sequence completion task, we train the LSTM-based seq2seq (encoder-decoder) model to accept the first half API sequence ( \(t_1, t_2, \dots , t_{n}\) ) and predict the last half of the sequence ( \(t_{n+1},t_{n+2}, \dots , t_{2n}\) ).

We filter our code dataset using CryptoGuard [42], which is a static cryptography API misuse detection tool. We exclude insecure cryptographic API usage to prevent the embeddings and models from learning these vulnerable patterns [44]. It also helps eliminate the situations that when the models predict secure APIs and the ground truth itself (from the original data) is insecure, such predictions are counted as incorrect answers. This step contributes to a more accurate and meaningful evaluation of model performance. We limit the maximum number of LSTM steps to 10. We use a batch size of 1,024 and a learning rate of 0.001. The highest accuracy achieved within 10 epochs is recorded. These hyperparameters are selected because no obvious accuracy improvement is observed by longer epochs, smaller batch size, or learning rate. We use the stacked LSTM architecture with vanilla LSTM cells for the LSTM-based models.

Next, we evaluate the effectiveness of sequence-level embedding (RQ2) from the comparison with token-level embedding on bytecode, slices, and dependence paths, respectively. We fine-tune the sequence-level embedding (i.e., byteBERT, sliceBERT, or depBERT) with the task-specific training before applying them to the API completion task. Then, the models are compared with unpretrained Transformer networks with token-level embeddings. We use two Transformer neural networks with different sizes, namely, Transformer-base and Transformer-small. The Transformer-base model has 12 hidden layers with size 768 and 12 attention heads. The Transformer-small model has 4 hidden layers with size 512 and 4 attention heads. Results are shown in Tables 7. We also show a comparison in Figure 4 and visualize the accuracy differences in Figure 9 in the Appendix.

Cross-app learning is a practical scenario in which we expect a pretrained model can be applied to other projects unseen in the training phase. Therefore, we conduct experiments to verify whether our conclusions still hold for new apps that never appear in the training.

Tables 8 and 9 show the API completion experiments on our advanced dataset (see Section 3.4), which follows the cross-app learning scenario. App sets 3, 4, 5, and 6 include apps that generate task-specific data. For every app category, we randomly select 80% apps of this category to generate training data and 20% apps to generate testing data. In other words, our training and testing data is cross-app but within a category.

Tables 8 and 9 compare sequence-level embeddings (i.e., depBERT and byteBERT) with the corresponding token-level embeddings (i.e., dep2vec and byte2vec). DepBERT and byteBERT are Transformer neural networks pretrained on app set 2 (see Table 3) with Masked Language Model (MLM). We use the small Transformer neural network for all the experiments. Figure 10 in the Appendix shows the accuracy differences achieved by program analysis and sequence-level embedding on App sets 3, 4, 5, and 6, respectively.

We observe similar conclusions with the basic dataset about program analysis. The experiments on API dependence paths (Table 8) again show significant advantages compared with bytecode sequences (Table 9). Program analysis preprocessing makes significant accuracy differences (16.95%, on average) in all situations.

A minor difference we observe is that sequence-level embedding brings more obvious improvement than on the basic dataset. As shown in Table 8, the average improvement of applying the sequence-level embedding is 2.17%. This indicates that sequence-level embedding is more significant when we train our models in the cross-app scenario. We observe that the sequence-level embedding substantially improves the accuracy for small data sizes. It achieves an accuracy 5.10% higher than the Transformer with dep2vec.

From Table 9, we also observe the improvement of applying sequence-level embedding byteBERT on bytecode sequences. However, without program analysis, the improvements (0.86%, on average) are quite small.

Besides the design choices we covered, we further experiment on two state-of-the-art sequence-level embeddings, GraphCodeBERT [22] and CodeBERT [20]. GraphCodeBERT and CodeBERT are general-purpose code embedding models pretrained by Microsoft. They adopt the Transformer-based neural architecture and pretrain it on CodeSearchNet dataset, which includes 2.3 million functions of six programming languages paired with natural language description. The differences between them are their code preprocessing parts and sequence-level embedding tasks. CodeBERT treats code as a sequence of tokens and is pretrained MLM. GraphCodeBERT uses program analysis to extract dataflow information as input and is pretrained by two extra structure-aware tasks introduced by the authors.

Table 10 shows the next API completion experiments on our app sets 4, 5, and 6. We decompiled .apk files into source code for the neural network inputs, although there might be lost information due to obfuscation. However, the amount of information loss caused by obfuscation is equal to our three methods (i.e., bytecode sequences, slices, and dependence paths), CodeBERT, and GraphCodeBERT. Therefore, we think it still forms a fair comparison. For each cryptographic API call, we extract two types of source code context for it: the method-level context and the class-level context. The former extracts the previous code within the wrapper method where the target call locates while the latter collects the previous code lines found in the same class of the target call. We fine-tune the two models with our data for 10 epochs with batch size 16. We use this setting because no substantial improvement is observed by longer epochs or smaller batch sizes.

We have three observations from Table 10. First, the best accuracy is achieved by GraphCodeBERT with the method-level context. However, the accuracy is still at a low level, an average of 59.94%. Second, GraphCodeBERT substantially outperforms CodeBERT in identical data and context settings. When using method-level context, GraphCodeBERT has an accuracy of 20.07% higher accuracy than CodeBERT, on average. When using class-level context, GraphCodeBERT achieves 6.34% higher accuracy, on average. This confirms our findings 1 and 3 that program analysis contributes a substantial improvement to the embeddings. Another observation is that method-level context is much better than class-level context. With GraphCodeBERT, the method-level context outperforms the class-level context by 22.29% accuracy improvement, on average. With CodeBERT, the method-level context results in an 8.80% higher accuracy, on average. The reason might be that the class-level context includes much more irrelevant information and makes the prediction worse.

Furthermore, with the rapid development of large language models, their application in code completion and code repair has been discussed widely. Recent work [45] evaluates ChatGPT, a conversational language model, on bug fixing and code repairing in Python. The results show that ChatGPT is able to fix 19 out of 40 simple bugs, comparable to other state-of-the-art solutions. However, while the results look promising, the queries are simple code snippets that have a few lines. It remains unclear how well ChatGPT can parse complex code context in large programs. Its performance in identifying vulnerable code (beyond simple syntactic and logic bugs) and providing secure code suggestions by itself could form an interesting research topic. Other large language model-powered code completion tools, such as Copilot, have also been published in recent years. These models are trained with a huge amount of source code without any program analysis preprocessing. Due to limited resources, we are unable to train comparable models from scratch on program analysis processed data. We leave those comparisons as our further work.

We summarize our major findings from experiments.

We perform the analogy tests to intuitively show the quality of token-level embeddings. Besides the impact on downstream tasks, good embedding vectors should also reflect the semantics of a token and its relationship with other tokens. In natural language processing, the quality of embedding is usually evaluated through analogous pairs (e.g., \(men- women \approx king- queen\) ) [31, 32, 33]. Therefore, following the practice in the natural language field, we design a few analogy tests to help understand the quality of API embeddings based on different program analysis methods. In our work, we define analogous pairs as two pairs of APIs or constants, (a and \(a^{\prime }\) ) with (b and \(b^{\prime }\) ), having a high degree of relational similarity (i.e., analogous) in terms of some programming property. For Java cryptographic code, we identify four categories of analogous pairs as follows. We show examples in Table 11.

Direct Dependency. For two APIs where one always accepts the other’s output, they form a pair having a direct dependency. For example, after a KeyGenerator instance is created by KeyGenerator.getInstance(.), it always needs to be initialized through KeyGenerator.init(.). The analogous relation could also be found between KeyStore.getInstance(.) and KeyStore.load(.) where the latter loads the required information to the KeyStore instance created by the former. We view the two pairs as analogous pairs under this category.

Semantic Symmetry. For two classes KeyGenerator and KeyPairGenerator, the former generates secret keys for symmetric cryptography while the latter generates keys for asymmetric cryptography. There is a symmetry relationship between their APIs. For example, they both have the APIs getInstance(String) to create instances and APIs to generate the key.

Argument Symmetry. There is an analogous relation between API - constant pairs. For example, symmetric cipher ”AES” can be passed to javax.crypto.KeyGenerator: javax.crypto.KeyGenerator getInstance(java.lang.String) as an argument. For asymmetric ciphers, ”RSA” and API java.security.KeyPairGenerator: java.security.KeyPairGenerator getInstance(java.lang.String) have a similar relation.

Syntactic Variants. Some APIs share the same name but differ in their full signatures. These APIs are functionally equivalent but have different types of arguments or return values. We name them syntactic variants. For example, there are several APIs with the same name doFinal(.) of the Java class Cipher and Java class MAC.

Based on the analogous pairs, we define 14 tests. We calculate the vector of the embedded object \(b^{\prime }\) based on the other three vectors of a, \(a^{\prime }\) , and b. If the actual embedding vector of \(b^{\prime }\) appears in the top k nearest list of the calculated one (ideal value of \(b^{\prime }\) ), then we say this analogy achieves rank k. Examples of how to calculate rank k are shown in Figure 5. The results of the 14 tests for dep2vec, slice2vec, and byte2vec are listed in Table 12.

In this small-scale analogous pairs evaluation, dep2vec performs the best. dep2vec achieves the best rank 12 times of the 14 test cases. slice2vec does well in some cases but performs poorly in the syntactic variants category. This is likely because the syntactic variant APIs usually appear in different contexts in slices, making slice2vec fail to recognize their similarity. For other more complicated relationships such as semantic symmetry or argument symmetry, the APIs and constants belonging to a pair often appear far away from each other in the code, increasing the difficulty of the test.

To help interpret how program analysis and embedding vectors help API completion, we show several case studies.

Case Study 1. This case study is on the effectiveness of the API dependence graph construction. Figure 6(a) shows a slice-based test case that is mispredicted by both slice2vec and its one-hot baseline. For digest calculation, it is common for MessageDigest.update(.) to be followed by MessageDigest.digest(.), appearing 6,697 times in training. However, Figure 6(a) shows a reverse order, which is caused by the if-else branch shown in Figure 6(b). When MessageDigest.update(.) appears in an if branch, there is no guarantee which branch would appear first in slices. This reverse order is less frequent, appearing 1,720 times in training. Thanks to the API dependence graph construction, this confusion is eliminated, which predicts this case correctly. Case Study 2. This case study is on the ability to recognize new previously unseen test cases. The slices in Figures 7(a) and 7(b) slightly differ in the arguments of the first API. slice2vec makes the correct predictions in both cases, while its one-hot baseline fails in Figure 7(a). MessageDigest.getInstance(String) appears much more frequent than MessageDigest.getInstance(String,Provider) in our dataset. Specifically, the former API appears 207,321 times, out of which 61,047 times are followed by the expected next token MessageDigest.digest(.). In contrast, the latter API—where one-hot fails—only appears 178 times, none of which is followed by MessageDigest.digest(). In slice2vec, the cosine similarity between MessageDigest.getInstance(String,Provider) and MessageDigest.getInstance(String) is 0.68.⁷ This similarity, as the result of slice2vec embedding, substantially improves the model’s ability to make inferences and recognize similar-yet-unseen cases.

Our findings empirically demonstrate that, among various design choices, applying program analysis brings the most significant improvement in the cryptographic code completion task, up to 45.8%. While upgrading to larger models and increasing model sizes also boost the accuracy, the significance of improvements is not comparable to applying semantically meaningful preprocessing. We observe only a 2.45% improvement when upgrading the Transformer model size. While large language models substantially changed the natural language processing field, applying them to programming languages as is may not be ideal. As shown in Section 4.2.2, pretraining provides only trivial enhancement to the model performance. This result suggests that instead of consuming high computational power for a slight boost, one should consider how to incorporate the most effective information for the prediction tasks.

Another key contribution of our article is the comparison between different program analysis strategies. The evaluation reveals that the dependence path brings the best accuracy. A possible reason could be that the analysis goes beyond the method boundary and collects dependence paths across the entire program. In this way, all information related to the target API is preserved in the path and gets embedded into the input. Therefore, a key step in building a code completion model is to incorporate program analysis, and the dependence path is a recommended method.

After deploying a code completion tool, a practical use scenario is to predict the next tokens on uncompleted programs. There exist more challenges when applying program analysis to code under development. Incorporating methods such as partial program analysis (PPA) [15, 16] is an important next step.

Soundness. Our conclusions are based on a rigorous approach with carefully controlled experiments. For the three dimensions, program analysis, token-level embedding, and sequence-level embedding, we measure the impact of a specific design by comparing the API completion trained with or without it.

Limitations. We briefly discuss our limitations and threats to validity. First, we perform security sanitization to filter insecure code in our dataset. However, security sanitization relies on a static analyzer that may not be perfect. Second, we apply static analysis to extract program slices and dependence paths. However, static analysis tends to overestimate execution paths. Thus, the slices and dependence paths used for learning might not necessarily occur. Third, we do not try other embedding techniques such as ELMo [40]. We met an incompatibility issue when adapting the published ELMo code for our API completion task. The published code requires outdated libraries such as TensorFlow v1.2 and CUDA 8, while our learning environment only supports CUDA 9 or later. We will consider adding more embedding models in the future. Last, our work focuses on Java cryptographic APIs. The generalizability to other languages, such as Python, is out of the scope of this work.

An internal threat to validity is that we use identical training hyperparameters for all the API completion experiments. When applying different program analysis and embedding techniques, we train neural networks with identical training hyperparameters. We did not tune hyperparameters to find the best practices for every case. An external threat to validity comes from the dataset we use in the measurement. We only perform API completion experiments with Java cryptographic API benchmark, although the embedding method is for general purposes. We choose Java cryptographic APIs, because it is complicated and the code completion task is more challenging. Our future work will extend the benchmark with more diverse APIs to confirm our results.

With specific embeddings, we need to train a neural network model (e.g., LSTM or Transformer) to perform API completion. For those comparisons, we choose the number of epochs, learning rate, and batch size through some preliminary experiments. We train the LSTM model with slice2vec with different batch sizes, and learning rates, and check their accuracies within 20 epochs.

Based on our observation in Table 13, the accuracy differences between batch sizes 1,024 and 512 are slight. We use batch size 1,024 to reduce the training time. We compare the LSTM trained with learning rates of 0.001, 0.01, and 0.1. Their difference is also small. Therefore, we use batch size 1,024 and a learning rate of 0.001 for all of the training tasks for API completion. For epoch, we found that the accuracy improvement after epoch 10 is negligible.

We fine-tune the pretrained model GraphCodeBERT and CodeBERT on our dataset. To determine the batch size, we try different batch size options in fine-tuning GraphCodeBERT on our dataset 6. The accuracy results after 10 epochs are shown in Table 14.

Table 14 suggests that a smaller batch size could result in higher accuracy. When we decrease the batch size from 512 to 16, the prediction accuracy keeps increasing. However, it also shows that batch size 16 is good enough as smaller batch size 8 results in no improvement. Therefore, we apply batch size 16 to all the comparative experiments on GraphCodeBERT and CodeBERT.

We report the improvement brought by different design choices, including embedding strategies, program analysis preprocessing, and model size, on the API completion task prediction accuracy (Figures 8–10).

In this section, we provide details of our design choices for deep learning models following the DL4SE guidelines in Reference [53].

Step 1: Preprocessing and Exploring Software Data. We extract API sequences from 16,048 Android apps for our experiments. Our program analysis preprocessing approaches (i.e., bytecode, program slices, and dependence graph) yield 708k, 927k, and 566k sequences, respectively (Section 3.4). The data size is sufficient for large deep learning models to learn from. We use 80% of the data for training and 20% for testing.

Step 2: Perform Feature Engineering. We use three different strategies to extract API sequences from Android apps and train embedding vectors representing the API. The corresponding API embedding sequences are used as input to the model. The dataset is labeled, as we have the ground truth for our API prediction tasks. For the token prediction task, the ground truth is the last API token in the sequence. For the sequence prediction task, the ground truth is the API sequence following the input sequence. Examples of API sequences generated from the three preprocessing strategies are shown in Figure 1.

Step 3: Select a Deep Learning Architecture. Because of the sequential nature of the data, we use deep learning models LSTM and Transformer in our experiment. Both models have been used for code completion tasks in previous works. We train the models for 10 epochs because, after 10 epochs, the accuracy improvement from each additional epoch is less than 0.001. We provide details about the hyperparameters used for each experiment in their corresponding section (i.e., Section 4.1 for RQ1, 4.2 for RQ2, 4.3 for RQ3, and 4.4 for RQ4).

Step 4: Check for Learning Principles. Our data are composed of Java cryptographic APIs extracted from Android apps, covering 3,739 unique APIs from various libraries. This variety provides enough representation of cryptographic API usage. We report the efficiency of program analysis-aided embedding through comparison with the naive approach (i.e., bytecode).

Step 5: Check for Generalizability. Our experiments are conducted on API sequences extracted from 12 different categories, covering a wide range of Android apps in practice. This proves that our results are generalizable to apps for diverse purposes. To confirm our models are not overfitted to the apps used in training, we further conduct a cross-app evaluation (i.e., 20% of apps are for testing only). The model accuracies in the cross-app setting are comparable with our basic setting, verifying there is no overfitting. Last, we compare our embedding approaches with the state-of-the-art models, namely, CodeBERT and GraphCodeBERT, on the same dataset using the same metric to support our results.