Microservice Security Metrics for Secure Communication, Identity Management, and Observability

Figure 1 shows the research steps of our study. For the initial data collection and analysis of microservice security tactics, we first consulted existing microservice-specific recommendations by industry organizations such as those of NIST [34], OWASP [37], or the Cloud Security Alliance [8], which represent aggregations of existing industry best practices on a broad level. These were independently analyzed by a team of industrial security experts, including the last three authors of this study before they got involved in this article. They also related the standards to their experiences from their industry projects. We confirmed and augmented these recommendations with the results of a multi-vocal literature study [15] which the author team conducted within the context of the AssureMOSS EU project.¹ Among other things, in this prior work, we reviewed the scientific literature summarized in Section 10 for microservice-related security practices and tactics. In addition, we are currently conducting a gray literature study [16] (i.e., guidelines, public repositories, recorded interviews, podcasts, videos, practitioner articles, presentations, and blog posts) on microservice-related security practices and tactics. In it, we so far studied 30 practitioner sources in-depth. In this article, we will not report more details of these other literature works, as this article is mainly based on the practices recommended in the existing microservice-specific recommendations (with a focus on the NIST SP 800-204 special publication [34]). We mention this additional work here to explain that the practices, tactics, and corresponding ADDs, studied in this article, are derived based on substantial empirical data.

From the initially collected data, we aimed to address the security tactics with our approach that (1) are concerning the core software architecture in the sense that they can be modeled in software architecture decomposition view (i.e., in the component and connector view); and (2) are discernible from the source code of existing microservice systems. As stated in the research questions, our goal is only to detect conformance to ADD options, not interdependences between them. This limitation of scope was necessary as the guidelines and literature contain many practices that are only discernible at runtime (such as runtime warnings after a number of failed login attempts at a location), are not visible in the source code (such as organization measures or human review activities), or can only be modeled in other views such as deployment views. All those other measures are considered as out-of-scope of our article and will be covered in our future work.

We studied 10 open source microservices systems as case studies line-by-line and manually annotated each security feature in their source code. Each is a system published by practitioners with microservice background (all are published on GitHub, see Tables 1 and 2). Next, we used our existing static code analysis approach for architecture reconstruction of polyglot microservice systems [36] (summarized in Section 2.6) as a foundation for the extraction of models from the source code of the 10 studied open source system cases, and the detection of security tactics in these models. For this approach and for the modeling of security tactics reported in this article, we began by performing an iterative study of a variety of microservice-related knowledge sources, and we gradually refined a meta-model that contains all the required elements to help us reconstruct existing microservice-based systems.

To increase our data set and explore the design space, we then created 20 additional models of system variants adapted from a published example according to discussions in the relevant literature in order to explore the possible decision space. Apart from the specific variations described in Tables 1 and 2, all other system aspects remained the same as in the base models. This resulted in a total of 30 models summarized in Tables 1 and 2. We assume that our evaluation systems are, or reflect, real-world practical examples of microservice architectures. As many of them are open source systems with the purpose of demonstrating practices or technologies, they are at most of the medium size and modest complexity, though. To ensure they represent industry practices, we have carefully compared them to commercial projects and industry guidelines. Those comparisons served as a guide to creating our 20 variants. We can assert that the found practices are representing industry practices, but in commercial systems, other combinations or interdependencies than in the studied open-source systems have been observed, too. To represent those combinations well in our data set, we created the 20 system variants following the practices observed in commercial systems and the literature/guidelines (see Section 2.1).

In parallel to and independent from the case study and model development, from our set of tactics, we selected a subset which the industrial security experts co-authoring our article, based on the collected data, judged as widely used security tactics for microservices. In particular, we selected the tactics on secure communication, identity management, and observability. We confirmed that these tactics are present or important concerns in the microservice open source systems studied. Based on both, the tactics and the open source code studies, we then defined a catalog of Architectural Design Decisions (ADDs) which use the found tactics as Design Options.

We then hypothesized a set of metrics that have the goal to automatically decide on each single decision point in our ADDs. These metrics are formally defined in Section 6. We developed and tested the metrics based on numerous small examples of our tactics and formalized them using a combination of set theory and first-order logic.

For case study-based analysis, we performed a systematic assessment on support or violation of the collected security tactics. Here, the three industrial security experts in the author team independently derived a recommendation based on the results of our tactics study which provides informal guidance for security experts to manually judge systems such as those in our models. Next, the other authors applied this recommendation as an ordinal rating scheme to each model variant summarized in Tables 1 and 2 to create a ground truth for our study. Then, the three industrial security experts in the author team and two experts from another company reviewed the rating scheme and the ratings in the ground truth. Please note that the academic authors all have substantial experiences in industrial software development and software security, and all of the industrial authors have substantial experience in IT risk management and decision analysis in numerous industry projects (consulting for many different customers). In case of inconsistency of the votes, we performed a discussion among the involved experts to resolve the conflict. We would have applied a majority vote, if the discussion would not have yielded consistent votes; but in all cases consensus was reached after the discussion.

Finally, in a statistical analysis, we first inspected how well the independent variables correlate with the dependent variable using Spearman rank correlation and then assessed how well the hypothesized metrics can possibly predict the ground truth data by performing an ordinal regression analysis. Spearman rank correlation [32] is a widely used correlation analysis method suitable for continuous and discrete ordinal variables. We used R’s cor.test function for this analysis. Ordinal regression is a widely used method for modeling an ordinal response’s dependence on a set of independent predictors, which is applicable in a variety of domains. For the ordinal regression analysis we used the lrm function from the rms package in R [13].

For modeling microservice architectures we followed the method reported in our previous work [60]. We used our existing CodeableModels tool,² a Python implementation for precisely specifying meta-models, models, and model instances in code. Based on CodeableModels, we realized automated code generators to generate graphical visualizations of all meta-models and models in PlantUML, detectors for detecting all relevant aspects of the metrics in the models, and generators for automatically calculating metrics.

As explained above, this article uses an approach developed in our prior work for automatic extraction of architecture models from the code [36]. In this section, we briefly explain this static code analysis approach, to make this article self-contained. Numerous architecture reconstruction approaches have been proposed to automatically or semi-automatically produce architecture models from the source code [10, 30, 31]. Unfortunately, these approaches usually involve a substantial effort to either manually maintain the reconstructed architecture model or repeat the reconstruction after the system has evolved (see [21]), meaning that they are not well suited for supporting the continuous evolution of systems. In addition, automated approaches have low accuracy (see [14]), and much additional, manual effort is needed for correcting and augmenting their results. Finally, most reconstruction approaches focus on a very limited number of programming languages and technologies (see [10]), meaning they are hard to use with modern systems, such as microservice-based systems, which use typically polyglot programming, persistence, and technologies, often in their latest iterations.

Instead of aiming at developing a one-size-fits-all, generic reconstruction method as many other reconstruction approaches do, our approach leverages the fact that system experts usually know a lot about their projects and thus, a generic, fully automated reconstruction may not be necessary. In particular, our work is an extension of the approach taken by Haitzer et al. [21]. We designed a detector-based approach that is capable of providing support for polyglot, continuously evolving systems, and can possibly use reusable detectors. Here, detectors are software components that continuously parse relevant parts of the source code and create model abstractions from the code. Reusable detectors are detectors that can be reused across different model abstraction tasks and projects.

Our approach presupposes that a system expert has identified the high-level architecture and the architectural security features of the system — with which the expert should be familiar either way — and modeled it in an execution script that iterates the detectors over each system element. In [36] we were able to show that this involves a modest initial effort per project, and then no or a relatively small per-release effort (such as for removal and addition of services and links between releases) is required.

When considering the Secure Communication category and our scope of architecture decomposition models, we mainly found various tactics related to the use of encrypted protocols. We need to consider if a connector is a backend connector or a connector on the paths from clients or UIs to system services. Here, the backend refers to those connectors inside of the system that are not linked to clients or UIs. As the connectors on the paths from clients or UIs to system services can be seen as more vulnerable, we treat them as a special case in our ADD on Secure Communication. Both cases are treated in one ADD, as the ideal option in this category is that all communication is encrypted. The resulting ADD is:

When considering the Identity Management category and our scope of architecture decomposition models, we mainly found various authentication-related tactics. In this context, it is important to note that authentication is crucial for all parts of the microservice architecture, but especially for services reachable directly or indirectly from the clients. As it makes in some case sense to only authenticate at the entry points of a system, we model the two resulting cases, backend authentication and authentication on paths from clients/UIs to services, using two separate ADDs with similar tactics, but a different decision scope:

Observability in microservices means ensuring that access to the data required to identify problems and detect defects is provided. This is critical to “detect attacks and identify factors for degradation of services (which may impact availability)” [34]. Please note that service-level degradation or availability issues may be caused by an attack, e.g., a denial-of-service attack. Monitoring for security should be performed at both the system level (e.g., gateways) and service level to detect, report, and respond to inappropriate behavior [34]. Whereas the previous tactics provide “protective” security controls, observability measures provide “detective” controls in the sense that they are used “to ensure that the system is operating as designed and the data is secure” [8]. It is usually performed through infrastructure services for performing monitoring, logging, and/or tracing [37]. Thus, when considering component-level observability, we can distinguish between ordinary components of the system and components facing clients or UIs, such as API Gateways [48], Backends for Frontends [48], or frontend services. The latter can be summarized as Facade components.

The metrics for the \(SC\) decision are shown in Figure 3. The first metric Secure Distributed Connectors \(SCO: \mathbb {P}{CN_M} \rightarrow [0, 1]\) sets the secure distributed connectors in relation to all distributed connectors. Both numerator and denominator of the ratio rely on the \(distributed\_connectors\hspace{-4.0pt}: \mathbb {P}{CN_M} \rightarrow \mathbb {P}{CN_M}\) function. A connector is a distributed connector if it is not of type \(InMemoryConnector\) . The detector \(secure\_connectors\) is \(successful\) , if a connector is of type \(EncryptedCommuniction\) (or \(HTTPS\) ). The next metric Secure Client Connectors \(SCC\hspace{-4.0pt}: \mathbb {P}{CN_M} \rightarrow [0, 1]\) is similar, but it uses the \(client\_connectors\hspace{-4.0pt}: \mathbb {P}{CN_M} \rightarrow \mathbb {P}{CN_M}\) function as its basis. This function selects the connectors with a \(cn\_source\) of type \(Client\) , or a \(cn\_source\) of type \(UI\) but only if this \(UI\) does not connect to another \(UI\) as target (i.e., it selects also system clients in UIs such as AJAX calls). Secure UI Connectors \(SUC\hspace{-4.0pt}: \mathbb {P}{CN_M} \rightarrow [0, 1]\) , in turn, uses \(ui\_connectors\hspace{-4.0pt}: \mathbb {P}{CN_M} \rightarrow \mathbb {P}{CN_M}\) as its and selects only those connectors that have a \(cn\_source\) of type \(UI\) .

In addition to these atomic metrics, we defined two metrics aggregating features from the others. Both again have the same form of metric definition as the previous ones (i.e., using \(secure\_connectors\) with varying basis sets). Secure External Client/UI Connectors (SEC) \(SEC\hspace{-4.0pt}: \mathbb {P}{CN_M} \rightarrow [0, 1]\) calculates the external connectors to clients/UIs simply as the set union of the client and UI connectors, and uses this set as the basis for its definition. Finally, Secure Internal Distributed Connectors (SIC) is based on the internal distributed connectors. We call these in the following backend connectors: A connector is a backend connector if it is not connected to a component of type \(Client\) or \(UI\) . We are interested in the distributed backend connectors which the function \(distributed\_backend\_connectors\hspace{-4.0pt}: CN_M \rightarrow \mathbb {P}{CN_M}\) selects by calculating the complement of the distributed connector set and the union of the client and UI connector sets.

The metrics for the \(BE\_AE\) decision are shown in Figure 4. They are all addressing distributed backend connectors that require authentication. To define this set of connectors formally, we first define a function distributed_backend_connectors_requiring_authentication: \(\mathbb {P}{CN_M} \rightarrow \mathbb {P}{CN_M}\) for constructing this connector set. The basis for this definition is the function distributed_backend_connectors defined in Section 6.1. By placing the type \(AuthenticationNotRequired\) onto a connector, an architect can indicate that a connector does not require to be authenticated (e.g., consider a connector to a Public API); we use the function \(connectors\_that\_require\_authentication\hspace{-4.0pt}: \mathbb {P}{CN_M} \rightarrow \mathbb {P}{CN_M}\) to select the connectors from a model \(M\) ’s connectors which are not of type \(AuthenticationNotRequired\) .

Based on the \(distributed\_backend\_connectors\_requiring\_authentication\) function, we define the metrics in the remainder of Figure 4. We first define the Authenticated Backend Connectors \(AEI\hspace{-4.0pt}: \mathbb {P}{CN_M} \rightarrow [0,1]\) metric which returns the number of authenticated distributed backend connectors requiring authentication in relation to the total number of distributed backend connectors requiring authentication. This way, we can measure the total level of authentication in a system without considering the specific authentication method used. Here, \(authenticated\_connectors\hspace{-4.0pt}: \mathbb {P}{CN_M} \rightarrow DR\) is a detector that detects the authenticated distributed backend connectors.

Next, we define the Securely Authenticated Backend Connectors \(AEI\_S: \mathbb {P}{CN_M} \rightarrow [0,1]\) which selects only the securely authenticated distributed backend connectors requiring authentication in relation to the total number of distributed backend connectors requiring authentication. Here, “securely” is measured through the detector securely_authenticated_connectors: \(\mathbb {P}{CN_M} \rightarrow DR\) which detects the authenticated connectors which are either of type ProtocolBasedSecureAuthentication or SecureAuthenticationToken. The Backend Connectors Authenticated with API Keys \(AEI\_K: \mathbb {P}{CN_M} \rightarrow [0,1]\) metric selects only the distributed backend connectors authenticated with API Keys, in relation to the total number of distributed backend connectors. This is done using the \(connectors\_authenticated\_with\_api\_keys: \mathbb {P}{CN_M} \rightarrow DR\) detector, which detects the distributed backend connectors of type \(AuthenticatedWithAPIKeys\) . The Backend Connectors Authenticated with Plaintext Credentials \(AEI\_P: \mathbb {P}{CN_M} \rightarrow [0,1]\) metric selects only the distributed backend connectors authenticated with plaintext credentials, in relation to the total number of distributed backend connectors. This is done using the \(connectors\_authenticated\_with\_plaintext\_credentials: CN_M \rightarrow DR\) detector, which detects the distributed backend connectors of type \(AuthenticatedWithPlaintextCredentials\) .

Authenticated Backend Connectors Over a Secure Connection \(AEI\_C: \mathbb {P}{CN_M} \rightarrow [0,1]\) selects only the authenticated distributed backend connectors which are sent over a secure connection, in relation to the total number of distributed backend connectors. It uses the \(secure\_connectors: \mathbb {P}{CN_M} \rightarrow DR\) detector defined above, and uses the successful results of this detector as input for the \(authenticated\_connectors: \mathbb {P}{CN_M} \rightarrow DR\) detector, defined at the beginning of this section. Finally, the Authenticated Backend Connectors Using a Secure Method or Transferred Over a Secure Connection \(AEI\_A: \mathbb {P}{CN_M} \rightarrow [0,1]\) metric selects only the distributed backend connectors which either use a secure authentication method, as defined above via the securely_authenticated_connectors: \(\mathbb {P}{CN_M} \rightarrow DR\) detector or are authenticated and use a secure connection.

The metrics for the \(CP\_AE\) decision are all addressing detectors on the path from clients or UIs to system services. They are defined in Figure 5. In them, we focus on the connectors between clients/UIs and system services that require authentication. This is calculated by the function \(client\_service\_path\_connectors\_requiring\_authentication: \mathbb {P}{CP\_M} \rightarrow \mathbb {P}{CN\_M}\) , which has the following ingredients:

Based on the \(client\_service\_path\_connectors\_requiring\_authentication\) function all metrics for measuring authentication on the paths from clients or UIs to system services are defined in much the same way as the metrics on backend authentication, defined in Section 6.2. The differences are that the components \(CP_M\) are used as inputs (i.e., they are all of the form \(AEC: \mathbb {P}{CP\_M} \rightarrow [0,1]\) ), the \(client\_service\_path\_connectors\_requiring\_authentication\) function is used instead of the \(distributed\_backend\_connectors\_requiring\_authentication\) function, and the AEC_A metric considers API Keys as well, as API Keys coming from clients are deemed an acceptable practice (see explanation in Table 3). To avoid repetition, we do not explain each metric in Figure 5 in detail again.

The metrics for the \(OBS\) decision are shown in Figure 6. The first metric Observed System Services \(OSS: \mathbb {P}{CP_M} \rightarrow [0, 1]\) sets the observed system services in relation to all system services. Both numerator and denominator of the ratio rely on the \(system\_services: \mathbb {P}{CP_M} \rightarrow \mathbb {P}{CP_M}\) function. A component is a system service, if it is not of the type \(ExternalComponent\) (i.e., no external services are system services), \(MiddlewareService\) (i.e., no middleware infrastructure services such as a Discovery Service), or \(Facade\) (i.e., no frontend services or gateways with the sole purpose of shielding the system from clients).

The detector \(observed\_by\_dedicated\_component\) is \(successful\) for a component \(c\) , if a dedicated observer component is connected to \(c\) , or formally: \(c \in\) \(\lbrace t \in CP_M :\) \((\exists \, o \in CP_M, l \in cp\_conn(t)\) : \(o \in\) observer(CP_M) \(\vee (cn\_source(l) = t \wedge cn\_target(l) = o) \vee\) \((cn\_source(l) = o \wedge cn\_target(l) = t))\rbrace\) . In this formula, the function \(observer: \mathbb {P}{CP_M} \rightarrow \mathbb {P}{CP_M}\) determines the dedicated observer components in a model, i.e., components of the types \(Logging\) , \(Monitoring\) , or \(Tracing\) .

The next metric Observed Facade Components \(OFA: \mathbb {P}{CP_M} \rightarrow [0, 1]\) is similar, but it uses the \(facades: \mathbb {P}{CP_M} \rightarrow \mathbb {P}{CP_M}\) function as its basis. A component is a facade, if it is of the type \(Facade\) , which is used directly for simple frontend services, or one of its subclasses such as \(APIGateway\) or \(Backends for Frontends Gateway\) .

Finally, Observed System Services and Facades \(OSF: \mathbb {P}{CP_M} \rightarrow [0, 1]\) is a simple integration of the two prior metrics by just using the union of the component sets returned by \(system\_services\) and \(facades\) as its basis.

To compare the independent variables in their association to their dependent variables, we first computed the Spearman rank correlations [32] in Table 6. In the context of this analysis, we first inspected the plots of each independent vs. its dependent variable for inspecting their monotonic relationship. As can be seen in Table 6, most of the independent variables alone have already a very strong correlation ( \(\rho\) > 0.8) or strong correlation ( \(\rho\) > 0.6) with the dependent variable, with very low p-values (<0.05), signaling high statistical significance. This indicates that those independent variables are well chosen, in the sense that they are highly associated with the dependent variable.

There are some notable discussion points in the correlation analysis. Firstly, the variables on API Keys-based authentication (AEI_K and AEC_K) and Plaintext-based Authentication (AEI_P and AEC_P) are neither highly correlated nor are the results statistically significant. Please note that this is not surprising: All other hypothesized variables in their ADDs have a plausible relation to the dependent variable that implies a high association with the dependent variable; this is not the case for these variables only measuring the fraction of one specific authentication method (API Keys or plaintext). But as the disambiguation of API Key or plaintext based methods is plausibly important to detect all ADD options fully and those aspects are not covered well in any of the other variables, we decided to include these variables nonetheless in our regression analysis, in the hope that they could introduce aspects in the models not covered well by the other variables. Indeed, in the regression analysis below, all the best models found for the two respective ADDs (BE_AE, CP_AE) use at least one of these variables. However, we also decided to never use them alone in regression models, but only with at least one other variable with a strong correlation.

Secondly, as many of our variables have a strong or very strong correlation with high significance, there was no clear indication of which variables to use in the regression analysis below. We thus decided to try all reasonable combinations of our dependent variables to find the models that best predict the dependent variable.

The objective of the ordinal logistic regression analysis is to predict the likelihood of the dependent outcome variable for each of the decisions by using the relevant metrics for each decision. Ordinal logistic regression is the recommended regression model in the case of ordinal response variables [13]. To enable comparison of the resulting regression models, we calculated many possible models for each ADD, and report the three best-performing models for each (see Columns 3–5 of Table 7).

Each resulting regression model consists of a baseline intercept and the independent variables multiplied by coefficients. There are different intercepts for each of the value transitions of the dependent variable ( \(\ge [-]\) : Insufficient support in a few places; \(\ge [\sim ]\) : Insufficient support in many places, \(\ge [+]\) : Acceptable support, \(\ge\) [ \(++\) ]: Very good support), while the coefficients reflect the impact of each independent variable on the outcome. For example, a positive coefficient, such as +5, indicates a corresponding five-fold increase in the dependent variable for each unit of increase in the independent variable; conversely, a coefficient of \(-\) 30 would indicate a thirty-fold decrease.

The statistical significance of each regression model is assessed by the p-value; the smaller the p-value, the stronger the model is. A p-value smaller than 0.05 is generally considered statistically significant. In Table 7, we report the p-values for the resulting models, which in all cases are very low, indicating that the sets of metrics we have defined are able to predict the ground truth assessment with high statistical significance.

Frequently, the C-index (which is also called concordance index and is equivalent to the area under the Receiver Operating Characteristic (ROC) curve) is reported in the statistical literature as a measure of the predictive power of ordinal regression models [1]. A C-index of 0.5 indicates random splitting, whereas a C-index of 1 indicates perfect prediction. Harrel [13] suggests bootstrapping as a method for obtaining nearly unbiased estimates of a model’s future performance based on re-sampling. It is a powerful model validation technique that supports estimating predictive accuracy without holding back data from the model development process [13]. This feature is important for our study, as our data set contains only 30 models. We used lrm’s validate function to perform bootstrapping and calculated the bias-corrected C-index in addition to the original C-index. The C-indexes, reported in Table 7, are all larger than 0.9, which indicates that the models are good enough for predicting the outcomes of individuals.

The Brier score is another commonly reported accuracy measure [17]. Even though somewhat less interpretable than the C-index, the Brier score overcomes some issues of the C-index, as it reflects both calibration and discrimination of a model [26]. In this sense, it can be better suited to assess overall model performance and compare models [52]. The value of the Brier score is always between 0.0 and 1.0. A score of 0 represents perfect accuracy, and a score of 1 represents perfect inaccuracy. Again, we do not only present the original Brier scores but also the bias-corrected scores after bootstrapping. Except for one model of CP_AE, all bias-corrected scores are below 0.1 (and the one model above 0.1 has also a very low Brier score of 0.1177); thus, for each ADD, we were able to find models that are comparatively good with a very low Brier score.

We used lrm’s function pentrace to assist in the selection of penalty factors for fitting regression models using penalized maximum likelihood estimation (see [13]). In the reported models, we generally used a simple penalty of 1 and a non-linear penalty of 5. Please note that the penalized regression models are offering slightly improved performance compared to non-penalized models.

The reported models in Table 7 do not always use the full set of our hypothesized metrics. We tried many combinations of independent variables and penalties. In the table, for each decision, we show the three models with the highest bias-corrected C-index values, which we found. It can be observed that we were able to reach for both original and bootstrapped C-indexes values above or very close to 0.9, and at the same time very low Brier scores. We can also observe that all our hypothesized independent variables are used in at least one of the models that are reported. That is, all hypothesized variables are relevant in our predictions, but in no decision all of them are needed to make a prediction with a high C-index, low Brier score, and low p-value.

The company EU-VRi, represented by three co-authors of this article, is currently developing an extension of its existing cyber-threat resilience assessment tool. The tool aims to enable developers, architects, or assessors to assess and optimize the security threat resilience (risk analysis, preparedness, absorption, recovery, adaption to adverse conditions, stresses, attacks, or compromises) of a system. To reach this goal, the tool calculates a cyber-threat resilience level index for a system, as a composite, multi-level indicator. It is supposed to be based on existing industry guidelines, such as the ones summarized in Section 2.1. The analysis of these guidelines performed by the last three co-authors of this article, explained in Section 2.1, was performed initially with the purpose to create manually verifiable metrics and indicators for this tool.

Our approach can be used as a building block for the cyber-threat resilience assessment tool to automatically calculate metrics for the software architectural parts of the composite resilience level index. The tool provides an API to integrate building blocks such as the metrics provided by our approach and use the composite indicator-based approach to calculate an aggregated score for resilience assessments. Further, the tool supports before/after analysis, multi-assessment monitoring over time, and decision support based on sensitivity analysis. As the tool is expected to perform automated assessments, we have designed our metrics and models to be integrated within this commercial tool as an automated component.

The company SEARCH-LAB⁷ is currently working on a new industrial tool for incremental and continuous security certification. This tool addresses the following problem: The rapid development in today’s microservice systems, with continuous integration and delivery being the expected norm rather than the exception, poses a substantial challenge to the certification of software security. Existing security certification schemes focus on certifying that software projects are following certain security best practices, and they largely focus on the software development process. Security concerns, such as the ADDs covered in this article, are usually checked manually, just as in our ground truth analysis, typically even without the support of (generated) architectural models, but instead directly in the source code.

In microservice projects that require certification before a release, this is highly problematic. In such projects, development processes are continuously adapted by developers, and thus process-based schemes are hard to apply. Architectural concerns are scattered throughout the source code and hard to find just by inspecting the source code. Changes and releases are expected to be happening with a high frequency, meaning that organizations might not have the necessary resources to perform certifications for each release or it would result in significant costs to maintain the certified status by paying independent evaluators due to the rapid release cycles.

The incremental and continuous security certification tool (called DeltAICert), being developed at SEARCH-LAB, takes a different view by focusing on artifacts rather than processes. It aims to certify only the required delta, by comparing evidence (source, code, logs, models, test results) and indicators (metrics from other analysis tools) from previous evaluations. For changed artifacts and their dependencies, lightweight and largely automated techniques for the continuous and incremental security evaluation shall be applied to enable the continuous re-certification of the developed software. The DeltAICert tool is able to make automated evaluations for the AssureMOSS certification scheme, which was also developed in the AssureMOSS project.⁸ In the context of this tool development, our approach is used as a building block to automate the (re)certification of the practices embodied in our ADDs based on the architectural models and the derived metrics. For this purpose, two industrial security experts from SEARCH-LAB have reviewed our models, metrics, and code, and designed an integration into SEARCH-LAB’s novel continuous certification scheme.

We initially performed a number of data collection and analysis steps, including a study of microservice architecture security guidelines, the gray literature, and the scientific literature, plus a manual inspection of the code of 10 open source system and in-depth reviews by five industrial security experts in order to find four industrial ADDs, each covering multiple security tactics. This enabled us to design our ADD model, reported in Section 4, which is a prerequisite to study our research questions. To answer RQ1 and RQ2, we proposed a set of generic, technology-independent metrics for each of the ADDs. Each of the decision options (aka security tactics) corresponds to at least one metric, but some aspects were covered by multiple metrics which we hypothesized could provide good predictions later on. We aimed to objectively assess for each model how well the security tactics are supported for establishing the ground truth. For this, the three industrial security experts on the author team and the team in their company created (independently of the work reported in this article, before they got involved in this article) a recommendation which serves as informal guidance for security experts to manually judge systems such as those in our models. The academic authors used this recommendation to assess each model manually, and then another independent review by the five industrial security experts was performed. We believe that these steps make it highly likely that our ADDs represent well-established practices in the industry, and that our ground truth assessment is very likely within the range of manual assessments to be expected from industrial security experts.

We formulated the metrics to numerically assess a security tactic’s implementation in each model, as well as automated detectors enabling the automatic computation of the models. In our previous work [36], we have developed an approach for automatically extracting the needed models from a system’s source code with modest manual specification effort. As this approach was applied to one of the larger models in our model data set which contained a highly polyglot microservice architecture (the Case RS0), we can assess that it is highly likely possible to extract all our models automatically from the source code with a small to medium specification effort in the same way. As a consequence, all stages in our work can be fully automated, once such an extraction specification has been realized. This means one can apply our approach e.g., in the context of a continuous delivery pipeline on each single code commit, which was one of the original motivations for our work (see Section 1). That is, our approach reported in Sections 4-6, together with the source code extraction technique reported in our earlier work [36] if manual modeling should be avoided, provide our answer to RQ1.

We performed an ordinal regression analysis using our metrics as independent variables to predict the ground truth assessment. Our results show that every set of decision-related metrics can predict with high statistical significance and high accuracy our objectively evaluated assessment, using a number of regression models (three regression models per ADD were reported). We also assessed the correlation of the individual independent variables to the dependent variable, and selected on models in which at least one variable with a strong or very strong correlation was present. This suggests that automatic metrics-based assessment of a system’s conformance to security tactics used as options in ADDs is possible with a high degree of confidence. The p-values, C-index values, and Brier scores provided in Section 7 provide concrete measurements showing that the metrics are accurate measures for assessing ADD conformance in our model data set, which answers RQ2.

Regarding RQ3, we consider that existing modeling practices can be easily mapped to our microservice meta-model (derived from [61]). Rather lightweight extensions for security tactics markup have been made. Both have been formally specified in Section 6. More specifically, for completing the modeling of our model data set, we needed to introduce 16 component types and 28 connector types, such as the OAuth2 Server component stereotype or Token-based Authorization connector stereotype (both shown as examples in Figure 2). In addition, we needed to introduce types for basic microservice modeling. In particular, 25 component types and 38 connector types are offered, ranging from general notions such as the Service component type, to very technology-specific classes such as the RESTful HTTP connector, which is a subclass of Service Connector. Our study shows that in each ADD context and the proposed metrics, only a small subset of the meta-model is required. We confirmed for each of the stereotypes that they can be extracted from the source code with the methods reported in our earlier work (i.e., [36]).

We deliberately relied on third-party systems as the basis for our study to increase internal validity, thus avoiding bias in system composition and structure. It is possible that our search procedures introduced some kind of unconscious exclusion of certain sources; we mitigated this by assembling an author team with many years of experience in the field (including the substantial industry experiences of three industrial experts as co-authors), and performing very general and broad searches. Given that our search was not exhaustive, and that most of the systems we found were made for demonstration purposes, i.e., relatively modestly sized, this means that some potential architecture elements were not included in our meta-model. In addition, this raises a possible threat to external validity of generalization to other, more complex, systems. We nevertheless feel confident that the systems documented are a representative cross-cut of current practices in the field, as the points of variance between them were limited and well attested in the literature.

It might, however, be the case that for substantially different kinds of systems, e.g., systems from other domains or substantially larger systems, the expert judgement for the ground truth would differ. Then it would be necessary to re-run our regression analysis with data from a few such systems in order to calibrate the prediction models to the changed circumstances.

To avoid threats with regard to the generalizability of our approach, we limited our scope to microservice-based systems, even though some aspects of our approach are likely applicable to other kinds of distributed systems than microservice-based systems as well. For instance, the security tactics and ADDs, as well as some of the related metrics, are likely applicable for many other kinds of systems as well. Reasons for the limitation to microservices are: Firstly, many aspects of the practices we investigated are specific to microservices and security recommendations for them (such as [34]). For instance, our investigation of observability practices is based on the concrete technologies currently being employed in the microservice field. Secondly, our evaluation is purely based on microservice-based systems. Thirdly, our meta-model and reconstruction approach focuses on microservice-specific abstractions, as explained in Section 2.6.

Another potential threat is the fact that the variant systems were derived by the author team. This is mitigated by the independent review of the variants by the industrial reviewers. Also, this was done according to existing practices documented in literature. We carefully made sure only to change specific aspects in a variant and keep all other aspects stable. That is, while the variants do not represent actual systems, they are reasonable evolutions of the original designs (e.g., representing possible architectural refactoring steps or deteriorations during system evolution). As a major goal of our approach was to be able to find issues during system evolution (e.g., in the context of continuous delivery), such as a mistake made in a refactoring step or deteriorations during system evolution, the modeling of variants actually introduces this aspect into our approach and our statistical analysis.

An aspect that might limit the validity of the reviews by our industrial co-authors is the shared context they have, as they work in the same company. We mitigated this threat firstly by involving two other industrial reviewers from another company. Secondly, we mitigated it by extensive reviews by the academic authors of this article, which all have substantial experiences in industrial software development and software security. Fourthly, please note that the company of the industrial co-authors consults other companies; that is, each of those industrial co-authors has experience from many different industry projects (across sizes, domains, software development methods). Yet another mitigation of the threat is that all practices and ADDs are directly derived from widely-used industry guidelines, which substantially limits the chance that only practices relevant for a particular company have been considered.

The modeling process is also considered as a source of an internal validity threat. The models of the systems were repeatedly and independently cross-checked by the author team that has considerable experience in similar methods, but the possibility of some interpretative bias remains: other researchers might have coded or modeled differently, leading to different models. As a mitigation, we also offer the whole models and the code as open access artifacts for review and reproducibility. Since we aimed only to find one model that is able to specify all observed phenomena, and this was achieved, we consider this threat not to be a major issue for our study. As mentioned, it is easily possible to change the modeling slightly and re-run our regression analysis to calibrate the prediction models to a different modeling approach.

The ground truth assessment might also be subject to different interpretations by different practitioners. For this purpose, we deliberately chose only a five-step ordinal scale, and given that the ground truth evaluation for each decision is fairly straightforward and based on industrial best practices and recommendations, we do not consider our interpretation controversial. Likewise, the individual metrics used to evaluate the presence of each security tactic were deliberately kept as simple as possible, so as to avoid false positives and enable a technology-independent assessment. As stated previously, generalization to more complex systems might not be possible without modification. But we consider that the basic approach taken when defining the metrics is validated by the success of the regression models.

Please note that we do not claim completeness of the metrics we present in this article. They are only complete in the sense that they cover all options of the ADDs they address. As discussed in Section 8, they are meant to be used as automatable building blocks in tools that cover a broader range of metrics (including e.g., organizational metrics that cannot be automatically extracted and calculated in the same way).

Much research has been conducted in collecting and systematizing microservice patterns. For instance, Richardson [48] collected microservice patterns related to major design and architectural practices. Zimmermann et al. [65] introduced microservice API related patterns. Skowronski [50] collected best practices for event-driven microservice architectures. Microservice fundamentals and best practices are also discussed by Fowler and Lewis [29], and are summarized in a mapping study by Pahl and Jamshidi [38]. Taibi and Lenarduzzi [53] studied microservice bad smells, i.e., practices that should be avoided, which would correspond to metrics violations in our work. In this article, we use such guidance as the foundation for microservice architecture modeling.

Likewise, attempts to define security patterns have been made [20, 49]. In the security field, recommendations by industry or standards organizations (such as the OWASP Top 10⁹ or the ISO 27002 standard¹⁰) are widely used. For microservices, microservice-specific recommendations by industry organizations such as those of NIST [34], OWASP [37], or the Cloud Security Alliance [8] are proposed, which represent aggregations of existing industry best practices on a broad level. In our work, we used such guidance for selecting the security practices we investigate.

Microservices [29, 33, 48] are, among many other things, a way to decompose an architecture based on services [62]. This is an area which has been studied intensively in recent years (see e.g., [38, 40, 63]). An important part of our approach is to model architecture component models and reconstruct such models via static code analysis, and analyze the resulting models later on. This part of our approach is related to architecture reconstruction and related modeling approaches. Architecture reconstruction focuses on automatically or semi-automatically producing architecture abstractions from the source code [10, 30, 31]. While such generic approaches fall short in being able to address the specific characteristics of microservice systems, a number of more specific modeling and reconstruction approaches have been proposed to address this gap.

Granchelli et al. [18] provide one of the few existing microservice-specific architecture reconstruction approaches. It statically analyses Docker and Docker Compose files for names and ports, and then the Docker containers and network bridges dynamically, to reconstruct the deployed microservices from the system’s communication logs. Alshuqayran et al. [2] present an approach that is intended as a groundwork for architecture reconstruction of microservices. From the analysis of eight open source projects, the approach derived a meta-model and possible mapping rules for microservices.

Vianden et al. [57] report on a study of a microservice-based reference architecture as a starting point for enterprise measurement infrastructures. This can be seen as an alternative to reconstruction, but it requires manual maintenance of the architecture in relation to the reference architecture. Rademacher et al. [46] suggest addressing the polyglot nature of microservices using an aspect-oriented modeling approach.

Our own prior work [36], used for reconstruction purposes in this article, aims to address issues of reconstructing microservice systems when facing polyglot programming, persistence, and technologies. As a consequence, our microservice architecture model explained in Section 6 is richer in this regard than the models used in the other related works. In this article, we use this approach as a foundation for extracting security tactics related models from the source code and perform automated metrics-based analysis of the resulting models. To the best of our knowledge, our approach is the first to consider microservice system architectures and their security aspects in a modeling-/reconstruction-based approach.

In addition to microservice architecture reconstruction, a second direction in which this article is related to architecture reconstruction approaches is the reconstruction and analysis of security features. This has been addressed before in a number of existing approaches. Sohr and Berger [51] use the Bauhaus tool to build security views. In particular, they extract a resource flow graph from Java code (J2EE or Android). The authors build security views that represent the RBAC permissions in the application. In a later work, Bunke and Sohr [6] extend the previous approach based on RFGs to extract security patterns from the code and check their correct implementation.

Concerning Android Java applications, Hamad et al. [22] proposed an approach to automatically extract the architectural design of an application starting from the APK (bytecode). The DelDroid tool is able to extract the application components (activities, services, broadcast receivers, content providers), the explicit and implicit communication flows (via intents), as well the granted permissions to each component. The tool also analyzes the code and computes the permissions that are actually used.

Vanciu and Abi-Antoun [56] proposed the Scoria approach. The approach is based on the extraction from Java code of an Ownership Object Graph (OOG). Objects are hierarchically organized in a tree according to ownership domains. Dataflow edges are automatically added on top of the hierarchy and represent information flows (i.e., read and write), as well as point-to and inheritance relationships among the objects. The resulting graph is called a Sec Graph and can be further enriched by software architects with security-related properties. The architect can run queries on the graph to identify security issues, such as insecure communication.

Peldszus et al. [42] defines an approach for the model-based security analysis of Java applications. The analysis is performed on a SecPL model, which is an extension of UMLsec. The SecPL model can be reverse engineered from the code, provided that the developers have enriched the code with SecPL annotations, which carry security information and properties.

Jasser [25] categorizes a number of ADDs for security. These security constraints are formalized as LTL formulas, which predicate over architectural concepts (e.g., components). The approach entails that the architect provides a manual mapping between the architectural concepts and the code (a DSL is provided to this aim). Finally, the tool checks the compliance between the security rules and the code.

Bauer et al. [4] focus on formally validating the security specification (as UMLsec diagrams) and then monitoring the system at run-time in order to check the compliance of the implementation with the security specification. To enable the latter step, the architect has to manually provide a mapping between code elements and model elements.

Peldszus et al. [43] present a semi-automated approach to map model elements from a SecDFD diagram to code elements (Java methods and types). The tool provides some suggestions that the architect can approve or discard. The architect can also suggest their own mappings. In an iterative way, the tool suggests new mappings by prioritizing the heuristics that have been ‘approved’ by the human. The heuristics work on an intermediate representation of the code called Program Model, which fuses concepts from the class diagram, the call graph, and the dependency graph.

The approaches discussed above have in common with our approach that they build a security-specific model from the source code as a foundation for later analysis. In contrast to our approach, they do not support typical microservice architecture characteristics in their models, such as various possible distributed systems invocations used throughout microservice systems, the link to continuous delivery approaches, or the use of polyglot programming, persistence, and technologies. This adds a considerable layer of complexity to the system models under investigation in this article, compared to those earlier works. In consequence, our proposed metrics and detectors need to consider the extra elements in the software architecture, too.

As microservice architectures rely on a level of assurance between the actors involved for engaging in interactions, the prospect of quantifying that level of assurance using security-relevant indicators is attractive. These indicators can enable organizations to assess architecture security [27]. Security metrics are then needed to understand the current state of security, and possibly improve that state [41]. While several organizations including Microsoft [23] and OWASP [59] propose processes and checklists for building secure architectures, there are very few tools that can automate these processes for tailor-made solutions due to the dynamics in and heterogeneity of the systems [39]. Our approach is to the best of our knowledge the first that supports automated security assessment based on an empirical investigation of existing (best) practices in the field of microservice architectures.

Ramos et al. [47] conducted a detailed review of the main existing model-based quantitative security metrics with a focus on network security metrics, analyzing their advantages and disadvantages in their use. Model-based security metrics use techniques to describe a system in terms of an abstract model that captures the necessary attributes of interest, based on the assumptions of the attacker and the system behavior. For example, Attack Graphs (AG) are a model commonly used to quantify network security. It uses the causal relationships between vulnerabilities which allows quantification of the likelihood of potential multi-step attacks that combine multiple vulnerabilities [58]. Noel et al. [35] describe a set of metrics for measuring network-wide cybersecurity risk based on a model of multi-step attack vulnerability with AGs. Their system for computing security metrics from vulnerability-based network AGs used the data imported from sources that are commonly deployed within enterprise networks, such as vulnerability scanners and firewall configuration files. They used a Topological Vulnerability Analysis (TVA) tool [24], Cauldron, which analyses network attack vulnerability from scanning tools and other data sources. The tool correlates cyber-vulnerabilities and environmental metadata and applies network access policy (firewall) rules.

Such general security metrics and indicators are a foundation for our work. But as none of them considers the specifics of microservice architectures yet, they cannot at all or without substantial adaptation be applied to our research problems. For this reason, we decided to design metrics based on existing recommendations (such as NIST [34], OWASP [37], or the Cloud Security Alliance [8]) specifically for microservice-based systems.

Many of the works on service metrics today are focused on runtime properties (see e.g., [44]). A number of studies have used metrics to assess microservice-based software architectures, e.g., [5, 40, 60], but each is focused on narrow sets of architecture-relevant tenets (e.g., loose coupling), and no general approach for an assessment across different microservice tenets exists. Pautasso and Wilde [40] propose a composite, facet-based metric for the assessment of loose coupling in service-oriented systems. Zdun et al. [60] study the independent deployment of microservices by defining metrics to assess architecture conformance to microservice patterns, focused on two aspects: independent deployment and shared dependencies of services. Bogner et al. [5] propose a maintainability quality model which combines eleven easily extracted code metrics into a broader quality assessment. Engel et al. [11] also propose a method of using real-time system communication traces to extract metrics on conformance to recommended microservice design principles such as loose coupling and small service size. Our work broadly follows the same approach, but does not focus on specific tenets only and adds notions for measuring the use of security tactics in security-related ADDs.

Once metrics can be checked automatically, our approach can be classified as a metrics-based, microservice-specific approach for software architecture conformance checking. In general, approaches for architecture conformance checking are often based on automated extraction techniques [19, 55]. Conformance to architecture patterns [19, 21] or other kinds of architectural rules [55] can often be checked by such approaches. Techniques that are based on a broad set of microservice-related metrics to cover multiple microservice tenets and security do not yet exist.

While a couple of works specifically focus on microservice security metrics, again many of those focus on runtime-related or non-architectural aspects [3, 12, 28, 54]. Chondamrongkul et al. [7] present an early approach for automatically investigating certain security flaws in a component and connector architecture, such as man-in-the-middle or denial-of-service attacks. As it is unclear why, if this information is modeled, the flaw is not fixed instead, in this article, we do rather not model potential flaws, but security tactics, which seem better suited for modeling and analysis at the architectural level. Also, our work is based on empirical data, whereas Chondamrongkul et al.’s work only uses modeling examples.