Microservices Security in Action

Microservices Security in Action

Prabath Siriwardena and Nuwan Dias

To comment go to liveBook

Manning

Shelter Island

For more information on this and other Manning titles go to

manning.com

Copyright

For online information and ordering of these and other Manning books, please visit manning.com. The publisher offers discounts on these books when ordered in quantity.

For more information, please contact

Special Sales Department

Manning Publications Co.

20 Baldwin Road

PO Box 761

Shelter Island, NY 11964

Email: orders@manning.com

©2020 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

♾ Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

ISBN: 9781617295959

dedication

To Dr. Sanjiva Weerawarana, our mentor for more than a decade and for many more years to come!

brief contents

Part 1. Overview

  1 Microservices security landscape

  2 First steps in securing microservices

Part 2. Edge security

  3 Securing north/south traffic with an API gateway

  4 Accessing a secured microservice via a single-page application

  5 Engaging throttling, monitoring, and access control

Part 3. Service-to-service communications

  6 Securing east/west traffic with certificates

  7 Securing east/west traffic with JWT

  8 Securing east/west traffic over gRPC

  9 Securing reactive microservices

Part 4. Secure deployment

10 Conquering container security with Docker

11 Securing microservices on Kubernetes

12 Securing microservices with Istio service mesh

Part 5. Secure development

13 Secure coding practices and automation

Appendixes:

Appendix A. OAuth 2.0 and OpenID Connect

Appendix B. JSON Web Token

Appendix C. Single-page application architecture

Appendix D. Observability in a microservices deployment

Appendix E. Docker fundamentals

Appendix F. Open Policy Agent

Appendix G. Creating a certificate authority and related keys with OpenSSL

Appendix H. Secure Production Identity Framework for Everyone

Appendix I. gRPC fundamentals

Appendix J. Kubernetes fundamentals

Appendix K. Service mesh and Istio fundamentals

contents

preface

acknowledgments

about this book

about the authors

about the cover illustration

Part 1. Overview

  1 Microservices security landscape

How security works in a monolithic application

Challenges of securing microservices

The broader the attack surface, the higher the risk of attack

Distributed security screening may result in poor performance

Deployment complexities make bootstrapping trust among microservices a nightmare

Requests spanning multiple microservices are harder to trace

Immutability of containers challenges how you maintain service credentials and access-control policies

The distributed nature of microservices makes sharing user context harder

Polyglot architecture demands more security expertise on each development team

Key security fundamentals

Authentication protects your system against spoofing

Integrity protects your system from data tampering

Nonrepudiation: Do it once, and you own it forever

Confidentiality protects your systems from unintended information disclosure

Availability: Keep the system running, no matter what

Authorization: Nothing more than you’re supposed to do

Edge security

The role of an API gateway in a microservices deployment

Authentication at the edge

Authorization at the edge

Passing client/end-user context to upstream microservices

Securing service-to-service communication

Service-to-service authentication

Service-level authorization

Propagating user context among microservices

Crossing trust boundaries

  2 First steps in securing microservices

Building your first microservice

Downloading and installing the required software

Clone samples repository

Compiling the Order Processing microservice

Accessing the Order Processing microservice

What is inside the source code directory?

Understanding the source code of the microservice

Setting up an OAuth 2.0 server

The interactions with an authorization server

Running the OAuth 2.0 authorization server

Getting an access token from the OAuth 2.0 authorization server

Understanding the access token response

Securing a microservice with OAuth 2.0

Security based on OAuth 2.0

Running the sample

Invoking a secured microservice from a client application

Performing service-level authorization with OAuth 2.0 scopes

Obtaining a scoped access token from the authorization server

Protecting access to a microservice with OAuth 2.0 scopes

Part 2. Edge security

  3 Securing north/south traffic with an API gateway

The need for an API gateway in a microservices deployment

Decoupling security from the microservice

The inherent complexities of microservice deployments make them harder to consume

The rawness of microservices does not make them ideal for external exposure

Security at the edge

Understanding the consumer landscape of your microservices

Delegating access

Why not basic authentication to secure APIs?

Why not mutual TLS to secure APIs?

Why OAuth 2.0?

Setting up an API gateway with Zuul

Compiling and running the Order Processing microservice

Compiling and running the Zuul proxy

Enforcing OAuth 2.0-based security at the Zuul gateway

Securing communication between Zuul and the microservice

Preventing access through the firewall

Securing the communication between the API gateway and microservices by using mutual TLS

  4 Accessing a secured microservice via a single-page application

Running a single-page application with Angular

Building and running an Angular application from the source code

Looking behind the scenes of a single-page application

Setting up cross-origin resource sharing

Using the same-origin policy

Using cross-origin resource sharing

Inspecting the source that allows cross-origin requests

Proxying the resource server with an API gateway

Securing a SPA with OpenID Connect

Understanding the OpenID Connect login flow

Inspecting the code of the applications

Using federated authentication

Multiple trust domains

Building trust between domains

  5 Engaging throttling, monitoring, and access control

Throttling at the API gateway with Zuul

Quota-based throttling for applications

Fair usage policy for users

Applying quota-based throttling to the Order Processing microservice

Maximum handling capacity of a microservice

Operation-level throttling

Throttling the OAuth 2.0 token and authorize endpoints

Privilege-based throttling

Monitoring and analytics with Prometheus and Grafana

Monitoring the Order Processing microservice

Behind the scenes of using Prometheus for monitoring

Enforcing access-control policies at the API gateway with Open Policy Agent

Running OPA as a Docker container

Feeding the OPA engine with data

Feeding the OPA engine with access-control policies

Evaluating OPA policies 132Next steps in using OPA

Part 3. Service-to-service communications

  6 Securing east/west traffic with certificates

Why use mTLS?

Building trust between a client and a server with a certificate authority

Mutual TLS helps the client and the server to identify each other

HTTPS is HTTP over TLS

Creating certificates to secure access to microservices

Creating a certificate authority

Generating keys for the Order Processing microservice

Generating keys for the Inventory microservice

Using a single script to generate all the keys

Securing microservices with TLS

Running the Order Processing microservice over TLS

Running the Inventory microservice over TLS

Securing communications between two microservices with TLS

Engaging mTLS

Challenges in key management

Key provisioning and bootstrapping trust

Certificate revocation

Key rotation

Monitoring key usage

  7 Securing east/west traffic with JWT

Use cases for securing microservices with JWT

Sharing user context between microservices with a shared JWT

Sharing user context with a new JWT for each service-to-service interaction

Sharing user context between microservices in different trust domains

Self-issued JWTs

Nested JWTs

Setting up an STS to issue a JWT

Securing microservices with JWT

Using JWT as a data source for access control

Securing service-to-service communications with JWT

Exchanging a JWT for a new one with a new audience

  8 Securing east/west traffic over gRPC

Service-to-service communications over gRPC

Securing gRPC service-to-service communications with mTLS

Securing gRPC service-to-service communications with JWT

  9 Securing reactive microservices

Why reactive microservices?

Setting up Kafka as a message broker

Developing a microservice to push events to a Kafka topic

Developing a microservice to read events from a Kafka topic

Using TLS to protect data in transit

Creating and signing the TLS keys and certificates for Kafka

Configuring TLS on the Kafka server

Configuring TLS on the microservices

Using mTLS for authentication

Controlling access to Kafka topics with ACLs

Enabling ACLs on Kafka and identifying the clients

Defining ACLs on Kafka

Setting up NATS as a message broker

Part 4. Secure deployment

10 Conquering container security with Docker

Running the security token service on Docker

Managing secrets in a Docker container

Externalizing secrets from Docker images

Passing secrets as environment variables

Managing secrets in a Docker production deployment

Using Docker Content Trust to sign and verify Docker images

The Update Framework

Docker Content Trust

Generating keys

Signing with DCT

Signature verification with DCT

Types of keys used in DCT

How DCT protects the client application from replay attacks

Running the Order Processing microservice on Docker

Running containers with limited privileges

Running a container with a nonroot user

Dropping capabilities from the root user

Running Docker Bench for security

Securing access to the Docker host

Enabling remote access to the Docker daemon

Enabling mTLS at the NGINX server to secure access to Docker APIs

Considering security beyond containers

11 Securing microservices on Kubernetes

Running an STS on Kubernetes

Defining a Kubernetes Deployment for the STS in YAML

Creating the STS Deployment in Kubernetes

Troubleshooting the Deployment

Exposing the STS outside the Kubernetes cluster

Managing secrets in a Kubernetes environment

Using ConfigMap to externalize configurations in Kubernetes

Defining a ConfigMap for application.properties file

Defining ConfigMaps for keystore.jks and jwt.jks files

Defining a ConfigMap for keystore credentials

Creating ConfigMaps by using the kubectl client

Consuming ConfigMaps from a Kubernetes Deployment

Loading keystores with an init container

Using Kubernetes Secrets

Exploring the default token secret in every container

Updating the STS to use Secrets

Understanding how Kubernetes stores Secrets

Running the Order Processing microservice in Kubernetes

Creating ConfigMaps/Secrets for the Order Processing microservice

Creating a Deployment for the Order Processing microservice

Creating a Service for the Order Processing microservice

Testing the end-to-end flow

Running the Inventory microservice in Kubernetes

Using Kubernetes service accounts

Creating a service account and associating it with a Pod

Benefits of running a Pod under a custom service account

Using role-based access control in Kubernetes

Talking to the Kubernetes API server from the STS

Associating a service account with a ClusterRole

12 Securing microservices with Istio service mesh

Setting up the Kubernetes deployment

Enabling Istio autoinjection

Clean up any previous work

Deploying microservices

Redeploying Order Processing and STS as NodePort Services

Testing end-to-end flow

Enabling TLS termination at the Istio Ingress gateway

Deploying TLS certificates to the Istio Ingress gateway

Deploying VirtualServices

Defining a permissive authentication policy

Testing end-to-end flow

Securing service-to-service communications with mTLS

Securing service-to-service communications with JWT

Enforcing JWT authentication

Testing end-to-end flow with JWT authentication

Peer authentication and request authentication

How to use JWT in service-to-service communications

A closer look at JSON Web Key

Enforcing authorization

A closer look at the JWT

Enforcing role-based access control

Testing end-to-end flow with RBAC

Improvements to role-based access control since Istio 1.4.0

Managing keys in Istio

Key provisioning and rotation via volume mounts

Limitations in key provisioning and rotation via volume mounts

Key provisioning and rotation with SDS

Part 5. Secure development

13 Secure coding practices and automation

OWASP API security top 10

Broken object-level authorization

Broken authentication

Excessive data exposure

Lack of resources and rate limiting

Broken function-level authorization

Mass assignment

Security misconfiguration

Injection

Improper asset management

Insufficient logging and monitoring

Running static code analysis

Integrating security testing with Jenkins

Setting up and running Jenkins

Setting up a build pipeline with Jenkins

Running dynamic analysis with OWASP ZAP

Passive scanning vs. active scanning

Performing penetration tests with ZAP

Appendixes

Appendix A. OAuth 2.0 and OpenID Connect

Appendix B. JSON Web Token

Appendix C. Single-page application architecture

Appendix D. Observability in a microservices deployment

Appendix E. Docker fundamentals

Appendix F. Open Policy Agent

Appendix G. Creating a certificate authority and related keys with OpenSSL

Appendix H. Secure Production Identity Framework for Everyone

Appendix I. gRPC fundamentals

Appendix J. Kubernetes fundamentals

Appendix K. Service mesh and Istio fundamentals

index

front matter

preface

While working at WSO2 for more than a decade, we’ve seen how the integration domain evolved over time from SOAP-based services to JSON/RESTful services and then to microservices. We spent most of our early days at WSO2 contributing to the Apache Axis2 project, which was a popular SOAP engine in those days, and to the Apache Rampart project, which implements many Organization for the Advancement of Structured Information Standards (OASIS) standards for web services security. Even though SOAP was quite promising in those days, it started to fade rapidly over time, and clearly JSON/RESTful services had won. Most of the microservice implementations we see today follow RESTful design principles.

In the last two to three years, we’ve seen a genuine interest from many companies we’ve worked with to move into microservices architecture, and projects starting from scratch are adopting microservices principles. Most of the early adopters of microservices just wanted to get things done, and worried mostly about implementing functional requirements. They didn’t worry too much about security, although they should have. In many cases, securing microservices would mean securing the interactions among microservices with Transport Layer Security (TLS), and may be, for some, enforcing mutual TLS for service-to-service authentication. But none of them are quite adequate. There are two main reasons many didn’t worry much about security: complexity and awareness.

Some time back, we found that most tools for securing microservices were not easy to use or couldn’t address the challenges specific to microservices deployments. This complexity was a barrier to securing microservices. At the same time, people who didn’t put much effort into security weren’t fully aware of the risks. We started hearing these stories from many of our customers as well as from the extended open source community we work with. That motivated us to write this book on securing microservices. Bringing an idea from inception to reality takes considerable time and effort. We lived with this idea of writing a book for more than two years until Manning reached out to us. During that period, with the increased adoption of microservices, the infrastructure around microservices security also evolved.

Writing a book about a rapidly evolving domain is bit of a challenge; you never know when your book will be obsolete. After discussing this challenge with the publisher, we decided to put more weight on principles and patterns, and use tools just to demonstrate how to apply those principles and patterns in practice. This was our ground rule in picking up the technology stack for the book. We use Spring Boot / Java to develop all the samples, though we don’t expect you to know either Java or Spring Boot in detail. If you have development experience in any programming language, you should be able to follow all the samples in the book with no difficulty.

Security itself is a larger domain. Securing microservices can mean different things to different people, based on their experiences and expectations. This fact was highlighted by one of the reviewers of the book, who comes from a security testing background. In our book, we wanted to focus on managing access to microservices. In other words, we wanted to focus on securing access to microservices with authentication and authorization. So, the book doesn’t talk about protecting microservices against different types of attacks, such as SQL injection, cross-site scripting (XSS), cross-site request forgery, and so on.

After a marathon effort that spanned slightly more than two years, we are glad to see that our book on microservices security is out. We are also excited that this is the very first book on securing microservices. We hope you will enjoy reading it!

acknowledgments

This book would not have been possible without the support of many amazing people:

Brian Sawyer, senior acquisitions editor at Manning, reached out to us and helped us structure our book proposal.

Marina Michaels, development editor at Manning, was very patient and tolerant of us throughout the publishing process and provided invaluable advice during the writing process.

To the rest of the staff at Manning: Deirdre Hiam, the project editor; Sharon Wilkey, the copyeditor; Keri Hales, the proofreader; and Ivan Martinovic´ ,the review editor.

All the Manning Early Access Program (MEAP) subscribers of the book.

Thorsten P. Weber, technical proofreader, who helped us review the code to make sure all the code samples work as expected.

Tim Hinrichs, one of the creators of the Open Policy Agent (OPA) project, and Andrew Jessup, one of the creators of the SPIFFE project, who helped us by reviewing the appendices on OPA and SPIFFE.

Sanjiva Weerawarana, the founder and CEO of WSO2, and Paul Fremantle, the CTO of WSO2, who have constantly mentored us for many years.

To all the reviewers: Andrew Bovill, Björn Nordblom, Bruno Vernay, Eros Pedrini, Evgeny Shmarnev, Gerd Koenig, Gustavo Gomes, Harinath Mallepally, Joel Holmes, John Guthrie, Jonas Medina, Jonathan Levine, Jorge Ezequiel Bo, Leonardo Gomes da Silva, Lukáš Hozda, Massimo Siani, Matthew Rudy Jacobs, Mostafa Siraj, Philip Taffet, Raushan Jha, Salvatore Campagna, Simeon Leyzerzon, Srihari Sridharan, Stephan Pirnbaum, Thilo Käsemann, Tim van Deurzen, Ubaldo Pescatore, Yurii Bodarev--your suggestions helped make this a better book.

Prabath Siriwardena: My wife, Pavithra, and my little daughter, Dinadi, supported me throughout the writing process. Thank you very much, Pavithra and Dinadi. My parents and my sister are with me all the time. I am grateful to them for everything they have done for me. And also, my wife’s parents--they were amazingly helpful.

Nuwan Dias: My family, including my wife, Kasun, and son, Jason. I would not have been able to make the effort required to contribute to this book if not for their consistent support and patience throughout. My parents and in-laws are always a strength to me and back me up in everything I do.

about this book

Microservices Security in Action teaches you how to secure your microservices applications code and infrastructure. After a straightforward introduction to the challenges of microservices security, you’ll learn fundamentals needed to secure both the application perimeter and service-to-service communications. Following a hands-on example, you’ll explore how to deploy and secure microservices behind an API gateway as well as how to access microservices via a single-page application (SPA).

Along the way, the book highlights important concepts including throttling, analytics gathering, access control at the API gateway, and microservice-to-microservice communications. You’ll also discover how to securely deploy microservices by using state-of-the-art technologies, including Kubernetes, Docker, and the Istio service mesh.

Lots of hands-on exercises secure your learning as you go, and this straightforward guide wraps up with a security process review and best practices. When you’re finished reading, you’ll be planning, designing, and implementing microservices applications with the priceless confidence that comes with knowing they’re secure!

Who should read this book

Microservices Security in Action is for developers who are well versed in microservices design principles and have a basic familiarity with Java. Even if you are not a Java developer, but are familiar with any object-oriented programming language such as C++ or C#, and understand basic programming constructs well, you’ll still get much out of this book. While some documents and blog posts exist online, this book brings together everything in a clear, easy-to-follow format that will benefit anyone wanting to get a thorough understanding of securing microservices.

How this book is organized: A roadmap

The book has five sections and 13 chapters. Part 1 takes you through the fundamentals in securing microservices:

Chapter 1 teaches you why securing microservices is challenging, and takes you through the key principles in securing a microservices deployment.

Chapter 2 teaches you how to build your first microservice in Spring Boot and secure it with OAuth 2.0. You will also learn how to set up an OAuth 2.0 token issuer.

Part 2 takes you through securing a microservice at the edge (or entry point) in a typical microservices deployment:

Chapter 3 takes you through the consumer landscape of your microservices and teaches you how to deploy a Spring Boot microservice behind the Zuul API gateway. You will also learn how to enforce OAuth 2.0-based security at the Zuul API gateway.

Chapter 4 teaches you how to develop a single-page application (SPA) with Angular. You will also learn how to secure a SPA with OpenID Connect.

Chapter 5 teaches you how to extend the use case you built in chapter 4 by engaging throttling, monitoring, and access control at the Zuul API gateway.

Part 3 takes you through the process of securing interactions among microservices once a request from a client application passes through the security at the edge and enters into your microservices deployment:

Chapter 6 teaches you how to secure communications among microservices that take place over HTTP, with mutual Transport Layer Security (mTLS).

In chapter 7, you learn how to share contextual data (for example, the end user context) among microservices by using JSON Web Token (JWT).

Chapter 8 teaches you how to secure communications among microservices that take place over gRPC, with mTLS and JWT.

Chapter 9 teaches you how to secure reactive microservices. It also teaches you how to set up Kafka as a message broker, and how to enforce access-control policies for Kafka topics.

Part 4 takes you through deploying and securing microservices in a containerized environment:

Chapter 10 teaches you how to deploy your microservices in Docker and to secure service-to-service interactions with mTLS and JWT. You also learn some of the built-in security features related to Docker.

Chapter 11 teaches you how to deploy your microservices as Docker containers in Kubernetes and to secure service-to-service communications with JWT over mTLS.

Chapter 12 teaches you how to offload the security processing overhead from your microservices by using the Istio service mesh.

Part 5 takes you through security testing in the development process:

Chapter 13 teaches you how to automate security testing of your microservices with SonarQube, Jenkins, and OWASP ZAP.

In general, you should be sure to read the first two chapters so that you have the right mindset to take on the challenges of securing microservices and that you’ve gotten your feet wet and are ready to build more complex security patterns, which the book teaches you. The appendices provide information on OAuth 2.0, JWT, gRPC, Docker, Kubernetes, Istio, Open Policy Agent (OPA), and SPIFFE. This information supplements the chapters.

About the code

This book contains many examples of source code both in numbered listings and in line with normal text. In both cases, source code is formatted in a fixed-width font like this to separate it from ordinary text.

In many cases, the original source code has been reformatted; we’ve added line breaks and reworked indentation to accommodate the available page space in the book. In rare cases, even this was not enough, and listings include line-continuation markers (\). Additionally, comments in the source code have often been removed from the listings when the code is described in the text. Code annotations highlight important concepts and significant lines of code in many of the listings.

Source code for the examples in this book is available for download from the publisher’s website at www.manning.com/books/microservices-security-in-action.

liveBook discussion forum

Purchase of Microservices Security in Action includes free access to a private web forum run by Manning Publications, where you can make comments about the book, ask technical questions, and receive help from the author and from other users. To access the forum and subscribe to it, point your web browser to www.manning.com/books/ microservices-security-in-action. You can also learn more about Manning's forums and the rules of conduct at https://livebook.manning.com/#!/discussion.

Manning’s commitment to our readers is to provide a venue where a meaningful dialogue between individual readers and between readers and the author can take place. It is not a commitment to any specific amount of participation on the part of the author, whose contribution to the forum remains voluntary (and unpaid). We suggest you try asking the authors some challenging questions lest their interest stray! The forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print.

Other online resources

Need additional help?

You can ask any questions related to the content of this book from the Microservices Security Slack channel: https://bit.ly/microservices-security.

The OAuth IETF working group is a good place to ask any questions on OAuth 2.0 and related standards. You can subscribe to the OAuth IETF working group mailing list with the information available at https://datatracker.ietf.org/wg /oauth/about.

The JOSE IETF working group is a good place to ask any questions on JSON Web Token (JWT) and the related standards. You can subscribe to the JOSE IETF working group mailing list with the information available at https:// datatracker.ietf.org/wg/jose/about.

You can ask any questions related to Kubernetes security from the Slack channel: https://slack.k8s.io/.

You can ask any questions related to the Open Policy Agent (OPA) project from the Slack channel: https://slack.openpolicyagent.org/.

You can ask any questions related to the SPIFFE project from the Slack channel: https://slack.spiffe.io/.

To get updates on the conference/meetup talks the authors of this book do regularly, you can subscribe to the YouTube channel: http://vlog.facilelogin.com/.

about the authors

Prabath Siriwardena is the vice president of security architecture at WSO2, and has been working in the identity management and security domain since 2007.

Nuwan Dias is the director of API architecture at WSO2 and has worked in the software industry since 2012, most of which he has spent focusing on the API management domain.

about the cover illustration

The figure on the cover of Microservices Security in Action is captioned Homme Islandois, or a man from Iceland. The illustration is taken from a collection of dress costumes from various countries by Jacques Grasset de Saint-Sauveur (1757-1810), titled Costumes de Différents Pays, published in France in 1797. Each illustration is finely drawn and colored by hand. The rich variety of Grasset de Saint-Sauveur’s collection reminds us vividly of how culturally apart the world’s towns and regions were just 200 years ago. Isolated from each other, people spoke different dialects and languages. In the streets or in the countryside, it was easy to identify where they lived and what their trade or station in life was just by their dress.

The way we dress has changed since then, and the diversity by region, so rich at the time, has faded away. It is now hard to tell apart the inhabitants of different continents, let alone different towns, regions, or countries. Perhaps we have traded cultural diversity for a more varied personal life--certainly for a more varied and fast-paced technological life.

At a time when it is hard to tell one computer book from another, Manning celebrates the inventiveness and initiative of the computer business with book covers based on the rich diversity of regional life of two centuries ago, brought back to life by Grasset de Saint-Sauveur’s pictures.

Part 1. Overview

Microservices are no longer a novelty. We’re seeing large-scale microservices deployments with thousands of services. But whether we have one or two services or thousands, security is a top priority. This part of the book takes you through the fundamentals in securing microservices.

Chapter 1 teaches you why securing microservices is challenging, and takes you through the key principles in securing a microservices deployment.

Chapter 2 teaches you how to build your first microservice in Spring Boot and secure it with OAuth 2.0. You will also learn how to set up an OAuth 2.0 token issuer.

When you’re finished with these two chapters, you’ll have the right mindset to take on the challenges of securing microservices. After getting your feet wet in this part of the book, you’ll be ready to build more complex security patterns (which we teach you in the rest of the book) on top of your first microservice.

1 Microservices security landscape

This chapter covers

Why microservices security is challenging

Principles and key elements of a microservices security design

Edge security and the role of an API gateway

Patterns and practices in securing service-to-service communications

Fail fast, fail often is a mantra in Silicon Valley. Not everyone agrees, but we love it! It’s an invitation to experiment with new things, accept failures, fix problems, and try again. Not everything in ink looks pretty in practice. Fail fast, fail often is only hype unless the organizational leadership, the culture, and the technology are present and thriving.

We find microservices to be a key enabler for fail fast, fail often. Microservices architecture has gone beyond technology and architectural principles to become a culture. Netflix, Amazon, Lyft, Uber, and eBay are the front-runners in building that culture. Neither the architecture nor the technology behind microservices--but the discipline practiced in an organizational culture--lets your team build stable products, deploy them in a production environment with less hassle, and introduce frequent changes with no negative impact on the overall system.

Speed to production and evolvability are the two key outcomes of microservices architecture. International Data Corporation (IDC) has predicted that by 2022, 90% of all apps will feature microservices architectures that improve the ability to design, debug, update, and leverage third-party code.1

We assume that you’re well versed in microservices design principles, applications, and benefits. If you’re new to microservices and have never been (or are only slightly) involved in development projects, we recommend that you read a book on microservices first, such as Spring Microservices in Action by John Carnell (Manning, 2017). Microservices Patterns by Chris Richardson (Manning, 2018) and Microservices in Action by Morgan Bruce and Paulo A. Pereira (Manning, 2018) are two other good books on the subject. Microservices for the Enterprise: Designing, Developing, and Deploying by Prabath Siriwardena (a coauthor of this book) and Kasun Indrasiri (Apress, 2018) is another beginner’s book on microservices.

In this book, we focus on microservices security. When you make the decision to go ahead with microservices architecture to build all your critical business operations, security is of topmost importance. A security breach could result in many unpleasant outcomes, from losing customer confidence to bankruptcy. Emphasis on security today is higher than at any time in the past. Microservices are becoming key enablers of digital transformation, so microservices security must be consciously planned, designed, and implemented.

This book introduces you to the key fundamentals, security principles, and best practices involved in securing microservices. We’ll be using industry-leading open source tools along with Java code samples developed with Spring Boot for demonstrations. You may pick better, competitive tools later in your production environment, of course.

This book will give you a good understanding of how to implement microservices security concepts in real life, even if you’re not a Java developer. If you’re familiar with any object-oriented programming language (such as C++ or C#) and understand basic programming constructs well, you’ll still enjoy reading the book, even though its samples are in Java. Then again, security is a broader topic. It’s a discipline with multiple subdisciplines. In this book, we mostly focus on application developers and architects who worry about managing access to their microservices. Access management itself is another broader subdiscipline of the larger security discipline. We do not focus on pen testing, developing threat models, firewalls, system-level configurations to harden security, and so on.

1.1 How security works in a monolithic application

A monolithic application has few entry points. An entry point for an application is analogous to a door in a building. Just as a door lets you into a building (possibly after security screening), an application entry point lets your requests in.

Think about a web application (see figure 1.1) running on the default HTTP port 80 on a server carrying the IP address 192.168.0.1. Port 80 on server 192.168.0.1 is an entry point to that web application. If the same web application accepts HTTPS requests on the same server on port 443, you have another entry point. When you have more entry points, you have more places to worry about securing. (You need to deploy more soldiers when you have a longer border to protect, for example, or to build a wall that closes all entry points.) The more entry points to an application, the broader the attack surface is.

Most monolithic applications have only a couple of entry points. Not every component of a monolithic application is exposed to the outside world and accepts requests directly.

In a typical Java Platform, Enterprise Edition (Java EE) web application such as the one in figure 1.1, all requests are scanned for security at the application level by a servlet filter.2 This security screening checks whether the current request is associated with a valid web session and, if not, challenges the requesting party to authenticate first.

Figure 1.1 A monolithic application typically has few entry points. Here, there are two: ports 80 and 443.

Further access-control checks may validate that the requesting party has the necessary permissions to do what they intend to do. The servlet filter (the interceptor) carries out such checks centrally to make sure that only legitimate requests are dispatched to the corresponding components. Internal components need not worry about the legitimacy of the requests; they can rightly assume that if a request lands there, all the security checks have already been done.

In case those components need to know who the requesting party (or user) is or to find other information related to the requesting party, such information can be retri-eved from the web session, which is shared among all the components (see figure 1.2). The servlet filter injects the requesting-party information into the web session during the initial screening process, after completing authentication and authorization.

Once a request is inside the application layer, you don’t need to worry about security when one component talks to another. When the Order Processing component talks to the Inventory component, for example, you don’t necessarily need to enforce any additional security checks (but, of course, you can if you need to enforce more granular access-control checks at the component level). These are in-process calls and in most cases are hard for a third party to intercept.

Figure 1.2 Multiple entry points (ports 80 and 443) are funneled to a single servlet filter. The filter acts as a centralized policy enforcement point.

In most monolithic applications, security is enforced centrally, and individual components need not worry about carrying out additional checks unless there is a desperate requirement to do so. As a result, the security model of a monolithic application is much more straightforward than that of an application built around microservices architecture.

1.2 Challenges of securing microservices

Mostly because of the inherent nature of microservices architecture, security is challenging. In this section, we discuss the challenges of securing microservices without discussing in detail how to overcome them. In the rest of the book, we discuss multiple ways to address these challenges.

1.2.1 The broader the attack surface, the higher the risk of attack

In a monolithic application, communication among internal components happens within a single process--in a Java application, for example, within the same Java Virtual Machine (JVM). Under microservices architecture, those internal components are designed as separate, independent microservices, and those in-process calls among internal components become remote calls. Also, each microservice now independently accepts requests or has its own entry points (see figure 1.3).

Figure 1.3 As opposed to a monolithic application with few entry points, a microservices-based application has many entry points that all must be secured.

Instead of a couple of entry points, as in a monolithic application, now you have a large number of entry points. As the number of entry points to the system increases, the attack surface broadens too. This situation is one of the fundamental challenges in building a security design for microservices. Each entry point to each microservice must be protected with equal strength. The security of a system is no stronger than the strength of its weakest link.

1.2.2 Distributed security screening may result in poor performance

Unlike in a monolithic application, each microservice in a microservices deployment has to carry out independent security screening. From the viewpoint of a monolithic application, in which the security screening is done once and the request is dispatched to the corresponding component, having multiple security screenings at the entry point of each microservice seems redundant. Also, while validating requests at each microservice, you may need to connect to a remote security token service (STS). These repetitive, distributed security checks and remote connections could contribute heavily to latency and considerably degrade the performance of the system.

Some do work around this by simply trusting the network and avoiding security checks at each and every microservice. Over time, trust-the-network has become an antipattern, and the industry is moving toward zero-trust networking principles. With zero-trust networking principles, you carry out security much closer to each resource in your network. Any microservices security design must take overall performance into consideration and must take precautions to address any drawbacks.

1.2.3 Deployment complexities make bootstrapping trust among microservices a nightmare

Security aside, how hard would it be to manage 10, 15, or hundreds of independent microservices instead of one monolithic application in a deployment? We have even started seeing microservices deployments with thousands of services talking to each other.

Capital One, one of the leading financial institutions in the United States, announced in July 2019 that its microservices deployment consists of thousands of microservices on several thousands of containers, with thousands of Amazon Elastic Compute Cloud (EC2) instances. Monzo, another financial institution based in the United Kingdom, recently mentioned that it has more than 1,500 services running in its microservices deployment. Jack Kleeman, a backend engineer at Monzo, explains in a blog (http://mng.bz/gyAx) how they built network isolation for 1,500 services to make Monzo more secure. The bottom line is, large-scale microservices deployments with thousands of services have become a reality.

Managing a large-scale microservices deployment with thousands of services would be extremely challenging if you didn’t know how to automate. If the microservices concept had popped up at a time when the concept of containers didn’t exist, few people or organizations would have the guts to adopt microservices. Fortunately, things didn’t happen that way, and that’s why we believe that microservices and containers are a match made in heaven. If you’re new to containers or don’t know what Docker is, think of containers as a way to make software distribution and deployment hassle-free. Microservices and containers (Docker) were born at the right time to complement each other nicely. We talk about containers and Docker later in the book, in chapter 10.

Does the deployment complexity of microservices architecture make security more challenging? We’re not going to delve deep into the details here, but consider one simple example. Service-to-service communication happens among multiple microservices. Each of these communication channels must be protected. You have many options (which we discuss in detail in chapters 6 and 7), but suppose that you use certificates.

Now each microservice must be provisioned with a certificate (and the corresponding private key), which it will use to authenticate itself to another microservice during service-to-service interactions. The recipient microservice must know how to validate the certificate associated with the calling microservice. Therefore, you need a way to bootstrap trust between microservices. Also, you need to be able to revoke certificates (in case the corresponding private key gets compromised) and rotate certificates (change the certificates periodically to minimize any risks in losing the keys unknowingly). These tasks are cumbersome, and unless you find a way to automate them, they’ll be tedious in a microservices deployment.

1.2.4 Requests spanning multiple microservices are harder to trace

Observability is a measure of what you can infer about the internal state of a system based on its external outputs. Logs, metrics, and traces are known as the three pillars of observability.

A log can be any event you record that corresponds to a given service. A log, for example, can be an audit trail that says that the Order Processing microservice accessed the Inventory microservice to update the inventory on April 15th, 2020, at 10:15.12 p.m. on behalf of the user Peter.

Aggregating a set of logs can produce metrics. In a way, metrics reflect the state of the system. In terms of security, the average invalid access requests per hour is a metric, for example. A high number probably indicates that the system is under attack or the first-level defense is weak. You can configure alerts based on metrics. If the number of invalid access attempts for a given microservice goes beyond a preset threshold value, the system can trigger an alert.

Traces are also based on logs but provide a different perspective of the system. Tracing helps you track a request from the point where it enters the system to the point where it leaves the system. This process becomes challenging in a microservices deployment. Unlike in a monolithic application, a request to a microservices deployment may enter the system via one microservice and span multiple microservices before it leaves the system.

Correlating requests among microservices is challenging, and you have to rely on distributed tracing systems like Jaeger and Zipkin. In chapter 5, we discuss how to use Prometheus and Grafana to monitor all the requests coming to a microservices deployment.

1.2.5 Immutability of containers challenges how you maintain service credentials and access-control policies

A server that doesn’t change its state after it spins up is called an immutable server. The most popular deployment pattern for microservices is container based. (We use the terms container and Docker interchangeably in this book, and in this context, both terms have the same meaning.) Each microservice runs in its own container, and as a best practice, the container has to be an immutable server.3 In other words, after the container has spun up, it shouldn’t change any of the files in its filesystem or maintain any runtime state within the container itself.

The whole purpose of expecting servers to be immutable in a microservices deployment is to make deployment clean and simple. At any point, you can kill a running container and create a new one with the base configuration without worrying about runtime data. If the load on a microservice is getting high, for example, you need more server instances to scale horizontally. Because none of the running server instances maintains any runtime state, you can simply spin up a new container to share the load.

What impact does immutability have on security, and why do immutable servers make microservices security challenging? In microservices security architecture, a microservice itself becomes a security enforcement point.4 As a result, you need to maintain a list of allowed clients (probably other microservices) that can access the given microservice, and you need a set of access-control policies.

These lists aren’t static; both the allowed clients and access-control policies get updated. With an immutable server, you can’t maintain such updates in the server’s filesystem. You need a way to get all the updated policies from some sort of policy administration endpoint at server bootup and then update them dynamically in memory, following a push or pull model. In the push model, the policy administration endpoint pushes policy updates to the interested microservices (or security enforcement points). In the pull model, each microservice has to poll the policy administration endpoint periodically for policy updates. Section 1.5.2 explains in detail service-level authorization.

Each microservice also has to maintain its own credentials, such as certificates. For better security, these credentials need to be rotated periodically. It’s fine to keep them with the microservice itself (in the container filesystem), but you should have a way to inject them into the microservice at the time it boots up. With immutable servers, maybe this process can be part of the continuous delivery pipeline, without baking the credentials into the microservice itself.

1.2.6 The distributed nature of microservices makes sharing user context harder

In a monolithic application, all internal components share the same web session, and anything related to the requesting party (or user) is retrieved from it. In microservices architecture, you don’t enjoy that luxury. Nothing is shared among microservices (or only a very limited set of resources), and the user context has to be passed explicitly from one microservice to another. The challenge is to build trust between two microservices so that the receiving microservice accepts the user context passed from the calling microservice. You need a way to verify that the user context passed among microservices isn’t deliberately modified.5

Using a JSON Web Token (JWT) is one popular way to share user context among microservices; we explore this technique in chapter 7. For now, you can think of a JWT as a JSON message that helps carry a set of user attributes from one microservice to another in a cryptographically safe manner.

1.2.7 Polyglot architecture demands more security expertise on each development team

In a microservices deployment, services talk to one another over the network. They depend not on each service’s implementation, but on the service interface. This situation permits each microservice to pick its own programming language and the technology stack for implementation. In a multiteam environment, in which each team develops its own set of microservices, each team has the flexibility to pick the optimal technology stack for its requirements. This architecture, which enables the various components in a system to pick the technology stack that is best for them, is known as a polyglot architecture.

A polyglot architecture makes security challenging. Because different teams use different technology stacks for development, each team has to have its own security experts. These experts should take responsibility for defining security best practices and guidelines, research security tools for each stack for static code analysis and dynamic testing, and integrate those tools into the build process. The responsibilities of a centralized, organization-wide security team are now distributed among different teams. In most cases, organizations use a hybrid approach, with a centralized security team and security-focused engineers on each team who build microservices.

1.3 Key security fundamentals

Adhering to fundamentals is important in any security design. There’s no perfect or unbreakable security. How much you should worry about security isn’t only a technical decision, but also an economic decision. There’s no point in having a burglar-alarm system to secure an empty garage, for example. The level of security you need depends on the assets you intend to protect. The security design of an e-commerce application could be different from that of a financial application.

In any case, adhering to security fundamentals is important. Even if you don’t foresee some security threats, following the fundamentals helps you protect your system against such threats. In this section, we walk you through key security fundamentals and show you how they’re related to microservices security.

1.3.1 Authentication protects your system against spoofing

Authentication is the process of identifying the requesting party to protect your system against spoofing. The requesting party can be a system (a microservice) or a system requesting access on behalf of a human user or another system (see figure 1.4). It’s rather unlikely that a human user will access a microservice directly, though. Before creating a security design for a given system, you need to identify the audience. The authentication method you pick is based on the audience.

Figure 1.4 A system (for example, a web/mobile application), just by being itself or on behalf of a human user or another system, can access microservices via an API gateway.

If you’re worried about a system accessing a microservice on behalf of a human user, you need to think about how to authenticate the system as well as the human user. In practice, this can be a web application, which is accessing a microservice, on behalf of a human user who logs into the web application. In these kinds of delegated use cases, in which a system requests access on behalf of another system or a human user, OAuth 2.0 is the de facto standard for security. We discuss OAuth 2.0 in detail in appendix A.

To authenticate the human user to a system (for example, a web application), you could request the username and password with another factor for multifactor authentication (MFA). Whether MFA is required is mostly a business decision, based on how critical your business assets are or how sensitive the data you want to share with users. The most popular form of MFA is the one-time passcode (OTP) sent over SMS. Even though it’s not the best method in terms of security, it’s the most usable form of MFA, mostly because a large portion of the world population has access to mobile phones (which don’t necessarily need to be smartphones). MFA helps reduce account breaches by almost 99.99%.6 Much stronger forms of MFA include biometrics, certificates, and Fast Identity Online (FIDO).

You have multiple ways to authenticate a system. The most popular options are certificates and JWTs. We discuss both these options in detail, with a set of examples, in chapters 6 and 7.

1.3.2 Integrity protects your system from data tampering

When you transfer data from your client application to a microservice or from one microservice to another microservice--depending on the strength of the communication channel you pick--an intruder could intercept the communication and change the data for their advantage. If the channel carries data corresponding to an order, for example, the intruder could change its shipping address to their own. Systems protected for integrity don’t ignore this possibility; they introduce measures so that if a message is altered, the recipient can detect and discard the request.

The most common way to protect a message for integrity is to sign it. Any data in transit over a communication channel protected with Transport Layer Security (TLS), for example, is protected for integrity. If you use HTTPS for communications among microservices (that communication is, in fact, HTTP over TLS), your messages are protected for integrity while in transit.

Along with the data in transit, the data at rest must be protected for integrity. Of all your business data, audit trails matter most for integrity checks. An intruder who gets access to your system would be happiest if they could modify your audit trails to wipe out any evidence. In a microservices deployment based on containers, audit logs aren’t kept at each node that runs the microservice; they’re published in some kind of a distributed tracing system like Jaeger or Zipkin. You need to make sure that the data maintained in those systems is protected for integrity.

One way is to periodically calculate the message digests of audit trails, encrypt them, and store them securely. In a research paper, Gopalan Sivathanu, Charles P. Wright, and Erez Zadok of Stony Brook University highlight the causes of integrity violations in storage and present a survey of available integrity assurance techniques.7 The paper explains several interesting applications of storage integrity checking; apart from security it also discusses implementation issues associated with those techniques.

1.3.3 Nonrepudiation: Do it once, and you own it forever

Nonrepudiation is an important aspect of information security that prevents you from denying anything you’ve done or committed. Consider a real-world example. When you lease an apartment, you agree to terms and conditions with the leasing company. If you leave the apartment before the end of the lease, you’re obliged to pay the rent for the remaining period or find another tenant to sublease the apartment. All the terms are in the leasing agreement, which you accept by signing it. After you sign it, you can’t dispute the terms and conditions to which you agreed. That’s nonrepudiation in the real world. It creates a legal obligation. Even in the digital world, a signature helps you achieve nonrepudiation; in this case, you use a digital signature.

In an e-commerce application, for example, after a customer places an order, the Order Processing microservice has to talk to the Inventory microservice to update inventory. If this transaction is protected for nonrepudiation, the Order Processing microservice can’t later deny that it updated inventory. If the Order Processing microservice signs a transaction with its private key, it can’t deny later that the transaction was initiated from that microservice. With a digital signature, only the owner of the corresponding private key can generate the same signature; so make sure that you never lose the key!

Validating the signature alone doesn’t help you achieve nonrepudiation, however. You also need to make sure that you record transactions along with the timestamp and the signature--and maintain those records for a considerable amount of time. In case the initiator disputes a transaction later, you’ll have it in your records.

1.3.4 Confidentiality protects your systems from unintended information disclosure

When you send order data from a client application to the Order Processing microservice, you expect that no party can view the data other than the Order Processing microservice itself. But based on the strength of the communication channel you pick, an intruder can intercept the communication and get hold of the data. Along with the data in transit, the data at rest needs to be protected for confidentiality (see figure 1.5). An intruder who gets access to your data storage or backups has direct access to all your business-critical data unless you’ve protected it for confidentiality.

Figure 1.5 To protect a system for confidentiality, both the data in transit and at rest must be protected. The data in transit can be protected with TLS, and data at rest can be protected by encryption.

Data in transit

Encryption helps you achieve confidentiality. A cryptographic operation makes sure that the encrypted data is visible only to the intended recipient. TLS is the most popular way of protecting data for confidentiality in transit. If one microservice talks to another over HTTPS, you’re using TLS underneath, and only the recipient microservice will be able to view the data in cleartext.

Then again, the protection provided by TLS is point to point. At the point where the TLS connection terminates, the security ends. If your client application connects to a microservice over a proxy server, your first TLS connection terminates at the proxy server, and a new TLS connection is established between the proxy server and the microservice. The risk is that anyone who has access to the proxy server can log the messages in cleartext as soon as the data leaves the first connection.

Most proxy servers support two modes of operation with respect to TLS: TLS bridging and TLS tunneling. TLS bridging terminates the first TLS connection at the proxy server, and creates a new TLS connection between the proxy server and the next destination of the message. If your proxy server uses TLS bridging, don’t trust it and possibly put your data at risk, even though you use TLS (or HTTPS). If you use TLS bridging, the messages are in cleartext while transiting through the proxy server. TLS tunneling creates a tunnel between your client application and the microservices, and no one in the middle will be able to see what’s going through, not even the proxy server. If you are interested in reading more about TLS, we recommend having a look at SSL and TLS: Designing and Building Secure Systems by Eric Rescorla (Addison-Wesley Professional, 2000).

NOTE Encryption has two main flavors: public-key encryption and symmetric-key encryption. With public-key encryption, the data is encrypted using the recipient’s public key, and only the party who owns the corresponding private key can decrypt the message and see what’s in it. With symmetric-key encryption, the data is encrypted with a key known to both the sender and the recipient. TLS uses both flavors. Symmetric-key encryption is used to encrypt the data, while public-key encryption is used to encrypt the key used in symmetric-key encryption. If you are interested in reading more about encryption and cryptography, we recommend having a look at Real-World Cryptography by David Wong (Manning, to be published in 2021).

Data at rest

Encryption should also apply to data at rest to protect it from intruders who get direct access to the system. This data can be credentials for other systems stored in the filesystem or business-critical data stored in a database. Most database management systems provide features for automatic encryption, and disk-level encryption features are available at the operating-system level. Application-level encryption is another option, in which the application itself encrypts the data before passing it over to the filesystem or to a database.

Of all these options, the one that best fits your application depends on the criticality of your business operations. Also keep in mind that encryption is a resource-intensive operation that would have considerable impact on your application’s performance unless you find the optimal solution.8

1.3.5 Availability: Keep the system running, no matter what

The whole point of building any kind of a system is to make it available to its users. Every minute (or even second) that the system is down, your business loses money. Amazon was down for 20 minutes in March 2016, and the estimated revenue loss was $3.75 million. In January 2017, more than 170 Delta Airlines flights were canceled because of a system outage, which resulted in an estimated loss of $8.5 million.

It’s not only the security design of a system that you need to worry about to keep a system up and running, but also the overall architecture. A bug in the core functionality of an application can take the entire system down. To some extent, these kinds of situations are addressed in the core design principles of microservices architecture. Unlike in monolithic applications, in a microservices deployment, the entire system won’t go down if a bug is found in one component or microservice. Only that microservice will go down; the rest should be able to function.

Of all the factors that can take a system down, security