The document provides an overview of Kubernetes networking concepts including single pod networking, pod to pod communication, service discovery and load balancing, external access patterns, network policies, Istio service mesh, multi-cluster networking, and best practices. It covers topics such as pod IP addressing, communication approaches like L2, L3, overlays, services, ingress controllers, network policies, multi-cluster use cases and deployment options.
1. Deep dive into
Kubernetes Networking
Sreenivas Makam
Partner Engineer @Google Cloud
@Container conference, 3rd Aug 2018
2. Agenda
● Single pod networking
● Pod to pod Networking(East-West traffic)
● Service discovery and load balancing
● External access(North-South traffic)
● Network policy
● Istio service mesh
● Multi-cluster and hybrid cloud
● Best practises
3. Pods
● Small group of tightly-coupled
containers
● Every pod has a unique IP
● Share same namespace and
volume
● Share same lifecycle
● Use cases for multi-container pods -
Sidecars, proxies/adapters
Example: File puller & web server
Consumers
Content
Manager
File
Puller
Web
Server
Volume
Pod
5. Communication across pods in different nodes
Kubernetes supports different networking approaches to do pod networking
Plugins can be added using CNI
Type/Features L2 L3 Overlay Cloud
Summary Pods
communicate
using L2
Pod traffic is
routed in underlay
network
Pod traffic is
encapsulated and uses
underlay for reachability
Pod traffic is routed
in cloud virtual
network
Underlying
technology
L2 arp, broadcast Routing protocol
like BGP
VXLAN Pre-programmed
fabric using
controller(eg: GKE)
Encapsulation No double
encapsulation
No double
encapsulation
Double encapsulation No double
encapsulation
Examples Calico Flannel, Weave GKE, ACS, EKS
9. Services
● A service is a group of endpoints (usually pods)
grouped by selector
● Services provide a stable VIP (except Headless)
● VIP automatically routes to backend pods
● VIP to backend pod mapping is managed by
kube-proxy, implemented using iptables
Client
Virtual IP
apiVersion: v1
kind: Service
metadata:
name: productpage
spec:
selector:
app: productpage
type: ClusterIP
ports:
- port: 80
targetPort: 8080
protocol: TCP
10. DNS
● Service Discovery achieved by SkyDNS
● SkyDNS runs as a pod in the cluster
apiserverwatch
etcd
kube2skyskyDNS
11. Service communication
● Service IP is accessible only from within the cluster
● Type “ClusterIP” is used for internal communication within the cluster
● Traffic sent to Service IP gets load balanced to pods that belong to service IP
● Source NAT is used for pods to communicate to external world
● IPtables are used extensively for load balancing and NAT
● Nodeport, Network load balancer and Ingress controller are service types for
external world to reach services inside the cluster.
12. Service external reachability options
Feature Nodeport Load balancer Ingress
Summary Service is exposed
using a reserved port in
all nodes of
cluster(Default:
32000-32767)
Typically implemented
as network load
balancer
Typically implemented
as http load balancer
IP address Node IP is used for
external communication
Each service needs to
have own external IP
Many services can
share same external IP,
uses path based demux
Use Case Demo purposes L3 services L7 services
Examples GKE Network load
balancer
GKE http load balancer,
nginx, Istio
16. Network control policy
Controls communication between Pods and Services
● Traffic is by default allowed without network policy
● Kubernetes network policies are allow policies, there is no deny policy
● Empty pod selector selects all pods
● Ingress and egress rules are available to control traffic
● Traffic is allowed if atleast 1 policy allows traffic
● Network plugins like Calico, Weavenet support network policies. GKE uses
calico
20. Istio is an open framework for connecting,
securing, managing and monitoring
services
Traffic control Observability Fault-injectionSecurity Hybrid cloud
What is Istio?
What can Istio do?
21. Istio Architecture
Traffic transparently proxied —
unaware of proxies
Pilot Mixer
Discovery & config
data to proxies
TLS certs
to proxies
Policy checks,
telemetry
Proxy
Frontend
Proxy
Payments
Citadel
Istio Control Plane
24. Multi-cluster
Use- cases:
● 1 cluster serving as DR for another cluster
● CI/CD between Dev and production clusters
● Global load balancing across multiple clusters
● Ability for 1 cluster to consume services from another cluster
Requirements:
● Common control plane across clusters
● Service discovery across clusters
● Load balancing traffic across replicas in different clusters
● Common policy management across clusters - RBAC, Quota etc
26. Multi-cluster deployment options
● Kubernetes Federated cluster
○ Have a single Kubernetes control plane that spans multiple clusters
○ Controlled using Kubefed
○ Service discovery works across clusters
● Istio Multicluster
○ Istio control plane that spans across multiple clusters
○ Kubernetes control plane limited to single cluster
○ Service discovery works across clusters
○ Istio ingress takes care of load balancing external traffic across clusters
● Managed service from cloud providers (eg: GCP Cloud services platform)
28. Kubernetes Networking - Best practises
● Use namespaces for multi-tenancy
● Use Ingress rather than Load balancer to expose
services. This is to preserve resources
● Do not expose cluster to outside world if its not needed
● Create security policies like IAM, RBAC, Network control
in the beginning rather than later
● Put emphasis on resource control including quotas,
priorities/preemption etc
29. Demo
Demo code and instructions:
https://github.com/smakam/gcp/tree/master/gke/containerconfbglr18
30. References
● The Ins and Outs of Kubernetes and GKE networking presentation
● Understanding Kubernetes Networking
● Kubernetes networking concepts
● Enforcing network policies with Kubernetes
● Building hybrid clouds with Istio
● Kubernetes Nodeport vs Loadbalancer vs Ingress
● https://github.com/thesandlord/Istio101
Note:
Some pictures used were borrowed from above blog posts