Deep dive networking
•Download as PPTX, PDF•
0 likes•172 views
This document summarizes the configuration and operation of Flannel CNI on a Kubernetes cluster. It shows logs from the Flannel pod running on the controller node, which sets up the overlay network and subnet leasing. It also shows logs from Flannel pods on worker nodes, which join the overlay network and configure iptables rules and IP masquerading for pod networking. The logs demonstrate how Flannel establishes connectivity between pods on different nodes using host-gateway mode.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Deep dive networking
Similar to Deep dive networking (20)
More from Victor Morales
More from Victor Morales (20)
Recently uploaded
Recently uploaded (20)
Deep dive networking
- 1. Deep dive K8s Networking Victor Morales
- 2. Victor Morales • +15 yrs as a Software Engineer • .NET, Java, python, Go programmer • OpenStack, OPNFV, ONAP and CNCF contributor. https://about.me/electrocucaracha
- 3. Multicore Crisis Named by Bob “SmoothSpan” Warfield in 2007, the situation in which the effects of Moore’s Law have changed: while the doubling of transistors per chip continues, the by-product is no longer faster processor speeds but more cores per chips instead. https://smoothspan.com/2007/09/06/a-picture-of-the-multicore-crisis/
- 5. Fallacies of distributed computing 1. The network is reliable 2. Latency is zero 3. Bandwidth is infinite 4. The network is secure 5. Topology doesn't change 6. There is one administrator 7. Transport cost is zero 8. The network is homogeneous https://blogs.oracle.com/developers/fallacies-of-distributed-systems
- 7. The Kubernetes network model • pods on a node can communicate with all pods on all nodes without NAT • agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node • pods in the host network of a node can communicate with all pods on all nodes without NAT https://kubernetes.io/docs/concepts/cluster- administration/networking/#the-kubernetes- network-model https://mrscriptkiddie.com/ what-is-network-address- translationnat-working- explained/
- 8. Kubernetes Demo Cluster controller eth0 eth1 10.0.2.184/24 10.10.16.3/24 worker01 eth0 eth1 10.0.2.254/24 10.10.16.4/24 management 10.0.16.0/24 administration 10.0.2.0/24 worker02 eth0 eth1 10.0.2.83/24 10.10.16.5/24 Flannel CNI (Host Gateway Backend)
- 9. Flannel CNI (controller) $ kubectl logs $(kubectl get pods -A -o wide | grep "flannel.*controller" | awk '{ print $2}') -n kube-system I0503 23:08:17.135980 1 main.go:518] Determining IP address of default interface I0503 23:08:17.136722 1 main.go:531] Using interface with name eth0 and address 10.0.2.56 I0503 23:08:17.136731 1 main.go:548] Defaulting external address to interface address (10.0.2.56) W0503 23:08:17.136748 1 client_config.go:517] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0503 23:08:17.143884 1 kube.go:119] Waiting 10m0s for node controller to sync I0503 23:08:17.143910 1 kube.go:306] Starting kube subnet manager I0503 23:08:18.144214 1 kube.go:126] Node controller sync successful I0503 23:08:18.144280 1 main.go:246] Created subnet manager: Kubernetes Subnet Manager - controller I0503 23:08:18.144295 1 main.go:249] Installing signal handlers I0503 23:08:18.144431 1 main.go:390] Found network config - Backend type: host-gw I0503 23:08:18.195402 1 main.go:355] Current network or subnet (10.233.64.0/18, 10.233.64.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules I0503 23:08:18.225336 1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN I0503 23:08:18.228056 1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully I0503 23:08:18.229526 1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN I0503 23:08:18.230803 1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j MASQUERADE --random-fully I0503 23:08:18.232086 1 main.go:305] Setting up masking rules I0503 23:08:18.232959 1 main.go:313] Changing default FORWARD chain policy to ACCEPT I0503 23:08:18.233059 1 main.go:321] Wrote subnet file to /run/flannel/subnet.env I0503 23:08:18.233069 1 main.go:325] Running backend. I0503 23:08:18.233082 1 main.go:343] Waiting for all goroutines to exit I0503 23:08:18.233114 1 route_network.go:53] Watching for new subnet leases I0503 23:08:18.234944 1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules I0503 23:08:18.234958 1 iptables.go:167] Deleting iptables rule: -s 10.233.64.0/18 -d 10.233.64.0/18 -j RETURN I0503 23:08:18.235938 1 iptables.go:167] Deleting iptables rule: -s 10.233.64.0/18 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully I0503 23:08:18.237037 1 iptables.go:167] Deleting iptables rule: ! -s 10.233.64.0/18 -d 10.233.64.0/24 -j RETURN I0503 23:08:18.238218 1 iptables.go:167] Deleting iptables rule: ! -s 10.233.64.0/18 -d 10.233.64.0/18 -j MASQUERADE --random-fully I0503 23:08:18.239041 1 iptables.go:155] Adding iptables rule: -s 10.233.64.0/18 -d 10.233.64.0/18 -j RETURN I0503 23:08:18.240543 1 iptables.go:155] Adding iptables rule: -s 10.233.64.0/18 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully I0503 23:08:18.242157 1 iptables.go:155] Adding iptables rule: ! -s 10.233.64.0/18 -d 10.233.64.0/24 -j RETURN I0503 23:08:18.243441 1 iptables.go:155] Adding iptables rule: ! -s 10.233.64.0/18 -d 10.233.64.0/18 -j MASQUERADE --random-fully I0503 23:08:18.245337 1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules I0503 23:08:18.245348 1 iptables.go:167] Deleting iptables rule: -s 10.233.64.0/18 -j ACCEPT I0503 23:08:18.245946 1 iptables.go:167] Deleting iptables rule: -d 10.233.64.0/18 -j ACCEPT I0503 23:08:18.246539 1 iptables.go:155] Adding iptables rule: -s 10.233.64.0/18 -j ACCEPT I0503 23:08:18.249302 1 iptables.go:155] Adding iptables rule: -d 10.233.64.0/18 -j ACCEPT I0503 23:08:20.442619 1 route_network.go:85] Subnet added: 10.233.65.0/24 via 10.0.2.14 I0503 23:08:20.466725 1 route_network.go:85] Subnet added: 10.233.66.0/24 via 10.0.2.254 controller eth0 10.0.2.56/24 Flannel Overlay network: 10.233.64.0/18 Pod’s Subnet: 10.233.64.0/24
- 10. Flannel CNI (worker01) $ kubectl logs $(kubectl get pods -A -o wide | grep "flannel.*worker01" | awk '{ print $2}') -n kube-system I0503 23:08:19.426175 1 main.go:518] Determining IP address of default interface I0503 23:08:19.426440 1 main.go:531] Using interface with name eth0 and address 10.0.2.254 I0503 23:08:19.426455 1 main.go:548] Defaulting external address to interface address (10.0.2.254) W0503 23:08:19.426470 1 client_config.go:517] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0503 23:08:19.433617 1 kube.go:119] Waiting 10m0s for node controller to sync I0503 23:08:19.433948 1 kube.go:306] Starting kube subnet manager I0503 23:08:20.433999 1 kube.go:126] Node controller sync successful I0503 23:08:20.434025 1 main.go:246] Created subnet manager: Kubernetes Subnet Manager - worker01 I0503 23:08:20.434030 1 main.go:249] Installing signal handlers I0503 23:08:20.434090 1 main.go:390] Found network config - Backend type: host-gw I0503 23:08:20.448682 1 main.go:355] Current network or subnet (10.233.64.0/18, 10.233.66.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules I0503 23:08:20.484395 1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN I0503 23:08:20.485015 1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully I0503 23:08:20.485531 1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN I0503 23:08:20.486022 1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j MASQUERADE --random-fully I0503 23:08:20.486554 1 main.go:305] Setting up masking rules I0503 23:08:20.486985 1 main.go:313] Changing default FORWARD chain policy to ACCEPT I0503 23:08:20.487040 1 main.go:321] Wrote subnet file to /run/flannel/subnet.env I0503 23:08:20.487045 1 main.go:325] Running backend. I0503 23:08:20.487052 1 main.go:343] Waiting for all goroutines to exit I0503 23:08:20.487064 1 route_network.go:53] Watching for new subnet leases I0503 23:08:20.487236 1 route_network.go:85] Subnet added: 10.233.64.0/24 via 10.0.2.56 I0503 23:08:20.487378 1 route_network.go:85] Subnet added: 10.233.65.0/24 via 10.0.2.14 I0503 23:08:20.488369 1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules I0503 23:08:20.488376 1 iptables.go:167] Deleting iptables rule: -s 10.233.64.0/18 -d 10.233.64.0/18 -j RETURN I0503 23:08:20.496685 1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules I0503 23:08:20.496696 1 iptables.go:167] Deleting iptables rule: -s 10.233.64.0/18 -j ACCEPT I0503 23:08:20.496766 1 iptables.go:167] Deleting iptables rule: -s 10.233.64.0/18 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully I0503 23:08:20.497367 1 iptables.go:167] Deleting iptables rule: -d 10.233.64.0/18 -j ACCEPT I0503 23:08:20.497888 1 iptables.go:155] Adding iptables rule: -s 10.233.64.0/18 -j ACCEPT I0503 23:08:20.498893 1 iptables.go:167] Deleting iptables rule: ! -s 10.233.64.0/18 -d 10.233.66.0/24 -j RETURN I0503 23:08:20.499455 1 iptables.go:167] Deleting iptables rule: ! -s 10.233.64.0/18 -d 10.233.64.0/18 -j MASQUERADE --random-fully I0503 23:08:20.500091 1 iptables.go:155] Adding iptables rule: -s 10.233.64.0/18 -d 10.233.64.0/18 -j RETURN I0503 23:08:20.501584 1 iptables.go:155] Adding iptables rule: -d 10.233.64.0/18 -j ACCEPT I0503 23:08:20.502204 1 iptables.go:155] Adding iptables rule: -s 10.233.64.0/18 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully I0503 23:08:20.508007 1 iptables.go:155] Adding iptables rule: ! -s 10.233.64.0/18 -d 10.233.66.0/24 -j RETURN I0503 23:08:20.509166 1 iptables.go:155] Adding iptables rule: ! -s 10.233.64.0/18 -d 10.233.64.0/18 -j MASQUERADE --random-fully worker01 eth0 10.0.2.254/24 Flannel Overlay network: 10.233.64.0/18 Pod’s Subnet: 10.233.66.0/24
- 11. Flannel CNI (worker02) $ kubectl logs $(kubectl get pods -A -o wide | grep "flannel.*worker02" | awk '{ print $2}') -n kube-system I0503 23:08:19.395547 1 main.go:518] Determining IP address of default interface I0503 23:08:19.395834 1 main.go:531] Using interface with name eth0 and address 10.0.2.14 I0503 23:08:19.395849 1 main.go:548] Defaulting external address to interface address (10.0.2.14) W0503 23:08:19.395859 1 client_config.go:517] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0503 23:08:19.403033 1 kube.go:119] Waiting 10m0s for node controller to sync I0503 23:08:19.403392 1 kube.go:306] Starting kube subnet manager I0503 23:08:20.403454 1 kube.go:126] Node controller sync successful I0503 23:08:20.403478 1 main.go:246] Created subnet manager: Kubernetes Subnet Manager - worker02 I0503 23:08:20.403482 1 main.go:249] Installing signal handlers I0503 23:08:20.403525 1 main.go:390] Found network config - Backend type: host-gw I0503 23:08:20.424202 1 main.go:355] Current network or subnet (10.233.64.0/18, 10.233.65.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules I0503 23:08:20.439644 1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN I0503 23:08:20.440832 1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully I0503 23:08:20.441848 1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN I0503 23:08:20.442882 1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j MASQUERADE --random-fully I0503 23:08:20.443786 1 main.go:305] Setting up masking rules I0503 23:08:20.444598 1 main.go:313] Changing default FORWARD chain policy to ACCEPT I0503 23:08:20.444686 1 main.go:321] Wrote subnet file to /run/flannel/subnet.env I0503 23:08:20.444695 1 main.go:325] Running backend. I0503 23:08:20.444706 1 main.go:343] Waiting for all goroutines to exit I0503 23:08:20.444729 1 route_network.go:53] Watching for new subnet leases I0503 23:08:20.444979 1 route_network.go:85] Subnet added: 10.233.64.0/24 via 10.0.2.56 I0503 23:08:20.445165 1 route_network.go:85] Subnet added: 10.233.66.0/24 via 10.0.2.254 I0503 23:08:20.446388 1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules I0503 23:08:20.446397 1 iptables.go:167] Deleting iptables rule: -s 10.233.64.0/18 -d 10.233.64.0/18 -j RETURN I0503 23:08:20.446864 1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules I0503 23:08:20.446872 1 iptables.go:167] Deleting iptables rule: -s 10.233.64.0/18 -j ACCEPT I0503 23:08:20.447285 1 iptables.go:167] Deleting iptables rule: -s 10.233.64.0/18 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully I0503 23:08:20.447701 1 iptables.go:167] Deleting iptables rule: -d 10.233.64.0/18 -j ACCEPT I0503 23:08:20.448317 1 iptables.go:167] Deleting iptables rule: ! -s 10.233.64.0/18 -d 10.233.65.0/24 -j RETURN I0503 23:08:20.448802 1 iptables.go:155] Adding iptables rule: -s 10.233.64.0/18 -j ACCEPT I0503 23:08:20.449289 1 iptables.go:167] Deleting iptables rule: ! -s 10.233.64.0/18 -d 10.233.64.0/18 -j MASQUERADE --random-fully I0503 23:08:20.450172 1 iptables.go:155] Adding iptables rule: -s 10.233.64.0/18 -d 10.233.64.0/18 -j RETURN I0503 23:08:20.450643 1 iptables.go:155] Adding iptables rule: -d 10.233.64.0/18 -j ACCEPT I0503 23:08:20.477083 1 iptables.go:155] Adding iptables rule: -s 10.233.64.0/18 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully I0503 23:08:20.478562 1 iptables.go:155] Adding iptables rule: ! -s 10.233.64.0/18 -d 10.233.65.0/24 -j RETURN I0503 23:08:20.480024 1 iptables.go:155] Adding iptables rule: ! -s 10.233.64.0/18 -d 10.233.64.0/18 -j MASQUERADE --random-fully worker02 eth0 10.0.2.14/24 Flannel Overlay network: 10.233.64.0/18 Pod’s Subnet: 10.233.65.0/24
- 13. Demo 1 worker01 eth0 pod1 10.233.66.4/24 eth0 client pod2 eth0 server 10.233.66.5/24 $ kubectl get pods -o custom-columns="NAME:metadata.name,IP:status.podIP,NODE:spec.nodeName" NAME IP NODE pod1 10.233.66.4 worker01 pod2 10.233.66.5 worker01
- 14. ssh worker01 (PIDs/net ns) worker01 eth0 pod1 10.233.66.4/24 eth0 client pause pod2 10.233.66.5/24 server pause $ for name in $(sudo docker ps --filter "name=pod*" --format "{{.Names}}"); do > echo "NAME:$name $(sudo docker inspect $name --format 'PID:{{.State.Pid}} CMD:{{.Path}}')" > done NAME:k8s_server_pod2_default_7e0da6ce-2693-448a-9def-55e25a53a9f8_0 PID:23231 CMD:sleep NAME:k8s_client_pod1_default_7f3063da-2e86-466e-812b-b45b8791af60_0 PID:23173 CMD:sleep NAME:k8s_POD_pod2_default_7e0da6ce-2693-448a-9def-55e25a53a9f8_0 PID:22936 CMD:/pause NAME:k8s_POD_pod1_default_7f3063da-2e86-466e-812b-b45b8791af60_0 PID:22902 CMD:/pause $ sudo lsns --type net NS TYPE NPROCS PID USER COMMAND 4026531993 net 139 1 root /sbin/init 4026532219 net 1 892 root /usr/sbin/haveged --Foreground --verbose=1 -w 1024 4026532321 net 2 17306 root /pause 4026532401 net 2 17675 root /pause 4026532481 net 2 22902 root /pause 4026532549 net 2 22936 root /pause 10.0.2.254/24 eth0 worker01 4026532481 10.233.66.4/24 eth0 23173 22902 4026532549 10.233.66.5/24 23231 22936 eth0 4026531993 eth0 10.0.2.254/24
- 15. ssh worker01 (veths) $ sudo nsenter -t 23173 -n ip a s eth0 3: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 22:3d:13:16:4d:91 brd ff:ff:ff:ff:ff:ff inet 10.233.65.5/24 brd 10.233.66.255 scope global eth0 valid_lft forever preferred_lft forever $ sudo nsenter -t 23231 -n ip a s eth0 3: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 4e:45:9e:07:aa:73 brd ff:ff:ff:ff:ff:ff inet 10.233.65.6/24 brd 10.233.66.255 scope global eth0 valid_lft forever preferred_lft forever $ ip add show vethd2ff1e8f 10: vethd2ff1e8f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cni0 state UP group default link/ether de:71:37:62:8c:81 brd ff:ff:ff:ff:ff:ff link-netnsid 2 $ ip add show vethedf4daba 11: vethedf4daba@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cni0 state UP group default link/ether 6e:38:5d:c3:1c:fe brd ff:ff:ff:ff:ff:ff link-netnsid 3 worker01 4026532481 10.233.66.4/24 eth0@if10 23173 22902 4026532549 10.233.66.5/24 23231 22936 4026531993 eth0 10.0.2.254/24 eth0@if11 vethd2ff1e8f vethedf4daba de:71:37:62:8c:81 6e:38:5d:c3:1c:fe
- 16. ssh worker01 (bridge) worker01 4026532481 10.233.66.4/24 eth0@if10 23173 22902 4026532549 10.233.66.5/24 23231 22936 4026531993 eth0 10.0.2.254/24 eth0@if11 vethd2ff1e8f vethedf4daba cni0 $ ip addr show cni0 7: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 2e:c5:8a:c4:ad:59 brd ff:ff:ff:ff:ff:ff inet 10.233.66.1/24 brd 10.233.66.255 scope global cni0 valid_lft forever preferred_lft forever $ brctl show cni0 bridge name bridge id STP enabled interfaces cni0 8000.1af5b56bb7bd no veth08082f56 vethd2ff1e8f vethe6c8889e vethedf4daba $ brctl showmacs cni0 port no mac addr is local? ageing timer 1 12:ec:de:82:22:e9 yes 0.00 1 12:ec:de:82:22:e9 yes 0.00 2 42:ed:db:49:ce:24 yes 0.00 2 42:ed:db:49:ce:24 yes 0.00 4 6e:38:5d:c3:1c:fe yes 0.00 4 6e:38:5d:c3:1c:fe yes 0.00 2 9e:93:b1:9b:10:1f no 1.16 3 de:71:37:62:8c:81 yes 0.00 3 de:71:37:62:8c:81 yes 0.00 1 ea:56:ef:33:2a:9a no 4.16 de:71:37:62:8c:81 6e:38:5d:c3:1c:fe 10.233.66.1/24
- 17. ssh worker01 (routes) worker01 4026532481 10.233.66.4/24 eth0@if10 23173 22902 4026532549 10.233.66.5/24 23231 22936 4026531993 eth0 10.0.2.254/24 eth0@if11 vethd2ff1e8f vethedf4daba cni0 $ sudo nsenter -t 23173 -n ping -c 1 10.233.66.5 PING 10.233.66.5 (10.233.66.5) 56(84) bytes of data. 64 bytes from 10.233.66.5: icmp_seq=1 ttl=64 time=0.154 ms --- 10.233.66.5 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.154/0.154/0.154/0.000 ms $ sudo nsenter -t 23173 -n ip route default via 10.233.66.1 dev eth0 10.233.64.0/18 via 10.233.66.1 dev eth0 10.233.66.0/24 dev eth0 proto kernel scope link src 10.233.66.4 $ ip route default via 10.0.2.1 dev eth0 proto dhcp src 10.0.2.254 metric 100 10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.254 10.0.2.1 dev eth0 proto dhcp scope link src 10.0.2.254 metric 100 10.10.16.0/24 dev eth1 proto kernel scope link src 10.10.16.4 10.233.64.0/24 via 10.0.2.56 dev eth0 10.233.65.0/24 via 10.0.2.14 dev eth0 10.233.66.0/24 dev cni0 proto kernel scope link src 10.233.66.1 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 10.233.66.1/24
- 19. Demo 2 worker01 eth0 pod1 10.233.66.6/24 eth0 client $ kubectl get pods -o custom-columns="NAME:metadata.name,IP:status.podIP,NODE:spec.nodeName" NAME IP NODE pod1 10.233.66.6 worker01 pod2 10.233.65.5 worker02 worker02 eth0 pod2 10.233.65.5/24 eth0 server
- 20. ssh worker01 (routes) worker01 4026532481 10.233.66.6/24 eth0@if12 24330 24200 4026531993 eth0 10.0.2.254/24 veth24e469fc cni0 $ sudo nsenter -t 24330 -n traceroute 10.233.65.5 traceroute to 10.233.65.5 (10.233.65.5), 30 hops max, 60 byte packets 1 10.233.66.1 (10.233.66.1) 2.161 ms 1.855 ms 1.775 ms 2 10.0.2.14 (10.0.2.14) 1.719 ms 1.603 ms 1.515 ms 3 10.233.65.5 (10.233.65.5) 1.447 ms 1.336 ms 3.424 ms $ sudo nsenter -t 23173 -n ip route default via 10.233.66.1 dev eth0 10.233.64.0/18 via 10.233.66.1 dev eth0 10.233.66.0/24 dev eth0 proto kernel scope link src 10.233.66.4 $ ip route default via 10.0.2.1 dev eth0 proto dhcp src 10.0.2.254 metric 100 10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.254 10.0.2.1 dev eth0 proto dhcp scope link src 10.0.2.254 metric 100 10.10.16.0/24 dev eth1 proto kernel scope link src 10.10.16.4 10.233.64.0/24 via 10.0.2.56 dev eth0 10.233.65.0/24 via 10.0.2.14 dev eth0 10.233.66.0/24 dev cni0 proto kernel scope link src 10.233.66.1 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown administration 10.0.2.0/24 worker02 4026532554 10.233.65.5/24 eth0@if11 22403 22281 4026531993 eth0 10.0.2.14/24 vethf6de8b6e cni0 10.233.66.1/24 10.233.65.1/24
- 22. worker01 4026532481 10.233.66.6/24 eth0@if12 24330 24200 4026531993 eth0 10.0.2.254/24 veth24e469fc cni0 administration 10.0.2.0/24 worker02 4026532554 10.233.65.5/24 eth0@if11 22403 22281 4026531993 eth0 10.0.2.14/24 vethf6de8b6e cni0 Host Gateway worker01 4026532481 10.233.66.6/24 eth0@if12 24330 24200 4026531993 eth0 10.0.2.254/24 veth24e469fc cni0 administration 10.0.2.0/24 worker02 4026532554 10.233.65.5/24 eth0@if11 22403 22281 4026531993 eth0 10.0.2.14/24 vethf6de8b6e cni0 VXLAN flannel.1 flannel.1 VXLAN tunneling
- 23. ssh worker01(routes) $ ip route default via 10.0.2.1 dev eth0 proto dhcp src 10.0.2.254 metric 100 10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.254 10.0.2.1 dev eth0 proto dhcp scope link src 10.0.2.254 metric 100 10.10.16.0/24 dev eth1 proto kernel scope link src 10.10.16.4 10.233.64.0/24 via 10.233.64.0 dev flannel.1 onlink 10.233.65.0/24 via 10.233.65.0 dev flannel.1 onlink 10.233.66.0/24 dev cni0 proto kernel scope link src 10.233.66.1 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown $ ip neigh show dev flannel.1 10.233.64.0 lladdr fe:f7:38:5b:0a:4d PERMANENT 10.233.65.0 lladdr a2:c1:bf:3d:c9:7b PERMANENT $ bridge fdb show dev flannel.1 fe:f7:38:5b:0a:4d dst 10.0.2.56 self permanent a2:c1:bf:3d:c9:7b dst 10.0.2.14 self permanent $ ip -d link show flannel.1 5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default link/ether f6:22:6b:4b:11:48 brd ff:ff:ff:ff:ff:ff promiscuity 0 vxlan id 1 local 10.0.2.207 dev eth0 srcport 0 0 dstport 8472 nolearning ttl inherit ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
- 24. Virtual eXtensible Local Area Networking (VXLAN) The VXLAN protocol is a tunnelling protocol designed to solve the problem of limited VLAN IDs (4096) in IEEE 802.1q. With VXLAN the size of the identifier is expanded to 24 bits (16777216). https://www.kernel.org/doc/html/latest/networking/vxlan.html https://www.beyondcli.com/101/vxlan-vsphere-vcns-vs-nsx-for-vsphere/
- 25. iperf results Host Gateway VXLAN Host Gateway offers 5x higher throughput than VXLAN backend.
- 26. Slow Responses https://pragprog.com/titles/mnee2/release-it-second-edition/ Let threads block for minutes before throwing exceptions. The blocked thread can’t process other transactions, so overall capacity is reduced. • Slow responses trigger Cascading Failures • For websites, slow responses cause more traffic • Consider Fail Fast • Hunt for memory leaks or resource contention
- 28. Backup
Editor's Notes
- https://superuser.openstack.org/articles/run-your-kubernetes-cluster-on-openstack-in-production/