Kubernetes网络分析中如何实现Container间通信
这篇文章主要介绍Kubernetes网络分析中如何实现Container间通信,文中介绍的非常详细,具有一定的参考价值,感兴趣的小伙伴们一定要看完!
成都创新互联主要企业基础官网建设,电商平台建设,移动手机平台,重庆小程序开发公司等一系列专为中小企业按需求定制开发产品体系;应对中小企业在互联网运营的各种问题,为中小企业在互联网的运营中保驾护航。
1. 引言##
上图显示了Kubernetes的基本结构图。
Master管理多个Slave节点
Slave节点上面可以运行多个Pod
Pod可以部署多个副本,多个副本可以运行在不同的Node上
一个Pod可以包含多个Container,一个Pod内的Container共享同样的网络地址空间
最重要的是最后一句话:一个Pod内的Container共享同样的网络地址空间
。这是通过Mapped Container做到的。
2. Mapped Container##
基本描述为下:
容器 A 的网络模式为正常docker的网络模式
容器 B 的网络模式为应用容器A的网络模式
###2.1 共享网络模式###
下面是一个例子来验证,我这里创建了一个busybox的Pod。
[root@centos7-node-221 ~]$ kubectl get po NAME READY STATUS RESTARTS AGE busybox 1/1 Running 224 9d [root@centos7-node-221 ~]$ kubectl describe po busybox Name: busybox Namespace: default Image(s): busybox Node: centos7-node-226/192.168.1.226 Labels:Status: Running Reason: Message: IP: 172.16.58.6 Replication Controllers: Containers: busybox: Image: busybox State: Running Started: Thu, 08 Oct 2015 08:20:30 -0400 Last Termination State: Terminated Exit Code: 0 Started: Thu, 08 Oct 2015 07:20:26 -0400 Finished: Thu, 08 Oct 2015 08:20:26 -0400 Ready: True Restart Count: 224 Variables: Conditions: Type Status Ready True Volumes: default-token-lv94w: Type: Secret (a secret that should populate this volume) SecretName: default-token-lv94w Events: FirstSeen LastSeen Count From SubobjectPath Reason Message 9d 37m 225 {kubelet centos7-node-226} spec.containers{busybox} pulled Container image "busybox" already present on machine 37m 37m 1 {kubelet centos7-node-226} spec.containers{busybox} Created Created with docker id fc8580292210 37m 37m 1 {kubelet centos7-node-226} spec.containers{busybox} Started Started with docker id fc8580292210
我们去192.168.1.226看下这个Pod和其Container.
[root@centos7-node-226 ~]$ docker ps | grep busybox fc8580292210 busybox "sleep 3600" 37 minutes ago Up 37 minutes k8s_busybox.62fa0587_busybox_default_86e98e8c-665f-11e5-af98-525400d7abb6_7f734c4d 02d259dc8ab5 gcr.io/google_containers/pause:0.8.0 "/pause" 9 days ago Up 9 days k8s_POD.7be6d81d_busybox_default_86e98e8c-665f-11e5-af98-525400d7abb6_ff9224f5
发现有两个容器,一个是pause容器,一个是busybox容器。其中pause容器为主网络容器,其他容器都共享pause容器的网络模式。我们分别看下其网络模式。下面是两个容器的网络模式。
[root@centos7-node-226 ~]$ docker inspect 02d259dc8ab5 | grep NetworkMode "NetworkMode": "bridge", [root@centos7-node-226 ~]$ docker inspect fc8580292210 | grep NetworkMode "NetworkMode": "container:02d259dc8ab59c1746d54d2df24d8733b2b9379a9fdfbfdc2066429b4a934a04", # 这个container的id号码就是上一个container的id的long形式
所以可以看到fc8580292210(busybox)
使用的是pause容器的网络空间。
让我们进一步验证。
2.2 IP地址和hostname、网络IO###
下面我在 192.168.1.224 搭建了一个DNS的pod,里面有4个容器,共享一个网络空间,我们采用查看其ip地址、hostname和网络IO的方式来鉴定。 下面是容器的id号
[root@centos7-node-224 ~]$ docker ps | grep dns b00a08d078d6 dockerimages.yinnut.com:15043/skydns:2015-03-11-001 "/skydns -machines=h 8 hours ago Up 8 hours k8s_skydns.c878079e_kube-dns-v9-y05vd_kube-system_12725077-64c0-11e5-9309-525400d7abb6_46f95e60 4e843585b938 dockerimages.yinnut.com:15043/exechealthz:1.0 "/exechealthz '-cmd= 11 days ago Up 11 days k8s_healthz.8ab20f84_kube-dns-v9-y05vd_kube-system_12725077-64c0-11e5-9309-525400d7abb6_f7c469e5 296ff779abb2 dockerimages.yinnut.com:15043/kube2sky:1.11 "/kube2sky -domain=c 11 days ago Up 11 days k8s_kube2sky.2a46d768_kube-dns-v9-y05vd_kube-system_12725077-64c0-11e5-9309-525400d7abb6_349c7246 f0118fac6952 dockerimages.yinnut.com:15043/etcd:2.0.9 "/usr/local/bin/etcd 11 days ago Up 11 days k8s_etcd.64e02c2f_kube-dns-v9-y05vd_kube-system_12725077-64c0-11e5-9309-525400d7abb6_9235054b f281dbf1ec41 gcr.io/google_containers/pause:0.8.0 "/pause" 11 days ago Up 11 days k8s_POD.6e934112_kube-dns-v9-y05vd_kube-system_12725077-64c0-11e5-9309-525400d7abb6_a8ea96d0
我们查看前三个 b00a08d078d6 4e843585b938 296ff779abb2
的上述属性。
2.2.1 dns设置和hostname####
[root@centos7-node-224 ~]$ for id in b00a08d078d6 4e843585b938 296ff779abb2 ; do echo $id; docker exec $id cat /etc/hosts ; docker exec $id cat /etc/resolv.conf ; echo "" ; done b00a08d078d6 172.16.60.4 kube-dns-v9-y05vd 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters nameserver 192.168.1.208 search 8.8.8.8 options ndots:5 4e843585b938 172.16.60.4 kube-dns-v9-y05vd 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters nameserver 192.168.1.208 search 8.8.8.8 options ndots:5 296ff779abb2 172.16.60.4 kube-dns-v9-y05vd 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters nameserver 192.168.1.208 search 8.8.8.8 options ndots:5
2.2.2 IP地址####
[root@centos7-node-224 ~]$ for id in b00a08d078d6 4e843585b938 296ff779abb2 ; do echo $id; docker exec $id ip a ; echo "" ; done b00a08d078d6 1: lo:mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 4e843585b938 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 296ff779abb2 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever
2.2.3 网络连接IO####
[root@centos7-node-224 ~]$ for id in b00a08d078d6 4e843585b938 296ff779abb2 ; do echo $id; docker exec $id netstat -lan ; echo "" ; done b00a08d078d6 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 127.0.0.1:4001 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:2380 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:7001 0.0.0.0:* LISTEN tcp 0 0 172.16.60.4:48582 10.254.0.1:443 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:35394 ESTABLISHED tcp 0 0 172.16.60.4:48584 10.254.0.1:443 ESTABLISHED tcp 0 0 127.0.0.1:60161 127.0.0.1:2379 ESTABLISHED tcp 0 0 127.0.0.1:51445 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:51445 ESTABLISHED tcp 0 0 127.0.0.1:35550 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:35394 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:2379 127.0.0.1:60161 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:51433 ESTABLISHED tcp 0 0 127.0.0.1:51433 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:35550 ESTABLISHED Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node Path 4e843585b938 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 127.0.0.1:4001 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:2380 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:7001 0.0.0.0:* LISTEN tcp 0 0 172.16.60.4:48582 10.254.0.1:443 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:35394 ESTABLISHED tcp 0 0 172.16.60.4:48584 10.254.0.1:443 ESTABLISHED tcp 0 0 127.0.0.1:60161 127.0.0.1:2379 ESTABLISHED tcp 0 0 127.0.0.1:51445 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:51445 ESTABLISHED tcp 0 0 127.0.0.1:35550 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:35394 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:2379 127.0.0.1:60161 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:51433 ESTABLISHED tcp 0 0 127.0.0.1:51433 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:35550 ESTABLISHED Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node Path 296ff779abb2 Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 127.0.0.1:4001 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:2380 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:7001 0.0.0.0:* LISTEN tcp 0 0 172.16.60.4:48582 10.254.0.1:443 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:35394 ESTABLISHED tcp 0 0 172.16.60.4:48584 10.254.0.1:443 ESTABLISHED tcp 0 0 127.0.0.1:60161 127.0.0.1:2379 ESTABLISHED tcp 0 0 127.0.0.1:51445 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:51445 ESTABLISHED tcp 0 0 127.0.0.1:35550 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:35394 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:2379 127.0.0.1:60161 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:51433 ESTABLISHED tcp 0 0 127.0.0.1:51433 127.0.0.1:4001 ESTABLISHED tcp 0 0 127.0.0.1:4001 127.0.0.1:35550 ESTABLISHED Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node Path
3. Container 之通信##
下面分析下最复杂的Container之间的通信。
3.1 Pod 之内 Container 通信
先说最简单的, Pod内的Container通信,由于共享网络地址空间,直接访问127.0.0.1即可。
3.2 跨机器之间Contianer通信###
3.2.1 背景引导####
例子: 192.168.1.224的fluentd-elasticsearch容器要连接192.168.1.223的elasticsearch-logging容器。
192.168.1.223的elasticsearch-logging容器及其IP地址:
[root@centos7-node-223 ~]$ docker ps |grep elasticsearch-logging 667cfd84c979 dockerimages.yinnut.com:15043/elasticsearch:1.7 "/run.sh" 12 days ago Up 12 days k8s_elasticsearch-logging.89fda9f_elasticsearch-logging-v1-i8x6q_kube-system_8b558d2c-62a3-11e5-9d7b-525400d7abb6_2a02a2c8 5201c8cbdebd gcr.io/google_containers/pause:0.8.0 "/pause" 12 days ago Up 12 days k8s_POD.8ecd2043_elasticsearch-logging-v1-i8x6q_kube-system_8b558d2c-62a3-11e5-9d7b-525400d7abb6_4022db35 [root@centos7-node-223 ~]$ docker inspect 5201c8cbdebd |grep IPAddress "IPAddress": "172.16.77.4", #IP地址 "SecondaryIPAddresses": null,
可以看到elasticsearch-logging的容器的Pod的IP地址为172.16.77.4
192.168.1.223的fluentd-elasticsearch容器及其IP地址:
[root@centos7-node-224 ~]$ docker ps |grep fluentd-elasticsearch d326d81468b5 gcr.io/google_containers/fluentd-elasticsearch:1.11 "td-agent -q" 12 days ago Up 12 days k8s_fluentd-elasticsearch.27a08aa3_fluentd-elasticsearch-centos7-node-224_kube-system_7dcc6ce562f3742190a876fda85e2359_58c54ef3 f9b76639d241 gcr.io/google_containers/pause:0.8.0 "/pause" 12 days ago Up 12 days k8s_POD.7be6d81d_fluentd-elasticsearch-centos7-node-224_kube-system_7dcc6ce562f3742190a876fda85e2359_333e52c0 [root@centos7-node-224 ~]$ docker inspect f9b76639d241 | grep IPAddress "IPAddress": "172.16.60.2", "SecondaryIPAddresses": null,
可以看到fluentd-elasticsearch的容器的Pod的IP地址为172.16.60.2
我们看下 fluentd-elasticsearch 的网络连接情况
[root@centos7-node-224 ~]$ docker exec d326d81468b5 netstat -nla | grep 172.16.77.4 tcp 0 0 172.16.60.2:56354 172.16.77.4:9200 TIME_WAIT tcp 0 0 172.16.60.2:56350 172.16.77.4:9200 TIME_WAIT tcp 0 0 172.16.60.2:56347 172.16.77.4:9200 TIME_WAIT tcp 0 0 172.16.60.2:56357 172.16.77.4:9200 TIME_WAIT tcp 0 0 172.16.60.2:56344 172.16.77.4:9200 TIME_WAIT tcp 0 0 172.16.60.2:56352 172.16.77.4:9200 TIME_WAIT
可以看到其的确是连接了 172.16.77.4 的9200端口。 而对方 elasticsearch-logging 容器的确开启了9200端口
[root@centos7-node-223 ~]$ docker exec 667cfd84c979 ss -l|grep LISTEN tcp LISTEN 0 50 :::9200 :::* tcp LISTEN 0 50 :::9300 :::*
那么这个过程是如何完成的呢???
3.2.2 Container间通信流程####
192.168.1.224/fluentd-elasticsearch -> 192.168.1.223/elasticsearch-logging
192.168.1.224/fluentd-elasticsearch 需要连接到elasticsearch-logging容器.
域名到IP对应。 elasticsearch-logging -> 解析为10.254.24.205
root@fluentd-elasticsearch-centos7-node-224:/$ dig elasticsearch-logging ; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> elasticsearch-logging ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 39181 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;elasticsearch-logging. IN A ;; Query time: 1 msec ;; SERVER: 10.254.0.10#53(10.254.0.10) ;; WHEN: Fri Oct 09 06:04:35 UTC 2015 ;; MSG SIZE rcvd: 39
访问该IP地址10.254.24.205:9200端口。根据路由,请求将会到达网关172.16.60.1,也就是这个docker的宿主机的docker0网卡地址。
# 容器内 root@fluentd-elasticsearch-centos7-node-224:/$ ip route default via 172.16.60.1 dev eth0 172.16.60.0/24 dev eth0 proto kernel scope link src 172.16.60.2 # 物理机 [root@centos7-node-224 ~]$ ifconfig docker0 docker0: flags=4163mtu 1450 inet 172.16.60.1 netmask 255.255.255.0 broadcast 0.0.0.0 inet6 fe80::5484:7aff:fefe:9799 prefixlen 64 scopeid 0x20 ether 56:84:7a:fe:97:99 txqueuelen 0 (Ethernet) RX packets 10182154 bytes 1777103288 (1.6 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 11195534 bytes 2271907616 (2.1 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
iptables负责转发请求到192.168.1.224:36967,而这个端口上kube-proxy进程在监听。
[root@centos7-node-224 ~]$ iptables-save | grep 10.254.24.205 |grep 9200 -A KUBE-PORTALS-CONTAINER -d 10.254.24.205/32 -p tcp -m comment --comment "kube-system/elasticsearch-logging:" -m tcp --dport 9200 -j REDIRECT --to-ports 36967 -A KUBE-PORTALS-HOST -d 10.254.24.205/32 -p tcp -m comment --comment "kube-system/elasticsearch-logging:" -m tcp --dport 9200 -j DNAT --to-destination 192.168.1.224:36967 [root@centos7-node-224 ~]$ netstat -nlp|grep 36967 tcp6 0 0 :::36967 :::* LISTEN 930/kube-proxy
谁负责响应10.254.24.205:9200的请求?由上述分析,看起来是kube-proxy,那么kube-proxy进程看起来是个proxy,那么被转发给谁处理?当然给Pod啦。可以到这个服务的Selector是
k8s-app=elasticsearch-logging
[root@centos7-node-224 ~]$ kubectl get svc --all-namespaces | grep 10.254.24.205 kube-system elasticsearch-logging 10.254.24.2059200/TCP k8s-app=elasticsearch-logging 14d
找到其对应的Pod为elasticsearch-logging-v1-gph5i和elasticsearch-logging-v1-i8x6q
[root@centos7-node-224 ~]$ kubectl get po -l k8s-app=elasticsearch-logging --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system elasticsearch-logging-v1-gph5i 1/1 Running 6 14d kube-system elasticsearch-logging-v1-i8x6q 1/1 Running 5 14d
我们查看其中的elasticsearch-logging-v1-i8x6q容器的IP地址,发现为172.16.77.4
[root@centos7-node-224 ~]$ kubectl describe po elasticsearch-logging-v1-i8x6q --namespace=kube-system | grep IP IP: 172.16.77.4
故而很明显kube-proxy 会把部分请求转发给 其中的一个Pod来处理,而这个Pod的IP地址是172.16.77.4 . 而 172.16.77.4 这个Pod 在 192.168.1.223 机器上.
[root@centos7-node-223 ~]$ docker inspect 5201c8cbdebd | grep IPAddress "IPAddress": "172.16.77.4", "SecondaryIPAddresses": null,
那如何与172.16.77.4进行通信呢?跨机器之间通信则采用flannel等诸如此类的overlay网络或者ovs等L2网络。
以上是“Kubernetes网络分析中如何实现Container间通信”这篇文章的所有内容,感谢各位的阅读!希望分享的内容对大家有帮助,更多相关知识,欢迎关注创新互联行业资讯频道!
本文题目:Kubernetes网络分析中如何实现Container间通信
分享地址:http://pwwzsj.com/article/ijepsh.html