openstack(pike)dvr中南北数据流向分析
1 在DVR环境中各组件的连接情况如图
2 带floatingip,从外部访问虚机，ping floatingip 192.168.100.190
此时数据流如图所示
2.1 获取MAC
外网中的机器首先要通过 ARP 获取虚机浮动 IP 对应的 MAC 地址。浮动 IP 并没有配置在 fip 的端口上，因此 fip 无法直接响应 ARP 请求，那怎么办呢？Neutron 在 fip NS 的 fg 端口上配置了 arp proxy，这样，fip 既可以响应它自己的 interface 上的 IP 地址的 ARP 请求，也可以响应能通过它路由到的 IP 地址的 ARP 请求,
fip netns上配置的proxy_arp
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e sysctl net.ipv4.conf.fg-42d94dcb-8b.proxy_arp
net.ipv4.conf.fg-42d94dcb-8b.proxy_arp = 1
fip netns 收到 ARP 请求后，将其 fg interface 的 MAC 地址返回,外网中的机器获取到虚机浮动 IP 的 MAC 地址后，发出 ICMP 网络包
2.2 fip netns 路由到下qrouter netns
网络包经过 br-ex，被 fip 的 fg 端口收到，查路由表，命中第一条路由，从其 fpr interface 发出，到达 169.254.106.114.
ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip route
169.254.106.114/31 dev fpr-b2ce2e13-1 proto kernel scope link src 169.254.106.115
192.168.100.0/24 dev fg-42d94dcb-8b proto kernel scope link src 192.168.100.184
192.168.100.190 via 169.254.106.114 dev fpr-b2ce2e13-1
fpr interface 和rfp interface 是一堆veth pair , fpr interface 发出的包被 veth 另一端的 qrouter 的 rfp-b2ce2e13-1 interface 收到。
可以看到qrouter netns 的 rfp-b2ce2e13-1口地址是169.254.106.114/31
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip addr
1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: rfp-b2ce2e13-1: mtu 1500 qdisc noqueue state UP qlen 1000
link/ether b2:e7:86:35:3d:ac brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 169.254.106.114/31 scope global rfp-b2ce2e13-1
valid_lft forever preferred_lft forever
inet6 fe80::b0e7:86ff:fe35:3dac/64 scope link
valid_lft forever preferred_lft forever
134: qr-d2b8d53f-64: mtu 1400 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:02:97:ab brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global qr-d2b8d53f-64
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe02:97ab/64 scope link
valid_lft forever preferred_lft forever
2.3 DNAT
在 qrouter 上，首先做 DNAT
-A neutron-l3-agent-PREROUTING -d 192.168.100.190/32 -i rfp-b2ce2e13-1 -j DNAT --to-destination 172.17.0.7
DNAT后，dest ip 改为172.17.0.7
2.4 再次路由
查 qrouter 的 main 路由表，命中第二条，从qr-d2b8d53f-64 发出
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route
169.254.106.114/31 dev rfp-b2ce2e13-1 proto kernel scope link src 169.254.106.114
172.17.0.0/16 dev qr-d2b8d53f-64 proto kernel scope link src 172.17.0.1
数据包经qr-d2b8d53f-64,发到br-int 到达虚机
3 带floatingip从虚机访问外网，从172.17.0.7 ping 8.8.8.8
3.1路由
进入qroute netns,查看路由规则走route 16表
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local
32766:from all lookup main
32767:from all lookup default
57483:from 172.17.0.7 lookup 16
2886795265:from 172.17.0.1/16 lookup 2886795265
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table 16
default via 169.254.106.115 dev rfp-b2ce2e13-1
3.2 SNAT
路由后会通过netfilter的POSTROUTING链中进行SNAT，将源IP改为192.168.100.190
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i rfp-b2ce2e13-1 ! -o rfp-b2ce2e13-1 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 192.168.100.190/32 -i rfp-b2ce2e13-1 -j DNAT --to-destination 172.17.0.7
-A neutron-l3-agent-float-snat -s 172.17.0.7/32 -j SNAT --to-source 192.168.100.190
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
从rfp-b2ce2e13-1口将包发到169.254.106.115，rfp-b2ce2e13-1和fip netns的fpr-b2ce2e13-1，是一对veth pair，fpr-b2ce2e13-1收到数据包
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip addr
1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: fpr-b2ce2e13-1: mtu 1500 qdisc noqueue state UP qlen 1000
link/ether aa:f7:5f:31:67:db brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 169.254.106.115/31 scope global fpr-b2ce2e13-1
valid_lft forever preferred_lft forever
inet6 fe80::a8f7:5fff:fe31:67db/64 scope link
valid_lft forever preferred_lft forever
135: fg-42d94dcb-8b: mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:c0:dc:94 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.184/24 brd 192.168.100.255 scope global fg-42d94dcb-8b
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fec0:dc94/64 scope link
valid_lft forever preferred_lft forever
3.3 数据包在fip netns 中再次路由
包通过fpr-b2ce2e13-1发到169.254.106.115，后再在fip netns中查路由规则，路由
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip rule
0:from all lookup local
32766:from all lookup main
32767:from all lookup default
2852022899:from all iif fpr-b2ce2e13-1 lookup 2852022899
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip route list table 2852022899
default via 192.168.100.1 dev fg-42d94dcb-8b
从fg-42d94dcb-8b 口将包发到192.168.100.1，再有192.168.100.1继续路由转发
4 不带浮动IP，虚机访问外网
4.1 计算节点qrouter 路由
查看路由规则
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local
32766:from all lookup main
32767:from all lookup default
2886795265:from 172.17.0.1/16 lookup 2886795265
查看路由表2886795265
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table 2886795265
default via 172.17.0.10 dev qr-d2b8d53f-64
只有一条策略，经过qr口，发到下一个路由172.17.0.10
172.17.0.10 路由是在network节点的snat netns sg口
[root@network ~]# ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip addr
1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
93: sg-935eff5f-b8: mtu 1400 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:bf:6d:b5 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.10/16 brd 172.17.255.255 scope global sg-935eff5f-b8
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:febf:6db5/64 scope link
valid_lft forever preferred_lft forever
94: qg-2b2169ee-a5: mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:9d:5c:bd brd ff:ff:ff:ff:ff:ff
inet 192.168.100.187/24 brd 192.168.100.255 scope global qg-2b2169ee-a5
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe9d:5cbd/64 scope link
valid_lft forever preferred_lft forever
4.2 SNAT
network 节点的snat netns 做SNAT
ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 iptables -t nat -S
-A neutron-l3-agent-snat -o qg-2b2169ee-a5 -j SNAT --to-source 192.168.100.187
4.3 snat netns路由
ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local
32766:from all lookup main
32767:from all lookup default
没有额外策略，走main表
ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table main
default via 192.168.100.1 dev qg-2b2169ee-a5
172.17.0.0/16 dev sg-935eff5f-b8 proto kernel scope link src 172.17.0.10
192.168.100.0/24 dev qg-2b2169ee-a5 proto kernel scope link src 192.168.100.187
根据目的地址，命中第一条，包从qg-2b2169ee-a5口出去，再从192.168.100.1网关上继续路由
网页标题：openstack(pike)dvr中南北数据流向分析
网页地址：http://pwwzsj.com/article/podohg.html
2 带floatingip,从外部访问虚机，ping floatingip 192.168.100.190
此时数据流如图所示
2.1 获取MAC
外网中的机器首先要通过 ARP 获取虚机浮动 IP 对应的 MAC 地址。浮动 IP 并没有配置在 fip 的端口上，因此 fip 无法直接响应 ARP 请求，那怎么办呢？Neutron 在 fip NS 的 fg 端口上配置了 arp proxy，这样，fip 既可以响应它自己的 interface 上的 IP 地址的 ARP 请求，也可以响应能通过它路由到的 IP 地址的 ARP 请求,
fip netns上配置的proxy_arp
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e sysctl net.ipv4.conf.fg-42d94dcb-8b.proxy_arp
net.ipv4.conf.fg-42d94dcb-8b.proxy_arp = 1
fip netns 收到 ARP 请求后，将其 fg interface 的 MAC 地址返回,外网中的机器获取到虚机浮动 IP 的 MAC 地址后，发出 ICMP 网络包
2.2 fip netns 路由到下qrouter netns
网络包经过 br-ex，被 fip 的 fg 端口收到，查路由表，命中第一条路由，从其 fpr interface 发出，到达 169.254.106.114.
ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip route
169.254.106.114/31 dev fpr-b2ce2e13-1 proto kernel scope link src 169.254.106.115
192.168.100.0/24 dev fg-42d94dcb-8b proto kernel scope link src 192.168.100.184
192.168.100.190 via 169.254.106.114 dev fpr-b2ce2e13-1
fpr interface 和rfp interface 是一堆veth pair , fpr interface 发出的包被 veth 另一端的 qrouter 的 rfp-b2ce2e13-1 interface 收到。
可以看到qrouter netns 的 rfp-b2ce2e13-1口地址是169.254.106.114/31
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip addr
1: lo:
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: rfp-b2ce2e13-1:
link/ether b2:e7:86:35:3d:ac brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 169.254.106.114/31 scope global rfp-b2ce2e13-1
valid_lft forever preferred_lft forever
inet6 fe80::b0e7:86ff:fe35:3dac/64 scope link
valid_lft forever preferred_lft forever
134: qr-d2b8d53f-64:
link/ether fa:16:3e:02:97:ab brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global qr-d2b8d53f-64
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe02:97ab/64 scope link
valid_lft forever preferred_lft forever
2.3 DNAT
在 qrouter 上，首先做 DNAT
-A neutron-l3-agent-PREROUTING -d 192.168.100.190/32 -i rfp-b2ce2e13-1 -j DNAT --to-destination 172.17.0.7
DNAT后，dest ip 改为172.17.0.7
2.4 再次路由
查 qrouter 的 main 路由表，命中第二条，从qr-d2b8d53f-64 发出
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route
169.254.106.114/31 dev rfp-b2ce2e13-1 proto kernel scope link src 169.254.106.114
172.17.0.0/16 dev qr-d2b8d53f-64 proto kernel scope link src 172.17.0.1
数据包经qr-d2b8d53f-64,发到br-int 到达虚机
3 带floatingip从虚机访问外网，从172.17.0.7 ping 8.8.8.8
3.1路由
进入qroute netns,查看路由规则走route 16表
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local
32766:from all lookup main
32767:from all lookup default
57483:from 172.17.0.7 lookup 16
2886795265:from 172.17.0.1/16 lookup 2886795265
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table 16
default via 169.254.106.115 dev rfp-b2ce2e13-1
3.2 SNAT
路由后会通过netfilter的POSTROUTING链中进行SNAT，将源IP改为192.168.100.190
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i rfp-b2ce2e13-1 ! -o rfp-b2ce2e13-1 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 192.168.100.190/32 -i rfp-b2ce2e13-1 -j DNAT --to-destination 172.17.0.7
-A neutron-l3-agent-float-snat -s 172.17.0.7/32 -j SNAT --to-source 192.168.100.190
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
从rfp-b2ce2e13-1口将包发到169.254.106.115，rfp-b2ce2e13-1和fip netns的fpr-b2ce2e13-1，是一对veth pair，fpr-b2ce2e13-1收到数据包
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip addr
1: lo:
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: fpr-b2ce2e13-1:
link/ether aa:f7:5f:31:67:db brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 169.254.106.115/31 scope global fpr-b2ce2e13-1
valid_lft forever preferred_lft forever
inet6 fe80::a8f7:5fff:fe31:67db/64 scope link
valid_lft forever preferred_lft forever
135: fg-42d94dcb-8b:
link/ether fa:16:3e:c0:dc:94 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.184/24 brd 192.168.100.255 scope global fg-42d94dcb-8b
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fec0:dc94/64 scope link
valid_lft forever preferred_lft forever
3.3 数据包在fip netns 中再次路由
包通过fpr-b2ce2e13-1发到169.254.106.115，后再在fip netns中查路由规则，路由
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip rule
0:from all lookup local
32766:from all lookup main
32767:from all lookup default
2852022899:from all iif fpr-b2ce2e13-1 lookup 2852022899
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip route list table 2852022899
default via 192.168.100.1 dev fg-42d94dcb-8b
从fg-42d94dcb-8b 口将包发到192.168.100.1，再有192.168.100.1继续路由转发
4 不带浮动IP，虚机访问外网
4.1 计算节点qrouter 路由
查看路由规则
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local
32766:from all lookup main
32767:from all lookup default
2886795265:from 172.17.0.1/16 lookup 2886795265
查看路由表2886795265
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table 2886795265
default via 172.17.0.10 dev qr-d2b8d53f-64
只有一条策略，经过qr口，发到下一个路由172.17.0.10
172.17.0.10 路由是在network节点的snat netns sg口
[root@network ~]# ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip addr
1: lo:
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
93: sg-935eff5f-b8:
link/ether fa:16:3e:bf:6d:b5 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.10/16 brd 172.17.255.255 scope global sg-935eff5f-b8
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:febf:6db5/64 scope link
valid_lft forever preferred_lft forever
94: qg-2b2169ee-a5:
link/ether fa:16:3e:9d:5c:bd brd ff:ff:ff:ff:ff:ff
inet 192.168.100.187/24 brd 192.168.100.255 scope global qg-2b2169ee-a5
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe9d:5cbd/64 scope link
valid_lft forever preferred_lft forever
4.2 SNAT
network 节点的snat netns 做SNAT
ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 iptables -t nat -S
-A neutron-l3-agent-snat -o qg-2b2169ee-a5 -j SNAT --to-source 192.168.100.187
4.3 snat netns路由
ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local
32766:from all lookup main
32767:from all lookup default
没有额外策略，走main表
ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table main
default via 192.168.100.1 dev qg-2b2169ee-a5
172.17.0.0/16 dev sg-935eff5f-b8 proto kernel scope link src 172.17.0.10
192.168.100.0/24 dev qg-2b2169ee-a5 proto kernel scope link src 192.168.100.187
根据目的地址，命中第一条，包从qg-2b2169ee-a5口出去，再从192.168.100.1网关上继续路由
网页标题：openstack(pike)dvr中南北数据流向分析
网页地址：http://pwwzsj.com/article/podohg.html
