Kubernetes的CNI网络插件之flannel

描述

1、简介

1.1前言

Kubernetes设计了网络模型,但却将它的实现讲给了网络插件,CNI网络插件最重要的功能就是实现Pod资源能够跨主机通信
常见的CNI网络插件如下:
Flannel;
Cacliao;
Canal;
Contiv;
OpenContrail;
NSX-T;
Kube-router。

1.2 Flannel的三种网络模型

host-gw模型:所有的node ip必须在同一个物理网关设备下才能使用,他的原理就是:给宿主机添加一个静态路由

Vxlan模型:当宿主机不在同一个网段下(不适用同一个网关),就可以使用此模型,相当于在每台宿主机上添加了一个虚拟网络设备,通过一个虚拟隧道进行通信。

直接路由模型:当node不在同一个物理网关,走vxlan模型,当在同一个路由下,走host-gw模型

2、集群规划

部署以10.4.7.21为例,10.4.7.22部署类似

 

主机名角色IP
hdss7-21Flannel10.4.7.21
hdss7-22Flannel10.4.7.22

 

3、下载软件,解压,做软连接

下载地址:https://github.com/flannel-io/flannel/

 

[root@hdss7-21 ~]# cd /opt/src/
[root@hdss7-21 src]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[root@hdss7-21 src]# mkdir /opt/flannel-v0.11.0
[root@hdss7-21 src]# tar -zxvf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel-v0.11.0
[root@hdss7-21 src]# ln -s /opt/flannel-v0.11.0  /opt/flannel

 

4、最终目录结构

 

[root@hdss7-21 src]# cd /opt/flannel
[root@hdss7-21 flannel]# ls
flanneld  mk-docker-opts.sh  README.md

 

5、拷贝client证书

作为etcd的客户端

 

[root@hdss7-21 flannel]# mkdir cert
[root@hdss7-21 flannel]# cd cert/
root@hdss7-21 cert]# scp hdss7-200:/opt/certs/ca.pem .
root@hdss7-21 cert]# scp hdss7-200:/opt/certs/client.pem .
root@hdss7-21 cert]# scp hdss7-200:/opt/certs/client-key.pem .
root@hdss7-21 cert]# ll
总用量 12
-rw-r--r-- 1 root root 1346 6月  13 21:55 ca.pem
-rw------- 1 root root 1679 6月  13 21:56 client-key.pem
-rw-r--r-- 1 root root 1363 6月  13 21:55 client.pem

 

6、创建配置

注意:flannel集群各主机的配置不同,部署其他节点时注意修改,10.4.7.22上FLANNEL_SUBNET=172.7.22.1/24

 

[root@hdss7-21 cert]# cd ..
[root@hdss7-21 flannel]# vim subnet.env
FLANNEL_NETWORK=172.7.0.0/16
FLANNEL_SUBNET=172.7.21.1/24
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false

 

7、创建启动脚本

注意IP(--public-ip=)与网络接口(--iface=)修改

 

[root@hdss7-21 flannel]# vim flanneld.sh
#!/bin/sh
./flanneld 
  --public-ip=10.4.7.21 
  --etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 
  --etcd-keyfile=./cert/client-key.pem 
  --etcd-certfile=./cert/client.pem 
  --etcd-cafile=./cert/ca.pem 
  --iface=ens33 
  --subnet-file=./subnet.env 
  --healthz-port=2401

 

8、检查配置、权限、创建日志目录

 

[root@hdss7-21 flannel]# chmod +x flanneld.sh
[root@hdss7-21 flannel]# mkdir -p /data/logs/flanneld

 

9、操作etcd,增加host-gw

在etcd集群中,可以在12,21,22上任意一台上操作

 

[root@hdss7-21 flannel]# cd /opt/etcd
注意网络地址
[root@hdss7-21 etcd]# ./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}'
{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}
查看
[root@hdss7-21 etcd]# ./etcdctl get /coreos.com/network/config
{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}
查看一下etcd集群信息
[root@hdss7-21 etcd]# ./etcdctl member list
988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=true

 

10、创建supervisor配置

 

[root@hdss7-21 etcd]# vim /etc/supervisord.d/flannel.ini
[program:flanneld-7-21]
command=/opt/flannel/flanneld.sh                             ; the program (relative uses PATH, can take args)
numprocs=1                                                   ; number of processes copies to start (def 1)
directory=/opt/flannel                                       ; directory to cwd to before exec (def no cwd)
autostart=true                                               ; start at supervisord start (default: true)
autorestart=true                                             ; retstart at unexpected quit (default: true)
startsecs=30                                                 ; number of secs prog must stay running (def. 1)
startretries=3                                               ; max # of serial start failures (default 3)
exitcodes=0,2                                                ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                              ; signal used to kill process (default TERM)
stopwaitsecs=10                                              ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                    ; setuid to this UNIX account to run the program
redirect_stderr=true                                         ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/flanneld/flanneld.stdout.log       ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                 ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                     ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                  ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                  ; emit events on stdout writes (default false)

 

11、启动服务并检查

 

[root@hdss7-21 etcd]# supervisorctl update
flanneld-7-21: added process group

[root@hdss7-21 etcd]# tail -100f /data/logs/flanneld/flanneld.stdout.log
I0613 22:58:53.387404   33642 main.go:527] Using interface with name ens33 and address 10.4.7.21
I0613 22:58:53.387528   33642 main.go:540] Using 10.4.7.21 as external address
2021-06-13 22:58:53.388681 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
I0613 22:58:53.388761   33642 main.go:244] Created subnet manager: Etcd Local Manager with Previous Subnet: 172.7.21.0/24
I0613 22:58:53.388772   33642 main.go:247] Installing signal handlers
I0613 22:58:53.396039   33642 main.go:587] Start healthz server on 0.0.0.0:2401
I0613 22:58:53.418509   33642 main.go:386] Found network config - Backend type: host-gw
I0613 22:58:53.432160   33642 local_manager.go:201] Found previously leased subnet (172.7.21.0/24), reusing
I0613 22:58:53.440005   33642 local_manager.go:220] Allocated lease (172.7.21.0/24) to current node (10.4.7.21)
I0613 22:58:53.440627   33642 main.go:317] Wrote subnet file to ./subnet.env
I0613 22:58:53.440644   33642 main.go:321] Running backend.
I0613 22:58:53.441000   33642 route_network.go:53] Watching for new subnet leases
I0613 22:58:53.452691   33642 main.go:429] Waiting for 22h59m59.980128025s to renew lease
I0613 22:58:53.452920   33642 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0613 22:58:53.452975   33642 iptables.go:167] Deleting iptables rule: -s 172.7.0.0/16 -j ACCEPT
I0613 22:58:53.464074   33642 iptables.go:167] Deleting iptables rule: -d 172.7.0.0/16 -j ACCEPT
I0613 22:58:53.471984   33642 iptables.go:155] Adding iptables rule: -s 172.7.0.0/16 -j ACCEPT
I0613 22:58:53.497924   33642 iptables.go:155] Adding iptables rule: -d 172.7.0.0/16 -j ACCEPT

 

12、部署集群其他节点,检查所有集群服务

上面的3-11(etcd不需要操作了)步骤在10.4.7.22上操作部署;
等22上的flannel服务起来之后检查路由,发现增加了到22容器网段通过10.4.7.22主机的路由(22也有到21的网络),这就是flannel做的事情(但是注意,物理主机需要在同一个网关下才能使用host-gw网络模型)

 

查看路由
[root@hdss7-21 etcd]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         0.0.0.0         UG    100    0        0 ens33
10.4.7.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.7.21.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
172.7.22.0      10.4.7.22       255.255.255.0   UG    0      0        0 ens33
22上有道21的网关路由:
[root@hdss7-21 etcd]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         0.0.0.0         UG    100    0        0 ens33
10.4.7.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.7.21.0      10.4.7.21       255.255.255.0   UG    0      0        0 ens33
172.7.22.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0

 

13、再次验证集群,POD网路互通

在21上访问22上的容器,已经可以互通了

 

[root@hdss7-21 etcd]# curl 172.7.22.2



Welcome to nginx!



Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

 

在22上访问21上的容器

 

[root@hdss7-22 flannel]#  curl 172.7.21.2



Welcome to nginx!



Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

 

14、在各个运算节点上优化iptables规则

为什么要做iptables规则优化:默认主机上是做好了snat的规则转换,就是容器访问到容器时,在目标容器内显示的源地址是主机地址,而不是源容器地址,如下所示,在10.4.7.21上的172.7.21.2容器访问172.7.22.2容器时,在172.7.22.2内查看到源地址的是10.4.7.21,这样就不知道到底是哪个具体容器进行访问的了,我们要做的就是把容器到容器的iptables的snat转换规则去掉。
以下操作在10.4.7.21上操作

14.1 容器添加附加curl功能

修改使用的镜像为curl镜像(之前在第四章部署harbor的时候创建过此镜像)

 

[root@hdss7-21 ~]# vim /opt/kubernetes/conf/nginx-ds.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
    name: nginx-ds
spec:
    template:
      metadata:
        labels:
          app: nginx-ds
      spec:
        containers:
        - name: my-nginx
          image: harbor.od.com/public/nginx:curl
          ports:
          - containerPort: 80

[root@hdss7-21 ~]# kubectl apply -f /opt/kubernetes/conf/nginx-ds.yaml
daemonset.extensions/nginx-ds configured

 

用新的镜像替换原来的容器

 

~]# kubectl get pod -o wide
NAME             READY   STATUS    RESTARTS   AGE    IP           NODE                NOMINATED NODE   READINESS GATES
nginx-ds-spdgm   1/1     Running   2          3d     172.7.21.2   hdss7-21.host.com              
nginx-ds-sx7hn   1/1     Running   0          111s   172.7.22.2   hdss7-22.host.com              
原来的容器被kill后会根据期望自动生成新的容器
[root@hdss7-21 ~]# kubectl delete pod nginx-ds-spdgm
pod "nginx-ds-spdgm" deleted
[root@hdss7-21 ~]# kubectl delete pod nginx-ds-sx7hn
pod "nginx-ds-sx7hn" deleted
[root@hdss7-21 ~]# kubectl get pod -o wide
NAME             READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
nginx-ds-97b8r   1/1     Running   0          12s   172.7.21.2   hdss7-21.host.com              
nginx-ds-ncpk8   1/1     Running   0          4s    172.7.22.2   hdss7-22.host.com              

 

14.2 优化前访问

进入172.7.21.2容器对172.7.22.2容器进行访问,可能会出现容器无法启动的情况(由于容器使用的dns是192.168.0.2,后面部署好dns就没问题了),那么换成原来的容器,参看本章16节的内容,使容器可以ping通外网后安装curl工具就行了

 

查看当前的pod信息
~]# kubectl get pod -owide
NAME             READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
nginx-ds-nqb57   1/1     Running   0          22m   172.7.21.2   hdss7-21.host.com              
nginx-ds-pmgm2   1/1     Running   0          22m   172.7.22.2   hdss7-22.host.com              
进入172.7.21.2容器访问172.7.22.2容器,查看172.7.22.2的日志,会发现源地址是172.7.21.2的宿主机即10.4.7.21访问的;
[root@hdss7-21 ~]# kubectl exec -it nginx-ds-nqb57 bash
root@nginx-ds-nqb57:/# curl 172.7.22.2
[root@hdss7-21 ~]#  kubectl logs -f nginx-ds-pmgm2
10.4.7.21 - - [17/Jun/2021:13:22:10 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"

 

14.3 开始优化iptables规则

以21为例,22网络地址不同

 

[root@hdss7-21 ~]# yum install iptables-services -y
[root@hdss7-21 ~]# systemctl start iptables && systemctl enable iptables
[root@hdss7-21 ~]# iptables-save | grep -i postrouting
:POSTROUTING ACCEPT [1:63]
:KUBE-POSTROUTING - [0:0]
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
[root@hdss7-21 ~]# iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
[root@hdss7-21 ~]# iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
[root@hdss7-21 ~]# iptables-save | grep -i postrouting
:POSTROUTING ACCEPT [53:3186]
:KUBE-POSTROUTING - [0:0]
-A POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
将iptables自带的拒绝规则删除
[root@hdss7-21 ~]# iptables-save | grep -i reject
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-21 ~]# iptables -t filter -D INPUT -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-21 ~]# iptables -t filter -D FORWARD -j REJECT --reject-with icmp-host-prohibited

 

15、各运算节点保存iptables规则

15.1 保存规则

 

[root@hdss7-21 ~]# iptables-save > /etc/sysconfig/iptables
[root@hdss7-21 ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  确定  ]
再次尝试容器之间访问
[root@hdss7-21 ~]# kubectl exec -it nginx-ds-nqb57 bash
root@nginx-ds-nqb57:/# curl 172.7.22.2
[root@hdss7-21 ~]# kubectl logs -f nginx-ds-pmgm2
10.4.7.21 - - [17/Jun/2021:13:16:55 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
10.4.7.21 - - [17/Jun/2021:13:22:10 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
172.7.21.2 - - [17/Jun/2021:13:33:07 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"

 

15.2 重启docker服务

注意:修改后会影响到docker原本的iptables链的规则,所以需要重启docker服务
docker重启后之前删除的nat规则会再次生效,filter不会

 

[root@hdss7-21 ~]# systemctl restart docker
[root@hdss7-21 ~]# iptables-save |grep -i postrouting|grep docker0
-A POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
可以用iptables-restore重新应用iptables规则,也可以直接再删
[root@hdss7-21 ~]# iptables-restore /etc/sysconfig/iptables
[root@hdss7-21 ~]# iptables-save |grep -i postrouting|grep docker0
-A POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE

 

16、报错排查

当你发现使用kubectl启动的容器无法ping通外网时,尝试使用本地的docker通过同一个镜像创建一个容器,如果此容器正常访问外网,那么可以说明宿主机没有问题,docker没有问题(如果有问题,网上的一些说法是重启一下docker),那么可能就是kubectl的配置有问题,比较一下两种容器的dns,发现不一样;

 

Docker启动的容器如下:
~]# docker exec -it f2be1d19ff19 bash
root@f2be1d19ff19:/# cat /etc/resolv.conf
# Generated by NetworkManager
search host.com
nameserver 10.4.7.11
root@f2be1d19ff19:/# ping baidu.com
PING baidu.com (39.156.69.79): 48 data bytes
56 bytes from 39.156.69.79: icmp_seq=0 ttl=53 time=10.950 ms
使用kubectl启动的容器如下:
~]# kubectl exec -it nginx-ds-gmhm7 bash
root@nginx-ds-gmhm7:/# cat /etc/resolv.conf
nameserver 192.168.0.2
search default.svc.cluster.local svc.cluster.local cluster.local host.com
options ndots:5
root@nginx-ds-gmhm7:/# ping baidu.com
ping: unknown host

 

发现,容器的ip地址和dns都不在同一个网段,将dns修改成正常docker启动的dns就行了

 

root@nginx-ds-gmhm7:/# echo "nameserver 10.4.7.11" > /etc/resolv.conf
root@nginx-ds-gmhm7:/# ping baidu.com
PING baidu.com (220.181.38.148): 48 data bytes
56 bytes from 220.181.38.148: icmp_seq=0 ttl=49 time=15.832 ms
root@nginx-ds-gmhm7:/# tee /etc/apt/sources.list << EOF
deb http://mirrors.163.com/debian/ jessie main non-free contrib
deb http://mirrors.163.com/debian/ jessie-updates main non-free contrib
EOF
root@nginx-ds-gmhm7:/# apt-get update && apt-get install curl -y
尝试curl百度首页
root@cc8ae2b47946:/# curl -k https://www.baidu.com

 

关于容器dns为192.168.0.2,无法访问外网,这个问题在后面部署好dns后就可以解决了

链接:https://www.cnblogs.com/wangyuanguang/p/15024812.html

                           

 

打开APP阅读更多精彩内容
声明:本文内容及配图由入驻作者撰写或者入驻合作网站授权转载。文章观点仅代表作者本人,不代表电子发烧友网立场。文章及其配图仅供工程师学习之用,如有内容侵权或者其他违规问题,请联系本站处理。 举报投诉

全部0条评论

快来发表一下你的评论吧 !

×
20
完善资料,
赚取积分