1、简介
1.1前言
Kubernetes设计了网络模型,但却将它的实现讲给了网络插件,CNI网络插件最重要的功能就是实现Pod资源能够跨主机通信
常见的CNI网络插件如下:
Flannel;
Cacliao;
Canal;
Contiv;
OpenContrail;
NSX-T;
Kube-router。
1.2 Flannel的三种网络模型
host-gw模型:所有的node ip必须在同一个物理网关设备下才能使用,他的原理就是:给宿主机添加一个静态路由
Vxlan模型:当宿主机不在同一个网段下(不适用同一个网关),就可以使用此模型,相当于在每台宿主机上添加了一个虚拟网络设备,通过一个虚拟隧道进行通信。
直接路由模型:当node不在同一个物理网关,走vxlan模型,当在同一个路由下,走host-gw模型
2、集群规划
部署以10.4.7.21为例,10.4.7.22部署类似
主机名角色IP hdss7-21Flannel10.4.7.21 hdss7-22Flannel10.4.7.22
3、下载软件,解压,做软连接
下载地址:https://github.com/flannel-io/flannel/
[root@hdss7-21 ~]# cd /opt/src/ [root@hdss7-21 src]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz [root@hdss7-21 src]# mkdir /opt/flannel-v0.11.0 [root@hdss7-21 src]# tar -zxvf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel-v0.11.0 [root@hdss7-21 src]# ln -s /opt/flannel-v0.11.0 /opt/flannel
4、最终目录结构
[root@hdss7-21 src]# cd /opt/flannel [root@hdss7-21 flannel]# ls flanneld mk-docker-opts.sh README.md
5、拷贝client证书
作为etcd的客户端
[root@hdss7-21 flannel]# mkdir cert [root@hdss7-21 flannel]# cd cert/ root@hdss7-21 cert]# scp hdss7-200:/opt/certs/ca.pem . root@hdss7-21 cert]# scp hdss7-200:/opt/certs/client.pem . root@hdss7-21 cert]# scp hdss7-200:/opt/certs/client-key.pem . root@hdss7-21 cert]# ll 总用量 12 -rw-r--r-- 1 root root 1346 6月 13 21:55 ca.pem -rw------- 1 root root 1679 6月 13 21:56 client-key.pem -rw-r--r-- 1 root root 1363 6月 13 21:55 client.pem
6、创建配置
注意:flannel集群各主机的配置不同,部署其他节点时注意修改,10.4.7.22上FLANNEL_SUBNET=172.7.22.1/24
[root@hdss7-21 cert]# cd .. [root@hdss7-21 flannel]# vim subnet.env FLANNEL_NETWORK=172.7.0.0/16 FLANNEL_SUBNET=172.7.21.1/24 FLANNEL_MTU=1500 FLANNEL_IPMASQ=false
7、创建启动脚本
注意IP(--public-ip=)与网络接口(--iface=)修改
[root@hdss7-21 flannel]# vim flanneld.sh #!/bin/sh ./flanneld --public-ip=10.4.7.21 --etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 --etcd-keyfile=./cert/client-key.pem --etcd-certfile=./cert/client.pem --etcd-cafile=./cert/ca.pem --iface=ens33 --subnet-file=./subnet.env --healthz-port=2401
8、检查配置、权限、创建日志目录
[root@hdss7-21 flannel]# chmod +x flanneld.sh [root@hdss7-21 flannel]# mkdir -p /data/logs/flanneld
9、操作etcd,增加host-gw
在etcd集群中,可以在12,21,22上任意一台上操作
[root@hdss7-21 flannel]# cd /opt/etcd 注意网络地址 [root@hdss7-21 etcd]# ./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}' {"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}} 查看 [root@hdss7-21 etcd]# ./etcdctl get /coreos.com/network/config {"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}} 查看一下etcd集群信息 [root@hdss7-21 etcd]# ./etcdctl member list 988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false 5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=true
10、创建supervisor配置
[root@hdss7-21 etcd]# vim /etc/supervisord.d/flannel.ini [program:flanneld-7-21] command=/opt/flannel/flanneld.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/flannel ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/flanneld/flanneld.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
11、启动服务并检查
[root@hdss7-21 etcd]# supervisorctl update flanneld-7-21: added process group [root@hdss7-21 etcd]# tail -100f /data/logs/flanneld/flanneld.stdout.log I0613 22:58:53.387404 33642 main.go:527] Using interface with name ens33 and address 10.4.7.21 I0613 22:58:53.387528 33642 main.go:540] Using 10.4.7.21 as external address 2021-06-13 22:58:53.388681 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated I0613 22:58:53.388761 33642 main.go:244] Created subnet manager: Etcd Local Manager with Previous Subnet: 172.7.21.0/24 I0613 22:58:53.388772 33642 main.go:247] Installing signal handlers I0613 22:58:53.396039 33642 main.go:587] Start healthz server on 0.0.0.0:2401 I0613 22:58:53.418509 33642 main.go:386] Found network config - Backend type: host-gw I0613 22:58:53.432160 33642 local_manager.go:201] Found previously leased subnet (172.7.21.0/24), reusing I0613 22:58:53.440005 33642 local_manager.go:220] Allocated lease (172.7.21.0/24) to current node (10.4.7.21) I0613 22:58:53.440627 33642 main.go:317] Wrote subnet file to ./subnet.env I0613 22:58:53.440644 33642 main.go:321] Running backend. I0613 22:58:53.441000 33642 route_network.go:53] Watching for new subnet leases I0613 22:58:53.452691 33642 main.go:429] Waiting for 22h59m59.980128025s to renew lease I0613 22:58:53.452920 33642 iptables.go:145] Some iptables rules are missing; deleting and recreating rules I0613 22:58:53.452975 33642 iptables.go:167] Deleting iptables rule: -s 172.7.0.0/16 -j ACCEPT I0613 22:58:53.464074 33642 iptables.go:167] Deleting iptables rule: -d 172.7.0.0/16 -j ACCEPT I0613 22:58:53.471984 33642 iptables.go:155] Adding iptables rule: -s 172.7.0.0/16 -j ACCEPT I0613 22:58:53.497924 33642 iptables.go:155] Adding iptables rule: -d 172.7.0.0/16 -j ACCEPT
12、部署集群其他节点,检查所有集群服务
上面的3-11(etcd不需要操作了)步骤在10.4.7.22上操作部署;
等22上的flannel服务起来之后检查路由,发现增加了到22容器网段通过10.4.7.22主机的路由(22也有到21的网络),这就是flannel做的事情(但是注意,物理主机需要在同一个网关下才能使用host-gw网络模型)
查看路由 [root@hdss7-21 etcd]# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default gateway 0.0.0.0 UG 100 0 0 ens33 10.4.7.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 172.7.21.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0 172.7.22.0 10.4.7.22 255.255.255.0 UG 0 0 0 ens33 22上有道21的网关路由: [root@hdss7-21 etcd]# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default gateway 0.0.0.0 UG 100 0 0 ens33 10.4.7.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 172.7.21.0 10.4.7.21 255.255.255.0 UG 0 0 0 ens33 172.7.22.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
13、再次验证集群,POD网路互通
在21上访问22上的容器,已经可以互通了
[root@hdss7-21 etcd]# curl 172.7.22.2Welcome to nginx! Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.Thank you for using nginx.
在22上访问21上的容器
[root@hdss7-22 flannel]# curl 172.7.21.2Welcome to nginx! Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.Thank you for using nginx.
14、在各个运算节点上优化iptables规则
为什么要做iptables规则优化:默认主机上是做好了snat的规则转换,就是容器访问到容器时,在目标容器内显示的源地址是主机地址,而不是源容器地址,如下所示,在10.4.7.21上的172.7.21.2容器访问172.7.22.2容器时,在172.7.22.2内查看到源地址的是10.4.7.21,这样就不知道到底是哪个具体容器进行访问的了,我们要做的就是把容器到容器的iptables的snat转换规则去掉。
以下操作在10.4.7.21上操作
14.1 容器添加附加curl功能
修改使用的镜像为curl镜像(之前在第四章部署harbor的时候创建过此镜像)
[root@hdss7-21 ~]# vim /opt/kubernetes/conf/nginx-ds.yaml apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: nginx-ds spec: template: metadata: labels: app: nginx-ds spec: containers: - name: my-nginx image: harbor.od.com/public/nginx:curl ports: - containerPort: 80 [root@hdss7-21 ~]# kubectl apply -f /opt/kubernetes/conf/nginx-ds.yaml daemonset.extensions/nginx-ds configured
用新的镜像替换原来的容器
~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ds-spdgm 1/1 Running 2 3d 172.7.21.2 hdss7-21.host.comnginx-ds-sx7hn 1/1 Running 0 111s 172.7.22.2 hdss7-22.host.com 原来的容器被kill后会根据期望自动生成新的容器 [root@hdss7-21 ~]# kubectl delete pod nginx-ds-spdgm pod "nginx-ds-spdgm" deleted [root@hdss7-21 ~]# kubectl delete pod nginx-ds-sx7hn pod "nginx-ds-sx7hn" deleted [root@hdss7-21 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ds-97b8r 1/1 Running 0 12s 172.7.21.2 hdss7-21.host.com nginx-ds-ncpk8 1/1 Running 0 4s 172.7.22.2 hdss7-22.host.com
14.2 优化前访问
进入172.7.21.2容器对172.7.22.2容器进行访问,可能会出现容器无法启动的情况(由于容器使用的dns是192.168.0.2,后面部署好dns就没问题了),那么换成原来的容器,参看本章16节的内容,使容器可以ping通外网后安装curl工具就行了
查看当前的pod信息 ~]# kubectl get pod -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ds-nqb57 1/1 Running 0 22m 172.7.21.2 hdss7-21.host.comnginx-ds-pmgm2 1/1 Running 0 22m 172.7.22.2 hdss7-22.host.com 进入172.7.21.2容器访问172.7.22.2容器,查看172.7.22.2的日志,会发现源地址是172.7.21.2的宿主机即10.4.7.21访问的; [root@hdss7-21 ~]# kubectl exec -it nginx-ds-nqb57 bash root@nginx-ds-nqb57:/# curl 172.7.22.2 [root@hdss7-21 ~]# kubectl logs -f nginx-ds-pmgm2 10.4.7.21 - - [17/Jun/2021:13:22:10 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
14.3 开始优化iptables规则
以21为例,22网络地址不同
[root@hdss7-21 ~]# yum install iptables-services -y [root@hdss7-21 ~]# systemctl start iptables && systemctl enable iptables [root@hdss7-21 ~]# iptables-save | grep -i postrouting :POSTROUTING ACCEPT [1:63] :KUBE-POSTROUTING - [0:0] -A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING -A POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE [root@hdss7-21 ~]# iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE [root@hdss7-21 ~]# iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE [root@hdss7-21 ~]# iptables-save | grep -i postrouting :POSTROUTING ACCEPT [53:3186] :KUBE-POSTROUTING - [0:0] -A POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE -A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE 将iptables自带的拒绝规则删除 [root@hdss7-21 ~]# iptables-save | grep -i reject -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited [root@hdss7-21 ~]# iptables -t filter -D INPUT -j REJECT --reject-with icmp-host-prohibited [root@hdss7-21 ~]# iptables -t filter -D FORWARD -j REJECT --reject-with icmp-host-prohibited
15、各运算节点保存iptables规则
15.1 保存规则
[root@hdss7-21 ~]# iptables-save > /etc/sysconfig/iptables [root@hdss7-21 ~]# service iptables save iptables: Saving firewall rules to /etc/sysconfig/iptables:[ 确定 ] 再次尝试容器之间访问 [root@hdss7-21 ~]# kubectl exec -it nginx-ds-nqb57 bash root@nginx-ds-nqb57:/# curl 172.7.22.2 [root@hdss7-21 ~]# kubectl logs -f nginx-ds-pmgm2 10.4.7.21 - - [17/Jun/2021:13:16:55 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-" 10.4.7.21 - - [17/Jun/2021:13:22:10 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-" 172.7.21.2 - - [17/Jun/2021:13:33:07 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
15.2 重启docker服务
注意:修改后会影响到docker原本的iptables链的规则,所以需要重启docker服务
docker重启后之前删除的nat规则会再次生效,filter不会
[root@hdss7-21 ~]# systemctl restart docker [root@hdss7-21 ~]# iptables-save |grep -i postrouting|grep docker0 -A POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE -A POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE 可以用iptables-restore重新应用iptables规则,也可以直接再删 [root@hdss7-21 ~]# iptables-restore /etc/sysconfig/iptables [root@hdss7-21 ~]# iptables-save |grep -i postrouting|grep docker0 -A POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
16、报错排查
当你发现使用kubectl启动的容器无法ping通外网时,尝试使用本地的docker通过同一个镜像创建一个容器,如果此容器正常访问外网,那么可以说明宿主机没有问题,docker没有问题(如果有问题,网上的一些说法是重启一下docker),那么可能就是kubectl的配置有问题,比较一下两种容器的dns,发现不一样;
Docker启动的容器如下: ~]# docker exec -it f2be1d19ff19 bash root@f2be1d19ff19:/# cat /etc/resolv.conf # Generated by NetworkManager search host.com nameserver 10.4.7.11 root@f2be1d19ff19:/# ping baidu.com PING baidu.com (39.156.69.79): 48 data bytes 56 bytes from 39.156.69.79: icmp_seq=0 ttl=53 time=10.950 ms 使用kubectl启动的容器如下: ~]# kubectl exec -it nginx-ds-gmhm7 bash root@nginx-ds-gmhm7:/# cat /etc/resolv.conf nameserver 192.168.0.2 search default.svc.cluster.local svc.cluster.local cluster.local host.com options ndots:5 root@nginx-ds-gmhm7:/# ping baidu.com ping: unknown host
发现,容器的ip地址和dns都不在同一个网段,将dns修改成正常docker启动的dns就行了
root@nginx-ds-gmhm7:/# echo "nameserver 10.4.7.11" > /etc/resolv.conf root@nginx-ds-gmhm7:/# ping baidu.com PING baidu.com (220.181.38.148): 48 data bytes 56 bytes from 220.181.38.148: icmp_seq=0 ttl=49 time=15.832 ms root@nginx-ds-gmhm7:/# tee /etc/apt/sources.list << EOF deb http://mirrors.163.com/debian/ jessie main non-free contrib deb http://mirrors.163.com/debian/ jessie-updates main non-free contrib EOF root@nginx-ds-gmhm7:/# apt-get update && apt-get install curl -y 尝试curl百度首页 root@cc8ae2b47946:/# curl -k https://www.baidu.com
关于容器dns为192.168.0.2,无法访问外网,这个问题在后面部署好dns后就可以解决了
链接:https://www.cnblogs.com/wangyuanguang/p/15024812.html
全部0条评论
快来发表一下你的评论吧 !