016.Kubernetes二进制部署所有节点kube-proxy
一部署 kube-proxy
kube-proxy 运行在所有节点上，它监听 apiserver 中 service 和 endpoint 的变化情况，创建路由规则以提供服务 IP 和负载均衡功能。
1.1安装kube-proxy
提示：k8smaster01节点已下载相应二进制，可直接分发至node节点。
1.2分发kube-proxy
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for all_ip in ${ALL_IPS[@]} 4 do 5 echo ">>> ${all_ip}" 6 scp kubernetes/server/bin/kube-proxy root@${all_ip}:/opt/k8s/bin/ 7 ssh root@${all_ip} "chmod +x /opt/k8s/bin/*" 8 done
1.3创建kube-scheduler证书和私钥
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > kube-proxy-csr.json <<EOF 3 { 4 "CN": "system:kube-proxy", 5 "key": { 6 "algo": "rsa", 7 "size": 2048 8 }, 9 "names": [ 10 { 11 "C": "CN", 12 "ST": "Shanghai", 13 "L": "Shanghai", 14 "O": "k8s", 15 "OU": "System" 16 } 17 ] 18 } 19 EOF 20 #创建kube-scheduler的CA证书请求文件
解释：
- CN：指定该证书的 User 为 system:kube-proxy；
- 预定义的 RoleBinding system:node-proxier 将User system:kube-proxy 与 Role system:node-proxier 绑定，该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限；
- 该证书只会被 kube-proxy 当做 client 证书使用，所以 hosts 字段为空。
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem 3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json 4 -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy#生成CA密钥（ca-key.pem）和证书（ca.pem）
1.4创建和分发kubeconfig
kube-proxy 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-proxy 证书：
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# kubectl config set-cluster kubernetes 4 --certificate-authority=/opt/k8s/work/ca.pem 5 --embed-certs=true 6 --server=${KUBE_APISERVER} 7 --kubeconfig=kube-proxy.kubeconfig 8 9 [root@k8smaster01 work]# kubectl config set-credentials kube-proxy 10 --client-certificate=kube-proxy.pem 11 --client-key=kube-proxy-key.pem 12 --embed-certs=true 13 --kubeconfig=kube-proxy.kubeconfig 14 15 [root@k8smaster01 work]# kubectl config set-context default 16 --cluster=kubernetes 17 --user=kube-proxy 18 --kubeconfig=kube-proxy.kubeconfig 19 20 [root@k8smaster01 work]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig 21 22 [root@k8smaster01 ~]# cd /opt/k8s/work 23 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 24 [root@k8smaster01 work]# for node_name in ${NODE_NAMES[@]} 25 do 26 echo ">>> ${node_name}" 27 scp kube-proxy.kubeconfig root@${node_name}:/etc/kubernetes/ 28 done
1.5创建kube-proxy 配置文件
从 v1.10 开始，kube-proxy 部分参数可以配置文件中配置。可以使用 --write-config-to 选项生成该配置文件。
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > kube-proxy-config.yaml.template <<EOF 3 kind: KubeProxyConfiguration 4 apiVersion: kubeproxy.config.k8s.io/v1alpha1 5 clientConnection: 6 burst: 200 7 kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig" 8 qps: 100 9 bindAddress: ##ALL_IP## 10 healthzBindAddress: ##ALL_IP##:10256 11 metricsBindAddress: ##ALL_IP##:10249 12 enableProfiling: true 13 clusterCIDR: ${CLUSTER_CIDR} 14 hostnameOverride: ##ALL_NAME## 15 mode: "ipvs" 16 portRange: "" 17 kubeProxyIPTablesConfiguration: 18 masqueradeAll: false 19 kubeProxyIPVSConfiguration: 20 scheduler: rr 21 excludeCIDRs: [] 22 EOF
解释：
- bindAddress: 监听地址；
- clientConnection.kubeconfig: 连接 apiserver 的 kubeconfig 文件；
- clusterCIDR: kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量，指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT；
- hostnameOverride: 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则；
- mode: 使用 ipvs 模式。
1.6分发配置文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for (( i=0; i < 6; i++ )) 4 do 5 echo ">>> ${ALL_NAMES[i]}" 6 sed -e "s/##ALL_NAME##/${ALL_NAMES[i]}/" -e "s/##ALL_IP##/${ALL_IPS[i]}/" kube-proxy-config.yaml.template > kube-proxy-config-${ALL_NAMES[i]}.yaml.template 7 scp kube-proxy-config-${ALL_NAMES[i]}.yaml.template root@${ALL_NAMES[i]}:/etc/kubernetes/kube-proxy-config.yaml 8 done
1.7创建kube-proxy的systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# cat > kube-proxy.service <<EOF 4 [Unit] 5 Description=Kubernetes Kube-Proxy Server 6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes 7 After=network.target 8 9 [Service] 10 WorkingDirectory=${K8S_DIR}/kube-proxy 11 ExecStart=/opt/k8s/bin/kube-proxy \ 12 --config=/etc/kubernetes/kube-proxy-config.yaml \ 13 --logtostderr=true \ 14 --v=2 15 Restart=on-failure 16 RestartSec=5 17 LimitNOFILE=65536 18 19 [Install] 20 WantedBy=multi-user.target 21 EOF
1.8分发kube-proxy systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for all_name in ${ALL_NAMES[@]} 4 do 5 echo ">>> ${all_name}" 6 scp kube-proxy.service root@${all_name}:/etc/systemd/system/ 7 done#分发system
二启动并验证
2.1启动kube-proxy 服务
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for all_ip in ${ALL_IPS[@]} 4 do 5 echo ">>> ${all_ip}" 6 ssh root@${all_ip} "mkdir -p ${K8S_DIR}/kube-proxy" 7 ssh root@${all_ip} "modprobe ip_vs_rr" 8 ssh root@${all_ip} "systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy" 9 done#启动服务前必须先创建工作目录
2.2检查kube-proxy 服务
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# for all_ip in ${ALL_IPS[@]} 3 do 4 echo ">>> ${all_ip}" 5 ssh root@${all_ip} "systemctl status kube-proxy|grep Active" 6 done
2.3查看监听端口
kube-proxy 监听 10249 和 10256 端口：
- 10249：对外提供 /metrics;
- 10256：对外提供 /healthz 的访问。
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for all_ip in ${ALL_IPS[@]} 4 do 5 echo ">>> ${all_ip}" 6 ssh root@${all_ip} "sudo netstat -lnpt|grep kube-prox" 7 done
2.4查看ipvs 路由规则
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for all_ip in ${ALL_IPS[@]} 4 do 5 echo ">>> ${all_ip}" 6 ssh root@${all_ip} "/usr/sbin/ipvsadm -ln" 7 done
可见所有通过 https 访问 K8S SVC kubernetes 的请求都转发到 kube-apiserver 节点的 6443 端口。
赞 (0)