009.Kubernetes二进制部署kube-apiserver

一部署master节点

1.1master节点服务

kubernetes master 节点运行如下组件：
  • kube-apiserver
  • kube-scheduler
  • kube-controller-manager
  • kube-nginx
kube-apiserver、kube-scheduler 和 kube-controller-manager 均以多实例模式运行：
kube-scheduler 和 kube-controller-manager 会自动选举产生一个 leader 实例，其它实例处于阻塞模式，当 leader 挂了后，重新选举产生新的 leader，从而保证服务可用性。
kube-apiserver 是无状态的，需要通过 kube-nginx 进行代理访问，从而保证服务可用性。

1.2安装Kubernetes

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# wget https://dl.k8s.io/v1.14.2/kubernetes-server-linux-amd64.tar.gz  3 [root@k8smaster01 work]# tar -xzvf kubernetes-server-linux-amd64.tar.gz  4 [root@k8smaster01 work]# cd kubernetes  5 [root@k8smaster01 kubernetes]# tar -xzvf  kubernetes-src.tar.gz

1.3分发Kubernetes

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}  4   do  5     echo ">>> ${master_ip}"  6     scp kubernetes/server/bin/{apiextensions-apiserver,cloud-controller-manager,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@${master_ip}:/opt/k8s/bin/  7     ssh root@${master_ip} "chmod +x /opt/k8s/bin/*"  8   done

二部署高可用kube-apiserver

2.1高可用apiserver介绍

本实验部署一个三实例 kube-apiserver 集群的步骤，它们通过 kube-nginx 进行代理访问，从而保证服务可用性。

2.2创建Kubernetes证书和私钥

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# cat > kubernetes-csr.json <<EOF  3 {  4   "CN": "kubernetes",  5   "hosts": [  6     "127.0.0.1",  7     "172.24.8.71",  8     "172.24.8.72",  9     "172.24.8.73", 10     "${CLUSTER_KUBERNETES_SVC_IP}", 11     "kubernetes", 12     "kubernetes.default", 13     "kubernetes.default.svc", 14     "kubernetes.default.svc.cluster", 15     "kubernetes.default.svc.cluster.local." 16   ], 17   "key": { 18     "algo": "rsa", 19     "size": 2048 20   }, 21   "names": [ 22     { 23       "C": "CN", 24       "ST": "Shanghai", 25       "L": "Shanghai", 26       "O": "k8s", 27       "OU": "System" 28     } 29   ] 30 } 31 EOF 32 #创建Kubernetes的CA证书请求文件
解释：
hosts 字段指定授权使用该证书的 IP 和域名列表，这里列出了 master 节点 IP、kubernetes 服务的 IP 和域名；
kubernetes 服务 IP 是 apiserver 自动创建的，一般是 --service-cluster-ip-range 参数指定的网段的第一个IP，后续可以通过下面命令获取：
1 # kubectl get svc kubernetes
1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem   3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json   4 -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes#生成CA密钥（ca-key.pem）和证书（ca.pem）

2.3分发证书和私钥

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}  4   do  5     echo ">>> ${master_ip}"  6     ssh root@${master_ip} "mkdir -p /etc/kubernetes/cert"  7     scp kubernetes*.pem root@${master_ip}:/etc/kubernetes/cert/  8   done

2.4创建加密配置文件

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# cat > encryption-config.yaml <<EOF  4 kind: EncryptionConfig  5 apiVersion: v1  6 resources:  7   - resources:  8       - secrets  9     providers: 10       - aescbc: 11           keys: 12             - name: key1 13               secret: ${ENCRYPTION_KEY} 14       - identity: {} 15 EOF

2.5分发加密配置文件

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}  4   do  5     echo ">>> ${master_ip}"  6     scp encryption-config.yaml root@${master_ip}:/etc/kubernetes/  7   done

2.6创建审计策略文件

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# cat > audit-policy.yaml <<EOF  4 apiVersion: audit.k8s.io/v1beta1  5 kind: Policy  6 rules:  7   # The following requests were manually identified as high-volume and low-risk, so drop them.  8   - level: None  9     resources: 10       - group: "" 11         resources: 12           - endpoints 13           - services 14           - services/status 15     users: 16       - 'system:kube-proxy' 17     verbs: 18       - watch 19  20   - level: None 21     resources: 22       - group: "" 23         resources: 24           - nodes 25           - nodes/status 26     userGroups: 27       - 'system:nodes' 28     verbs: 29       - get 30  31   - level: None 32     namespaces: 33       - kube-system 34     resources: 35       - group: "" 36         resources: 37           - endpoints 38     users: 39       - 'system:kube-controller-manager' 40       - 'system:kube-scheduler' 41       - 'system:serviceaccount:kube-system:endpoint-controller' 42     verbs: 43       - get 44       - update 45  46   - level: None 47     resources: 48       - group: "" 49         resources: 50           - namespaces 51           - namespaces/status 52           - namespaces/finalize 53     users: 54       - 'system:apiserver' 55     verbs: 56       - get 57  58   # Don't log HPA fetching metrics. 59   - level: None 60     resources: 61       - group: metrics.k8s.io 62     users: 63       - 'system:kube-controller-manager' 64     verbs: 65       - get 66       - list 67  68   # Don't log these read-only URLs. 69   - level: None 70     nonResourceURLs: 71       - '/healthz*' 72       - /version 73       - '/swagger*' 74  75   # Don't log events requests. 76   - level: None 77     resources: 78       - group: "" 79         resources: 80           - events 81  82   # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes 83   - level: Request 84     omitStages: 85       - RequestReceived 86     resources: 87       - group: "" 88         resources: 89           - nodes/status 90           - pods/status 91     users: 92       - kubelet 93       - 'system:node-problem-detector' 94       - 'system:serviceaccount:kube-system:node-problem-detector' 95     verbs: 96       - update 97       - patch 98  99   - level: Request100     omitStages:101       - RequestReceived102     resources:103       - group: ""104         resources:105           - nodes/status106           - pods/status107     userGroups:108       - 'system:nodes'109     verbs:110       - update111       - patch112 113   # deletecollection calls can be large, don't log responses for expected namespace deletions114   - level: Request115     omitStages:116       - RequestReceived117     users:118       - 'system:serviceaccount:kube-system:namespace-controller'119     verbs:120       - deletecollection121 122   # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,123   # so only log at the Metadata level.124   - level: Metadata125     omitStages:126       - RequestReceived127     resources:128       - group: ""129         resources:130           - secrets131           - configmaps132       - group: authentication.k8s.io133         resources:134           - tokenreviews135   # Get repsonses can be large; skip them.136   - level: Request137     omitStages:138       - RequestReceived139     resources:140       - group: ""141       - group: admissionregistration.k8s.io142       - group: apiextensions.k8s.io143       - group: apiregistration.k8s.io144       - group: apps145       - group: authentication.k8s.io146       - group: authorization.k8s.io147       - group: autoscaling148       - group: batch149       - group: certificates.k8s.io150       - group: extensions151       - group: metrics.k8s.io152       - group: networking.k8s.io153       - group: policy154       - group: rbac.authorization.k8s.io155       - group: scheduling.k8s.io156       - group: settings.k8s.io157       - group: storage.k8s.io158     verbs:159       - get160       - list161       - watch162 163   # Default level for known APIs164   - level: RequestResponse165     omitStages:166       - RequestReceived167     resources:168       - group: ""169       - group: admissionregistration.k8s.io170       - group: apiextensions.k8s.io171       - group: apiregistration.k8s.io172       - group: apps173       - group: authentication.k8s.io174       - group: authorization.k8s.io175       - group: autoscaling176       - group: batch177       - group: certificates.k8s.io178       - group: extensions179       - group: metrics.k8s.io180       - group: networking.k8s.io181       - group: policy182       - group: rbac.authorization.k8s.io183       - group: scheduling.k8s.io184       - group: settings.k8s.io185       - group: storage.k8s.io186 187   # Default level for all other requests.188   - level: Metadata189     omitStages:190       - RequestReceived191 EOF

2.7分发策略文件

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}  4   do  5     echo ">>> ${master_ip}"  6     scp audit-policy.yaml root@${master_ip}:/etc/kubernetes/audit-policy.yaml  7   done

2.8创建访问 metrics-server的证书和密钥

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# cat > proxy-client-csr.json <<EOF  3 {  4   "CN": "aggregator",  5   "hosts": [],  6   "key": {  7     "algo": "rsa",  8     "size": 2048  9   }, 10   "names": [ 11     { 12       "C": "CN", 13       "ST": "Shanghai", 14       "L": "Shanghai", 15       "O": "k8s", 16       "OU": "System" 17     } 18   ] 19 } 20 EOF 21 #创建metrics-server的CA证书请求文件
解释：
CN 名称需要位于 kube-apiserver 的 --requestheader-allowed-names 参数中，否则后续访问 metrics 时会提示权限不足。
1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem   3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json   4 -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client#生成CA密钥（ca-key.pem）和证书（ca.pem）

2.9分发证书和私钥

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}  4   do  5     echo ">>> ${master_ip}"  6     scp proxy-client*.pem root@${master_ip}:/etc/kubernetes/cert/  7   done

2.10创建kube-apiserver的systemd

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# cat > kube-apiserver.service.template <<EOF  4 [Unit]  5 Description=Kubernetes API Server  6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes  7 After=network.target  8   9 [Service] 10 WorkingDirectory=${K8S_DIR}/kube-apiserver 11 ExecStart=/opt/k8s/bin/kube-apiserver \ 12   --advertise-address=##MASTER_IP## \ 13   --default-not-ready-toleration-seconds=360 \ 14   --default-unreachable-toleration-seconds=360 \ 15   --feature-gates=DynamicAuditing=true \ 16   --max-mutating-requests-inflight=2000 \ 17   --max-requests-inflight=4000 \ 18   --default-watch-cache-size=200 \ 19   --delete-collection-workers=2 \ 20   --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \ 21   --etcd-cafile=/etc/kubernetes/cert/ca.pem \ 22   --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \ 23   --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \ 24   --etcd-servers=${ETCD_ENDPOINTS} \ 25   --bind-address=##MASTER_IP## \ 26   --secure-port=6443 \ 27   --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \ 28   --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \ 29   --insecure-port=0 \ 30   --audit-dynamic-configuration \ 31   --audit-log-maxage=15 \ 32   --audit-log-maxbackup=3 \ 33   --audit-log-maxsize=100 \ 34   --audit-log-mode=batch \ 35   --audit-log-truncate-enabled \ 36   --audit-log-batch-buffer-size=20000 \ 37   --audit-log-batch-max-size=2 \ 38   --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \ 39   --audit-policy-file=/etc/kubernetes/audit-policy.yaml \ 40   --profiling \ 41   --anonymous-auth=false \ 42   --client-ca-file=/etc/kubernetes/cert/ca.pem \ 43   --enable-bootstrap-token-auth \ 44   --requestheader-allowed-names="aggregator" \ 45   --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \ 46   --requestheader-extra-headers-prefix="X-Remote-Extra-" \ 47   --requestheader-group-headers=X-Remote-Group \ 48   --requestheader-username-headers=X-Remote-User \ 49   --service-account-key-file=/etc/kubernetes/cert/ca.pem \ 50   --authorization-mode=Node,RBAC \ 51   --runtime-config=api/all=true \ 52   --enable-admission-plugins=NodeRestriction \ 53   --allow-privileged=true \ 54   --apiserver-count=3 \ 55   --event-ttl=168h \ 56   --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \ 57   --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \ 58   --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \ 59   --kubelet-https=true \ 60   --kubelet-timeout=10s \ 61   --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \ 62   --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \ 63   --service-cluster-ip-range=${SERVICE_CIDR} \ 64   --service-node-port-range=${NODE_PORT_RANGE} \ 65   --logtostderr=true \ 66   --v=2 67 Restart=on-failure 68 RestartSec=10 69 Type=notify 70 LimitNOFILE=65536 71  72 [Install] 73 WantedBy=multi-user.target 74 EOF
解释：
  • --advertise-address：apiserver 对外通告的 IP（kubernetes 服务后端节点 IP）；
  • --default-*-toleration-seconds：设置节点异常相关的阈值；
  • --max-*-requests-inflight：请求相关的最大阈值；
  • --etcd-*：访问 etcd 的证书和 etcd 服务器地址；
  • --experimental-encryption-provider-config：指定用于加密 etcd 中 secret 的配置；
  • --bind-address： https 监听的 IP，不能为 127.0.0.1，否则外界不能访问它的安全端口 6443；
  • --secret-port：https 监听端口；
  • --insecure-port=0：关闭监听 http 非安全端口(8080)；
  • --tls-*-file：指定 apiserver 使用的证书、私钥和 CA 文件；
  • --audit-*：配置审计策略和审计日志文件相关的参数；
  • --client-ca-file：验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书；
  • --enable-bootstrap-token-auth：启用 kubelet bootstrap 的 token 认证；
  • --requestheader-*：kube-apiserver 的 aggregator layer 相关的配置参数，proxy-client & HPA 需要使用；
  • --requestheader-client-ca-file：用于签名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的证书；在启用了 metric aggregator 时使用；
  • --requestheader-allowed-names：不能为空，值为逗号分割的 --proxy-client-cert-file 证书的 CN 名称，这里设置为 "aggregator"；
  • --service-account-key-file：签名 ServiceAccount Token 的公钥文件，kube-controller-manager 的 --service-account-private-key-file 指定私钥文件，两者配对使用；
  • --runtime-config=api/all=true： 启用所有版本的 APIs，如 autoscaling/v2alpha1；
  • --authorization-mode=Node,RBAC、--anonymous-auth=false： 开启 Node 和 RBAC 授权模式，拒绝未授权的请求；
  • --enable-admission-plugins：启用一些默认关闭的 plugins；
  • --allow-privileged：运行执行 privileged 权限的容器；
  • --apiserver-count=3：指定 apiserver 实例的数量；
  • --event-ttl：指定 events 的保存时间；
  • --kubelet-*：如果指定，则使用 https 访问 kubelet APIs；需要为证书对应的用户(上面 kubernetes*.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则，否则访问 kubelet API 时提示未授权；
  • --proxy-client-*：apiserver 访问 metrics-server 使用的证书；
  • --service-cluster-ip-range： 指定 Service Cluster IP 地址段；
  • --service-node-port-range： 指定 NodePort 的端口范围。
提示：如果 kube-apiserver 机器没有运行 kube-proxy，则还需要添加 --enable-aggregator-routing=true 参数。
注意：requestheader-client-ca-file 指定的 CA 证书，必须具有 client auth and server auth；
如果 --requestheader-allowed-names 为空，或者 --proxy-client-cert-file 证书的 CN 名称不在 allowed-names 中，则后续查看 node 或 pods 的 metrics 失败，提示：
1 [root@zhangjun-k8s01 1.8+]# kubectl top nodes  2 Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "aggregator" cannot list resource "nodes" in API group "metrics.k8s.io" at the cluster scope  3

2.11分发systemd

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# for (( i=0; i < 3; i++ ))  4   do  5     sed -e "s/##MASTER_NAME##/${MASTER_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${MASTER_IPS[i]}.service  6   done  7 [root@k8smaster01 work]# ls kube-apiserver*.service#替换相应的IP  8 [root@k8smaster01 ~]# cd /opt/k8s/work  9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 10 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 11   do 12     echo ">>> ${master_ip}" 13     scp kube-apiserver-${master_ip}.service root@${master_ip}:/etc/systemd/system/kube-apiserver.service 14   done                #分发systemd

三启动并验证

3.1启动kube-apiserver服务

1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh  2 [root@k8smaster01 ~]# for master_ip in ${MASTER_IPS[@]}  3   do  4     echo ">>> ${master_ip}"  5     ssh root@${master_ip} "mkdir -p ${K8S_DIR}/kube-apiserver"  6     ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"  7   done

3.2检查kube-apiserver服务

1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh  2 [root@k8smaster01 ~]# for master_ip in ${MASTER_IPS[@]}  3   do  4     echo ">>> ${master_ip}"  5     ssh root@${master_ip} "systemctl status kube-apiserver |grep 'Active:'"  6   done

3.3查看kube-apiserver写入etcd的数据

1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh  2 [root@k8smaster01 ~]# ETCDCTL_API=3 etcdctl   3     --endpoints=${ETCD_ENDPOINTS}   4     --cacert=/opt/k8s/work/ca.pem   5     --cert=/opt/k8s/work/etcd.pem   6     --key=/opt/k8s/work/etcd-key.pem   7     get /registry/ --prefix --keys-only

3.4检查集群信息

1 [root@k8smaster01 ~]# kubectl cluster-info  2 [root@k8smaster01 ~]# kubectl get all --all-namespaces  3 [root@k8smaster01 ~]# kubectl get componentstatuses  4 [root@k8smaster01 ~]# sudo netstat -lnpt|grep kube#检查 kube-apiserver 监听的端口
提示：
如果执行 kubectl 命令式时输出如下错误信息，则说明使用的 ~/.kube/config 文件不对，请切换到正确的账户后再执行该命令：
The connection to the server localhost:8080 was refused - did you specify the right host or port?
执行 kubectl get componentstatuses 命令时，apiserver 默认向 127.0.0.1 发送请求。当 controller-manager、scheduler 以集群模式运行时，有可能和 kube-apiserver 不在一台机器上，这时 controller-manager 或 scheduler 的状态为 Unhealthy，但实际上它们工作正常；
6443: 接收 https 请求的安全端口，对所有请求做认证和授权；
由于关闭了非安全端口，故没有监听 8080。

3.5授权

授予 kube-apiserver 访问 kubelet API 的权限。
在执行 kubectl exec、run、logs 等命令时，apiserver 会将请求转发到 kubelet 的 https 端口。本实验定义 RBAC 规则，授权 apiserver 使用的证书（kubernetes.pem）用户名（CN：kuberntes）访问 kubelet API 的权限：
1 [root@k8smaster01 ~]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
赞 (0)

相关推荐