企业项目实战k8s篇（十六）Helm部署使用及chart仓库搭建
Helm部署使用及chart仓库搭建
- 一.helm概述
- 二.helm安装部署
- 1.helm安装
- 2.helm部署应用
- 补充：删除之前部署pvc
- 三.chart仓库及应用发布
- 1.应用创建
- 2.本地chart仓库及push插件
- 3.应用上传
- 4.应用安装
- 5.应用升级
- 6.应用卸载
一.helm概述
Helm是Kubernetes 应用的包管理工具，主要用来管理 Charts，类似Linux系统的yum。
Helm Chart 是用来封装 Kubernetes 原生应用程序的一系列 YAML 文件。可以在你部署应用的时候自定义应用程序的一些 Metadata，以便于应用程序的分发。
对于应用发布者而言，可以通过 Helm 打包应用、管理应用依赖关系、管理应用版本并发布应用到软件仓库。
对于使用者而言，使用 Helm 后不用需要编写复杂的应用部署文件，可以以简单的方式在 Kubernetes 上查找、安装、升级、回滚、卸载应用程序。
Helm V3 与 V2 最大的区别在于去掉了tiller
二.helm安装部署
1.helm安装
Helm 官网：https://helm.sh/docs/intro/
下载helm helm-v3.4.1-linux-amd64.tar.gz
创建helm目录，用于进行helm相关实验，解压压缩包到目录
[root@server1 ~]# mkdir helm
[root@server1 ~]# cd helm/
[root@server1 helm]# ls
[root@server1 helm]# mv ../helm
helm/ helm-v3.4.1-linux-amd64.tar.gz
[root@server1 helm]# mv ../helm-v3.4.1-linux-amd64.tar.gz .
[root@server1 helm]# ls
helm-v3.4.1-linux-amd64.tar.gz
[root@server1 helm]# tar zxf helm-v3.4.1-linux-amd64.tar.gz
[root@server1 helm]# ls
helm-v3.4.1-linux-amd64.tar.gz linux-amd64
将二进制可执行文件 helm 复制到 /usr/local/bin/，测试helm命令 helm help
[root@server1 helm]# cd linux-amd64/
[root@server1 linux-amd64]# ls
helm LICENSE README.md
[root@server1 linux-amd64]# cp helm /usr/local/bin/
[root@server1 linux-amd64]# ls
helm LICENSE README.md
[root@server1 linux-amd64]# helm help
The Kubernetes package manager
Common actions for Helm:
- helm search: search for charts
- helm pull: download a chart to your local directory to view
- helm install: upload the chart to Kubernetes
- helm list: list releases of charts
设置helm命令补齐
[root@server1 linux-amd64]# cd
[root@server1 ~]# echo 'source <(helm completion bash)' >> ~/.bashrc
[root@server1 ~]# source .bashrc
[root@server1 ~]# helm
completion dependency get install list plugin repo search status test upgrade version
create env history lint package pull rollback show template uninstall verify
Helm 添加第三方 Chart 库
[root@server1 ~]# helm repo add bitnami https://charts.bitnami.com/bitnami
'bitnami' has been added to your repositories
[root@server1 ~]# helm repo list
NAME URL
bitnamihttps://charts.bitnami.com/bitnami
2.helm部署应用
查询redis应用
[root@server1 ~]# helm search repo redis
NAME CHART VERSIONAPP VERSIONDESCRIPTION
bitnami/redis 14.8.7 6.2.5 Open source, advanced key-value store. It is of...
bitnami/redis-cluster6.3.2 6.2.5 Open source, advanced key-value store. It is of...
拉取应用
[root@server1 ~]# helm pull bitnami/redis-cluster
解压压缩包redis-cluster-6.3.2.tgz，进入目录，values.yaml内为所有变量 Chart.yaml 描述Chart的基本信息,包括名称版本等
[root@server1 ~]# mv redis-cluster-6.3.2.tgz helm/
[root@server1 ~]# cd helm/
[root@server1 helm]# tar zxf redis-cluster-6.3.2.tgz
[root@server1 helm]# cd redis-cluster/
[root@server1 redis-cluster]# ls
Chart.lock charts Chart.yaml img README.md templates values.yaml
[root@server1 redis-cluster]# ll values.yaml
-rw-r--r-- 1 root root 34207 Jul 30 08:12 values.yaml
[root@server1 redis-cluster]# vim values.yaml
拉取部署应用所需镜像redis-cluster:6.2.5-debian-10-r0，更改tag并上传至集群harbor仓库
[root@server3 ~]# docker pull bitnami/redis-cluster:6.2.5-debian-10-r0
6.2.5-debian-10-r0: Pulling from bitnami/redis-cluster
86a19151e740: Already exists
57a1d2d33896: Pull complete
205fbdb2c4df: Pull complete
b50f4b53d89e: Pull complete
daec4ad90bad: Pull complete
3a2283fca554: Pull complete
795c1eb71802: Pull complete
4084d80f7f4f: Pull complete
Digest: sha256:0adf6a9c86f820646455894283af74205209ceaa1dd1c3751e811b4c8ef02652
Status: Downloaded newer image for bitnami/redis-cluster:6.2.5-debian-10-r0
docker.io/bitnami/redis-cluster:6.2.5-debian-10-r0
[root@server3 ~]# docker tag docker.io/bitnami/redis-cluster:6.2.5-debian-10-r0 reg.westos.org/bitnami/redis-cluster:6.2.5-debian-10-r0
[root@server3 ~]# docker push reg.westos.org/bitnami/redis-cluster:6.2.5-debian-10-r0
The push refers to repository [reg.westos.org/bitnami/redis-cluster]
265c6517575c: Pushed
27927d559e88: Pushed
f96e0118c822: Pushed
1615551dea1b: Pushed
c65559991f66: Pushed
8a9b93540aad: Pushed
566ccae6ccab: Pushed
3fa01eaf81a5: Mounted from library/metrics-server
6.2.5-debian-10-r0: digest: sha256:0adf6a9c86f820646455894283af74205209ceaa1dd1c3751e811b4c8ef02652 size: 1996
执行安装命令
[root@server1 redis-cluster]# helm install redis-cluster .
NAME: redis-cluster
LAST DEPLOYED: Wed Aug 4 02:57:59 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **
To get your password run:
export REDIS_PASSWORD=$(kubectl get secret --namespace 'default' redis-cluster -o jsonpath='{.data.redis-password}' | base64 --decode)
You have deployed a Redis™ Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis™ cluster:
1. Run a Redis™ pod that you can use as a client:
kubectl run --namespace default redis-cluster-client --rm --tty -i --restart='Never' \
--env REDIS_PASSWORD=$REDIS_PASSWORD \
--image reg.westos.org/bitnami/redis-cluster:6.2.5-debian-10-r0 -- bash
2. Connect using the Redis™ CLI:
redis-cli -c -h redis-cluster -a $REDIS_PASSWORD
查看已安装应用
[root@server1 redis-cluster]# helm list
NAME NAMESPACEREVISIONUPDATED STATUS CHART APP VERSION
redis-clusterdefault 1 2021-08-04 02:57:59.283039665 -0400 EDTdeployedredis-cluster-6.3.26.2.5
[root@server1 redis-cluster]# kubectl get pod
NAME READY STATUS RESTARTS AGE
php-apache-6cc67f7957-crtvk 1/1 Running 0 3h16m
redis-cluster-0 1/1 Running 1 100s
redis-cluster-1 1/1 Running 1 100s
redis-cluster-2 1/1 Running 1 100s
redis-cluster-3 1/1 Running 2 99s
redis-cluster-4 1/1 Running 1 99s
redis-cluster-5 1/1 Running 0 99s
补充：删除之前部署pvc
[root@server1 redis-cluster]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-037cf46b-d7f5-41f6-b156-17799899daaa 1Gi RWO Delete Bound default/www-web-2 managed-nfs-storage 4d2h
pvc-22524877-c04c-498e-955a-5f121a1e40cc 100Mi RWX Delete Bound default/nfs-pv1 managed-nfs-storage 4d2h
pvc-5a835670-8962-4714-8311-c8a441bf98eb 1Gi RWO Delete Bound default/www-web-0 managed-nfs-storage 4d2h
pvc-b227ba79-3f52-4d34-a1fd-655ff1516592 1Gi RWO Delete Bound default/www-web-1 managed-nfs-storage 4d2h
[root@server1 redis-cluster]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-pv1 Bound pvc-22524877-c04c-498e-955a-5f121a1e40cc 100Mi RWX managed-nfs-storage 4d2h
www-web-0 Bound pvc-5a835670-8962-4714-8311-c8a441bf98eb 1Gi RWO managed-nfs-storage 4d2h
www-web-1 Bound pvc-b227ba79-3f52-4d34-a1fd-655ff1516592 1Gi RWO managed-nfs-storage 4d2h
www-web-2 Bound pvc-037cf46b-d7f5-41f6-b156-17799899daaa 1Gi RWO managed-nfs-storage 4d2h
[root@server1 redis-cluster]# kubectl delete pvc --all
persistentvolumeclaim 'nfs-pv1' deleted
persistentvolumeclaim 'www-web-0' deleted
persistentvolumeclaim 'www-web-1' deleted
persistentvolumeclaim 'www-web-2' deleted
[root@server1 redis-cluster]# kubectl get pvc
No resources found in default namespace.
三.chart仓库及应用发布
1.应用创建
构建一个 Helm Chart，查看mychart 结构
[root@server1 helm]# helm create mychart
Creating mychart
[root@server1 helm]# tree mychart/
mychart/
├── charts
├── Chart.yaml
├── templates
│ ├── deployment.yaml
│ ├── _helpers.tpl
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── NOTES.txt
│ ├── serviceaccount.yaml
│ ├── service.yaml
│ └── tests
│ └── test-connection.yaml
└── values.yaml
3 directories, 10 files
编写mychart的应用描述信息 编写应用部署信息：
[root@server1 helm]# cd mychart/
[root@server1 mychart]# ls
charts Chart.yaml templates values.yaml
[root@server1 mychart]# vim values.yaml
[root@server1 mychart]# vim Chart.yaml
[root@server1 mychart]# vim values.yaml
[root@server1 mychart]# vim Chart.yaml
[root@server1 mychart]# cd ..
检查依赖和模板配置是否正确:
$ helm lint mychart/
==> Linting mychart/
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
将应用打包
[root@server1 helm]# helm package mychart/
Successfully packaged chart and saved it to: /root/helm/mychart-0.1.0.tgz
[root@server1 helm]# ls
mychart mychart-0.1.0.tgz redis-cluster redis-cluster-6.3.2.tgz
2.本地chart仓库及push插件
建立本地chart仓库，本地仓库以harbor仓库为基础，在安装harbor时指定chart模块，建立的harbor也可提供chart仓库服务
harbor搭建详见之前博客 企业项目实战docker篇（四）企业级仓库Harbor搭建
harbor内创建charts项目
添加本地chart库westos（自指定仓库名，但路径一定要是真实有效路径），添加失败，显示未通过认证
[root@server1 helm]# helm repo add westos https://reg.westos.org/chartrepo/charts
Error: looks like 'https://reg.westos.org/chartrepo/charts' is not a valid chart repository or cannot be reached: Get 'https://reg.westos.org/chartrepo/charts/index.yaml': x509: certificate signed by unknown authority
将认证文件 ca.crt复制到 /etc/pki/ca-trust/source/anchors/，执行更新命令 update-ca-trust
[root@server1 helm]# cd /etc/pk
pkcs11/ pki/
[root@server1 helm]# cd /etc/pki/ca-trust/source/anchors/
[root@server1 anchors]# ls
[root@server1 anchors]# pwd
/etc/pki/ca-trust/source/anchors
[root@server1 anchors]# cp /etc/docker/certs.d/reg.westos.org/ca.crt .
[root@server1 anchors]# ls
ca.crt
[root@server1 anchors]# update-ca-trust
再次添加本地chart库 westos
[root@server1 anchors]# helm repo add westos https://reg.westos.org/chartrepo/charts
'westos' has been added to your repositories
[root@server1 anchors]# helm repo list
NAME URL
bitnamihttps://charts.bitnami.com/bitnami
westos https://reg.westos.org/chartrepo/charts
[root@server1 anchors]# cd
仓库搭建完毕，安装push插件 helm-push_0.9.0_linux_amd64.tar.gz
插件压缩包
[root@server1 ~]# mv helm-push_0.9.0_linux_amd64.tar.gz helm
[root@server1 ~]# cd helm/
[root@server1 helm]# ls
helm-push_0.9.0_linux_amd64.tar.gz
查看环境变量
[root@server1 helm]# helm env
HELM_BIN='helm'
HELM_CACHE_HOME='/root/.cache/helm'
HELM_CONFIG_HOME='/root/.config/helm'
HELM_DATA_HOME='/root/.local/share/helm'
HELM_DEBUG='false'
HELM_KUBEAPISERVER=''
HELM_KUBEASGROUPS=''
HELM_KUBEASUSER=''
HELM_KUBECONTEXT=''
HELM_KUBETOKEN=''
HELM_MAX_HISTORY='10'
HELM_NAMESPACE='default'
HELM_PLUGINS='/root/.local/share/helm/plugins'
HELM_REGISTRY_CONFIG='/root/.config/helm/registry.json'
HELM_REPOSITORY_CACHE='/root/.cache/helm/repository'
HELM_REPOSITORY_CONFIG='/root/.config/helm/repositories.yaml'
/root/.local/share/helm/plugins不存在，手动创建
[root@server1 helm]# cd /root/.local/share/helm/plugins
-bash: cd: /root/.local/share/helm/plugins: No such file or directory
进入创建好的目录，创建push目录，将push插件解压至该目录下/root/.local/share/helm/plugins/push/
[root@server1 helm]# cd /root/.local/share/helm/plugins
[root@server1 plugins]# ls
[root@server1 plugins]# mkdir push
[root@server1 plugins]# cd
[root@server1 ~]# cd helm/
[root@server1 helm]# tar zxf helm-push_0.9.0_linux_amd64.tar.gz -C /root/.local/share/helm/plugins/push/
可以看到push命令可以使用
[root@server1 helm]# helm push
completion dependency get install list plugin repo search status test upgrade version
create env history lint package pull rollback show template uninstall verify
3.应用上传
上传应用，应用使用myapp：v1镜像，tag：v1 版本号：0.1.0
[root@server1 helm]# helm push mychart-0.1.0.tgz westos --insecure -u admin -p westos
Pushing mychart-0.1.0.tgz to westos...
Done.
上传应用后要更新helm repo，可以查找到mychart
[root@server1 helm]# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the 'westos' chart repository
[root@server1 helm]# helm search repo mychart
NAME CHART VERSIONAPP VERSIONDESCRIPTION
westos/mychart0.1.0 v1 A Helm chart for Kubernetes
查看mychart应用的values
[root@server1 helm]# helm show values westos/mychart
# Default values for mychart.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: myapp
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: 'v1'
imagePullSecrets: []
nameOverride: ''
fullnameOverride: ''
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ''
podAnnotations: {}
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000
service:
type: ClusterIP
port: 80
ingress:
enabled: false
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: 'true'
hosts:
- host: chart-example.local
paths: []
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
nodeSelector: {}
tolerations: []
affinity: {}
4.应用安装
应用安装helm install mychart westos/mychart
[root@server1 helm]# helm install mychart westos/mychart
NAME: mychart
LAST DEPLOYED: Wed Aug 4 03:55:34 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l 'app.kubernetes.io/name=mychart,app.kubernetes.io/instance=mychart' -o jsonpath='{.items[0].metadata.name}')
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath='{.spec.containers[0].ports[0].containerPort}')
echo 'Visit http://127.0.0.1:8080 to use your application'
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
查看部署，可以看到一切都按照values内配置自动部署完成
[root@server1 helm]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/mychart-6675bd6ffd-qxgj8 1/1 Running 0 93s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 10d
service/mychart ClusterIP 10.109.119.97 <none> 80/TCP 93s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/mychart 1/1 1 1 93s
NAME DESIRED CURRENT READY AGE
replicaset.apps/mychart-6675bd6ffd 1 1 1 93s
[root@server1 helm]# ls
helm-push_0.9.0_linux_amd64.tar.gz mychart mychart-0.1.0.tgz redis-cluster redis-cluster-6.3.2.tgz
测试访问
[root@server1 helm]# curl 10.109.119.97
Hello MyApp | Version: v1 | <a href='hostname.html'>Pod Name</a>
5.应用升级
进入mychart目录，更换镜像myapp：v2，tag：v1 版本号 0.2.0，检测无误打包为mychart-0.2.0.tgz
[root@server1 helm]# cd mychart/
[root@server1 mychart]# vim values.yaml
[root@server1 mychart]# vim Chart.yaml
[root@server1 mychart]# cd ..
[root@server1 helm]# helm lint mychart
==> Linting mychart
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
[root@server1 helm]# helm package mychart
Successfully packaged chart and saved it to: /root/helm/mychart-0.2.0.tgz
[root@server1 helm]# ls
helm-push_0.9.0_linux_amd64.tar.gz mychart mychart-0.1.0.tgz mychart-0.2.0.tgz redis-cluster redis-cluster-6.3.2.tgz
push mychart-0.2.0.tgz至本地chart仓库 westos
[root@server1 helm]# helm push mychart-0.2.0.tgz westos --insecure -u admin -p westos
Pushing mychart-0.2.0.tgz to westos...
Done.
更新repo，查询mychart，默认显示最新版本
[root@server1 helm]# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the 'westos' chart repository
^C
[root@server1 helm]# helm search repo mychart
NAME CHART VERSIONAPP VERSIONDESCRIPTION
westos/mychart0.2.0 v2 A Helm chart for Kubernetes
查看mychart所有版本
[root@server1 helm]# helm search repo mychart -l
NAME CHART VERSIONAPP VERSIONDESCRIPTION
westos/mychart0.2.0 v2 A Helm chart for Kubernetes
westos/mychart0.1.0 v1 A Helm chart for Kubernetes
执行更新，当前部署mychart自动升级至0.2.0 v2版本
[root@server1 helm]# helm upgrade mychart westos/mychart
Release 'mychart' has been upgraded. Happy Helming!
NAME: mychart
LAST DEPLOYED: Wed Aug 4 04:00:52 2021
NAMESPACE: default
STATUS: deployed
REVISION: 2
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l 'app.kubernetes.io/name=mychart,app.kubernetes.io/instance=mychart' -o jsonpath='{.items[0].metadata.name}')
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath='{.spec.containers[0].ports[0].containerPort}')
echo 'Visit http://127.0.0.1:8080 to use your application'
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
获取访问ip，测试访问，证实版本为v2，升级成功
[root@server1 helm]# kubectl get pod
NAME READY STATUS RESTARTS AGE
mychart-57fb89f5cc-h69xr 1/1 Running 0 27s
[root@server1 helm]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 10d
mychart ClusterIP 10.109.119.97 <none> 80/TCP 5m52s
[root@server1 helm]# curl 10.109.119.97
Hello MyApp | Version: v2 | <a href='hostname.html'>Pod Name</a>
[root@server1 helm]#
6.应用卸载
[root@server1 nfs-client]# helm list --all-namespaces
NAME NAMESPACEREVISIONUPDATED STATUS CHART APP VERSION
mychartdefault 2 2021-08-04 04:00:52.542465557 -0400 EDTdeployedmychart-0.2.0v2
[root@server1 nfs-client]# helm uninstall mychart
release 'mychart' uninstalled
[root@server1 nfs-client]# helm list --all-namespaces
NAMENAMESPACEREVISIONUPDATEDSTATUSCHARTAPP VERSION
赞 (0)