IT
Kubernetes Cluster Version 업그레이드
concave
2021. 1. 1. 13:26
반응형
CKA 준비 과정에서 명확하게 정리하지 못했던 클러스터 업그레이드 방법을 정리하였다.
버전 업그레이드 절차는 크게 3가지 과정으로 이루어진다.
- Control plane node upgrade
- Additional control plane node upgrade
- worker node upgrade
테스트된 환경은 아래와 같다.
- OS : CentOS 8
- Kubernetes Version : 1.18.5
- External ETCD 3ea, Control Plane 3ea, Worker 2ea
- Cilium (CNI)
- Rook Ceph (CSI)
먼저 업그레이드 할 버전을 확인한다. CentOS에서는 yum list 명령으로 확인 가능하다. 자세한 명령은 Kubernetes.io에 친절하게 설명되어 있다. 현재 버전은 1.18.5-0으로 1.18.6-0으로 버전 업그레이드해 보기로 한다.
[root@k8smaster1 ~]# yum list --showduplicates kubeadm --disableexcludes=kubernetes
Last metadata expiration check: 0:23:04 ago on Fri Jan 1 11:42:27 2021.
Installed Packages
kubeadm.x86_64 1.18.5-0 @kubernetes
Available Packages
kubeadm.x86_64 1.18.4-1 kubernetes
kubeadm.x86_64 1.18.5-0 kubernetes
kubeadm.x86_64 1.18.6-0 kubernetes
kubeadm.x86_64 1.18.8-0 kubernetes
kubeadm.x86_64 1.18.9-0 kubernetes
kubeadm.x86_64 1.18.10-0 kubernetes
kubeadm.x86_64 1.18.12-0 kubernetes
kubeadm.x86_64 1.18.13-0 kubernetes
kubeadm.x86_64 1.18.14-0 kubernetes
kubeadm.x86_64 1.19.0-0 kubernetes
kubeadm.x86_64 1.19.1-0 kubernetes
kubeadm.x86_64 1.19.2-0 kubernetes
kubeadm.x86_64 1.19.3-0 kubernetes
kubeadm.x86_64 1.19.4-0 kubernetes
kubeadm.x86_64 1.19.5-0 kubernetes
kubeadm.x86_64 1.19.6-0 kubernetes
kubeadm.x86_64 1.20.0-0 kubernetes
kubeadm.x86_64 1.20.1-0 kubernetes
[root@k8smaster1 ~]#세부 정확한 버전을 지정하여 업그레이드 한다. 설치가 완료되면 kubeadm의 버전을 확인한다. 정확한 버전이 설치되었는지 확인하자.
[root@k8smaster1 ~]# yum install -y kubeadm-1.18.6-0 --disableexcludes=kubernetes
Last metadata expiration check: 0:37:55 ago on Fri Jan 1 11:42:27 2021.
Dependencies resolved.
=============================================================================================================================================
Package Architecture Version Repository Size
=============================================================================================================================================
Upgrading:
kubeadm x86_64 1.18.6-0 kubernetes 8.8 M
Transaction Summary
=============================================================================================================================================
Upgrade 1 Package
Total download size: 8.8 M
Downloading Packages:
20eefd52d2aee73b3c52abc3d43ed689cb1d79387f5d627faa4a1acc7b4406f9-kubeadm-1.18.6-0.x86_64.rpm 1.8 MB/s | 8.8 MB 00:04
---------------------------------------------------------------------------------------------------------------------------------------------
Total 1.8 MB/s | 8.8 MB 00:04
warning: /var/cache/dnf/kubernetes-33343725abd9cbdc/packages/20eefd52d2aee73b3c52abc3d43ed689cb1d79387f5d627faa4a1acc7b4406f9-kubeadm-1.18.6-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY
Kubernetes 11 kB/s | 3.6 kB 00:00
Importing GPG key 0xA7317B0F:
Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
Fingerprint: D0BC 747F D8CA F711 7500 D6FA 3746 C208 A731 7B0F
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Key imported successfully
Importing GPG key 0xBA07F4FB:
Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
Fingerprint: 54A6 47F9 048D 5688 D7DA 2ABE 6A03 0B21 BA07 F4FB
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Key imported successfully
Importing GPG key 0x836F4BEB:
Userid : "gLinux Rapture Automatic Signing Key (//depot/google3/production/borg/cloud-rapture/keys/cloud-rapture-pubkeys/cloud-rapture-signing-key-2020-12-03-16_08_05.pub) <glinux-team@google.com>"
Fingerprint: 59FE 0256 8272 69DC 8157 8F92 8B57 C5C2 836F 4BEB
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Key imported successfully
Kubernetes 2.4 kB/s | 975 B 00:00
Importing GPG key 0x3E1BA8D5:
Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>"
Fingerprint: 3749 E1BA 95A8 6CE0 5454 6ED2 F09C 394C 3E1B A8D5
From : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: kubeadm-1.18.6-0.x86_64 1/1
Upgrading : kubeadm-1.18.6-0.x86_64 1/2
Cleanup : kubeadm-1.18.5-0.x86_64 2/2
Running scriptlet: kubeadm-1.18.5-0.x86_64 2/2
Verifying : kubeadm-1.18.6-0.x86_64 1/2
Verifying : kubeadm-1.18.5-0.x86_64 2/2
Upgraded:
kubeadm-1.18.6-0.x86_64
Complete!
[root@k8smaster1 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:56:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
[root@k8smaster1 ~]#Control plane 노드를 드레인 한다. 드레인할 노드 이름은 kubectl get nodes 명령으로 확인 가능하다.
[root@k8smaster1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster1.homelab.local Ready master 12h v1.18.5
k8smaster2.homelab.local Ready master 12h v1.18.5
k8smaster3.homelab.local Ready master 12h v1.18.5
k8sworker1.homelab.local Ready <none> 12h v1.18.5
k8sworker2.homelab.local Ready <none> 12h v1.18.5
[root@k8smaster1 ~]# kubectl drain k8smaster1.homelab.local --ignore-daemonsets
node/k8smaster1.homelab.local cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/cilium-pfnj4, kube-system/kube-proxy-lcrts
evicting pod kube-system/coredns-66bff467f8-dfp5k
evicting pod kube-system/coredns-66bff467f8-p84vw
pod/coredns-66bff467f8-dfp5k evicted
pod/coredns-66bff467f8-p84vw evicted
node/k8smaster1.homelab.local evicted
[root@k8smaster1 ~]#노드 드레인이 완료되면 업그레이드 세부 계획을 확인한다.
[root@k8smaster1 ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.18.5
[upgrade/versions] kubeadm version: v1.18.6
I0101 12:26:24.265437 60620 version.go:252] remote version is much newer: v1.20.1; falling back to: stable-1.18
[upgrade/versions] Latest stable version: v1.18.14
[upgrade/versions] Latest stable version: v1.18.14
[upgrade/versions] Latest version in the v1.18 series: v1.18.14
[upgrade/versions] Latest version in the v1.18 series: v1.18.14
External components that should be upgraded manually before you upgrade the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Etcd 3.4.3 3.4.3-0
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 5 x v1.18.5 v1.18.14
Upgrade to the latest version in the v1.18 series:
COMPONENT CURRENT AVAILABLE
API Server v1.18.5 v1.18.14
Controller Manager v1.18.5 v1.18.14
Scheduler v1.18.5 v1.18.14
Kube Proxy v1.18.5 v1.18.14
CoreDNS 1.6.7 1.6.7
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.18.14
Note: Before you can perform this upgrade, you have to update kubeadm to v1.18.14.
_____________________________________________________________________
[root@k8smaster1 ~]#현재 업그레이드 가능한 버전은 v1.18.14로 표시되지만 위에서 v1.18.6으로 패키지를 업그레이드 하였으므로 v1.18.6으로 버전 업그레이드를 진행한다.
[root@k8smaster1 ~]# kubeadm upgrade apply v1.18.6
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.18.6"
[upgrade/versions] Cluster version: v1.18.5
[upgrade/versions] kubeadm version: v1.18.6
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler]
[upgrade/prepull] Prepulling image for component kube-scheduler.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.18.6"...
Static pod: kube-apiserver-k8smaster1.homelab.local hash: a4f0665fb56db40aca9954d5a965c8b8
Static pod: kube-controller-manager-k8smaster1.homelab.local hash: 73c3b2c91edc9c9ec292a8640b4099eb
Static pod: kube-scheduler-k8smaster1.homelab.local hash: 3415bde3e2a04810cc416f7719a3f6aa
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests928750774"
W0101 12:28:54.011092 63274 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-01-12-28-53/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-k8smaster1.homelab.local hash: a4f0665fb56db40aca9954d5a965c8b8
Static pod: kube-apiserver-k8smaster1.homelab.local hash: a4f0665fb56db40aca9954d5a965c8b8
Static pod: kube-apiserver-k8smaster1.homelab.local hash: a4f0665fb56db40aca9954d5a965c8b8
Static pod: kube-apiserver-k8smaster1.homelab.local hash: aefb945015c6406dd67fd9042bf1ba2a
[apiclient] Found 3 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-01-12-28-53/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-k8smaster1.homelab.local hash: 73c3b2c91edc9c9ec292a8640b4099eb
Static pod: kube-controller-manager-k8smaster1.homelab.local hash: d405c8788f24e20e8a38ce798df7cf87
[apiclient] Found 3 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-01-12-28-53/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-k8smaster1.homelab.local hash: 3415bde3e2a04810cc416f7719a3f6aa
Static pod: kube-scheduler-k8smaster1.homelab.local hash: 3dd66788a2c7782d910d05ea37b91678
[apiclient] Found 3 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.18.6". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
[root@k8smaster1 ~]#Control plane 노드가 업그레이드되면 CNI Plugin을 업그레이드 한다. 여기서는 CNI로 cilium을 사용하고 cilium은 daemon set으로 실행되므로 추가적인 CNI Plugin 업그레이드는 필요하지 않다.
[root@k8smaster1 ~]# kubectl get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
cilium 5 5 5 5 5 <none> 11h
kube-proxy 5 5 5 5 5 kubernetes.io/os=linux 12h
[root@k8smaster1 ~]#Control plane 노드에 적용된 cordon을 해제한다.
[root@k8smaster1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster1.homelab.local Ready,SchedulingDisabled master 12h v1.18.5
k8smaster2.homelab.local Ready master 12h v1.18.5
k8smaster3.homelab.local Ready master 12h v1.18.5
k8sworker1.homelab.local Ready <none> 12h v1.18.5
k8sworker2.homelab.local Ready <none> 12h v1.18.5
[root@k8smaster1 ~]# kubectl uncordon k8smaster1.homelab.local
node/k8smaster1.homelab.local uncordoned
[root@k8smaster1 ~]#추가적인 control plane 노드 업그레이드를 진행한다. 첫번째 control plane 노드와 동일한 업그레이드가 필요하지만 apply 대신 node 명령을 사용하여 업그레이드 진행한다. (upgrade node에서는 버전 지정을 하면 오류가 발생하닌 버전을 생략하고 업그레이드한다.)
- kubeadm install
- drain control plane
- upgrade node
- uncordon control plane
[root@k8smaster2 ~]# yum install -y kubeadm-1.18.6-0 --disableexcludes=kubernetes
Last metadata expiration check: 1:00:23 ago on Fri Jan 1 11:42:26 2021.
Dependencies resolved.
=============================================================================================================================================
Package Architecture Version Repository Size
=============================================================================================================================================
Upgrading:
kubeadm x86_64 1.18.6-0 kubernetes 8.8 M
Transaction Summary
=============================================================================================================================================
Upgrade 1 Package
Total download size: 8.8 M
Downloading Packages:
20eefd52d2aee73b3c52abc3d43ed689cb1d79387f5d627faa4a1acc7b4406f9-kubeadm-1.18.6-0.x86_64.rpm 1.8 MB/s | 8.8 MB 00:04
---------------------------------------------------------------------------------------------------------------------------------------------
Total 1.8 MB/s | 8.8 MB 00:04
warning: /var/cache/dnf/kubernetes-33343725abd9cbdc/packages/20eefd52d2aee73b3c52abc3d43ed689cb1d79387f5d627faa4a1acc7b4406f9-kubeadm-1.18.6-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY
Kubernetes 8.7 kB/s | 3.6 kB 00:00
Importing GPG key 0xA7317B0F:
Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
Fingerprint: D0BC 747F D8CA F711 7500 D6FA 3746 C208 A731 7B0F
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Key imported successfully
Importing GPG key 0xBA07F4FB:
Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
Fingerprint: 54A6 47F9 048D 5688 D7DA 2ABE 6A03 0B21 BA07 F4FB
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Key imported successfully
Importing GPG key 0x836F4BEB:
Userid : "gLinux Rapture Automatic Signing Key (//depot/google3/production/borg/cloud-rapture/keys/cloud-rapture-pubkeys/cloud-rapture-signing-key-2020-12-03-16_08_05.pub) <glinux-team@google.com>"
Fingerprint: 59FE 0256 8272 69DC 8157 8F92 8B57 C5C2 836F 4BEB
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Key imported successfully
Kubernetes 2.2 kB/s | 975 B 00:00
Importing GPG key 0x3E1BA8D5:
Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>"
Fingerprint: 3749 E1BA 95A8 6CE0 5454 6ED2 F09C 394C 3E1B A8D5
From : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: kubeadm-1.18.6-0.x86_64 1/1
Upgrading : kubeadm-1.18.6-0.x86_64 1/2
Cleanup : kubeadm-1.18.5-0.x86_64 2/2
Running scriptlet: kubeadm-1.18.5-0.x86_64 2/2
Verifying : kubeadm-1.18.6-0.x86_64 1/2
Verifying : kubeadm-1.18.5-0.x86_64 2/2
Upgraded:
kubeadm-1.18.6-0.x86_64
Complete!
[root@k8smaster2 ~]# kubectl drain k8smaster2.homelab.local --ignore-daemonsets
node/k8smaster2.homelab.local cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/cilium-r4tw9, kube-system/kube-proxy-6w77h
node/k8smaster2.homelab.local drained
[root@k8smaster2 ~]# kubeadm upgrade node v1.18.6
unknown command "v1.18.6" for "kubeadm upgrade node"
To see the stack trace of this error execute with --v=5 or higher
[root@k8smaster2 ~]# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Upgrading your Static Pod-hosted control plane instance to version "v1.18.6"...
Static pod: kube-apiserver-k8smaster2.homelab.local hash: dc02b2d16a200dc98fa76d6bf4dc773a
Static pod: kube-controller-manager-k8smaster2.homelab.local hash: 73c3b2c91edc9c9ec292a8640b4099eb
Static pod: kube-scheduler-k8smaster2.homelab.local hash: 3415bde3e2a04810cc416f7719a3f6aa
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests533003659"
W0101 12:47:49.200126 32189 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-01-12-47-49/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-k8smaster2.homelab.local hash: dc02b2d16a200dc98fa76d6bf4dc773a
Static pod: kube-apiserver-k8smaster2.homelab.local hash: 7482cb4608713323692fc61d99f9ef7b
[apiclient] Found 3 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-01-12-47-49/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-k8smaster2.homelab.local hash: 73c3b2c91edc9c9ec292a8640b4099eb
Static pod: kube-controller-manager-k8smaster2.homelab.local hash: d405c8788f24e20e8a38ce798df7cf87
[apiclient] Found 3 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-01-12-47-49/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-k8smaster2.homelab.local hash: 3415bde3e2a04810cc416f7719a3f6aa
Static pod: kube-scheduler-k8smaster2.homelab.local hash: 3dd66788a2c7782d910d05ea37b91678
[apiclient] Found 3 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade] The control plane instance for this node was successfully updated!
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
[root@k8smaster2 ~]# kubectl uncordon k8smaster2.homelab.local
node/k8smaster2.homelab.local uncordoned
[root@k8smaster2 ~]#추가 3번째 control plane은 2번과 동일하다. 동일한 과정이므로 로그는 생략하였다.
control plane에서 업그레이드 해도 버전이 동일하게 v1.18.5로 표시되는것을 확인할 수 있다. 업그레이드된 버전 확인은 kubelet과 kubectl 패키지 업데이트 후 kubelet 서비스를 재 시작해야 버전 업그레이드된 것으로 확인 가능하다.
[root@k8smaster1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster1.homelab.local Ready master 13h v1.18.5
k8smaster2.homelab.local Ready master 13h v1.18.5
k8smaster3.homelab.local Ready master 13h v1.18.5
k8sworker1.homelab.local Ready <none> 13h v1.18.5
k8sworker2.homelab.local Ready <none> 13h v1.18.5
[root@k8smaster1 ~]#
모든 control plane에서 버전 업그레이드가 완료되면 kubelet과 kubectl 업그레이드를 진행한다.
kubelet과 kubectl 패키지 업그레이드 후 kubelet을 재 시작해야 한다.
[root@k8smaster1 ~]# yum install -y kubelet-1.18.6-0 kubectl-1.18.6-0 --disableexcludes=kubernetes
Last metadata expiration check: 1:19:47 ago on Fri Jan 1 11:42:27 2021.
Dependencies resolved.
=============================================================================================================================================
Package Architecture Version Repository Size
=============================================================================================================================================
Upgrading:
kubectl x86_64 1.18.6-0 kubernetes 9.5 M
kubelet x86_64 1.18.6-0 kubernetes 21 M
Transaction Summary
=============================================================================================================================================
Upgrade 2 Packages
Total download size: 30 M
Downloading Packages:
(1/2): 9fe14ad1137ad4e42eca5df1df99c735098e1ef43ead5184ee2af108d31ecb44-kubectl-1.18.6-0.x86_64.rpm 2.8 MB/s | 9.5 MB 00:03
(2/2): 155c953863e5dc40f1d0cd5010d4df91b45d8c62edc5e93f7fc508516015fcb1-kubelet-1.18.6-0.x86_64.rpm 3.1 MB/s | 21 MB 00:06
---------------------------------------------------------------------------------------------------------------------------------------------
Total 4.5 MB/s | 30 MB 00:06
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: kubelet-1.18.6-0.x86_64 1/1
Upgrading : kubelet-1.18.6-0.x86_64 1/4
Upgrading : kubectl-1.18.6-0.x86_64 2/4
Cleanup : kubectl-1.18.5-0.x86_64 3/4
Cleanup : kubelet-1.18.5-0.x86_64 4/4
Running scriptlet: kubelet-1.18.5-0.x86_64 4/4
Verifying : kubectl-1.18.6-0.x86_64 1/4
Verifying : kubectl-1.18.5-0.x86_64 2/4
Verifying : kubelet-1.18.6-0.x86_64 3/4
Verifying : kubelet-1.18.5-0.x86_64 4/4
Upgraded:
kubectl-1.18.6-0.x86_64 kubelet-1.18.6-0.x86_64
Complete!
[root@k8smaster1 ~]# systemctl daemon-reload
[root@k8smaster1 ~]# systemctl restart kubelet
[root@k8smaster1 ~]#kubelet까지 재 시작 후 node 버전을 확인해 보면 최종 버전이 v1.18.6으로 업그레이드 된것을 확인 가능하다.
[root@k8smaster1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster1.homelab.local Ready master 13h v1.18.6
k8smaster2.homelab.local Ready master 13h v1.18.5
k8smaster3.homelab.local Ready master 13h v1.18.5
k8sworker1.homelab.local Ready <none> 13h v1.18.5
k8sworker2.homelab.local Ready <none> 13h v1.18.5
[root@k8smaster1 ~]#추가적인 control plane 노드 업그레이드 작업도 진행한다. 첫번째 control plane와 동일하다. 모든 control plane의 버전 업그레이드가 완료되면 kubectl get nodes 명령으로 버전을 확인하자.
[root@k8smaster1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster1.homelab.local Ready master 13h v1.18.6
k8smaster2.homelab.local Ready master 13h v1.18.6
k8smaster3.homelab.local Ready master 13h v1.18.6
k8sworker1.homelab.local Ready <none> 13h v1.18.5
k8sworker2.homelab.local Ready <none> 13h v1.18.5
[root@k8smaster1 ~]#
worker 노드 2대에서 버전 업그레이드를 진행한다. 업그레이드 절차는 추가 control plane 방법과 동일한다. 하지만 drain, uncordon 명령어는 일반적으로 master 노드에서만 실행되도록 kubeconfig 파일이 설정되어 있는게 일반적이므로 해당 명령어는 master에 실행하도록 한다.
- kubeadm package upgrade
- drain worker node
- upgrade node
- kubelet, kubectl package upgrade
- daemon-reload
- restart kubelet
- uncordon worker node
[root@k8sworker1 ~]# yum install -y kubeadm-1.18.6-0 --disableexcludes=kubernetes
Last metadata expiration check: 1:32:45 ago on Fri Jan 1 11:42:28 2021.
Dependencies resolved.
=============================================================================================================================================
Package Architecture Version Repository Size
=============================================================================================================================================
Upgrading:
kubeadm x86_64 1.18.6-0 kubernetes 8.8 M
Transaction Summary
=============================================================================================================================================
Upgrade 1 Package
Total download size: 8.8 M
Downloading Packages:
20eefd52d2aee73b3c52abc3d43ed689cb1d79387f5d627faa4a1acc7b4406f9-kubeadm-1.18.6-0.x86_64.rpm 2.7 MB/s | 8.8 MB 00:03
---------------------------------------------------------------------------------------------------------------------------------------------
Total 2.7 MB/s | 8.8 MB 00:03
warning: /var/cache/dnf/kubernetes-33343725abd9cbdc/packages/20eefd52d2aee73b3c52abc3d43ed689cb1d79387f5d627faa4a1acc7b4406f9-kubeadm-1.18.6-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY
Kubernetes 7.1 kB/s | 3.6 kB 00:00
Importing GPG key 0xA7317B0F:
Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
Fingerprint: D0BC 747F D8CA F711 7500 D6FA 3746 C208 A731 7B0F
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Key imported successfully
Importing GPG key 0xBA07F4FB:
Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
Fingerprint: 54A6 47F9 048D 5688 D7DA 2ABE 6A03 0B21 BA07 F4FB
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Key imported successfully
Importing GPG key 0x836F4BEB:
Userid : "gLinux Rapture Automatic Signing Key (//depot/google3/production/borg/cloud-rapture/keys/cloud-rapture-pubkeys/cloud-rapture-signing-key-2020-12-03-16_08_05.pub) <glinux-team@google.com>"
Fingerprint: 59FE 0256 8272 69DC 8157 8F92 8B57 C5C2 836F 4BEB
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Key imported successfully
Kubernetes 2.3 kB/s | 975 B 00:00
Importing GPG key 0x3E1BA8D5:
Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>"
Fingerprint: 3749 E1BA 95A8 6CE0 5454 6ED2 F09C 394C 3E1B A8D5
From : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: kubeadm-1.18.6-0.x86_64 1/1
Upgrading : kubeadm-1.18.6-0.x86_64 1/2
Cleanup : kubeadm-1.18.5-0.x86_64 2/2
Running scriptlet: kubeadm-1.18.5-0.x86_64 2/2
Verifying : kubeadm-1.18.6-0.x86_64 1/2
Verifying : kubeadm-1.18.5-0.x86_64 2/2
Upgraded:
kubeadm-1.18.6-0.x86_64
Complete!
[root@k8sworker1 ~]# [root@k8smaster1 ~]# kubectl drain k8sworker1.homelab.local --ignore-daemonsets
node/k8sworker1.homelab.local cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/cilium-8vrkc, kube-system/kube-proxy-xqlgt, rook-ceph/rook-discover-gjnbg
evicting pod kube-system/coredns-66bff467f8-49shv
pod/coredns-66bff467f8-49shv evicted
node/k8sworker1.homelab.local evicted
[root@k8smaster1 ~]#[root@k8sworker1 ~]# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Skipping phase. Not a control plane node.
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
[root@k8sworker1 ~]# yum install -y kubelet-1.18.6-0 kubectl-1.18.6-0 --disableexcludes=kubernetes
Last metadata expiration check: 1:32:54 ago on Fri Jan 1 11:42:28 2021.
Dependencies resolved.
=============================================================================================================================================
Package Architecture Version Repository Size
=============================================================================================================================================
Upgrading:
kubectl x86_64 1.18.6-0 kubernetes 9.5 M
kubelet x86_64 1.18.6-0 kubernetes 21 M
Transaction Summary
=============================================================================================================================================
Upgrade 2 Packages
Total download size: 30 M
Downloading Packages:
(1/2): 9fe14ad1137ad4e42eca5df1df99c735098e1ef43ead5184ee2af108d31ecb44-kubectl-1.18.6-0.x86_64.rpm 3.1 MB/s | 9.5 MB 00:03
(2/2): 155c953863e5dc40f1d0cd5010d4df91b45d8c62edc5e93f7fc508516015fcb1-kubelet-1.18.6-0.x86_64.rpm 4.7 MB/s | 21 MB 00:04
---------------------------------------------------------------------------------------------------------------------------------------------
Total 6.8 MB/s | 30 MB 00:04
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: kubelet-1.18.6-0.x86_64 1/1
Upgrading : kubelet-1.18.6-0.x86_64 1/4
Upgrading : kubectl-1.18.6-0.x86_64 2/4
Cleanup : kubectl-1.18.5-0.x86_64 3/4
Cleanup : kubelet-1.18.5-0.x86_64 4/4
Running scriptlet: kubelet-1.18.5-0.x86_64 4/4
Verifying : kubectl-1.18.6-0.x86_64 1/4
Verifying : kubectl-1.18.5-0.x86_64 2/4
Verifying : kubelet-1.18.6-0.x86_64 3/4
Verifying : kubelet-1.18.5-0.x86_64 4/4
Upgraded:
kubectl-1.18.6-0.x86_64 kubelet-1.18.6-0.x86_64
Complete!
[root@k8sworker1 ~]# systemctl daemon-reload
[root@k8sworker1 ~]# systemctl restart kubelet[root@k8smaster1 ~]# kubectl uncordon k8sworker1.homelab.local
node/k8sworker1.homelab.local uncordoned
[root@k8smaster1 ~]#업그레이드 완료되었으면 kubectl get nodes 명령으로 최종 버전을 확인한다.
[root@k8smaster1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster1.homelab.local Ready master 13h v1.18.6
k8smaster2.homelab.local Ready master 13h v1.18.6
k8smaster3.homelab.local Ready master 13h v1.18.6
k8sworker1.homelab.local Ready <none> 13h v1.18.6
k8sworker2.homelab.local Ready <none> 13h v1.18.5
[root@k8smaster1 ~]#
추가 worker만 업그레이드 첫번째 worker 업그레이드와 동일한다. 출력 로그는 생략하였다.
전체 노드에 대해서 위 대로 실행하였다면 아래와 같이 v1.18.5 -> v1.18.6으로 업그레이드 된것을 확인 가능하다.
[root@k8smaster1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster1.homelab.local Ready master 13h v1.18.6
k8smaster2.homelab.local Ready master 13h v1.18.6
k8smaster3.homelab.local Ready master 13h v1.18.6
k8sworker1.homelab.local Ready <none> 13h v1.18.6
k8sworker2.homelab.local Ready <none> 13h v1.18.6
[root@k8smaster1 ~]#
Reference
- https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
Upgrading kubeadm clusters
This page explains how to upgrade a Kubernetes cluster created with kubeadm from version 1.19.x to version 1.20.x, and from version 1.20.x to 1.20.y (where y > x). Skipping MINOR versions when upgrading is unsupported. To see information about upgrading cl
kubernetes.io
반응형