As part of best practices we should keep our software with latest stable released version. As responsible developers, we have followed this and updated our AKS version in production. Sorry, I missed to inform we are using Kubernetes to manage our application scaling needs and helm to easily manage our releases.
All good till now. Now when I try to deploy new changes using helm upgrade
I keep getting this error, even when I am passing the updated Kubernetes manifests. So what's going on here? Let's try to understand the problem and finally work on fixing it.
Error: UPGRADE FAILED: current release manifest contains removed kubernetes api(s)
for this kubernetes version and it is therefore unable to build the kubernetes
objects for performing the diff. error from kubernetes: unable to recognize "":
no matches for kind "Deployment" in version "apps/v1beta1"
NOTE: This error can happen for other resources as well like ConfigMap
, Secret
, Service
, Deployment
and Ingress
etc.
Why this error?
Helm always upgrade the chart by creating a diff patch between the currently deployed release against the new manifest passed. When we upgraded our cluster we have missed to update the deprecated APIs of Kubernetes. Updated AKS version is now missing these APIs. As these API versions have been removed from Kubernetes, the Kuberenetes Go client library cannot parse these deprecated objects. Hence, helm fails when it tries to call the library and unable to manage the release.
How to fix this error?
Helm release object stores the manifest details inside the data
field of a Secret(default) or ConfigMap in the cluster. The data field contains a gzipped object which is base64 encoded. Secrets are encode one more time with base64 encoding. This is exactly where the problem lies. This release object contains the old API version and we need to update it.
Option 1
Use helm-mapkubeapis plugin to fix the issue. However, this didn't work for us. It kept showing there is no deprecated API in the already deployed release.
Now what? Do we delete the entire namespace and recreate everything? But that will cause a severe downtime and our business cannot afford that. So, let's look into the other option.
Option 2
It's a bit of a manual process but it worked for us.
Grab the release name from the currently deployed resources
helm list -n <namespace>
Get the name of Secret or ConfigMap that is currently present in the deployed release
Secretkubectl get secret -l owner=helm,status=deployed,name=<release_name> --namespace <namespace> | awk '{print $1}' | grep -v NAME
ConfigMap
kubectl get configmap -l owner=helm,status=deployed,name=<release_name> --namespace <namespace> | awk '{print $1}' | grep -v NAME
Above command should result in a Secret or ConfigMap name. We will use this in the next command
Secretkubectl get secret <secret_name> -n <namespace> -o yaml > release.yaml
OR
ConfigMapkubectl get configmap <configmap_name> -n <namespace> -o yaml > release.yaml
Previous command should have created a
release.yaml
file in the current working directory. Let's create a backup of this file, in case anything goes wrong.cp release.yaml release.bak
Optional: In case anything goes wrong, we can restore the previous release
kubectl apply -f release.bak -n <namespace>
Let's decode the release object
Secretcat release.yaml | ggrep -oP '(?<=release: ).*' | base64 -d | base64 -d | gzip -d > release.data.decoded
OR
ConfigMapcat release.yaml | ggrep -oP '(?<=release: ).*' | base64 -d | gzip -d > release.data.decoded
In the
release.data.decoded
file, find the deprecated APIs and update itLet's encode the updated release object
Secretcat release.data.decoded | gzip | base64 | base64
OR
ConfigMapcat release.data.decoded | gzip | base64
Copy the output of the previous command and update
data.release
property value in therelease.yaml
fileApply the new release file
kubectl apply -f release.yaml -n <namespace>
Run
helm upgrade
command with a version of chart containing the supported Kubernetes API versionsAdd appropriate description in the update, to avoid rolling back to an older version prior to the current version
Learnings
Above steps can be safely followed to fix broken helm release in Kubernetes cluster. Another important learning, always upgrade the deprecated API versions to supported API versions, before upgrading Kubernetes cluster that removes those API versions, so that you don't end up in this situation at the first place. :P But as always let's focus on the learning from this mistake, it paved a way to learn the internals of helm and fix the issue.
Thank you for reading this. We can have a discussion in the comment section. Please share your valuable feedback.