Fixing broken release manifest in helm after AKS upgrade

Fixing broken release manifest in helm after AKS upgrade

As part of best practices we should keep our software with latest stable released version. As responsible developers, we have followed this and updated our AKS version in production. Sorry, I missed to inform we are using Kubernetes to manage our application scaling needs and helm to easily manage our releases.

All good till now. Now when I try to deploy new changes using helm upgrade I keep getting this error, even when I am passing the updated Kubernetes manifests. So what's going on here? Let's try to understand the problem and finally work on fixing it.

Error: UPGRADE FAILED: current release manifest contains removed kubernetes api(s)
for this kubernetes version and it is therefore unable to build the kubernetes
objects for performing the diff. error from kubernetes: unable to recognize "":
no matches for kind "Deployment" in version "apps/v1beta1"

NOTE: This error can happen for other resources as well like ConfigMap, Secret, Service, Deployment and Ingress etc.

Why this error?

Helm always upgrade the chart by creating a diff patch between the currently deployed release against the new manifest passed. When we upgraded our cluster we have missed to update the deprecated APIs of Kubernetes. Updated AKS version is now missing these APIs. As these API versions have been removed from Kubernetes, the Kuberenetes Go client library cannot parse these deprecated objects. Hence, helm fails when it tries to call the library and unable to manage the release.

How to fix this error?

Helm release object stores the manifest details inside the data field of a Secret(default) or ConfigMap in the cluster. The data field contains a gzipped object which is base64 encoded. Secrets are encode one more time with base64 encoding. This is exactly where the problem lies. This release object contains the old API version and we need to update it.

Option 1

Use helm-mapkubeapis plugin to fix the issue. However, this didn't work for us. It kept showing there is no deprecated API in the already deployed release.

Now what? Do we delete the entire namespace and recreate everything? But that will cause a severe downtime and our business cannot afford that. So, let's look into the other option.

Option 2

It's a bit of a manual process but it worked for us.

  1. Grab the release name from the currently deployed resources

    helm list -n <namespace>
    
  2. Get the name of Secret or ConfigMap that is currently present in the deployed release
    Secret

    kubectl get secret -l owner=helm,status=deployed,name=<release_name> --namespace <namespace> | awk '{print $1}' | grep -v NAME
    

    ConfigMap

    kubectl get configmap -l owner=helm,status=deployed,name=<release_name> --namespace <namespace> | awk '{print $1}' | grep -v NAME
    
  3. Above command should result in a Secret or ConfigMap name. We will use this in the next command
    Secret

    kubectl get secret <secret_name> -n <namespace> -o yaml > release.yaml
    

    OR
    ConfigMap

    kubectl get configmap <configmap_name> -n <namespace> -o yaml > release.yaml
    
  4. Previous command should have created a release.yaml file in the current working directory. Let's create a backup of this file, in case anything goes wrong.

    cp release.yaml release.bak
    

    Optional: In case anything goes wrong, we can restore the previous release

    kubectl apply -f release.bak -n <namespace>
    
  5. Let's decode the release object
    Secret

    cat release.yaml | ggrep -oP '(?<=release: ).*' | base64 -d | base64 -d | gzip -d > release.data.decoded
    

    OR
    ConfigMap

    cat release.yaml | ggrep -oP '(?<=release: ).*' | base64 -d | gzip -d > release.data.decoded
    
  6. In the release.data.decoded file, find the deprecated APIs and update it

  7. Let's encode the updated release object
    Secret

    cat release.data.decoded | gzip | base64 | base64
    

    OR
    ConfigMap

    cat release.data.decoded | gzip | base64
    
  8. Copy the output of the previous command and update data.release property value in the release.yaml file

  9. Apply the new release file

    kubectl apply -f release.yaml -n <namespace>
    
  10. Run helm upgrade command with a version of chart containing the supported Kubernetes API versions

  11. Add appropriate description in the update, to avoid rolling back to an older version prior to the current version

Learnings

Above steps can be safely followed to fix broken helm release in Kubernetes cluster. Another important learning, always upgrade the deprecated API versions to supported API versions, before upgrading Kubernetes cluster that removes those API versions, so that you don't end up in this situation at the first place. :P But as always let's focus on the learning from this mistake, it paved a way to learn the internals of helm and fix the issue.

Thank you for reading this. We can have a discussion in the comment section. Please share your valuable feedback.

Resources

  1. Helm Docs: Deprecated Kubernetes APIs