We run a GitOps based infrastructure for our Kubernetes cluster. When setting this up, keeping secrets was one of the first challenges to figure out. How do we make our secret values available to the automated deployments within the cluster, but also easy for developers to work with and edit? Database credentials, API keys, and passwords all need to be available within the infrastructure’s git repository, and must be kept secure.

Here is a developer and GitOps friendly workflow for working with secrets, using a combination of tools: sops, SealedSecrets, Husky, and Bash.

Secrets aren’t secret

Kubernetes uses Secret objects to store sensitive information in the cluster. So we store Kubernetes Secret resources in our git repository, right? No. Astute readers will remember that Kubernetes Secret manifests aren’t secret. Secret values within bare manifests are base 64 encoded strings, and decoding them is trivial. We never want to commit Kubernetes Secrets to git. I think of Secrets as a platform for secrecy within the Kubernetes cluster, but securing access to the data within the cluster and at-rest within our infrastructure repository is still up to us.

Sealed Secrets

Because we’re using GitOps, we want to commit all our files to git, but we can’t simply commit a bare Secret manifest, they’re not really secret, after all. Instead, we create a SealedSecret resource. This is an encrypted object that is decrypted by a controller in the cluster to produce a normal Kubernetes Secret. SealedSecrets are safe to commit to git.

Fine, but if we want to update the values in our SealedSecret, we need to get all its secret values somehow, edit them, recreate the Kubernetes Secret manifest, then re-encrypt it into a SealedSecret. So we still need to keep our actual secret values somewhere that developers can access and amend.

sops

Thankfully sops solves this problem for us. sops provides line-by-line value encryption, so we can see the key for each encrypted value in plain text, and we can even use git-diff to see which keys changed from commit to commit (though obviously not their values).

Encryption can be managed with PGP keys or key management systems provided by AWS KMS, GCP KMS or Azure Key Vault. This makes it easy for developers with the right KMS credentials to be given access to the keys necessary to work with these files.

The sops files are the source-of-truth for our secrets, and can be committed to git. To edit a sops encrypted file in your default editor:

sops path/to/secrets.sops.yaml

From sops to SealedSecrets

Awesome, so we have developer-friendly sops files as a source-of-truth for secret values, and GitOps/Kubernetes friendly SealedSecret resources. You’ve edited and saved a sops file, how do you create the corresponding SealedSecret for the cluster?

The procedure is as follows:

1. Decrypt secrets.sops.yaml with sops
2. Create a Kubernetes Generic Secret using the decrypted file as input
3. Use kubeseal to create a SealedSecret using the Secret we created
4. Write the SealedSecret to a file

This SealedSecret file can now be committed so gitops and the SealedSecrets controller can make if available as a Secret within the cluster.

This is a bit long-winded, so we usually put commands like this into a Bash script:

#!/bin/bash

APP=$1
sealedsecret="$(sops -d vars/"${APP}"/secrets.sops.yaml \
  | kubectl create secret generic \
      "${APP}"-secrets \
      --dry-run=client \
      --from-file=secrets.yaml=/dev/stdin \
      -o yaml \
  | kubeseal --format=yaml --cert=./sealedsecrets-pub-cert.pem)" \
&& echo "${sealedsecret}" > ./"${APP}".sealedsecret.yaml

So each time a sops file has been edited, we run our script to generate the corresponding SealedSecret: ./secret.sh myapp

Trigger SealedSecrets generation with Husky

Remembering to run that command to generate SealedSecrets gets old fast. It’s too easy to forget to do it after you’ve edited a sops file, and be left wondering why the state of your app hasn’t changed after committing.

We use Husky to add a post-commit git hook to ensure that every time a sops file in vars/ has been committed, the corresponding SealedSecret file is generated and added to the commit too.

All a developer has to do is run npm install to install Husky, and the githook defined in package.json will be run after each commit:

  ...
  "devDependencies": {
    "husky": "^4.0.0",
  },
  "husky": {
    "hooks": {
      "post-commit": "./amend_with_secrets.sh --silent"
    }
  }

amend_with_secrets.sh is another bash script we’ve created to regenerate and add sealed-secrets to the last git commit. It looks like this:

#!/bin/bash

REG_EXP="^vars\/(.+)\.secrets\.sops\.yaml"

git show --pretty="" --name-only --no-commit-id | \
{ grep -E "${REG_EXP}" || test $? = 1; } | \

while IFS= read -r line; do
  if [[ $line =~ $REG_EXP ]]
  then
    app="${BASH_REMATCH[1]}"
    ./secret.sh ${app}
    git add "./$app.sealedsecret.yaml"
  fi
done

STAGED=$(git diff --name-only --cached)
if [ "$STAGED" ]; then
  HUSKY_SKIP_HOOKS=1 git commit --amend --no-edit
fi

1. Call git show to list the files in the last commit that match the regex.
2. For each file path that matches the regex, e.g. vars/myapp/secrets.sops.yaml, call our secret.sh script
3. git add the generated SealedSecret
4. git commit --amend to add them to the previous commit

With this in place, all a developer has to do is work with sops files, and doesn’t have to worry about generating SealedSecrets at all. Husky takes care of that for us.

Summary

We now have a workflow so a team of developers can work with secrets using the developer-friendly sops tool, but automatically generate the corresponding GitOps/Kubernetes-friendly SealedSecret.