Guide to recovering lost SECRETS_SYSTEM

[lorenzoranucci]

We are running hydra on Kubernetes. Somehow, for a reason under investigation, we lost the SECRETS_SYSTEM value.

When we deployed the first time using the helm chart the expected SECRETS_SYSTEM value supplied in the hydra.config.secrets.system variable was probably ignored and a random one was used.

We didn't discover this issue until we enabled the capability to "refresh" our kube secrets (see here) for internal reasons. After this event the SECRETS_SYSTEM for the new pods started to be valued with the expected hydra.config.secrets.system determining a mismatch with the initial one generated randomly on the first deployment.

We started to have a fatal error on new pods startup like the following one:
level=fatal msg="Could not ensure that signing keys for \"hydra.openid.id-token\" exists. If you are running against a persistent SQL database this is most likely because your \"secrets.system\" (\"SECRETS_SYSTEM\" environment variable) is not set or changed. When running with an SQL database backend you need to make sure that the secret is set and stays the same, unless when doing key rotation. This may also happen when you forget to run \"hydra migrate sql\".."

Fortunately, some "old" pods kept running so we tried to recover the initial random SECRETS_SYSTEM value from them so that we could reuse it as hydra.config.secrets.system value. We tried the following:

1. using a shell to print the SECRETS_SYSTEM env var, but unfortunately, we were using the hydra scratch image, so we had no shell to use
2. using a kubectl describe, but it mask secret values
3. kubectl cp to inject a binary to print the value inside the running container. It failed because cp tar the given binary and the scratch image has no tool to untar
4. looking at old helm logs, but the oldest we had were still too recent
5. at the end we succeeded, accessing one of our cluster nodes, and using directly the container runtime (docker in this case) to run a docker inspect and take SECRETS_SYSTEM env var value.

I hope it can help and that someone could have a smarter way to solve the same problem. Having a way to provide a new SECRETS_SYSTEM without losing existing tokens would be the best :)

[aeneasr]

Awesome, thank you for the write up!