Ensure NGINX reload actually happens

[kate-osborn]

When we reload NGINX, we do not check whether the reload succeeds. We just sleep for a second to wait for the new workers to spin up. We need to verify that a reload was successful and return an error if it fails. Additionally, we should retry on failure.

Acceptance:

• When a NGINX reload occurs, NKG will wait for the reload to finish, and if an error occurs, the error is reported to the user.
• When NGINX fails to reload, NKG will retry reloading NGINX.

This work should resolve the following FIXME: https://github.com/nginxinc/nginx-kubernetes-gateway/blob/b00216eeac0f8edfb054d45ad073c9c004788762/internal/nginx/runtime/manager.go#L52

[mpstefan]

Logs could be the solution to this problem.

[brianehlert]

Logs should not be written to disk. Log rotate cron jobs do not run in containers. So tailing a log should be avoided.
Watching log output stream is possible but could be problematic.

[sjberman]

Our k8s Golang client has the ability to get a Pod's logs at any point (we would need to instantiate a client-go client vs the current controller-runtime client, though, since the latter doesn't support this yet). We'd likely want to look for the string signal 1 (SIGHUP) received which tells us a reload happened. We have to figure out how to tie it to the last reload call though, because there are going to be many instances of that string. Use a timestamp somehow.

[brianehlert]

What if, instead of looking to logs, you noticed the NGINX workers changed?
Which indicates a successful reload (the config was determined "good" by NGINX).

It does not solve the problem of applying configurations through the N+ API. Which would not have either of these signals.

[sjberman]

If the PIDs change, then maybe. I'm not sure if they do though. (I guess if the reload creates new workers, then they probably do)

EDIT: verified that at least the single worker process PID in my deployment did change on a reload.