node-collector.go's Collect function all-or-nothing

[jshufro]

Collect spins up a bunch of goroutines in a waitgroup to gather stats, then waits for them, exiting early if any one fails.

Subsequently it publishes them all synchronously. This means if any one metric can't be computed due to some transient error, none are.

This decision seems to be a result of the fact some metrics are derived from other stats, and those stats need to be observed before the derived metrics can be published.

One option would be to compute all the derived metrics in grafana, and only publish raw metrics to prometheus. Alternatively, the raw metrics could asynchronously produce their results on the output channel (which is thread-safe), and the derived metrics could synchronously produce their results on the channel iff all the raw metrics succeed.