Send Prometheus metrics to Grafana Cloud #2994
Open
+42
−17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ndr-ds
force-pushed
the
12-02-send_prometheus_metrics_to_grafana_cloud
branch
from
00f28a0 to
4aa0e5d
Compare
ndr-ds
force-pushed
the
11-28-avoid_build_on_linera_net_up_--kubernetes
branch
from
a872d17 to
37758e3
Compare
ndr-ds
force-pushed
the
12-02-send_prometheus_metrics_to_grafana_cloud
branch
from
4aa0e5d to
e11c5b7
Compare
This was referenced Dec 3, 2024
christos-h
requested changes
Dec 3, 2024
There was a problem hiding this comment.
Thanks for the cleanup.
I'm a bit confused on two things:
- Do we not still want the internal Grafana which is validator specific? In which case why is the dashboard disabled? If you don't want the dashboard why not remove Grafana entirely?
- The reference to the GCP secret should not be hard-coded in
linera-protocolbut instead be parameterised and injected by the caller inlinera-infra(probablylineractl).
|
ndr-ds
force-pushed
the
11-28-avoid_build_on_linera_net_up_--kubernetes
branch
from
37758e3 to
1b117c3
Compare
ndr-ds
force-pushed
the
12-02-send_prometheus_metrics_to_grafana_cloud
branch
from
e11c5b7 to
188dc33
Compare
ndr-ds
force-pushed
the
11-28-avoid_build_on_linera_net_up_--kubernetes
branch
2 times, most recently
from
6266ccd to
3650595
Compare
ndr-ds
force-pushed
the
12-02-send_prometheus_metrics_to_grafana_cloud
branch
from
188dc33 to
17b1c8c
Compare
ndr-ds
changed the base branch from
11-28-avoid_build_on_linera_net_up_--kubernetes
to
graphite-base/2994
ndr-ds
force-pushed
the
12-02-send_prometheus_metrics_to_grafana_cloud
branch
from
17b1c8c to
ec0f9af
Compare
ndr-ds
force-pushed
the
12-02-send_prometheus_metrics_to_grafana_cloud
branch
from
ec0f9af to
e9fee28
Compare
ndr-ds
force-pushed
the
12-02-send_prometheus_metrics_to_grafana_cloud
branch
from
e9fee28 to
cc8f74c
Compare
ndr-ds
force-pushed
the
12-02-send_prometheus_metrics_to_grafana_cloud
branch
from
cc8f74c to
f682033
Compare
christos-h
requested changes
Jan 10, 2025
| name: grafana-cloud-auth-secret | ||
| type: kubernetes.io/basic-auth | ||
| stringData: | ||
| username: {{ .Values.grafanaCloudUsername | quote }} |
There was a problem hiding this comment.
What happens if this isn't specified (for example external validators which use Kuberentes?).
There was a problem hiding this comment.
I'm currently changing this to only be created if writeToGrafanaCloud is true, which will only happen for internal validators
| @@ -1,12 +1,12 @@ | |||
| # Values for charts linera-validator for local validators. | |||
|
|
|||
| # Linera | |||
| lineraImage: "" # Is set by helmfile. | |||
| lineraImage: {{ env "LINERA_HELMFILE_LINERA_IMAGE" | default "linera:latest" }} | |||
There was a problem hiding this comment.
Thanks for this. I'm not entirely sure if this is better or worse to be honest. On the one hand there is less indirection on the other hand configurable variables are now in more places.
There was a problem hiding this comment.
I think it definitely looks better/less confusing than just an empty string, tbh 🤔 at least now you know what the default for this is supposed to be
|
It's related, actually. Working on a fix right now! |
ndr-ds
force-pushed
the
12-02-send_prometheus_metrics_to_grafana_cloud
branch
from
b8d7e85 to
2001d5c
Compare
ndr-ds
force-pushed
the
12-02-send_prometheus_metrics_to_grafana_cloud
branch
from
2001d5c to
d158401
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
We're migrating our internal validators to send data to Grafana Cloud.
Proposal
Starting with the local setup. This was useful to test the required changes that we'll likely need for the actual internal validator setup. We probably won't want to send metrics to Grafana Cloud from local runs all the time (specially for cost reasons), so this is disabled by default.
Grafana Cloud credentials are stored in a secret within GCP's Secret Manager. Since Grafana Cloud will be for our internal validators, then I figured storing the secret on GCP made sense.
Test Plan
Ran locally with
linera net up --kubernetes, and saw the metrics in our Grafana Cloud instance:Release Plan
If we want to start sending metrics to Grafana Cloud from our devnet/testnet, then:
devnetbranchtestnetbranch, then