Optimize and align host monitoring config with newrelic-opentelemetry-examples

[jack-berg]

Optimizations:

• Reduce collection interval to default 60s
• Disable process metrics to align with NRI defaults
• Add cumulativetodelta processor to align with New Relic OTLP preferences
• Drop metric description and unit

Separate cleanup including:

• Use otlphttp exporter instead of otlp to align with New Relic OTLP preferences
• Split out dedicated pipelines for host metrics and logs

Corresponding synchronization PR in newrelic-opentelemetry-examples: newrelic/newrelic-opentelemetry-examples#636

cc @alanwest

[rubenruizdegauna]

I used this config and I saw some errors:

Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"staging-otlp.nr-data.net:4317/v1/metrics\": unsupported protocol scheme \"staging-otlp.nr-data.net\"", "interval": "5.150597003s"}

and metrics are not being sent.

cc @josemore to be aware of this change

[josemore]

some context / notes:

• "Disable process metrics to align with NRI defaults": the guided install experience for the infra agent (main entry point to install the agent) does enable process monitoring by default and it's surfaced in the host entity detail view. I'm OK disabling this as it's too verbose (high-cost vs value) via OTel at the moment until we have a better solution (might require alignment with Infra experience stakeholders).
• "interval to default 60s": I understand we defined same interval as default for the same metrics in the infra agent (20s), similarly as above, ok to change this to offer better defaults to OTel customers.
• As this is entering Preview soon with related new capabilities, we need to ensure the config is valid and we don't break the Infra experiences.

[jack-berg]

"Disable process metrics to align with NRI defaults": the guided install experience for the infra agent (main entry point to install the agent) does enable process monitoring by default and it's surfaced in the host entity detail view. I'm OK disabling this as it's too verbose (high-cost vs value) via OTel at the moment until we have a better solution (might require alignment with Infra experience stakeholders).

My understanding is that the answer of whether process metrics are enabled by default or not varies based on the install workflow. I used newrelic/nri-bundle, which I believe uses nri-kubernetes as a subchart, which I believe disables process metrics by default here.

Let me know if I'm misunderstanding something. It seems like a pretty complex piece of config and not exactly clear to an outsider what does what.

As this is entering Preview soon with related new capabilities, we need to ensure the config is valid and we don't break the Infra experiences.

Yup.

[jack-berg]

As this is entering Preview soon with related new capabilities, we need to ensure the config is valid and we don't break the Infra experiences.

This repo really needs a smoke test as part of the build 😉

[jack-berg]

Anything required to get this merged?

It appears that snyk validation is failing due to an authentication error unrelated to this PR.

[rubenruizdegauna]

LGTM

[jack-berg]

This repo has a merge requirement that all commits are signed, so I recommitted signed commits and force pushed.

@rubenruizdegauna do you have merge permissions?