Customize resource attributes reported by Prometheus.

[h0cheung]

Component(s)

receiver/prometheus

Is your feature request related to a problem? Please describe.

As for now, the Prometheus receiver will add as many resource attributes as possible.

It causes some problems:

• The number of attributes is too large. With k8s discovery, 11 resource attributes will be added, while the default label limit of VictoriaMetrics is 30, more than 1/3 of which has been used by this. Some labels are useless, for example, server.address and server.port are duplicated with service.instance.id.
• Some resource attributes often get wrong value. For example, due to when discovered labels has conflicts, for example __meta_kubernetes_pod_container_name, we should be drop them instead of preserve the last one. prometheus/prometheus#15171, the value of k8s.container.name will be wrong in many cases.

Describe the solution you'd like

Add a configuration to set which resource attributes should be put.
Consider use a small set of attributes by default. And k8s.container.name should be disabled by default as the source of it is not reliable.

Describe alternatives you've considered

transform processor has ability to drop unwanted attributes. But it's not as convenient and sufficient as directly configuring the Prometheus receiver.

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Some resource attributes often get wrong value.

The issue you linked to sounds like a bug. I think fixing the bug is probably the appropriate way forward, rather than adding config to work around the issue.

I don't think we can change the default without breaking existing users, and we shouldn't do that without good reason.

11 resource attributes seems fairly average by OTel standards. Out of curiosity, which resource attributes that we currently collect would you drop? Just server.address and server.port, or others too?

As for now, the Prometheus receiver will add as many resource attributes as possible.

The main use-case I could see for the kind of config you are proposing is that we could allow users to add even more resource attributes if they were opt-in. For example, we could add pod labels or annotations.