[receiver/splunkenterprise] Add Attributes to track Build & Version Data

[michael-burt]

Component(s)

No response

Is your feature request related to a problem? Please describe.

I would like to attach splunkd_build and splunkd_version attributes to metrics being emitted by the Splunk Enterprise Receiver. I am unsure whether these should be added as "Resource Attributes" or just normal "Attributes". These peices of metadata about the Splunk host are available on the /services/server/info API endpoint.

Describe the solution you'd like

I think these pieces of data should be added as either attributes or resource attributes.

Describe alternatives you've considered

I have implemented a new collector to emit this data as a metric, see here: #36118

Upon reflection, this does not seem like the correct approach. This data is useful when processing all metrics emitted by this receiver, not just the info metric.

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/splunkenterprise: @shalper2 @MovieStoreGuy @greatestusername

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[shalper2]

I think this will be a good enhancement. A couple notes: I think to begin with we will want to have this set to off by default. This way existing users who don't care about these attributes don't suddenly get flooded with a bunch more data. Second, I don't think it would make sense to have these attributes be updated every time the scrape function is called, however we do have to imagine that version and build info could change during the lifetime of the collector instance so we must update it occasionally. I propose we scrape this information on a hardcoded, relatively lengthy interval. Perhaps every 60m? Thoughts on this? Any other considerations?

[michael-burt]

For my use case, 60m may be too long to wait. Could this be a configurable setting perhaps? I don't think it would put too much load on Splunk to fetch this data on every scrape by the receiver, either.

[shalper2]

I think that is definitely possible. Good to know re: load on Splunk

[shalper2]

so because of the way the client is designed (i.e. it may contain several endpoints) I think I'm going to have to go back on my word here and just have build and version attached as attributes