Update OTLP exporter to send different signals to different endpoints

[MrAlias]

The OTLP exporter specification states the exporter needs to be able to send different signals (traces and metric events) to different endpoints. The current exporter design only supports sending to a single endpoint for both signals.

[stefanprisca]

@MrAlias I would like to work on this.

Maybe a clean approach to this would be to separate the exporter logic (from otlp.go in two: MetricsExporter and TracesExporter. These are both internal structures, will have specific connection and export methods and can be configured separately. The existing OTLP Exporter then maintains the same client interface (together with the added options for configuring traces and metrics separately), and under the hood it delegates the actions to the two different exporters.

What do you think? Is there another design that you have in mind for this?

[MrAlias]

Maybe a clean approach to this would be to separate the exporter logic (from otlp.go in two: MetricsExporter and TracesExporter. These are both internal structures, will have specific connection and export methods and can be configured separately. The existing OTLP Exporter then maintains the same client interface (together with the added options for configuring traces and metrics separately), and under the hood it delegates the actions to the two different exporters.

I think I would need to see a prototype to really follow this, or if you are able to join the SIG meeting in an hour we could talk about this.

The important thing I would look for in a design would be the reusability of connection handling logic. The connections to the collector need to support things like back-off, retries, and likely more features in the future. It is important that this logic doesn't need to be duplicated in every signal that is configured.

[stefanprisca]

I pushed a prototype on my own repo: https://github.com/stefanprisca/opentelemetry-go/tree/otlp-exporters-1121/exporters/otlp/1121
Here are some flow diagrams, and notes, showing the start logic for the connections: https://github.com/stefanprisca/opentelemetry-go/blob/otlp-exporters-1121/exporters/otlp/1121/OtlpExporterDesign-Page-1.png
There is also a functional example here, configuring two separate connections for traces and metrics: https://github.com/stefanprisca/opentelemetry-go/blob/otlp-exporters-1121/exporters/otlp/1121/example_test.go

The idea of the design is to have a configurable connection, which can be used by both metrics and traces.
By moving all the connection logic into a new otlpConnection object, the Exporter can make use of this to configure different connections for traces and metrics. Moreover, by using a newConnectionHandler callback, the Exporter can create specific metric/traces clients from the corresponding otlpConnections. These clients are then used to export the metrics and traces.
For example, creating the metrics connection is done as such:

e.metricsConnection = newOtlpConnection(e.handleNewMetricsConnection, config.metrics)
The handleNewMetricsConnection is the newConnectionHandler for metrics, which will create a colmetricpb.NewMetricsServiceClient grpc client for sending metrics. The second argument is the metrics specific configuration.

This design is a bit simpler than the initial one I proposed, as there are no separate metrics/traces exporters. Looking forward to hearing what you think!

[stefanprisca]

@MrAlias should I continue with this design, and put together a pull request?

[MrAlias]

@stefanprisca sorry about the delay. Thank you for digging into this, your design docs are great! This looks like the approach I was thinking about and I think with your prototype it's clear that it is viable.

The only feedback I have on a potential change would the the options might be better abstracted. We could collapse the With*Address, With*Insecure, and With*GRPCDialOption options into something like WithMetricsConnectionConfig(c ConnectionConfig) and WithTracesConnectionConfig(c ConnectionConfig). But this is minor. I think your changes are definitely ready for a PR.

[stefanprisca]

@MrAlias no worries, just wanted to check if you got a chance to look over the design :)

I submitted a PR, but it still needs a bit of polishing (build failing and CHANGELOG.md update) so it's in progress for now.