Does Micrometer support GRPC Opentelemetry metrics?

[alexFilichkin]

Hi,

I'm trying to configure Micrometer for metrics to send custom metrics to OTEL collector in GRPC format. Everything works with HTTP protocol, but doesn't work with GRPC.
I even cannot see it documentation https://docs.micrometer.io/micrometer/reference/implementations/otlp.html

[jonatan-ivanov]

Hi,
Thanks for the issue!

OTel supports 3 transports:

• HTTP/json
• HTTP/protobuf
• gRPC(/protobuf)

According to the specification, clients need to support one of these but backends/collectors must support all. So from the backend perspective, it should not matter which one you use. Micrometer supports HTTP/protobuf.

gRPC is not the "format" but the "protocol". gRPC uses protobuf to encode/serialize the data and it uses HTTP/2 to transfer the data. So HTTP/protobuf (what Micrometer uses) is very similar to gRPC (same encoding over HTTP).

With this in mind, could you please tell us why you need gRPC and why HTTP/protobuf does not work for you?

It also worth to mention that if you already have a gRPC client and Micrometer would bring you another one, that could case issues (as it did in the OTel SDK iirc).

[alexFilichkin]

Hi @jonatan-ivanov thank you for the comment.
Our OTEL collector is configured only for grpc and with this setup we cannot use micrometer, and have to use OTEL SDK 😞

[jonatan-ivanov]

Can you add http to the collector config?
Something like this:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

What do you use the collector for? Are you pushing metrics to a backend that Micrometer supports?

[alexFilichkin]

I see your point. It's clear that we can expose the HTTP protocol in the OTEL collector or send the metric directly to the backend, bypassing the OTEl collector.
But my initial request was about the Micrometer. Why the most popular Observability library doesn't support GRPC?
gRPC offers better performance(compared to HTTP with Protobuf), especially for high-throughput systems, thanks to persistent connections and multiplexing.

[shakuzen]

especially for high-throughput systems

This issue is about publishing metrics, with a payload size that should not depend on the throughput of your system, and that by default is published every 60 seconds. I fail to see how gRPC is going to substantially improve the performance of metrics publication, but perhaps I'm missing something. Have you noticed a performance issue with metrics publication that will be solved by gRPC?

Why the most popular Observability library doesn't support GRPC?

We have to consider priorities, maintainability, compatibility, and tradeoffs for all of our users when we choose on what to spend our efforts supporting. You may be interested in #5690 under consideration. Similar to that, we have some ideas we plan to explore about making our HttpSender abstraction more generic, which may make it easier for us to enable a gRPC-based sender implementation for OTLP metrics. So, I think it is likely that we will eventually support OTLP via gRPC.

[alexFilichkin]

Hi @shakuzen , I don't have a real performance comparison between gRPC vs HTTP for OTEL metrics. But I expect that gRPC latency will be better due to HTTP/2 optimizations and persistent connections (HTTP/2). I may be wrong :)

But anyway it would be good to support OTLP with gRPC.

[jonatan-ivanov]

gRPC offers better performance(compared to HTTP with Protobuf), especially for high-throughput systems, thanks to persistent connections and multiplexing.

I would like to echo what Tommy wrote above and also, I think it worth mentioning that HTTP/1.1 uses persistent connections too (by default). HTTP/2 though allows multiple concurrent requests to be multiplexed over a single connection but this might not be happening in case of Micrometer since we don't send requests concurrently to an OTLP backend.

But I expect that gRPC latency will be better due to HTTP/2 optimizations and persistent connections (HTTP/2).

As I mentioned above, those optimizations might not be applicable in this case so I would expect they being similar. Since publishing metrics is asynchronous to your application, you will not notice slight latency improvements or degradations.

Please let us know if you end-up doing a perf test, I asked the OTel community if anyone did such a test before.

[jonatan-ivanov]

From the conversation with the OTel community (you can find the whole discussion in the OTel Java Slack channel):

OTel has its own serialization and grpc implementation and they use OkHttp for both grpc and http/protobuf (the code is almost the same for these two transports). One of the OTel Java maintainers says perf differences between them should be negligible because the code is almost identical. They have perf differences based on which client they use (OkHttp vs JDK 11+ HTTP client) not based on the protocol. I think this would be applicable to Micrometer too.

This is also an interesting aspect:

I have struggled to produce or find any compelling evidence suggesting an advantage of grpc over http/protobuf. On the contrary, as an employee of a company that accepts OTLP natively, we have routinely run into issues with grpc. Mostly proxys and load balancers not being grpc aware and returning error scenario responses which clients don’t correctly interpret as retriable.

OTel recommends using http/protobuf (this is the default):

If no configuration is provided the default transport SHOULD be http/protobuf unless SDKs have good reasons to choose grpc as the default (e.g. for backward compatibility reasons when grpc was already the default in a stable SDK release).

New Relic also recommends http/protobuf:

Additionally, you should configure your OTLP exporter to use the OTLP/HTTP binary protobuf version of the protocol if available. While New Relic supports all versions of OTLP, OTLP/HTTP binary protobuf has proved to be more robust than gRPC without any apparent reduction in performance.

[github-actions]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[github-actions]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open.