Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exporter for Metrics Defaults to gRPC Even When Explicitly Set #13024

Open
jeffNedley opened this issue Jan 9, 2025 · 3 comments
Open

Exporter for Metrics Defaults to gRPC Even When Explicitly Set #13024

jeffNedley opened this issue Jan 9, 2025 · 3 comments
Labels
bug Something isn't working

Comments

@jeffNedley
Copy link

jeffNedley commented Jan 9, 2025 •
edited
Loading

Describe the bug
Using the Spring Boot Zero-Code Implementation and following the documentation closely, I cannot seem to get the implementation to work without seeing this error message.

From what it seems, even after setting the endpoint and protocol values for what my team needs (http/protobuf), we always receive the error saying, "Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317". This is incorrect for two reasons. One, because I have set the endpoint using the application.yml and two, because I have set the protocol using the application.yml.

Here is our config (application.yml) below:

otel:
  service:
    name: <redacted-for-security>
  exporter:
    otlp:
      traces:
        endpoint:  http://<redacted-for-security>:4318/v1/traces
        protocol: http/protobuf
      metrics:
        endpoint:  http://<redacted-for-security>:4318/v1/metrics 
        protocol: http/protobuf
      logs:
        endpoint:  http://<redacted-for-security>:4318/v1/logs
        protocol: http/protobuf
  resource:
    attributes:
      deployment:
        environment: <redacted-for-security>

Here is our dependency config (pom.xml):

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>3.1.12</version>
		<relativePath/>
	</parent>
	<groupId><redacted-for-security></groupId>
	<artifactId><redacted-for-security></artifactId>
	<version>0.1.6-SNAPSHOT</version>
	<name><redacted-for-security></name>
	<properties>
		<project.version>0.1.8-SNAPSHOT</project.version>
		<java.version>21</java.version>
		<argLine>-Dnet.bytebuddy.experimental=true</argLine>
		<!-- Remove when Mockito gets updated
        for Java 21 -->
		<confluent.version>7.4.0</confluent.version>
		<avro.version>1.11.3</avro.version>
		<avro.base.directory>src/main/resources/avro/</avro.base.directory>
		<avro.schema.directory>src/main/resources/avro/schemas</avro.schema.directory>
		<avro.idl.directory>src/main/resources/avro/idl</avro.idl.directory>
		<redacted-for-security>
	</properties>
	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>io.opentelemetry.instrumentation</groupId>
				<artifactId>opentelemetry-instrumentation-bom</artifactId>
				<version>2.11.0</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>
	<dependencies>
		<!-- OTEL DEPENDENCY -->
		<dependency>
			<groupId>io.opentelemetry.instrumentation</groupId>
			<artifactId>opentelemetry-spring-boot-starter</artifactId>
		</dependency>
		
		<!-- INTERNAL DEPENDENCIES /////////////////////////////////////////////// -->
		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<version>1.18.32</version>
			<scope>provided</scope>
		</dependency>
		<!-- SPRING DEPENDENCIES ///////////////////////////////////////////////// -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.retry</groupId>
			<artifactId>spring-retry</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-aspects</artifactId>
		</dependency>
		<!-- ELASTICSEARCH DEPENDENCIES ////////////////////////////////////////// -->
		<dependency>
			<groupId>co.elastic.clients</groupId>
			<artifactId>elasticsearch-java</artifactId>
			<version>8.16.0</version>
		</dependency>
		<dependency>
			<groupId>jakarta.json</groupId>
			<artifactId>jakarta.json-api</artifactId>
			<version>2.0.1</version>
		</dependency>
		<dependency>
			<groupId>jakarta.annotation</groupId>
			<artifactId>jakarta.annotation-api</artifactId>
			<version>3.0.0</version>
		</dependency>
		<dependency>
			<groupId>com.google.code.gson</groupId>
			<artifactId>gson</artifactId>
			<version>2.10.1</version>
		</dependency>
		<dependency>
			<groupId>com.fasterxml.jackson.datatype</groupId>
			<artifactId>jackson-datatype-jsr310</artifactId>
		</dependency>
		<!-- LEADERSHIP DEPENDENCIES/////////////////////////////////////////////////// -->		
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-kubernetes-fabric8-leader</artifactId>
			<version>3.0.2</version>
		</dependency>
		<dependency>
			<groupId>org.awaitility</groupId>
			<artifactId>awaitility</artifactId>
			<version>4.2.2</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.integration</groupId>
			<artifactId>spring-integration-core</artifactId>
			<version>6.3.5</version>
		</dependency>
		<!-- KAFKA DEPENDENCIES/////////////////////////////////////////////////// -->
		<dependency>
			<groupId>org.springframework.kafka</groupId>
			<artifactId>spring-kafka</artifactId>
			<version>3.2.4</version>
		</dependency>
		<dependency>
			<groupId>org.apache.kafka</groupId>
			<artifactId>kafka-streams</artifactId>
			<version>3.9.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.kafka</groupId>
			<artifactId>kafka-storage-api</artifactId>
		</dependency>
		<dependency>
			<groupId>org.apache.avro</groupId>
			<artifactId>avro</artifactId>
			<version>${avro.version}</version>
		</dependency>
		<dependency>
			<groupId>org.apache.avro</groupId>
			<artifactId>avro-maven-plugin</artifactId>
			<version>${avro.version}</version>
		</dependency>
		<dependency>
			<groupId>io.confluent</groupId>
			<artifactId>kafka-avro-serializer</artifactId>
			<version>${confluent.version}</version>
		</dependency>
		<dependency>
			<groupId>io.confluent</groupId>
			<artifactId>kafka-streams-avro-serde</artifactId>
			<version>${confluent.version}</version>
		</dependency>
		<dependency>
			<groupId>io.confluent</groupId>
			<artifactId>kafka-schema-registry-client</artifactId>
			<version>${confluent.version}</version>
		</dependency>
		<!-- TEST DEPENDENCIES /////////////////////////////////////////////////// -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.kafka</groupId>
			<artifactId>spring-kafka-test</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.junit.jupiter</groupId>
			<artifactId>junit-jupiter-api</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.mockito</groupId>
			<artifactId>mockito-core</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>
	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
				<configuration>
					<excludes>
						<exclude>
							<groupId>org.projectlombok</groupId>
							<artifactId>lombok</artifactId>
						</exclude>
					</excludes>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.apache.avro</groupId>
				<artifactId>avro-maven-plugin</artifactId>
				<version>${avro.version}</version>
				<executions>
					<execution>
						<id>idl-protocol</id>
						<phase>generate-sources</phase>
						<goals>
							<goal>idl-protocol</goal>
						</goals>
						<configuration>
							<stringType>String</stringType>
							<sourceDirectory>${avro.idl.directory}</sourceDirectory>
							<outputDirectory>${project.basedir}/src/main/java</outputDirectory>
							<includes>
								<include>**/*.avdl</include>
							</includes>
						</configuration>
					</execution>
				</executions>
				<configuration>
					<javaSourceDirectories>
						<redacted-for-security>
					</javaSourceDirectories>
					<avroOutputDirectory>${avro.schema.directory}</avroOutputDirectory>
				</configuration>
			</plugin>
			<!-- CMD: 'mvn schema-registry:register' -->
			<plugin>
				<groupId>io.confluent</groupId>
				<artifactId>kafka-schema-registry-maven-plugin</artifactId>
				<version>${confluent.version}</version>
				<redacted-for-security>
				<goals>
					<goal>register</goal>
				</goals>
			</plugin>
			<plugin>
				<groupId>com.google.cloud.tools</groupId>
				<artifactId>jib-maven-plugin</artifactId>
				<version>3.3.2</version>
				<redacted-for-security>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-javadoc-plugin</artifactId>
				<version>3.6.0</version>
				<configuration>
					<source>21</source>
					<target>21</target>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.jacoco</groupId>
				<artifactId>jacoco-maven-plugin</artifactId>
				<version>0.8.12</version>
				<configuration>
					<excludes>
						<exclude>META-INF/**</exclude>
						<exclude>${avro.java.shared.directory}</exclude>
						<exclude>${avro.java.events.directory}</exclude>
						<exclude>${avro.base.directory}</exclude>
					</excludes>
				</configuration>
				<executions>
					<execution>
						<goals>
							<goal>prepare-agent</goal>
						</goals>
					</execution>
					<execution>
						<id>report</id>
						<phase>prepare-package</phase>
						<goals>
							<goal>report</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-clean-plugin</artifactId>
				<configuration>
					<filesets>
						<fileset>
							<directory>${avro.java.events.directory}</directory>
							<includes>
								<include>**/*</include>
							</includes>
							<followSymlinks>false</followSymlinks>
						</fileset>
						<fileset>
							<directory>${avro.java.shared.directory}</directory>
							<includes>
								<include>**/*</include>
							</includes>
							<followSymlinks>false</followSymlinks>
						</fileset>
						<fileset>
							<directory>${avro.schema.directory}</directory>
							<includes>
								<include>**/*</include>
							</includes>
							<followSymlinks>false</followSymlinks>
						</fileset>
					</filesets>
				</configuration>
			</plugin>
		</plugins>
	</build>
	<distributionManagement>
		<redacted-for-security>
	</distributionManagement>
</project>

Error that comes up constantly:

2025-01-09 15:24:39.910ERROR --- i.o.exporter.internal.grpc.GrpcExporter  : Failed to export metrics. The request could not be executed. Error message: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317

java.net.ConnectException: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:na]
        at io.opentelemetry.exporter.sender.okhttp.internal.RetryInterceptor.intercept(RetryInterceptor.java:91) ~[opentelemetry-exporter-sender-okhttp-1.44.1.jar:1.44.1]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) ~[okhttp-4.10.0.jar:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
        Suppressed: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:4317
                ... 21 common frames omitted
        Caused by: java.net.ConnectException: Connection refused
                at java.base/sun.nio.ch.Net.pollConnect(Native Method)
                at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
                at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
                at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592)
                at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
                at java.base/java.net.Socket.connect(Socket.java:751)
                at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128)
                at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
                ... 20 common frames omitted
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[na:na]
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[na:na]
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542) ~[na:na]
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592) ~[na:na]
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[na:na]
        at java.base/java.net.Socket.connect(Socket.java:751) ~[na:na]
        at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295) ~[okhttp-4.10.0.jar:na]
        ... 20 common frames omitted

Steps to reproduce
Follow OpenTelemetry documentation for Zero-Code Implementation under the Spring Boot Starter section.

What did you expect to see?
Integration and all telemetry data to show up in my Collector with no errors in the logs

What did you see instead?
All telemetry except a log of an error was shown repeatedly

What version and what artifacts are you using?
Artifacts: opentelemetry-instrumentation-bom, opentelemetry-spring-boot-starter
Version: 2.11.0
How did you reference these artifacts?
pom.xml Maven

Environment
Compiler: openjdk 21.0.4 2024-07-16 LTS
OS: RedHat Linux
Runtime: Corretto-21.0.4.7.1 (build 21.0.4+7-LTS)
OS RedHat Linux
@jeffNedley jeffNedley added the bug Something isn't working label Jan 9, 2025
@jeffNedley
Copy link
Author

jeffNedley commented Jan 10, 2025 •
edited
Loading

Found additional information on this issue!

I was able to mitigate the issue by instead using environment variables rather than the spring application.yml. I don't count this as a non-issue though since the documentation clearly states that I should be able to use either.

I found that the spring boot starter implementation was not reading ANY of my configurations in the application.yml!

@trask
Copy link
Member

trask commented Jan 10, 2025

transferring to opentelemetry-java-instrumentation repo where the opentelemetry-spring-boot-starter lives

cc @jeanbisutti @zeitlinger

@trask trask transferred this issue from open-telemetry/opentelemetry-java Jan 10, 2025
@laurit
Copy link
Contributor

laurit commented Jan 11, 2025

@jeffNedley a minimal sample application that reproduces the issue could help.
Perhaps the issue is that there are 2 OpenTelemetry instances. One that is configured by the spring starter and another one that uses the default configuration?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@trask @laurit @jeffNedley