graal/native compiled spring cloud function for aws requires GLIBC at Runtime

[cforce]

OS: Linux
JDK: OpenJDK 17
Spring Boot: 3.0.0
Spring Cloud Function: 4.0.0-SNAPSHOT
Using todays (SNAPSHOT) graalvm-reachability-metadata

Step to reproduce:

• mvn -Pnative,lambda,nativeRS package
• deploy zip file to aws lamda
• execute lambda function

cloud-function-dynamodb-lambda: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./cloud-function-dynamodb-lambda)
./cloud-function-dynamodb-lambda: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./cloud-function-dynamodb-lambda)
./cloud-function-dynamodb-lambda: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./cloud-function-dynamodb-lambda)
./cloud-function-dynamodb-lambda: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./cloud-function-dynamodb-lambda)
START RequestId: 054fd135-79e2-4ec7-b080-11820822a134 Version: $LATEST
RequestId: 054fd135-79e2-4ec7-b080-11820822a134 Error: Runtime exited with error: exit status 1
Runtime.ExitError
END RequestId: 054fd135-79e2-4ec7-b080-11820822a134
REPORT RequestId: 054fd135-79e2-4ec7-b080-11820822a134	Duration: 75.74 ms	Billed Duration: 76 ms	Memory Size: 128 MB	Max Memory Used: 4 MB	
XRAY TraceId: 1-639994f6-6fc8489d57c0cd106bdbaee8	SegmentId: 11040f061038b2f9	Sampled: true	

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.0.0</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.demo</groupId>
    <artifactId>cloud-function-dynamodb-lambda</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>cloud-function-dynamodb-lambda</name>
    <description>Native Spring Cloud Function for AWS Lambda, with DynamoDB demo</description>
    <properties>
        <java.version>17</java.version>
        <spring-cloud.version>2021.0.4</spring-cloud.version>
        <software.amazon.awssdk>2.18.27</software.amazon.awssdk>
        <testcontainers.version>1.17.6</testcontainers.version>
        <native.buildtools.version>0.9.19</native.buildtools.version>
        <org.junit.platform.verison>1.9.1</org.junit.platform.verison>
        <spring-cloud-function.version>4.0.0-SNAPSHOT</spring-cloud-function.version>
    </properties>
    <build>
        <finalName>${project.artifactId}</finalName>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-function-context</artifactId>
            <version>${spring-cloud-function.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-function-adapter-aws</artifactId>
            <version>${spring-cloud-function.version}</version>
        </dependency>

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>dynamodb</artifactId>
            <version>${software.amazon.awssdk}</version>
        </dependency>

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>protocol-core</artifactId>
            <version>${software.amazon.awssdk}</version>
        </dependency>

        <!-- Testing -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>junit-jupiter</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-function-dependencies</artifactId>
                <version>${spring-cloud-function.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>org.testcontainers</groupId>
                <artifactId>testcontainers-bom</artifactId>
                <version>${testcontainers.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${software.amazon.awssdk}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <repositories>
        <repository>
            <id>jitpack.io</id>
            <url>https://jitpack.io</url>
        </repository>
        <repository>
            <id>spring-release</id>
            <name>Spring release</name>
            <url>https://repo.spring.io/release</url>
        </repository>
        <repository>
            <id>spring-snapshots</id>
            <name>Spring Snapshots</name>
            <url>https://repo.spring.io/snapshot</url>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>spring-milestones</id>
            <name>Spring Milestones</name>
            <url>https://repo.spring.io/milestone</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>spring-release</id>
            <name>Spring release</name>
            <url>https://repo.spring.io/release</url>
        </pluginRepository>
    </pluginRepositories>

    <profiles>
        <profile>
            <id>nativeRS</id><!-- add this profile to use local oracle reachability -->
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.graalvm.buildtools</groupId>
                        <artifactId>native-maven-plugin</artifactId>
                        <configuration>
                            <classesDirectory>${project.build.outputDirectory}</classesDirectory>
                            <metadataRepository>
                                <enabled>true</enabled>
                                <url>
                                    file:/git/serverless/graalvm-reachability-metadata/build/graalvm-reachability-metadata-0.2.6-SNAPSHOT.zip
                                </url>
                            </metadataRepository>
                            <requiredVersion>22.3</requiredVersion>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>


        <profile>
            <id>lambda</id> <!-- package a lambda zip -->
            <build>
                <plugins>
                    <!-- Builds the native image -->
                    <plugin>
                        <groupId>org.graalvm.buildtools</groupId>
                        <artifactId>native-maven-plugin</artifactId>
                        <version>${native.buildtools.version}</version>
                        <extensions>true</extensions>
                        <executions>
                            <!--execution>
                                <id>test-native</id>
                                <goals>
                                    <goal>test</goal>
                                </goals>
                                <phase>test</phase>
                            </execution-->
                            <execution>
                                <id>build-native</id>
                                <goals>
                                    <goal>build</goal>
                                </goals>
                                <phase>package</phase>
                            </execution>
                        </executions>
                        <configuration>
                            <buildArgs>--enable-url-protocols=http</buildArgs>
                            <buildArgs>-H:ConfigurationFileDirectories=target\native\agent-output</buildArgs>
                            <buildArgs>--no-fallback</buildArgs>
                            <buildArgs>-H:DashboardDump=fortune</buildArgs>
                            <buildArgs>-H:+DashboardAll</buildArgs>
                        </configuration>

                    </plugin>
                    <!-- Creates the .zip with bootstrap and native image -->
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-assembly-plugin</artifactId>
                        <version>3.4.2</version>
                        <executions>
                            <execution>
                                <id>native</id>
                                <phase>package</phase>
                                <goals>
                                    <goal>single</goal>
                                </goals>
                                <inherited>false</inherited>
                            </execution>
                        </executions>
                        <configuration>
                            <descriptors>
                                <descriptor>src/assembly/native.xml</descriptor>
                            </descriptors>
                        </configuration>
                    </plugin>
                    <plugin>
                        <groupId>org.springframework.boot</groupId>
                        <artifactId>spring-boot-maven-plugin</artifactId>
                        <configuration>
                            <classifier>exec</classifier>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>

        <profile>
            <id>nativeTest</id>
            <dependencies>
                <dependency>
                    <groupId>org.junit.platform</groupId>
                    <artifactId>junit-platform-launcher</artifactId>
                    <version>${org.junit.platform.verison}</version>
                    <scope>test</scope>
                </dependency>
            </dependencies>
        </profile>

    </profiles>
</project>

Above GLIBC issue is not reproduceable locally by just executing the native compiled linux exe on ubuntu 22.04.
As i don't yet have a glue howto call the same function from e.g. cmd line instead via deployed lambda i can not rile out that it would also and only also happen when run my function triggered by an event like done on aws console.

I suspect that the Linux which is used as base for lambda is providing only GLIBC <2.32The aws lambda website mentioned above pretends to use this image to run amazon/amzn-ami-hvm-2018.03.0.20220802.0-x86_64-gp2 which I was not able to download yet somewhere but initial release year 2018 sounds pretty old. I found this docs for it

• https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/?nc1=h_ls
• https://aws.amazon.com/amazon-linux-ami/?nc1=h_ls

The last update seems to be some months ago and is Amazon Linux 2018.03.0.20220419.0
Updated Packages:
• glibc-2.17-324.189.amzn1.x86_64

I will try to build with amazonlinux and same or lower glibc version. However a buildpack which is sync with lambda execution environment ants its GLIBC versions where i can build a fully compatible lambda native executable without setting up own docker container would make sense especially for spring cloud function aws adapter.
Another approach would be to get independent of dynamic lib loading.
https://github.com/orgs/paketo-buildpacks/discussions/58#discussioncomment-4419519

[VisionaryAppDev]

I am also facing the same issue. It seem like AWS Lamda environment is using older of ldd (GNU libc) 2.26 while our build environment is using something much newer ldd (Ubuntu GLIBC 2.35-0ubuntu3.1) 2.35 (Github action). I am thinking if we could use older ldd version to build the project or use something like customer layer to run the project and see if it's working?

[VisionaryAppDev]

I have tried building the native image on Docker and upload it to AWS Lambda and it's working! @cforce, you may try run this docker image and then build the native image from the inside of the container and upload the file you got to AWS.

Plz also change arm64v8 -> amd or sth as your environment.

[cforce]

Yes, of course i could dockerize my complete build env, this way i am reimplementing a concept like buildpacks which brings setting up the build env out of the box by intension, however the spring-boot-maven plugins which does only comes with builder images based on recent images which won't fit to the execution runtime of aws lambda (being to far behind)
see aws/aws-lambda-base-images#59

Spring boot maven plugin uses buildpacks to provide build and runtime for native image compilation and execution in a docker image. The stack is a name for the build and run images. These are the base images for the builder and your application image.

A builder is an image that contains three components:

• a set of buildpacks, which provide your app’s dependencies
• a stack, which provides the OS layer for your app image
• the CNB lifecycle(opens in a new tab), which puts everything together to produce your final app image
  For more information about builders, see buildpacks.io

The builder is the image that forms the container environment when you run the actual build. It includes the build image plus all the buildpacks. The stack provides a glibc . At build time, it's the build time and at run time, it's the run image.

Below a list of used buildpacks which come out of the box with spring boot (>3.x)

[INFO]     [creator]     paketo-buildpacks/ca-certificates   3.5.1
[INFO]     [creator]     paketo-buildpacks/bellsoft-liberica 9.10.1
[INFO]     [creator]     paketo-buildpacks/syft              1.23.0
[INFO]     [creator]     paketo-buildpacks/executable-jar    6.5.0
[INFO]     [creator]     paketo-buildpacks/spring-boot       5.20.0
[INFO]     [creator]     paketo-buildpacks/native-image      5.6.0

libc vesions per env:

docker run --interactive --tty --entrypoint /bin/sh amazon/aws-codebuild-local:latest

ldd --version # 2.17

docker run --interactive --tty --entrypoint /bin/sh amazonlinux:2

ldd --version # 2.26

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws
docker run --interactive --tty --entrypoint /bin/sh public.ecr.aws/amazonlinux/amazonlinux:latest

ldd --version # 2.26
docker run --interactive --tty --entrypoint /bin/sh public.ecr.aws/amazon/amzn-ami-hvm-2018.03.0.20220802.0-x86_64-gp2

docker run --interactive --tty --entrypoint /bin/sh bellsoft/liberica-runtime-container:jdk-17-stream-glibc-20221019

/lib/ld-linux-x86-64.so.2 --version # 2.34

docker run --interactive --tty --entrypoint /bin/sh bellsoft/liberica-runtime-container:jdk-11-stream-glibc-20221019

/lib/ld-linux-x86-64.so.2 --version # 2.34

The full list of builders: https://paketo.io/docs/reference/builders-reference/

It looks like aws lambda execution runtime is still based on glibc which comes with Ubuntu 18.04, while spring boot maveb builder already used 22.04 and a provides a glibc at runtime which is not compatible with the lambda glibc from execution runtime. So either

1. you have to reconfigure (if that possible) to use another stack on spring-boot maven plugin to build with almost static linked glibc from 18.04 and maintain along lambda target execution runtime (wasn't yet successful here)
   Your may configure your custom setup:

<plugin>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-maven-plugin</artifactId>
  <configuration>
    <image>
      <builder>paketobuildpacks/builder:tiny</builder>
        <buildpacks>
          <buildpack>gcr.io/paketo-buildpacks/java-native-image</buildpack>
        </buildpacks>
        <env>
          <BP_NATIVE_IMAGE>true</BP_NATIVE_IMAGE>
          <BP_JVM_VERSION>${java.version}</BP_JVM_VERSION>
        </env>
      </image>
    </configuration>
  </plugin>

or

2. build with musl to create a full static (no libc runtime dep) binary (did that successfully, however a lot of effort on top and also no buildpacks support yet - see Add support for static compilation with native-image paketo-buildpacks/java-native-image#650 ) or
3. custom dockerize the build env and completely bypass spring boot buildpacks (from my opinion the most unattractive) or
4. adopt the spring-boot buildpack setup in spring-cloud-function-aws to fully support aws lambda build env via the spring-boot-buildpacks (feature wish) so it just works and produces a zip file which can be uploaded to lambda api and executes on lambda runtime
5. Build a container image based on "mvn -Pnative package spring-boot:build-image" for lambda which afaik shall allow any embedded glibc (also the one from recent Jammy), however i am not sure which downsides we have here on lamda as this deployment is the wale of all the options. Not to mention that native images are build by default based on the build env for arm(aarch) or amd(x86) architecture and on lambda you may choose both. To take advantage of the Graviton processors you might want to build on amd and target arm.
   That shall as well be supported using "docker buildx build --platform linux/arm64,linux/amd64" used in packet buildpacks as multiarch support

[olegz]

Have you tried https://github.com/spring-cloud/spring-cloud-function/tree/main/spring-cloud-function-samples/function-sample-aws-native

[olegz]

Closing it as it appears that it has nothing to do with spring-cloud-function