AWT graphics with AWS Lambda not working in native #23998

felixosth · 2022-02-27T21:29:59Z

Describe the bug

Combining the two guides https://quarkus.io/guides/amazon-lambda-http and https://github.com/quarkusio/quarkus-quickstarts/tree/main/awt-graphics-rest-quickstart doesn't work with native-image.

The error /var/task/bootstrap: /lib64/libm.so.6: version 'GLIBC_2.29' not found (required by /var/task/bootstrap) is thrown when testing using ./mvnw package -Dnative and sam local invoke --template target/sam.native.yaml. awt-quarkus is added to pom.xml.
Sample project attached: source.zip

Expected behavior

No error thrown and the application starts successfully.

Actual behavior

The following error is visible in the terminal:

$ sudo sam local invoke --template target/sam.native.yaml
Invoking not.used.in.provided.runtimei (provided)
Decompressing /home/x/Source/test/target/function.zip
Skip pulling image and use local one: public.ecr.aws/sam/emulation-provided:rapid-1.40.0-x86_64.

Mounting /tmp/tmpjs_qxczo as /var/task:ro,delegated inside runtime container
START RequestId: 133f829d-5b03-4fda-bb70-1707922fed61 Version: $LATEST
/var/task/bootstrap: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /var/task/bootstrap)
27 Feb 2022 21:16:23,204 [ERROR] (rapid) Init failed error=Runtime exited with error: exit status 1 InvokeID=
/var/task/bootstrap: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /var/task/bootstrap)
END RequestId: 9124214f-1a01-4b9d-835f-1708e5597a6c
REPORT RequestId: 9124214f-1a01-4b9d-835f-1708e5597a6c	Init Duration: 0.13 ms	Duration: 7.61 ms	Billed Duration: 8 ms	Memory Size: 128 MB	Max Memory Used: 128 MB

How to Reproduce?

Download the attached project above and run the following commands:

./mvnw package -Dnative
sam local invoke --template target/sam.native.yaml

Output of `uname -a` or `ver`

Linux my-ubuntu 5.13.0-30-generic #33~20.04.1-Ubuntu SMP Mon Feb 7 14:25:10 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Output of `java -version`

openjdk version "11.0.14" 2022-01-18 OpenJDK Runtime Environment GraalVM CE 21.3.1 (build 11.0.14+9-jvmci-21.3-b09) OpenJDK 64-Bit Server VM GraalVM CE 21.3.1 (build 11.0.14+9-jvmci-21.3-b09, mixed mode, sharing)

GraalVM version (if different from Java)

GraalVM CE 21.3.1 (build 11.0.14+9-jvmci-21.3-b09)

Quarkus version or git rev

2.7.2.Final

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Apache Maven 3.8.4

Additional information

No response

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2022-02-27T21:30:02Z

/cc @matejvasek, @patriot1burke

geoand · 2022-02-28T06:17:47Z

cc @Karm

Karm · 2022-02-28T09:15:07Z

Hmm, weird, I looks as if MLib wasn't baked in, i.e. before https://bugs.openjdk.java.net/browse/JDK-8254024.
I will take a look.

gsmet · 2022-02-28T18:44:37Z

@Karm couldn't it be that the native image is built with a Glibc that is incompatible with the one of the runtime image?

Karm · 2022-03-01T14:29:10Z

@gsmet It is weird, because @felixosth 's system in not anything obscure, it's just a modern Ubuntu.

It works for me with my GLibc 2.28:

$ export JAVA_HOME=~/workspaceRH/mandrel-release/CPU/mandrel-java11-22.0.0.2-Final/;export GRAALVM_HOME=${JAVA_HOME};export PATH=${JAVA_HOME}/bin:${PATH}
$ ./mvnw package -Dnative
$ sam local invoke --template target/sam.native.yaml

Starting Quarkus native just fine:

Invoking not.used.in.provided.runtimei (provided)
Decompressing /home/karm/workspaceRH/test-aws-lambda-awt/target/function.zip
Image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-provided
Building image.............................................................................................................................................................................................................................................................................................................................................................................................
Skip pulling image and use local one: public.ecr.aws/sam/emulation-provided:rapid-1.40.0-x86_64.

Mounting /tmp/tmpqa_cl719 as /var/task:ro,delegated inside runtime container
START RequestId: 23553e3b-aaf2-43a2-83a7-4c325ffa2a61 Version: $LATEST
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2022-03-01 12:52:31,611 INFO  [io.quarkus] (main) test 1.0-SNAPSHOT native (powered by Quarkus 2.7.2.Final) started in 0.009s. 
2022-03-01 12:52:31,612 INFO  [io.quarkus] (main) Profile prod activated. 
2022-03-01 12:52:31,612 INFO  [io.quarkus] (main) Installed features: [amazon-lambda, awt, cdi, resteasy, security, smallrye-context-propagation, vertx]
2022-03-01 12:52:31,612 ERROR [qua.ama.lam.http] (Lambda Thread (NORMAL)) Request Failure: java.lang.IllegalStateException: Missing HTTP method in request event
    at io.quarkus.amazon.lambda.http.LambdaHttpHandler.nettyDispatch(LambdaHttpHandler.java:176)
    at io.quarkus.amazon.lambda.http.LambdaHttpHandler.handleRequest(LambdaHttpHandler.java:63)
    at io.quarkus.amazon.lambda.http.LambdaHttpHandler.handleRequest(LambdaHttpHandler.java:43)
    at io.quarkus.amazon.lambda.runtime.AmazonLambdaRecorder$1.processRequest(AmazonLambdaRecorder.java:164)
    at io.quarkus.amazon.lambda.runtime.AbstractLambdaPollLoop$1.run(AbstractLambdaPollLoop.java:126)
    at java.lang.Thread.run(Thread.java:829)
    at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:597)
    at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:194)

{"statusCode":500,"headers":null,"multiValueHeaders":{"Content-Type":["application/json"]},"cookies":null,"body":"{ \"message\": \"Internal Server Error\" }","isBase64Encoded":false}END RequestId: 23553e3b-aaf2-43a2-83a7-4c325ffa2a61
REPORT RequestId: 23553e3b-aaf2-43a2-83a7-4c325ffa2a61  Init Duration: 0.12 ms  Duration: 12.65 ms  Billed Duration: 13 ms  Memory Size: 128 MB Max Memory Used: 128 MB

The error at the end is related to something else. Reading the guide:

Responses for binary types will be automatically encoded with base64. This is different than the behavior
using quarkus:dev which will return the raw bytes. Amazon’s API has additional restrictions requiring the
base64 encoding. In general, client code will automatically handle this encoding but in certain custom
situations, you should be aware you may need to manually manage that encoding.

My system:
Centos 8 Stream, kernel: 4.18.0-365.el8.x86_64
GLibC: GNU C Library (GNU libc) stable release version 2.28.

I will try with Ubuntu...

felixosth · 2022-03-01T14:41:36Z

Might be worth mentioning that we build our app in Github Actions using the command ./mvnw package -Dnative -Dquarkus.native.container-build=true and that's where I found the problem initially. After that I installed Ubuntu (using Windows usually) and built it there.

Karm · 2022-03-01T18:35:27Z

I can reproduce it on my Ubuntu VM:

Mounting /tmp/tmp027xwngq as /var/task:ro,delegated inside runtime container
START RequestId: d26ca004-2b76-4410-a2ad-cebfc4528209 Version: $LATEST
/var/task/bootstrap: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /var/task/bootstrap)
01 Mar 2022 17:25:24,791 [ERROR] (rapid) Init failed error=Runtime exited with error: exit status 1 InvokeID=
/var/task/bootstrap: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /var/task/bootstrap)

System:

Linux karm-KVM 5.13.0-28-generic #31~20.04.1-Ubuntu SMP 
GNU C Library (Ubuntu GLIBC 2.31-0ubuntu9.2) stable release version 2.31.

Karm · 2022-03-02T14:59:46Z

IMHO Ubuntu's glibc is too new for the Linux AWS Lambda executes on. I will try to confirm...

gsmet · 2022-03-02T15:01:50Z

It's not really about being too new or too old. It's more that you need consistent Glibc (or at least compatible Glibc) in the builder image and in the runtime image.

Karm · 2022-03-02T15:11:00Z

Well, the version (backward vs forward) matters though. e.g. executable built on Centos 8 works just fine on Ubuntu 20, but not vice versa.

$ scp ubuntu20://home/karm/test-aws-lambda-awt/target/test-1.0-SNAPSHOT-runner /tmp/

$ /tmp/test-1.0-SNAPSHOT-runner 
/tmp/test-1.0-SNAPSHOT-runner: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /tmp/test-1.0-SNAPSHOT-runner)

Karm · 2022-03-02T15:35:17Z

I also suspect that if I make it work (the base64 / no binary in API AWS Lambda limitation thing), it would hit possibly missing libraries such as fontconfig etc. I will try to toy with AWS Lambda to make a working example.
I don't think this is the last time we are seeing an issue of this kind, i.e. moving a Linux executable built on Linux X to execute on Linux Y. And we cannot statically link everything...

felixosth · 2022-03-07T12:23:57Z

Please let me know if I can do anything to help!

knuspertante · 2022-05-12T07:39:58Z

I run in the same issue with lambda...

You must provide a libm.so.6 file in zip.native under resources folder (See: https://quarkus.io/guides/amazon-lambda#modifying-function-zip)

If your host system isn't linux you could get the libm.so.6 file from docker as described here: https://quarkus.io/guides/amazon-lambda#additional-requirements-for-client-ssl

felixosth · 2022-05-12T07:46:53Z

I run in the same issue with lambda...

You must provide a libm.so.6 file in zip.native under resources folder (See: https://quarkus.io/guides/amazon-lambda#modifying-function-zip)

If your host system isn't linux you could get the libm.so.6 file from docker as described here: https://quarkus.io/guides/amazon-lambda#additional-requirements-for-client-ssl

So you managed to fix it with libm.so.6 only? I remember troubleshooting by adding libz, libfreetype, libfontconfig and libpng16 but that didn't work.

knuspertante · 2022-05-12T07:59:53Z

Ah sorry, I only read your stacktrace. Sure I also provide some additional libs like libz2, libexpat, libfreetype, libfontconfig, ...

Did you use Amazon Linux 1 or 2?

The error is because libm use a newer version of libz.

I managed this, that I tell the lambda (through environment properties) to preload the provided libz.

JAVA CDK-Script:

Function lambdaFunctionRunTask =
        Function.Builder.create(this, "NAME")
            .runtime(Runtime.PROVIDED_AL2)
            .handler("not.used.in.provided.runtime")
            .memorySize(2048)
            .architecture(Architecture.ARM_64)
            .environment(
                new HashMap<>() {
                  {
                    put("LD_PRELOAD", "/var/task/libz.so.1");
                    put("DISABLE_SIGNAL_HANDLERS", "true");
                  }
                })
            ...
            .build();

Karm · 2023-11-15T08:33:15Z

Closing. Not a Quarkus/Mandrel issue any more; one needs to pick compatible runtime for the deployment (libc, dependencies).

felixosth added the kind/bug Something isn't working label Feb 27, 2022

quarkus-bot bot added the area/amazon-lambda label Feb 27, 2022

geoand added the area/native-image label Feb 28, 2022

Karm self-assigned this Feb 28, 2022

Karm mentioned this issue Jun 1, 2022

AWS Lambda REST throws an exception in native mode #25897

Closed

Karm closed this as completed Nov 15, 2023

Karm mentioned this issue Jun 3, 2024

Native executable quarkus-amazon-lambda images built on Mac M1 Pro not working on AWS #38715

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWT graphics with AWS Lambda not working in native #23998

AWT graphics with AWS Lambda not working in native #23998

felixosth commented Feb 27, 2022

quarkus-bot bot commented Feb 27, 2022

geoand commented Feb 28, 2022

Karm commented Feb 28, 2022

gsmet commented Feb 28, 2022

Karm commented Mar 1, 2022

felixosth commented Mar 1, 2022

Karm commented Mar 1, 2022 •

edited

Loading

Karm commented Mar 2, 2022

gsmet commented Mar 2, 2022

Karm commented Mar 2, 2022 •

edited

Loading

Karm commented Mar 2, 2022

felixosth commented Mar 7, 2022

knuspertante commented May 12, 2022

felixosth commented May 12, 2022

knuspertante commented May 12, 2022

Karm commented Nov 15, 2023

AWT graphics with AWS Lambda not working in native #23998

AWT graphics with AWS Lambda not working in native #23998

Comments

felixosth commented Feb 27, 2022

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

quarkus-bot bot commented Feb 27, 2022

geoand commented Feb 28, 2022

Karm commented Feb 28, 2022

gsmet commented Feb 28, 2022

Karm commented Mar 1, 2022

felixosth commented Mar 1, 2022

Karm commented Mar 1, 2022 • edited Loading

Karm commented Mar 2, 2022

gsmet commented Mar 2, 2022

Karm commented Mar 2, 2022 • edited Loading

Karm commented Mar 2, 2022

felixosth commented Mar 7, 2022

knuspertante commented May 12, 2022

felixosth commented May 12, 2022

knuspertante commented May 12, 2022

Karm commented Nov 15, 2023

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Karm commented Mar 1, 2022 •

edited

Loading

Karm commented Mar 2, 2022 •

edited

Loading