Using Jenkins Docker Slaves on Amazon Linux 2 AMI

[cvrooman]

I'm running into a strange issue using the standard docker images provided by Jenkins - only when Jenkins is running on Amazon Linux 2 though. Jenkins on Ubuntu 18.04 works fine.

The problem is that Jenkins is not able to launch these slave containers.

Connect method - "Attach Docker Container" since the docker engine is running locally on the Jenkins server.

Interestingly, I've been able to get these containers to successfully run for a build on Amazon Linux if I set USER root in a new DockerFile (not required on Ubuntu). However, I need these to be running as a non-root user (like jenkins user) in order to run unit testing for postgres.

Any thoughts on how I can achieve that for Amazon Linux 2? Could I be missing a docker setting or security setting on the OS?

Docker images I've tested:

• https://hub.docker.com/r/jenkins/agent/
• https://hub.docker.com/r/jenkins/jnlp-slave
• https://hub.docker.com/r/jenkins/inbound-agent

If you are sure that this plugin is the source of your troubles then please report:

• [1.2.1 ] docker-plugin version
• [2.266 ] jenkins version
• [19.03.13-ce ] docker engine version
• ["Attach Docker Container" ] details of the docker container(s) involved and details of how the docker-plugin is connecting to them

The errors aren't super helpful:

Connecting to docker container 4486c8f2696f40c7de32160d78d6e2438abb1ff92517c71cf66b93baf3d0877f, running command java -jar /home/jenkins/remoting-4.6.jar -noReconnect -noKeepAlive -slaveLog /home/jenkins/agent.log HTTP/1.1 101 UPGRADED Content-Type: application/vnd.docker.raw-stream Connection: Upgrade Upgrade: tcp Api-Version: 1.40 Docker-Experimental: false Ostype: linux Server: Docker/19.03.13-ce (Linux)

ERROR: Unexpected error in launching an agent. This is probably a bug in Jenkins

Basic Setup to Reproduce:

Docker Template:

[pjdarton]

I suspect that you're hitting bug #818 and this has nothing to do with Amazon-specific stuff.
Take a look at the workaround detailed in #818 and see if that fixes it.

[cvrooman]

@pjdarton thanks for your help! But this seems to have the same result upon launching a container. I've updated the
-slaveLog parameter to -agentLog to test.

Anything I can do to better debug this?

Full Log:

Connecting to docker container 9e92bacb6866a9f3335894d76415a1e2561ed629ca80578ae4d7d13fa6193dc6, running command java -jar //remoting-4.5.jar -noReconnect -noKeepAlive -agentLog //agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
Api-Version: 1.40
Docker-Experimental: false
Ostype: linux
Server: Docker/19.03.6-ce (linux)

ERROR: Unexpected error in launching an agent. This is probably a bug in Jenkins
Also:   java.lang.Throwable: launched here
	at hudson.slaves.SlaveComputer._connect(SlaveComputer.java:283)
	at hudson.model.Computer.connect(Computer.java:435)
	at com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy.start(DockerOnceRetentionStrategy.java:80)
	at com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy.start(DockerOnceRetentionStrategy.java:35)
	at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:162)
	at hudson.model.AbstractCIBase.access$000(AbstractCIBase.java:44)
	at hudson.model.AbstractCIBase$2.run(AbstractCIBase.java:224)
	at hudson.model.Queue._withLock(Queue.java:1401)
	at hudson.model.Queue.withLock(Queue.java:1278)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:207)
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1634)
	at jenkins.model.Nodes$2.run(Nodes.java:139)
	at hudson.model.Queue._withLock(Queue.java:1401)
	at hudson.model.Queue.withLock(Queue.java:1278)
	at jenkins.model.Nodes.addNode(Nodes.java:135)
	at jenkins.model.Jenkins.addNode(Jenkins.java:2157)
	at com.nirima.jenkins.plugins.docker.DockerCloud.robustlyAddNodeToJenkins(DockerCloud.java:445)
	at com.nirima.jenkins.plugins.docker.DockerCloud.access$000(DockerCloud.java:68)
	at com.nirima.jenkins.plugins.docker.DockerCloud$1.run(DockerCloud.java:381)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
java.io.EOFException: unexpected stream termination
	at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:415)
	at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:360)
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:425)
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:392)
	at io.jenkins.docker.connector.DockerComputerAttachConnector$DockerAttachLauncher.launch(DockerComputerAttachConnector.java:319)
	at hudson.slaves.DelegatingComputerLauncher.launch(DelegatingComputerLauncher.java:64)
	at io.jenkins.docker.connector.DockerDelegatingComputerLauncher.launch(DockerDelegatingComputerLauncher.java:37)
	at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:294)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)