java.lang.ClassNotFoundException: com.github.dockerjava.netty.WebTarget on 'Test Connection'

[FeynmanFan]

I'm using latest version of the plugin - 1.1.9, upgraded as part of the new Jenkins LTS (2.222.1). When attempting to configure a new Docker cloud, I'm attempting to test the connection to the host (which I have verified is running with a containers/json call in the browser), and I receive this error:

java.lang.ClassNotFoundException: com.github.dockerjava.netty.WebTarget
	at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1387)
	at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1342)
	at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1089)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
Caused: java.lang.NoClassDefFoundError: com/github/dockerjava/netty/WebTarget
	at io.jenkins.docker.client.NettyDockerCmdExecFactory.init(NettyDockerCmdExecFactory.java:235)
	at com.github.dockerjava.core.DockerClientImpl.withDockerCmdExecFactory(DockerClientImpl.java:193)
	at com.github.dockerjava.core.DockerClientBuilder.build(DockerClientBuilder.java:45)
	at io.jenkins.docker.client.DockerAPI.makeClient(DockerAPI.java:249)
	at io.jenkins.docker.client.DockerAPI.getOrMakeClient(DockerAPI.java:199)
	at io.jenkins.docker.client.DockerAPI.getClient(DockerAPI.java:168)
	at io.jenkins.docker.client.DockerAPI.getClient(DockerAPI.java:151)
	at io.jenkins.docker.client.DockerAPI$DescriptorImpl.doTestConnection(DockerAPI.java:358)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
Caused: javax.servlet.ServletException
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:797)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:748)

I believe this actually may be due to a problem in the Docker-API plug-in, which has changed with the LTS as well. This all worked perfectly when Jenkins LTS was 2.204.6:.

[FeynmanFan]

Confirmed that downgrading the Docker-API plugin from 3.1.5 to 3.0.14 fixes the problem...should I close the issue, or is this still a Docker plugin issue since it relies on it?

[pjdarton]

This plugin can only specify a minimum version of plugins it depends on; it can't say "don't use > v1.2.3" so there's no code changes possible that'd fix it here. This needs fixing in the docker Java API plugin (and ASAP).
Raise the issue there and reference this one.

[FeynmanFan]

It looks like that project has bug reports disabled...

[FeynmanFan]

Is there some way to contact Eric Citaire directly?

[pjdarton]

Hmm... @ericcitaire did you get access to docker-java-api in the end? Are you aware of this unwanted side effect?

[pjdarton]

This is probably all to do with PR #777 here so, yes, I'm fairly sure Eric will notice this.

[pjdarton]

FYI it's now 1am where I am (UK) and I really should be going to sleep now ... but I'll check my email etc tomorrow morning.
It may be we need to do a fast release of this plugin with #777 merged ... but that'll have to wait until (my) tomorrow.

[FeynmanFan]

That would be most appreciated. I'm creating a Jenkins-Docker course for Pluralsight, and that will get me unblocked.

If you guys have anything you wish people knew about this stuff, let me know and I'll include it.

[abakhru]

Any ETA for release of fix for this issue. Its blocking me. ASAP release would be appreciated.

[ericcitaire]

Hi all. Thank you for reporting this. As stated by @pjdarton, there's no way to specify a maximum version in plugin dependencies.

docker-java-api-plugin is just a wrapper around the docker-java lib. If breaking changes are introduced from one version to another (ie. moving a class), I'm afraid I cannot do anything about it.

I created a PR (#777) to upgrade docker-java-api in docker-plugin. Feel free to join the discussion.

In the meantime, the only thing I can suggest you is to downgrade to docker-java-api-plugin 3.0.14.

[pjdarton]

Docker plugin 1.2.0 is available; that works with docker-java-api-plugin version 3.1.5 (but not with version 3.0.14)

[FeynmanFan]

Verified - this is working correctly now - thanks for fixing this so quickly.

[ericcitaire]

I just released docker-java-api-plugin 3.1.5.1, with an incompatibility warning (see screenshot below). I hope this will limit the damage for those who didn't update the plugin yet.

Sorry for the inconvenience and thanks a lot to @pjdarton for fixing my mistake.

[roman-vynar]

Not fixed.
Now there is a different problem after upgrading those 2 plugins above ^^
There is no error but it is stuck on this:

2020-04-03 08:17:36.913+0000 [id=47]	INFO	c.n.j.plugins.docker.DockerCloud#canAddProvisionedSlave: Provisioning 'docker.repo/service-builder:latest' number 1 (of 1) on 'local-docker'; Total containers: 0 (of 100)
2020-04-03 08:17:36.914+0000 [id=47]	INFO	c.n.j.plugins.docker.DockerCloud#provision: Will provision 'docker.repo/service-builder:latest', for label: 'local-builder', in cloud: 'local-docker'
2020-04-03 08:17:36.914+0000 [id=41]	INFO	c.n.j.p.docker.DockerTemplate#pullImage: Pulling image 'docker.repo/service-builder:latest'. This may take awhile...

Repeating those messages every 10s and nothing happens... The job is still stuck.
Reverted those 2 plugins and all back to normal.

[geraintwhite]

I had the same issue as @roman-vynar when upgrading the two plugins and have also had to revert to the previous versions.

[ericcitaire]

Not fixed.
Now there is a different problem after upgrading those 2 plugins above ^^
There is no error but it is stuck on this:

2020-04-03 08:17:36.913+0000 [id=47]	INFO	c.n.j.plugins.docker.DockerCloud#canAddProvisionedSlave: Provisioning 'docker.repo/service-builder:latest' number 1 (of 1) on 'local-docker'; Total containers: 0 (of 100)
2020-04-03 08:17:36.914+0000 [id=47]	INFO	c.n.j.plugins.docker.DockerCloud#provision: Will provision 'docker.repo/service-builder:latest', for label: 'local-builder', in cloud: 'local-docker'
2020-04-03 08:17:36.914+0000 [id=41]	INFO	c.n.j.p.docker.DockerTemplate#pullImage: Pulling image 'docker.repo/service-builder:latest'. This may take awhile...

Repeating those messages every 10s and nothing happens... The job is still stuck.
Reverted those 2 plugins and all back to normal.

This is not the same issue. Can you create a separate issue please?

Also, it would be great to tell us how to reproduce this, because I don't have this behavior. Thanks.

[roman-vynar]

Not the same but directly related.

I am just building a job pointed to the jenkins cloud template which therefore is pointed to a local docker socket. Nothing special. The job is stuck in the queue and not progressing. The behaviour is described above and one other user confirmed.

[roman-vynar]

I think this is the same issue ^^

[pjdarton]

@roman-vynar @grit96 What you're reporting is a new issue. It's probably got the same underlying trigger (the change to the docker-java-api-plugin or, more specifically, it's change from docker-java version 3.0.14 to version 3.1.5), but the solution will be very different.
I suggest all discussion of that issue move to #784 - if you can assist Eric by testing any PRs he raises (they should auto-build) then that'll help everyone and result in a speedier solution.

For anyone else watching, or anyone in future finding this issue after encountering other issues seemingly caused by the upgrade from docker-java-api-plugin 3.0.14 to 3.1.5, please raise a new issue and include a mention of this one (just use the text #782) in your description.