fresh minikub v0.23.0 fails to start

[bpasson]

This is a bug report.
Environment:

Minikube version: v0.23.0

• OS: OSX 10.11
• VM Driver: vmwarefusion
• ISO version: minikube-v0.23.6.iso
• Install tools: curl
• Others: nothing

What happened:
A newly created minikube failed to start.

What you expected to happen:
A newly creatted minikube started successfully

How to reproduce it (as minimally and precisely as possible):
minikube -v10 start --vm-driver=vmwarefusion

Output of minikube logs (if applicable):
Nov 01 15:20:11 minikube localkube[3644]: Starting apiserver...
Nov 01 15:20:11 minikube localkube[3644]: Waiting for apiserver to be healthy...
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.270847 3644 server.go:114] Version: v1.8.0
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271174 3644 interface.go:360] Looking for default routes with IPv4 addresses
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271186 3644 interface.go:365] Default route transits interface "eth0"
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271258 3644 interface.go:174] Interface eth0 is up
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271297 3644 interface.go:222] Interface "eth0" has 2 addresses :[10.0.2.15/24 fe80::a00:27ff:fec0:f6cc/64].
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271311 3644 interface.go:189] Checking addr 10.0.2.15/24.
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271318 3644 interface.go:196] IP found 10.0.2.15
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271324 3644 interface.go:228] Found valid IPv4 address 10.0.2.15 for interface "eth0".
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271330 3644 interface.go:371] Found active IP 10.0.2.15
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271349 3644 services.go:51] Setting service IP to "10.0.0.1" (read-write).
Nov 01 15:20:11 minikube localkube[3644]: W1101 15:20:11.271363 3644 authentication.go:380] AnonymousAuth is not allowed with the AllowAll authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271740 3644 server.go:644] Initializing deserialization cache size based on 0MB limit
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271751 3644 server.go:663] Initializing cache sizes based on 0MB limit
Nov 01 15:20:11 minikube localkube[3644]: I1101 15:20:11.271772 3644 plugins.go:101] No cloud provider specified.
Nov 01 15:20:11 minikube localkube[3644]: setting up the initial cluster version to 3.1
Nov 01 15:20:11 minikube localkube[3644]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
Nov 01 15:20:11 minikube localkube[3644]: set the initial cluster version to 3.1
Nov 01 15:20:11 minikube localkube[3644]: enabled capabilities for version 3.1
Nov 01 15:20:11 minikube localkube[3644]: apiserver: Exit with error: unable to load client CA file: unable to load client CA file: open /var/lib/localkube/certs/proxy-client-ca.crt: no such file or directory
Nov 01 15:20:12 minikube localkube[3644]: I1101 15:20:12.134547 3644 server.go:114] Version: v1.8.0
Nov 01 15:20:12 minikube localkube[3644]: I1101 15:20:12.134722 3644 services.go:51] Setting service IP to "10.0.0.1" (read-write).
Nov 01 15:20:12 minikube localkube[3644]: I1101 15:20:12.134746 3644 server.go:663] Initializing cache sizes based on 0MB limit
Nov 01 15:20:12 minikube localkube[3644]: I1101 15:20:12.134791 3644 plugins.go:101] No cloud provider specified.
Nov 01 15:20:12 minikube localkube[3644]: I1101 15:20:12.270979 3644 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
Nov 01 15:20:12 minikube localkube[3644]: E1101 15:20:12.272100 3644 ready.go:40] Error performing healthcheck: Get https://localhost:8443/healthz: dial tcp 127.0.0.1:8443: getsockopt: connection refused
Nov 01 15:20:12 minikube localkube[3644]: apiserver: Exit with error: unable to load client CA file: unable to load client CA file: open /var/lib/localkube/certs/proxy-client-ca.crt: no such file or directory

Anything else do we need to know:
nothing

[r2d4]

Can you try a delete and a start? If you've upgraded from an old version of minikube, you need to recreate the VM.

[bpasson]

As you requested I performed a delete and a start. The first time I got a VM but not all certificates, proxy-* certs were missing. Tried it again, but then minikube got stuck at the point where userdata.tar is copied. That failed for some reason. Manually fixing that and then calling minikube start again gave me the correct certs. Might be my VMWare Fusion version (6.0.6) that is the culprit here. I need to do a few more runs and do alle the minikube steps manually to see what is causing the problem.

[jonpharry]

Failure is due to the command "tar xvf userdata.tar ...". The working directory for the /bin/sh is / and so file is not found. Command should be "tar xvf /home/docker/userdata.tar ...". The same issue has already been fixed in docker-machine.

[bpasson]

I did some checking and it indeed seems to be the issue like @jonpharry mentions. Problem is in the following part

Creating Tar key bundle...
executing: /Applications/VMware Fusion.app/Contents/Library/vmrun -gu docker -gp tcuser directoryExistsInGuest /Users/basp/.minikube/machines/minikube/minikube.vmx /var/lib/boot2docker
executing: /Applications/VMware Fusion.app/Contents/Library/vmrun -gu docker -gp tcuser CopyFileFromHostToGuest /Users/basp/.minikube/machines/minikube/minikube.vmx /Users/basp/.minikube/machines/minikube/userdata.tar /home/docker/userdata.tar
executing: /Applications/VMware Fusion.app/Contents/Library/vmrun -gu docker -gp tcuser runScriptInGuest /Users/basp/.minikube/machines/minikube/minikube.vmx /bin/sh sudo sh -c "tar xvf userdata.tar -C /home/docker > /var/log/userdata.log 2>&1 && chown -R docker:staff /home/docker"
executing: /Applications/VMware Fusion.app/Contents/Library/vmrun -gu docker -gp tcuser runScriptInGuest /Users/basp/.minikube/machines/minikube/minikube.vmx /bin/sh sudo /bin/mv /home/docker/userdata.tar /var/lib/boot2docker/userdata.tar

[bpasson]

Is there anything known about when the fix from docker/machine will be included in minikube? I tried to update the go dependency on docker/machine myself for a local build, but that breaks the code.

[anfernee]

@bpasson can you try v0.25.0, #2126 got fixed there. it looks like the same issue.

[mikeroySoft]

Confirm this looks fixed in 0.25.0

[bpasson]

It seems to work now in 0.25.0

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close