minikube 1.11.0: PVC stuck in pending, storage provisioner parse error

[wcjohnson]

Issue Description
Persistent Volume Claims created by various operators (in this example, the Strimzi Kafka operator) remain stuck in the Pending state. Upon examination of the output of the storage-provisioner pod in kube-system, there appears to be a JSON parse error. The JSON appears to be valid, and the same operator-generated volume claim works as expected on other Kubernetes implementations (GKE, Docker for Mac)

Steps to reproduce the issue:

1. Start Minikube: minikube start --driver=parallels (Mac OS 10.15, Parallels 15)
2. Install Strimzi Kafka operator:
   helm repo add strimzi https://strimzi.io/charts/
helm install strimzi strimzi/strimzi-kafka-operator --namespace kafka --set watchAnyNamespace=true
3. Deploy a Kafka instance:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  namespace: development
  name: kafka
spec:
  kafka:
    version: 2.4.0
    replicas: 1
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.3"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 1Gi
        deleteClaim: true
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 1Gi
      deleteClaim: true

Full output of failed command:
The following error is repeatedly emitted by the storage-provisioner pod:

First line: [storage-provisioner storage-provisioner] E0530 04:50:23.157132 1 reflector.go:205] k8s.io/minikube/vendor/github.com/r2d4/external-storage/lib/controller/controller.go:411: Failed to list *v1.PersistentVolumeClaim: v1.PersistentVolumeClaimList: Items: []v1.PersistentVolumeClaim: v1.PersistentVolumeClaim: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 5301 ...:{},"k:{\"... at {"kind"

Full error emitted by storage-provisioner

``` [storage-provisioner storage-provisioner] E0530 04:50:23.157132 1 reflector.go:205] k8s.io/minikube/vendor/github.com/r2d4/external-storage/lib/controller/controller.go:411: Failed to list *v1.PersistentVolumeClaim: v1.PersistentVolumeClaimList: Items: []v1.PersistentVolumeClaim: v1.PersistentVolumeClaim: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 5301 ...:{},"k:{\"... at {"kind":"PersistentVolumeClaimList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/persistentvolumeclaims","resourceVersion":"2745"},"items":[{"metadata":{"name":"data-mysql-0","namespace":"development","selfLink":"/api/v1/namespaces/development/persistentvolumeclaims/data-mysql-0","uid":"b9bdebe8-117f-49ad-bb07-0eb1492192fe","resourceVersion":"1246","creationTimestamp":"2020-05-30T04:38:36Z","labels":{"kubedb.com/kind":"MySQL","kubedb.com/name":"mysql"},"annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"1d078d69-a22f-11ea-a37d-001c42492410\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-05-30T04:38:36Z\",\"renewTime\":\"2020-05-30T04:39:06Z\",\"leaderTransitions\":0}","pv.kubernetes.io/bind-completed":"yes","pv.kubernetes.io/bound-by-controller":"yes","volume.beta.kubernetes.io/storage-provisioner":"k8s.io/minikube-hostpath"},"finalizers":["kubernetes.io/pvc-protection"],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:38:36Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{}},"f:labels":{".":{},"f:kubedb.com/kind":{},"f:kubedb.com/name":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{},"f:volumeName":{}},"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}}},{"manager":"storage-provisioner","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:39:06Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:control-plane.alpha.kubernetes.io/leader":{}}}}}]},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"2Gi"}},"volumeName":"pvc-b9bdebe8-117f-49ad-bb07-0eb1492192fe","storageClassName":"standard","volumeMode":"Filesystem"},"status":{"phase":"Bound","accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"}}},{"metadata":{"name":"data-redis-0","namespace":"development","selfLink":"/api/v1/namespaces/development/persistentvolumeclaims/data-redis-0","uid":"4e064ada-dfb7-48de-9093-111dc45e0651","resourceVersion":"1299","creationTimestamp":"2020-05-30T04:38:55Z","labels":{"kubedb.com/kind":"Redis","kubedb.com/name":"redis"},"annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"1d078d69-a22f-11ea-a37d-001c42492410\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-05-30T04:38:55Z\",\"renewTime\":\"2020-05-30T04:39:25Z\",\"leaderTransitions\":0}","pv.kubernetes.io/bind-completed":"yes","pv.kubernetes.io/bound-by-controller":"yes","volume.beta.kubernetes.io/storage-provisioner":"k8s.io/minikube-hostpath"},"finalizers":["kubernetes.io/pvc-protection"],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:38:55Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{}},"f:labels":{".":{},"f:kubedb.com/kind":{},"f:kubedb.com/name":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{},"f:volumeName":{}},"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}}},{"manager":"storage-provisioner","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:39:25Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:control-plane.alpha.kubernetes.io/leader":{}}}}}]},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"volumeName":"pvc-4e064ada-dfb7-48de-9093-111dc45e0651","storageClassName":"standard","volumeMode":"Filesystem"},"status":{"phase":"Bound","accessModes":["ReadWriteOnce"],"capacity":{"storage":"1Gi"}}},{"metadata":{"name":"data-kafka-zookeeper-0","namespace":"development","selfLink":"/api/v1/namespaces/development/persistentvolumeclaims/data-kafka-zookeeper-0","uid":"fabbaafe-f1d9-40e3-a342-0e2e95e5c83f","resourceVersion":"2745","creationTimestamp":"2020-05-30T04:43:44Z","labels":{"app.kubernetes.io/instance":"kafka","app.kubernetes.io/managed-by":"strimzi-cluster-operator","app.kubernetes.io/name":"zookeeper","app.kubernetes.io/part-of":"strimzi-kafka","strimzi.io/cluster":"kafka","strimzi.io/kind":"Kafka","strimzi.io/name":"kafka-zookeeper"},"annotations":{"strimzi.io/delete-claim":"true","volume.beta.kubernetes.io/storage-provisioner":"k8s.io/minikube-hostpath"},"ownerReferences":[{"apiVersion":"kafka.strimzi.io/v1beta1","kind":"Kafka","name":"kafka","uid":"599d91ef-9823-47ba-a67c-bd1c447c6d6f","controller":false,"blockOwnerDeletion":false}],"finalizers":["kubernetes.io/pvc-protection"],"managedFields":[{"manager":"okhttp","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:43:44Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:strimzi.io/delete-claim":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:strimzi.io/cluster":{},"f:strimzi.io/kind":{},"f:strimzi.io/name":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"599d91ef-9823-47ba-a67c-bd1c447c6d6f\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}},"f:status":{"f:phase":{}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:48:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:volume.beta.kubernetes.io/storage-provisioner":{}},"f:finalizers":{".":{},"v:\"kubernetes.io/pvc-protection\"":{}}}}}]},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"standard","volumeMode":"Filesystem"},"status":{"phase":"Pending"}}]} ```

Pretty-printed JSON:

{
  "kind": "PersistentVolumeClaimList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/persistentvolumeclaims",
    "resourceVersion": "2745"
  },
  "items": [
    {
      "metadata": {
        "name": "data-mysql-0",
        "namespace": "development",
        "selfLink": "/api/v1/namespaces/development/persistentvolumeclaims/data-mysql-0",
        "uid": "b9bdebe8-117f-49ad-bb07-0eb1492192fe",
        "resourceVersion": "1246",
        "creationTimestamp": "2020-05-30T04:38:36Z",
        "labels": {
          "kubedb.com/kind": "MySQL",
          "kubedb.com/name": "mysql"
        },
        "annotations": {
          "control-plane.alpha.kubernetes.io/leader": "{\"holderIdentity\":\"1d078d69-a22f-11ea-a37d-001c42492410\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-05-30T04:38:36Z\",\"renewTime\":\"2020-05-30T04:39:06Z\",\"leaderTransitions\":0}",
          "pv.kubernetes.io/bind-completed": "yes",
          "pv.kubernetes.io/bound-by-controller": "yes",
          "volume.beta.kubernetes.io/storage-provisioner": "k8s.io/minikube-hostpath"
        },
        "finalizers": [
          "kubernetes.io/pvc-protection"
        ],
        "managedFields": [
          {
            "manager": "kube-controller-manager",
            "operation": "Update",
            "apiVersion": "v1",
            "time": "2020-05-30T04:38:36Z",
            "fieldsType": "FieldsV1",
            "fieldsV1": {
              "f:metadata": {
                "f:annotations": {
                  ".": {},
                  "f:pv.kubernetes.io/bind-completed": {},
                  "f:pv.kubernetes.io/bound-by-controller": {},
                  "f:volume.beta.kubernetes.io/storage-provisioner": {}
                },
                "f:labels": {
                  ".": {},
                  "f:kubedb.com/kind": {},
                  "f:kubedb.com/name": {}
                }
              },
              "f:spec": {
                "f:accessModes": {},
                "f:resources": {
                  "f:requests": {
                    ".": {},
                    "f:storage": {}
                  }
                },
                "f:volumeMode": {},
                "f:volumeName": {}
              },
              "f:status": {
                "f:accessModes": {},
                "f:capacity": {
                  ".": {},
                  "f:storage": {}
                },
                "f:phase": {}
              }
            }
          },
          {
            "manager": "storage-provisioner",
            "operation": "Update",
            "apiVersion": "v1",
            "time": "2020-05-30T04:39:06Z",
            "fieldsType": "FieldsV1",
            "fieldsV1": {
              "f:metadata": {
                "f:annotations": {
                  "f:control-plane.alpha.kubernetes.io/leader": {}
                }
              }
            }
          }
        ]
      },
      "spec": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "resources": {
          "requests": {
            "storage": "2Gi"
          }
        },
        "volumeName": "pvc-b9bdebe8-117f-49ad-bb07-0eb1492192fe",
        "storageClassName": "standard",
        "volumeMode": "Filesystem"
      },
      "status": {
        "phase": "Bound",
        "accessModes": [
          "ReadWriteOnce"
        ],
        "capacity": {
          "storage": "2Gi"
        }
      }
    },
    {
      "metadata": {
        "name": "data-redis-0",
        "namespace": "development",
        "selfLink": "/api/v1/namespaces/development/persistentvolumeclaims/data-redis-0",
        "uid": "4e064ada-dfb7-48de-9093-111dc45e0651",
        "resourceVersion": "1299",
        "creationTimestamp": "2020-05-30T04:38:55Z",
        "labels": {
          "kubedb.com/kind": "Redis",
          "kubedb.com/name": "redis"
        },
        "annotations": {
          "control-plane.alpha.kubernetes.io/leader": "{\"holderIdentity\":\"1d078d69-a22f-11ea-a37d-001c42492410\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-05-30T04:38:55Z\",\"renewTime\":\"2020-05-30T04:39:25Z\",\"leaderTransitions\":0}",
          "pv.kubernetes.io/bind-completed": "yes",
          "pv.kubernetes.io/bound-by-controller": "yes",
          "volume.beta.kubernetes.io/storage-provisioner": "k8s.io/minikube-hostpath"
        },
        "finalizers": [
          "kubernetes.io/pvc-protection"
        ],
        "managedFields": [
          {
            "manager": "kube-controller-manager",
            "operation": "Update",
            "apiVersion": "v1",
            "time": "2020-05-30T04:38:55Z",
            "fieldsType": "FieldsV1",
            "fieldsV1": {
              "f:metadata": {
                "f:annotations": {
                  ".": {},
                  "f:pv.kubernetes.io/bind-completed": {},
                  "f:pv.kubernetes.io/bound-by-controller": {},
                  "f:volume.beta.kubernetes.io/storage-provisioner": {}
                },
                "f:labels": {
                  ".": {},
                  "f:kubedb.com/kind": {},
                  "f:kubedb.com/name": {}
                }
              },
              "f:spec": {
                "f:accessModes": {},
                "f:resources": {
                  "f:requests": {
                    ".": {},
                    "f:storage": {}
                  }
                },
                "f:volumeMode": {},
                "f:volumeName": {}
              },
              "f:status": {
                "f:accessModes": {},
                "f:capacity": {
                  ".": {},
                  "f:storage": {}
                },
                "f:phase": {}
              }
            }
          },
          {
            "manager": "storage-provisioner",
            "operation": "Update",
            "apiVersion": "v1",
            "time": "2020-05-30T04:39:25Z",
            "fieldsType": "FieldsV1",
            "fieldsV1": {
              "f:metadata": {
                "f:annotations": {
                  "f:control-plane.alpha.kubernetes.io/leader": {}
                }
              }
            }
          }
        ]
      },
      "spec": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "resources": {
          "requests": {
            "storage": "1Gi"
          }
        },
        "volumeName": "pvc-4e064ada-dfb7-48de-9093-111dc45e0651",
        "storageClassName": "standard",
        "volumeMode": "Filesystem"
      },
      "status": {
        "phase": "Bound",
        "accessModes": [
          "ReadWriteOnce"
        ],
        "capacity": {
          "storage": "1Gi"
        }
      }
    },
    {
      "metadata": {
        "name": "data-kafka-zookeeper-0",
        "namespace": "development",
        "selfLink": "/api/v1/namespaces/development/persistentvolumeclaims/data-kafka-zookeeper-0",
        "uid": "fabbaafe-f1d9-40e3-a342-0e2e95e5c83f",
        "resourceVersion": "2745",
        "creationTimestamp": "2020-05-30T04:43:44Z",
        "labels": {
          "app.kubernetes.io/instance": "kafka",
          "app.kubernetes.io/managed-by": "strimzi-cluster-operator",
          "app.kubernetes.io/name": "zookeeper",
          "app.kubernetes.io/part-of": "strimzi-kafka",
          "strimzi.io/cluster": "kafka",
          "strimzi.io/kind": "Kafka",
          "strimzi.io/name": "kafka-zookeeper"
        },
        "annotations": {
          "strimzi.io/delete-claim": "true",
          "volume.beta.kubernetes.io/storage-provisioner": "k8s.io/minikube-hostpath"
        },
        "ownerReferences": [
          {
            "apiVersion": "kafka.strimzi.io/v1beta1",
            "kind": "Kafka",
            "name": "kafka",
            "uid": "599d91ef-9823-47ba-a67c-bd1c447c6d6f",
            "controller": false,
            "blockOwnerDeletion": false
          }
        ],
        "finalizers": [
          "kubernetes.io/pvc-protection"
        ],
        "managedFields": [
          {
            "manager": "okhttp",
            "operation": "Update",
            "apiVersion": "v1",
            "time": "2020-05-30T04:43:44Z",
            "fieldsType": "FieldsV1",
            "fieldsV1": {
              "f:metadata": {
                "f:annotations": {
                  ".": {},
                  "f:strimzi.io/delete-claim": {}
                },
                "f:labels": {
                  ".": {},
                  "f:app.kubernetes.io/instance": {},
                  "f:app.kubernetes.io/managed-by": {},
                  "f:app.kubernetes.io/name": {},
                  "f:app.kubernetes.io/part-of": {},
                  "f:strimzi.io/cluster": {},
                  "f:strimzi.io/kind": {},
                  "f:strimzi.io/name": {}
                },
                "f:ownerReferences": {
                  ".": {},
                  "k:{\"uid\":\"599d91ef-9823-47ba-a67c-bd1c447c6d6f\"}": {
                    ".": {},
                    "f:apiVersion": {},
                    "f:blockOwnerDeletion": {},
                    "f:controller": {},
                    "f:kind": {},
                    "f:name": {},
                    "f:uid": {}
                  }
                }
              },
              "f:spec": {
                "f:accessModes": {},
                "f:resources": {
                  "f:requests": {
                    ".": {},
                    "f:storage": {}
                  }
                },
                "f:volumeMode": {}
              },
              "f:status": {
                "f:phase": {}
              }
            }
          },
          {
            "manager": "kube-controller-manager",
            "operation": "Update",
            "apiVersion": "v1",
            "time": "2020-05-30T04:48:45Z",
            "fieldsType": "FieldsV1",
            "fieldsV1": {
              "f:metadata": {
                "f:annotations": {
                  "f:volume.beta.kubernetes.io/storage-provisioner": {}
                },
                "f:finalizers": {
                  ".": {},
                  "v:\"kubernetes.io/pvc-protection\"": {}
                }
              }
            }
          }
        ]
      },
      "spec": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "resources": {
          "requests": {
            "storage": "1Gi"
          }
        },
        "storageClassName": "standard",
        "volumeMode": "Filesystem"
      },
      "status": {
        "phase": "Pending"
      }
    }
  ]
}

Analysis
It looks like the JSON parser in the storage provisioner doesn't like the escape sequence in the managedFields.f:ownerReferences section.

[wcjohnson]

I did some searching and discovered the following comment on another issue:

#7828 (comment)

minikube config set kubernetes-version v1.16.0 as suggested by the commenter did resolve this issue for me. This may therefore be a duplicate, though I am not certain the cause is the same in both cases.

[afbjorklund]

Unfortunately I think this is the same as in #7218, the updated storage-provisioner didn't make it in v1.11.0 (which is still running the previous v1.8.1, from two years ago).

[hadim]

Got the same issue and switching to 1.16.0 seems to work.

[tstromberg]

Some recent work has gone into the storage-provisioner, so it's feasible that we may be able to fix this in a near-term release.

[wcjohnson]

For those looking for a temporary fix for this and some related issues (particularly the persistent storage being lost due to being in tmpfs), I have forked the newest storage provisioner (which does work with Kubernetes 1.18) and made a small change allowing the hostpath location to be changed.

Source code is at https://github.com/wcjohnson/minikube/tree/storage-provisioner-pvdir-arg

Kubernetes resources for replacing the storage provisioner: https://raw.githubusercontent.com/wcjohnson/minikube/storage-provisioner-pvdir-arg/hack/storage_provisioner_fixes/storage-provisioner.yaml

To deploy:

# STEP 1: get updated storage provisioner yaml
wget https://raw.githubusercontent.com/wcjohnson/minikube/storage-provisioner-pvdir-arg/hack/storage_provisioner_fixes/storage-provisioner.yaml

# STEP 2: start a minikube
minikube config set driver docker
minikube start

# STEP 3: disable bugged provisioner
minikube addons disable storage-provisioner

# STEP 4: create persistent storage path inside the docker container
# (can be skipped if you aren't bitten by the persistent data loss bug)
minikube ssh

sudo mkdir /persistent_volumes
sudo chmod 777 /persistent_volumes
exit

# STEP 5: apply the fixed storage provisioner
# (remove the -pvdir option from the command line if you didn't do step 4)
kubectl apply -f ./storage-provisioner.yaml

For me this fixed all my storage-related issues. Looking forward to when it lands in the mainline.

[medyagh]

@wcjohnson we updated storage provsioner in minikube v1.12.2 do you mind trying with that see if it was fixed?

if not I would be happy to accept a PR that fixes this

[wcjohnson]

@medyagh I can confirm the new storage provisioner resolves this problem.

[medyagh]

great to hear that ! thank you for confirming

[roncemer]

great to hear that ! thank you for confirming

@medyagh How is it "completed" when the problem persists more than two years later? Has the updated storage provisioner been merged into the main branch, or not? I'm running Minikube 1.27.0 on an Intel Mac, and this bug is still there. Additionally, Minikube 1.27.0 refuses to run Kubernetes 1.16.0, so that "fix" is a non-starter. This is rendering Minikube pretty much worthless for developing and testing Kubernetes deployments on the desktop. It's sad, because two or three years ago, when I was developing my last k8s application deployment, Minikube worked great for exactly this task. If Minikube is no longer actively maintained, I guess we're stuck with Microk8s or K3s.

Attached is a YAML file which most definitely hangs when trying to create a persistent volume claim on a persistent volume which exposes a directory on the host machine. This bug definitely persists, and should be prioritized and fixed.

Also, the workaround posted by @wcjohnson doesn't work on Minikube 1.27.0.
spark-master.yaml.txt

[yngfoxx]

I had issue with PVC so I ran kubectl describe pvc and discovered it was a problem with the storageClassName I used
so I updated the storageClassName in my statefulset yaml to use the default available in minikube. To see the default run

kubectl get storageclass

Then I replaced accordingly

Now it works for me