KongCredentials are not associated immediately or never with KongConsumers

[angelshila]

NOTE: GitHub issues are reserved for bug reports only.
For anything else, please join the conversation
in Kong Nation https://discuss.konghq.com/c/kubernetes.

---

Summary

When a KongConsumer is created along with an associated KongCredential, it can take anywhere between 0 to infinite time in sec for the KongCredential records to get stored in Cassandra. The KongConsumer info is immediately stored whereas the credential info shows an unacceptable amount of latency almost every time.

As a result, Authentication fails as Kong can't find a valid consumer-credential pair.

Note:
Latency is mostly seen from creation of the second consumer and credential onwards or if CRDs are created back to back one after the other.

Kong Ingress controller version
0.3.0

Kong or Kong Enterprise version
1.0.3

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:59:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:50:45Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment

• Cloud provider or hardware configuration: Vmware Vsphere/K8s
• OS (e.g. from /etc/os-release): CentOS 7
• Kernel (e.g. uname -a): Linux ****-master-01 3.10.0-514.el7.x86_64 Status of controller? #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: YAML/Jenkins
• Others:

What happened

Unacceptable latency occurred whenever a new consumer and credential crd objects were created and authentication kept failing.

I created the following CRDs :

apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: consumer-pqr
username: pqr
---

apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: credential-pqr
consumerRef: consumer-pqr
type: basic-auth
config:
  username: pqr
  password: pqr

I then checked cassandra records and there is no credential for consumer with username 'pqr'.

API call result:

Expected behavior

KongConsumers and KongCredentials should get created at the same time and authentication plugins should be effective immediately.

Steps To Reproduce

1. Create a KongConsumer and KongCredential and apply/create deployment.
2. Make sure both the CRDs are created without any error.
3. Check cassandra records or try authenticating
4. Authentication Failed due to "Invalid authentication credentials"

[HsinHeng]

I found that if we recreated consumer, it will sync the credentials.

Hope developer team to fix it.
Thx.

[angelshila]

I found that if we recreated consumer, it will sync the credentials.

Hope developer team to fix it.
Thx.

@HsinHeng
Hey. So, I tried recreating the consumer. I simply deleted the previous consumer and created another with the same config and name using kubectl create. It doesn't sync the credentials. I also tried kubectl apply without deleting the KongConsumer crd. However, that doesn't help either.

What steps did you follow to make the sync work?

[HsinHeng]

I found that if we recreated consumer, it will sync the credentials.
Hope developer team to fix it.
Thx.

@HsinHeng
Hey. So, I tried recreating the consumer. I simply deleted the previous consumer and created another with the same config and name using kubectl create. It doesn't sync the credentials. I also tried kubectl apply without deleting the KongConsumer crd. However, that doesn't help either.

What steps did you follow to make the sync work?

versions:

1. kong-ingress-controller:0.3.0 for "kong-ingress-controller" Deployment.
2. kong:1.0.3-alpine for "kong-proxy" Deployment.

Steps for me:
kubectl delete -f kong-consumer.yaml --wait=true --ignore-not-found=true
// use option --wait=true to perform sync operation to completed deletion.
kubectl apply -f kong-consumer.yaml

please check you ingress-controller has any error message or not.

Hope it could help you.

[srleyva]

Also experiencing this issue. I find the credentials do eventually populate but it takes a couple of hours. Is this expected behavior?

[hbagdi]

Also experiencing this issue. I find the credentials do eventually populate but it takes a couple of hours. Is this expected behavior?

@srleyva Can you share your steps to reproduce?
I've not been able to reproduce this locally yet.

[srleyva]

Thanks for the response @hbagdi I'll give some context as well. I am working with an existing set of users that I am attempting to import into kong via a utility script I've written.

1. My script generates k8s resource definitions for each consumer and credentials associated with each. (70 for our current environment)

   # Example of one consumer but 70 are created
   apiVersion: configuration.konghq.com/v1
   kind: KongConsumer
   metadata:
     name: johndoe
     namespace: kong
   username: johndoe
   custom_id:  some-custom-id 

   # Example of one credential but 70 are created
   apiVersion: configuration.konghq.com/v1
   kind: KongCredential
   metadata:
    name: johndoe-jwt
    namespace: kong
   consumerRef: johndoe
   type: jwt
   config:
    secret: some-base64-encoded-key
    key: johndoeiss

2. kubectl apply -f consumers/ at this point all the consumers exist in Kong

3. kubectl apply -f credentials/ k8s successfully creates all 70 credential resources and they exist in k8s. However, only the first credential exists in kong with all subsequent credentials being created over next few hours.

To reproduce locally I would try creating bulk consumers and credentials. I would use the Kong API directly but waiting on #246

[julianguinard]

I reproduced this issue above with a significant number of kong consumers (2750 consumers), using kong-ingress-controller v0.3.0

When I create the 2751th consumer, it is added to Kong's postgresql database by kong-ingress-controller pod but its jwt and basic auth credentials are never added, although I can list the credentials as custom resources from /apis/configuration.konghq.com/v1/namespaces/APP-NAMESPACE/kongcredentials API route.

No error message is given either.

The only way to have the 2751 consumers along with the matching 2751 JWTs and 2751 basic auths is to delete everything from Kong's consumers, jwt_secrets and basicauth_credential tables and wait for kong-ingress-controller to sync custom resources again

i was wondering if this had something to to with kong consumers synchronisation not being over before its next round starts?

[hbagdi]

An update here:
The way consumers and credentials are synced with 0.5.0 have significantly changed in 0.5.0.

I ran the tests ranging up to 4000 consumers and was not successful in reproducing this problem.

Please test it with 0.5.0 (expect at least a release candidate to be out next week) and let us know if you run into this yet again. I'll leave the issue open for now.

[hbagdi]

Closing this issue.
Please re-open if the issue persists for Kong Ingress Controller >=0.5.0.

[lionel1993]

An update here:
The way consumers and credentials are synced with 0.5.0 have significantly changed in 0.5.0.

I ran the tests ranging up to 4000 consumers and was not successful in reproducing this problem.

Please test it with 0.5.0 (expect at least a release candidate to be out next week) and let us know if you run into this yet again. I'll leave the issue open for now.

However，I reproduce this problem with 0.5.0，

postgresql database ：

k8s kongcredentials：

k8s Kong Ingress Controller：

Is there a problem with my configuration?

echo "
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: user2
username: user2
" | kubectl apply -n ai-test -f - 
  
echo "
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: user2-key1
consumerRef: user2
type: key-auth
config:
  key: user2-secret1
  " | kubectl apply -n ai-test -f -  

echo "
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: user2-key2
consumerRef: user2
type: key-auth
config:
  key: user2-secret2
  " | kubectl apply -n ai-test -f -  

And api call failed :

Ask for help , thx

[lionel1993]

An update here:
The way consumers and credentials are synced with 0.5.0 have significantly changed in 0.5.0.
I ran the tests ranging up to 4000 consumers and was not successful in reproducing this problem.
Please test it with 0.5.0 (expect at least a release candidate to be out next week) and let us know if you run into this yet again. I'll leave the issue open for now.

However，I reproduce this problem with 0.5.0，

postgresql database ：

k8s kongcredentials：

k8s Kong Ingress Controller：

Is there a problem with my configuration?

echo "
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: user2
username: user2
" | kubectl apply -n ai-test -f - 
  
echo "
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: user2-key1
consumerRef: user2
type: key-auth
config:
  key: user2-secret1
  " | kubectl apply -n ai-test -f -  

echo "
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: user2-key2
consumerRef: user2
type: key-auth
config:
  key: user2-secret2
  " | kubectl apply -n ai-test -f -  

And api call failed :

Ask for help , thx

While ,After about seven minutes , postgresql has credential record,

[shugydw]

I am facing the same issue with 0.5.0 and Kong 1.3.0 with key-auth and Postgres.

It's been over 10 minutes, but the credential didn't sync.

However, if I first delete the keys (all keys for now) from Kong directly and then deleting (in K8s) one of the Credentials that synced, then the missing credential appears in Kong. Unfortunately, the deleted Credential (in K8s) also appears in Kong. (I know the Update and Delete are not yet implemented).

I hope I am clear. Please ask if it's confusing.

Thanks for your help! @hbagdi

[hbagdi]

@shugydw With #371 merged in, the syncing of credentials is completely revamped and that should address this issue. Also, see #230.

[dcherniv]

@hbagdi i'm testing the new 0.6.0 with 1.3.0 kong and somehow the issue is now much worse. I'm not sure how i arrived at this problem but it seems that kong no longer picks up any consumers that are created either via kubectl or helm.
Thinking i somehow botched the upgrade from older kong-ingress i wiped everything. pulled the latest helm chart and reinstalled everything from scratch. When i try to create a consumer with the following:

apiVersion: v1
items:
- apiVersion: configuration.konghq.com/v1
  kind: KongConsumer
  metadata:
    name: dev-ml-trainer-user
    namespace: dev
  username: dev-ml-trainer-user
kind: List

Nothing happens at all. Kong ingress logs don't show anything at all. No error message of any kind.
**EDIT:
It worked initially after kong upgrade, but now i can't seem to get to get unstuck. One thing that i saw was at some point i created two consumers with the same api key but even when i fixed that no consumers can be created anymore at all.

[hbagdi]

@dcherniv Are you using the default ingress.class or a custom one?

[dcherniv]

@hbagdi kong-internal is the class. These are startup params:

      - args:
        - /kong-ingress-controller
        - --publish-service=kong-internal/kong-internal-kong-ingress-kong-proxy
        - --ingress-class=kong-internal
        - --election-id=kong-ingress-controller-leader-kong-internal
        - --kong-url=http://localhost:8444

[hbagdi]

Okay, then you need to specify kubernetes.io/ingress.class: "kong-internal" annotation on the KongConsumer (and every other custom) resources.

[dcherniv]

@hbagdi oh "duh" moment. Is there documentation that needs updating? i couldn't find anything relating to annotations. I can submit a PR, just point me where you want it.

[hbagdi]

This is documented here: https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/references/annotations.md#kubernetesioingressclass

PR to make it clearer welcome!

[ismailbaskin]

@hbagdi What do you think about namespace level annotation? We need to change two many stuff's annotation when using different staging namespaces.

[hbagdi]

@hbagdi What do you think about namespace level annotation? We need to change two many stuff's annotation when using different staging namespaces.

Not totally against the idea. Another idea would be to watch a bunch of namespace. PRs/Issues to discuss this at length are welcome.
Let's not further discuss this on this closed issue.