Memory leak caused by frequent GET requests to admin API in Kong API Gateway 3.0

[surenraju-careem]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.0

Environment

Kubernetes

Mode

DBless/Declarative

Current Behavior

We identified a memory leak issue in Kong API Gateway 3.0 running on Kubernetes platform with declarative configuration. We use the GET /config admin API every 10 seconds to fetch routing configuration and check for any changes in the configuration from our Git repo. If there is any difference in the configuration, We use POST /config admin API to update the declarative configuration in Kong.

We have identified that accessing GET /config admin API every 10 seconds causes the Nginx worker process memory usage to grow to maximum allocated capacity, leading to OOM_killed by the kernel. Upon reducing the interval between GET requests, the memory usage of workers kept growing indefinitely, making the issue worse.

Further research on Kong documentation led us to the configuration check_hash, which compares the hash of the input config data against that of the previous one. If the configuration is identical, Kong does not reload it and returns HTTP 304. Enabling the check_hash parameter to 1 using the POST /config admin API to update the configuration resolved the issue.

However, there should not be constant increase in memory usage when GET /config admin API invoked every 10 seconds.

Expected Behavior

There should not be constant increase in memory usage when GET /config admin API invoked every 10 seconds.

Steps To Reproduce

Just run the attached kubernetes manifest and monitor memory usage for several hours. There is no need to generate requests though kong.

deployment.yaml

Memory Usage Metrics

[ADD-SP]

Hello @surenraju-careem , could you please try to reproduce this issue for Kong Gateway v3.2.2 ?

[surenraju-careem]

Seems the issue is not reproducible in 3.2.2.

[ADD-SP]

Seems the issue is not reproducible in 3.2.2.

If the latest version does not have this issue, I recommend upgrading to the latest version.

[surenraju-careem]

Closing the ticket as the issue is not reproducible in kong 3.2.2.