Memory consumption constantly increasing when running hammerdbcli

[filiptanic98]

Hello all,

I noticed that memory consumption is constantly increasing when running HammerDB TPCC benchmarks through hammerdbcli and using .tcl scripts.
My infrastructure consists of containers created inside of Azure VMs. From those containers I am targeting my MS SQL Server instance.
To run hammerdb from containers I use .tcl scripts and hammerdbcli.
I noticed that sometimes when I run TPCC I was getting out of memory errors in containers.
After that I investigated and found out it was correlated with the benchmark duration.
I continued my investigation and found out that memory usage of tclsh86t process is constantly increasing when benchmark is running.
I also tried reproducing this from outside container on my local machine and behavior is same.
I tried with running HammerDB from GUI (which runs wish86t.exe process) and there is no memory usage increase.
Image1 - reproduced issue on my local machine:

Image2 - shows constant memory increase for the container running inside kubectl environment:

Did someone encounter similar issues with hammerdbcli?

[sm-shaw]

If you provide some more details on the test environment, then we can try and reproduce, details such as system memory, database version, driver version and also the test being run, the configuration and the number of virtual users etc the more details you can provide the better to reproduce.

From the outset when running a workload, you do initially expect the memory consumption to increase as each virtual user is a thread and when it is ramping up it loads a separate interpreter (this is how HammerDB runs completely in parallel) and also loads the ODBC driver, so each time it does this the memory will increase and the more virtual users the more memory it will use - you would expect this to level out after the ramp-up phase is complete, however.

Now, one thing you do mention that is of interest here is that you say that the effect is not seen in the GUI only the CLI. So the workload running under both, driver loaded etc is identical except for just one thing, Tclsh creates three additional threads per Virtual User when a thread does any i/o command (puts etc.) - one for stdin, stderr, stdout to the CMD console. Wish does not do this because there is no Windows console attached for GUI apps. So for Windows and CLI only you are going to see 3X the number of threads and therefore will use more memory (but not a huge amount) than the GUI.

For further investigation, you can use process explorer and Windows inspection toolkit:

https://learn.microsoft.com/en-gb/sysinternals/downloads/process-explorer
https://wits.magicsplat.com/

as per the images below, you can see 3X the threads and more virtual memory (about 100MB) with the CLI:

So if the amount of memory is small you may see higher consumption with the CLI, but in both cases it should level out after rampup and not keep increasing. It's not likely to be in the workload itself if it works for the GUI. As noted, what is running is identical in both cases - the CLI is a thin layer over the GUI code.

If it is thread count related, the good news is that this was investigated previously and we asked the Tcl core team. Even though it is working correctly by design with an increase in thread counts the design was not optimal, and therefore they advised us that this has now been redesigned in Tcl/Tk 9 https://www.tcl.tk/software/tcltk/9.0.html, so there is not the increase in threads on Windows any more, and we are currently working on porting HammerDB to Tcl/Tk 9.

If it does keep increasing the memory throughout the workload after all the virtual users have been loaded, this will need more investigation.

[filiptanic98]

Hey @sm-shaw , so this is configuration that I am using:

• MS SQL Server (I am targeting SQL Managed Instance server)
• Using ODBC 18
• Using SQL auth

I tried investigating a little bit more throughout the weekend, and what I can confirm - it does not matter what number of virtual users is.
When there is higher number of virtual users, initial load is larger, but whatever the number of virtual users is - memory consumption is always constantly increasing.

I was able to reproduce this locally on SQL VMs, just by using scripts that are in the official HammerDB 4.11 release (scripts located in C:\Program Files\HammerDB-4.11-Original\scripts\tcl\mssqls\tprocc directory).

Here are more images:
Script from official HammerDB release that I used to reproduce the issue:

And here you can see memory constantly increasing (this is on 50 virtual users):

This is for 5 virtual users:

I guess this large increase at the beginning is due to virtual users loading one by one, but after the virtual users are loaded, I can still see memory increase.

Also, I want to note that by using python it is similar situation, but in this case, memory is increasing for python process, and not for tclsh86t:

Python process memory usage:

tclsh86t process memory usage:

[sm-shaw]

Many thanks for the additional information. I will create an issue from this discussion and investigate.
It may take a little longer than usual as we are in the middle of porting to Tcl/Tk9 for the next major release. This for example involves porting all the extensions in C for both Windows and Linux and I want to test this issue against the new version.
As it doesn't occur with the GUI only CLI - I am suspecting that this is related to one of the commands in the CLI in the higher level script rather than anything that is running in a virtual user against the database - however I will check and update.

[sm-shaw]

Issue #765 created