Why is the GPU utilization low?

[LemonTwoL]

Hi, thanks for releasing the nice work.
I running the code, but find that the GPU utilization is very low, is your GPU utilization also low?

Training speed is a little slow.

[ruizhaocv]

Hi Xiaomin,

I tested the code on A5000, and the GPU usage is nearly 100%. And the training speed is around 1.6s/it. Have you modified parameters other than the file path in the config file?

[LemonTwoL]

Let me check it, thanks.

[ruizhaocv]

Hi Xiaomin. Have you found the problem?

[LemonTwoL]

Hi Xiaomin. Have you found the problem?

hi, I trained on modelscope, and disable gradient_checkpointing, the GPU usage is always unstable.