Why the GPU usage is so weird at the starting moment?

When I start the program, it is of a very high cost and fluctuation as you can see from the attached screenshot. However, it finally become stable at around 3G. How can I avoid this? Because my network structure is not complicated, sometimes it will be out of GPU memory despite the fact that it actually only cost 3G. How to avoid the fluctuation at the starting moment? Thanks.

This is likely happening because of the auto-tuning, which MXnet does at start-up . You could try setting MXNET_CUDNN_AUTOTUNE_DEFAULT=0
You can find more information about this environment variable here: https://mxnet.incubator.apache.org/faq/env_var.html

Thanks for your reply. Is it possible to set MXNET_CUDNN_AUTOTUNE_DEFAULT under windows os, since I am using mxnet on windows 10?

You could either set it when executing your python script e.g. MXNET_CUDNN_AUTOTUNE_DEFAULT=0 python ... But not sure this works under Windows. Or you overwrite the environment variable from within your python script:

import os