To accelerate the training process, I did some adjustments to the source code of MXNet (src/kvstore/kvstore_dist.h/PullImpl()). The pull_first_time records how many Pull operations have been conducted, while the parameter_count keeps the number of parameters. I used the first 600 batches for warming up, and when I pull back the updated parameters every 3 times, the training result converges to the target accuracy. However, when I pull back the parameters every 4 times, the training accuracy decreases to ~0.1, even though the accuracy increases in the warming up phase. Does anyone know the reason for this phenomenon?
Note: As only one picture can be uploaded, so I upload three figures in one picture.)
I uploaded the wrong screenshot about the source code and see my next discussion topic to check the code.