I would like to discuss the best option for kvstore parameter that is passed to the fit method of module API.
For single machine, single GPU:
If we don’t have GPU memory constraints, is it always faster to use
device instead of
For single machine, multiple GPU:
Again if we don’t have GPU memory constraints, is
device always better? The documentation says “When using a large number of GPUs, e.g. >=4, we suggest using device for better performance.”
For multiple machines, multiple GPU:
For synchronous updates, which is better? dist_sync or dist_device_sync?