Also, for debugging purpose, you can run MXNet with the env var MXNET_ENGINE_TYPE=NaiveEngine which will MXNet use a synchronous engine instead of the default ThreadedEnginePerDevice which gives you the asynchronicity.
More info about it here https://mxnet.incubator.apache.org/versions/master/faq/env_var.html#engine-type 