I’ve trained a simple net with batch normalization before activations but when I try to predict with context = gpu() I get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/ndarray/ndarray.py", line 1972, in asnumpy
ctypes.c_size_t(data.size)))
File "/home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/base.py", line 252, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [15:34:32] src/operator/nn/./cudnn/cudnn_batch_norm-inl.h:157: Check failed: e == CUDNN_STATUS_SUCCESS (9 vs. 0) cuDNN: CUDNN_STATUS_NOT_SUPPORTED
Stack trace returned 10 entries:
[bt] (0) /home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x36161a) [0x7fcc1f92461a]
[bt] (1) /home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x361c31) [0x7fcc1f924c31]
[bt] (2) /home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x335345f) [0x7fcc2291645f]
[bt] (3) /home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x33571b7) [0x7fcc2291a1b7]
[bt] (4) /home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2a6fe6f) [0x7fcc22032e6f]
[bt] (5) /home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2a76aec) [0x7fcc22039aec]
[bt] (6) /home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2a55354) [0x7fcc22018354]
[bt] (7) /home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2a59623) [0x7fcc2201c623]
[bt] (8) /home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2a59876) [0x7fcc2201c876]
[bt] (9) /home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2a55a64) [0x7fcc22018a64]
I’m calling the model with:
dataiter = mx.io.NDArrayIter(data=data, label=label, batch_size=data.shape[0])
model = mx.module.Module.load(prefix=modelPrefix, epoch=epoch, context = mx.gpu())
model.bind(data_shapes=dataiter.provide_data,label_shapes=dataiter.provide_label)
preds = model.predict(dataiter)