mxnet.base.MXNetError: [22:46:02] src/operator/nn Check failed: e == CUDNN_STATUS_SUCCESS (8 vs. 0) cuDNN: CUDNN_STATUS_EXECUTION_FAILED

Discussion
1

Hi
I was running the mxnet example with ’ io@ubuntu:~/mxnet/example/image-classification$ python train_mnist.py --gpus 0,1 ’ , and there are errors ‘mxnet.base.MXNetError: [11:10:04] src/operator/nn/./cudnn/cudnn_activation-inl.h:205: Check failed: e == CUDNN_STATUS_SUCCESS (8 vs. 0) cuDNN: CUDNN_STATUS_EXECUTION_FAILED’

My software configuration is cuda9.0, cudnn7, mxnet1.3.0, python2.7, ubuntu 16.04.4.
Two 2080ti are used.

I have no idea about that. What is wrong? What Happened?

When I did not use CUDNN, it is runing.

But I want to use CUDNN, so, how can I solve this problems?

2

Sorry due to the limit, I have to post again.
The error is
Traceback (most recent call last):
File “train_mnist.py”, line 98, in
fit.fit(args, sym, get_mnist_iter)
File “/home/io/lilu/mxnet/example/image-classification/common/fit.py”, line 333, in fit
monitor=monitor)
File “/home/io/lilu/mxnet/python/mxnet/module/base_module.py”, line 539, in fit
self.update_metric(eval_metric, data_batch.label)
File “/home/io/lilu/mxnet/python/mxnet/module/module.py”, line 773, in update_metric
self.exec_group.update_metric(eval_metric, labels, pre_sliced)
File “/home/io/lilu/mxnet/python/mxnet/module/executor_group.py”, line 639, in update_metric
eval_metric.update_dict(labels, preds)
File “/home/io/lilu/mxnet/python/mxnet/metric.py”, line 304, in update_dict
metric.update_dict(labels, preds)
File “/home/io/lilu/mxnet/python/mxnet/metric.py”, line 132, in update_dict
self.update(label, pred)
File “/home/io/lilu/mxnet/python/mxnet/metric.py”, line 418, in update
pred_label = pred_label.asnumpy().astype(‘int32’)
File “/home/io/lilu/mxnet/python/mxnet/ndarray/ndarray.py”, line 1980, in asnumpy
ctypes.c_size_t(data.size)))
File “/home/io/lilu/mxnet/python/mxnet/base.py”, line 252, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [10:49:12] src/operator/nn/./cudnn/cudnn_activation-inl.h:129: Check failed: e == CUDNN_STATUS_SUCCESS (8 vs. 0) cuDNN: CUDNN_STATUS_EXECUTION_FAILED

3

Oh, It is me here again.
I have solved the problem.
It needs compatible with version.

4

@ LucinyaLi I meet the same issue now. How did you solve it? Thank you!!
we use two 2080ti too.

5

Is the compatible here specific what be point to?

6

@LucinyaLi Can you tell us your cuda cudnn version detail. I have encountered the same error with you!
Thank you!

7

the version of cuda and that of cudnn

8

cuda 9.2 and cudnn 7.3
have a try
It should be OK