Symbolic Faster-RCNN meets CUDA illegal memory access error

Discussion
1

Hi,

I tried to convert the Faster-RCNN model from gluoncv model zoo into symbol API format. When I do inference with CPU, everything is well. However, when inferring with GPU, it raised Ignore CUDA Error

The full code as following:

import time

import mxnet as mx
from gluoncv import model_zoo
from tqdm import tqdm


ctx = mx.gpu()
model = model_zoo.get_model('faster_rcnn_resnet50_v1b_voc', pretrained=True)
model.hybridize(static_alloc=True, static_shape=True)
x = mx.nd.random.normal(shape=[1, 3, 300, 300])
_ = model(x)

model.export('temp', 0)
sym, args, aux = mx.model.load_checkpoint('temp', 0)

for k, v in args.items():
    args[k] = v.as_in_context(ctx)
args['data'] = x.as_in_context(ctx)

executor = sym.bind(ctx, args=args, aux_states=aux, grad_req='null')
start = time.time()
for i in tqdm(range(100)):
    executor.forward(is_train=False)
    mx.nd.waitall()
end = time.time()
print('Elapsed time:', end-start)

Here is the error message:

[02:19:17] src/resource.cc:279: Ignore CUDA Error [02:19:17] src/storage/./pooled_storage_manager.h:97: CUDA: an illegal memory access was encountered

Did anyone meet this error?