SSD Finetuning with Resnet50

I am trying to finetune a model using resnet50 512. I have around ~850 training images, and when I run, the validation/mAP starts low and quickly gets to around .25-.30 around 70 epochs. Then it seems to stay there indefinitely.

I am using the official apache incubator repo, but I had to make some code changes to get it to work.
Specifically I modified this block of code in to match this version in order to remove layers which seem to not match with the pretrained model.
I used this command to start the training

python --network resnet50 --train-path data/train.rec --val-path data/test.rec --class-names text_block --num-class 1 --data-shape 512 --lr 0.0001 --finetune 1 --end-epoch 1000 --gpus 0 --val-list test

I stopped the training at epoch 457. Here is the full log

sample output from my logs during training.

INFO:root:Epoch[216] Validation-text_block=0.277103
INFO:root:Epoch[216] Validation-mAP=0.277103
INFO:root:Epoch[217] Batch [20] Speed: 92.03 samples/sec CrossEntropy=0.573019 SmoothL1=0.505079
INFO:root:Epoch[217] Train-CrossEntropy=0.568360
INFO:root:Epoch[217] Train-SmoothL1=0.480534
INFO:root:Epoch[217] Time cost=8.733
INFO:root:Saved checkpoint to “/home/aschu/development/apache-incubator/mxnet/incubator-mxnet/example/ssd/model/ssd_resnet50_512-0218.params”
INFO:root:Epoch[217] Validation-text_block=0.266455
INFO:root:Epoch[217] Validation-mAP=0.266455
INFO:root:Epoch[218] Batch [20] Speed: 97.72 samples/sec CrossEntropy=0.570998 SmoothL1=0.501688
INFO:root:Epoch[218] Train-CrossEntropy=0.566719
INFO:root:Epoch[218] Train-SmoothL1=0.494521
INFO:root:Epoch[218] Time cost=7.926
INFO:root:Saved checkpoint to “/home/aschu/development/apache-incubator/mxnet/incubator-mxnet/example/ssd/model/ssd_resnet50_512-0219.params”
INFO:root:Epoch[218] Validation-text_block=0.237516
INFO:root:Epoch[218] Validation-mAP=0.237516
INFO:root:Epoch[219] Batch [20] Speed: 95.20 samples/sec CrossEntropy=0.570789 SmoothL1=0.487176
INFO:root:Epoch[219] Train-CrossEntropy=0.570646
INFO:root:Epoch[219] Train-SmoothL1=0.475513
INFO:root:Epoch[219] Time cost=8.406
INFO:root:Saved checkpoint to “/home/aschu/development/apache-incubator/mxnet/incubator-mxnet/example/ssd/model/ssd_resnet50_512-0220.params”
INFO:root:Epoch[219] Validation-text_block=0.259088
INFO:root:Epoch[219] Validation-mAP=0.259088

Hi Alex,

I never worked with this implementation of SSD, but most probably you have confused 2 mutually exclusive parameters.

In your code you are using finetune parameter, which is, according to help doc “finetune from epoch n, rename the model before doing this”. I guess what you really want is to use a pretrained network and then fine-tune it. To do that you need to use pretrained parameter, which loads a base network in pretrained mode . Take a look into this line to see what it does

So, I would recommend to change the parameter and see if it helped you (you probably want to do it without your code modifications, just to control the number of possible errors).

Hey, thanks for the reply. I have tried this and unfortunately it does not seem to work. I also found this github issue from a while back which explains the use of the pretrained flag vs finetune flag.

This is the error I get when I use that flag

python --network resnet50 --train-path data/train.rec --val-path data/test.rec --class-names text_block --num-class 1 --data-shape 512 --pretrained model/ssd_resnet50_512 --end-epoch 1000 --gpus 0 --val-list test
[22:28:18] src/io/ ImageDetRecordIOParser: data/train.rec, use 11 threads for decoding…
[22:28:18] src/io/ ImageDetRecordIOParser: data/train.rec, label padding width: 350
[22:28:19] src/io/ ImageDetRecordIOParser: data/test.rec, use 11 threads for decoding…
[22:28:19] src/io/ ImageDetRecordIOParser: data/test.rec, label padding width: 350
INFO:root:Start training with (gpu(0)) from pretrained model model/ssd_resnet50_512
[22:28:22] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while… (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
Traceback (most recent call last):
File “”, line 156, in
File “/home/aschu/development/apache-incubator/mxnet/incubator-mxnet/example/ssd/train/”, line 367, in train_net
File “/home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/module/”, line 464, in fit
allow_missing=allow_missing, force_init=force_init)
File “/home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/module/”, line 308, in init_params
_impl(desc, arr, arg_params)
File “/home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/module/”, line 296, in _impl
File “/home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/ndarray/”, line 1876, in copyto
return _internal._copyto(self, out=other)
File “”, line 25, in _copyto
File “/home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/_ctypes/”, line 92, in _imperative_invoke
File “/home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/”, line 146, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [22:28:29] src/operator/nn/./…/tensor/…/elemwise_op_common.h:123: Check failed: assign(&dattr, (*vec)[i]) Incompatible attr in node at 0-th output: expected [84,128,3,3], got [8,128,3,3]

Stack trace returned 10 entries:
[bt] (0) /home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/ [0x7f41963afe78]
[bt] (1) /home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/ [0x7f41963b0288]
[bt] (2) /home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/ [0x7f41966166d6]
[bt] (3) /home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/ [0x7f419661792b]
[bt] (4) /home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/ [0x7f41985c50ea]
[bt] (5) /home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/ [0x7f41985c7c19]
[bt] (6) /home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/ [0x7f4198509a7b]
[bt] (7) /home/aschu/development/apache-incubator/mxnet/lib/python3.5/site-packages/mxnet/ [0x7f4198509fe3]
[bt] (8) /home/aschu/development/apache-incubator/mxnet/lib/python3.5/lib-dynload/ [0x7f41d98b7e20]
[bt] (9) /home/aschu/development/apache-incubator/mxnet/lib/python3.5/lib-dynload/ [0x7f41d98b788b]

I received a similar error with the finetune flag before I modified