Ok, I didn’t understand what you wanted to do precisely, now I think it’s clear!
Honestly, I never tried what you’re doing here, but I guess you could:
create a new dataset with the new “difficult” images;
load the same custom model you used during the first training, and load the last .param file you saved during the training, something like:
classes = ["pen", "pencil"]
net = gcv.model_zoo.get_model('ssd_512_mobilenet1.0_custom', classes=classes, pretrained_base=False, ctx=ctx)
At that point, you can run the training again on the new dataset, so that ti doesn’t start from scratch, but it goes on from where you left it.
Again, I never really tried, so you can either try or wait for someone else’s opinion as well.