Is there any example of changing a parameter during testing (and not training)? Im specifically thinking of AlphaDropout - the noise should be added during the training but not testing… Just wanted to see if there was an example
@piiswrong Any ideas here or examples I can look at?
@eric-haibin-lin We’re still not sure how to do this
Could you be more specific regarding what parameter what want to change? Did you add a custom operator for the activation layer?
No, I did (in gluon)
if mx.autograd.is_training(): noise = F.random.uniform() else: noise = 1 output = output * noise
I’d like to do this only during training - and not testing… How can I do that in mxnet symbol? In gluon I just use
mx.autograd.is_training() to check whether I should forward through an operation
@eric-haibin-lin is that clearer?
Couldn’t really see this in the documentation either @eric-haibin-lin
It’s a bit tricky to express that using Symbol. Is there any reason why Gluon is not used?
Legacy codebase - is there anything I can look at to mimic the behavior?
You can create a new module and bind with shared_module arg.
For example, let’s say the parameters/variables in
net2 are all present in
def net1(...): weight = mx.sym.var('w') score = mx.sym.FullyConnected(data, weight) out = mx.sym.SoftmaxOutput(score, label) def net2(...): weight = mx.sym.var('w') score = mx.sym.FullyConnected(data, weight) noise = mx.sym.random_uniform(...) new_score = noise + score out = mx.sym.SoftmaxOutput(new_score, label)
Then net2 can share the parameter in net1 with the following code:
module1 = mx.module.Module(net1, ...) module2 = mx.module.Module(net2, ...) module2.bind(ctx, ..., shared_module=module1)
How would that turn off the noise at test time?
Would this work for you:
train_test_gate = mx.sym.Variable('train_test_gate') # an array with (2,) shape output = output * mx.sym.random.uniform() * train_test_gate + output * train_test_gate
Then you’d want to include
train_test_gate in your inputs in your DataIter and set it to
mx.nd.array([1, 0]) for train dataset and
mx.nd.array([0, 1]) for test dataset.
You can also do this:
is_training = mx.sym.Variable('is_training') noise = mx.sym.random.uniform() output = output * mx.sym.where(is_training, noise, mx.sym.ones_like(noise))
is_training to 1 (or 0) in your training (or testing) dataset.
I should mention that topic of control flow blocks (e.g. if condition, for-loop, while-loop) in MXNet have been discussed in multiple threads before (like here and here) and the current recommendation is to use imperative if possible.