Add some input-based parameters in BatchNorm

1

Hi ,
I want to add epsilon in BatchNorm which is not a constant instead exp(-mul(x,x)).
where x is the weights should be batchnormed.
But seems that nn.BatchNorm only accept a constant as epsilon
I tyr to :

1:
class Guassian_Batchnorm(BatchNorm):
    def __init__(self):
        super(Guassian_Batchnorm, self).__init__()
    def hybrid_forward(self, F, x, gamma, beta, running_mean, running_var):
        eps = F.exp(-F.elemwise_mul(x, x))
        return F.BatchNorm(x, gamma, beta, running_mean, running_var, eps,
                           name='fwd', **self._kwargs)

2:

class Guassian_Batchnorm(BatchNorm):
    def __init__(self):
        super(Guassian_Batchnorm, self).__init__()

module.add(nn.BatchNorm(out_channels,epsilon=Gaussian()))

Both of them get errors:
ValueError: Deferred initialization failed because shape cannot be inferred. BatchNorm() got multiple values for argument ‘eps’ for 1

ValueError: Deferred initialization failed because shape cannot be inferred. Invalid Parameter format for eps expect double but value=‘Gaussian()’, in operator BatchNorm(name="", axis=“64”, fix_gamma=“False”, momentum=“0.9”, use_global_stats=“False”, eps=“Gaussian()”) for 2

How to do that?
thank you for your time and considerations.
2

eps in BatchNorm has to be a constant float value. What you want requires eps to be an NDArray, which is not possible in current implementation.

You can, however, implement your own custom BatchNorm layer (gluon Blocks are easy to implement) and use your gaussian eps instead. Let me know if you need help with that.

3

In 2,I have try to do what you said above but get erros. Please let me know your solution. Thank you.:grin:

4 
class Guassian_Batchnorm(BatchNorm):

    def __init__(self):

        super(Guassian_Batchnorm, self).__init__()

    def hybrid_forward(self, F, x, gamma, beta, running_mean, running_var):

        eps = F.exp(-F.elemwise_mul(x, x))

        return F.BatchNorm(x, gamma, beta, running_mean, running_var, eps,

                           name='fwd', **self._kwargs)

and here is my code
ValueError: Deferred initialization failed because shape cannot be inferred. BatchNorm() got multiple values for argument ‘eps’
I have try to do what you said above but get erros. Please let me know your solution. Thank you.:grin: