How to apply F.softmax to Gluon Parameters

Gluon
1

I am trying to implement task-specific weighting of multiple embeddings as in Elmo.
Currently, I initialized weights for multiple embeddings using self.param.get. However, it throws me the error.
AssertionError: Argument data must be Symbol instances, but got Parameter elmoembedding0_weights (shape=(3,), dtype=<class 'numpy.float32'>).

I can call x.data() for non hybridized or x.var() for hybridized version for the parameters. Is there a way to simply apply Softmax to parameters and work with both versions? Thanks!

My code looks something like this

import mxnet as mx
import mxnet.gluon as gluon

class ElmoEmbedding(gluon.HybridBlock):
    def __init__(self):
        super(ElmoEmbedding, self).__init__()

        with self.name_scope():
            self.weights = self.params.get('weights',
                                           shape=(3,),
                                           init=mx.init.Constant(1.0))
            self.scales = self.params.get('scales',
                                      shape=(1,0),
                                      init=mx.init.Constant(1.0))

    def hybrid_forward(self, F, x, *args, **kwargs):
        normalized_weights = F.softmax(self.weights)
        weighted_x = F.dot(normalized_weights, x)
        output = F.broadcast_mul(self.scales, weighted_x)
        return output

net = ElmoEmbedding()
net.hybridize()

# create input
x = mx.ndarray.random.randn(3,100)
output = net(x)
print("output", output.shape)
2

hi,
initialize your network before calling net.hybridize() and see what happens

3

hybrid_forward passes the parameters as part of the function arguments, so you would access them by defining them or using kwargs. Then we can use them as usual for both hybridized and non-hybridized versions.

class ElmoEmbedding(gluon.HybridBlock):
    def __init__(self):
        ...
    def hybrid_forward(self, F, x, *args, **kwargs):
        weights = kwargs['weights']
        scales = kwargs['scales']
        normalized_weights = F.softmax(weights)
        ...