Including a constant matrix multiply in a HybridSequential

1

I have a model based on HybridSequential that I need also to include a fixed matrix mutiply. It looks like something like this:

class MyModel_MX(gluon.nn.HybridSequential):

    def __init__(...):
        super(DeformNetModel_MX, self).__init__()

        with self.name_scope():
            self.inputlayer = nn.Dense(units)
            self.batchnorm0 = nn.BatchNorm()
            self.dropout0 = nn.Dropout(dropout)
            
            # add some more layers...

            self.outputlayer = nn.Dense(outputdim)

            # there is a parameter called abc which has the matrix under its 'components_' attribute
            self.thematrix = gluon.Constant('abc_components', mx.nd.array(abc.components_, dtype=np.float32))


    def hybrid_forward(self, F, input_tensor, abc_components):
        x = self.inputlayer(input_tensor)
        x = self.batchnorm0(x)
        x = self.dropout0(x)

        # process the other layers in the same way
 
        x = self.outputlayer(x)

        # neither of these work:
        # x = F.linalg.gemm2(x, abc_components, transpose_b=True)
        x = F.linalg.gemm2(x, self.thematrix, transpose_b=True)

        return x

I’ve tried a LOT of variations of how to specify the matrix in the init (everything from don’t specify it at all, to making it a constant as above, to making it a parameter, to initializing the data directly or in the hybrid_forward, and so on). But it fails in the forward either as:

type <class ‘mxnet.gluon.parameter.Parameter’> not supported

Or in a variety of other ways.

Does anyone have a working example of, say, a 1-layer MLP with a matrix multiply by a const matrix at the end? Huge thanks if so!!

2

Hi, this will get you started:

import mxnet as mx
from mxnet import gluon, nd
from mxnet.gluon import HybridBlock
import mxnet.gluon.nn as nn

class MyModel_MX(HybridBlock):
    def __init__(self,nunits,dropout,outputdim=3):
        super().__init__()

        with self.name_scope():
            self.inputlayer = nn.Dense(nunits)
            self.batchnorm0 = nn.BatchNorm()
            self.dropout0 = nn.Dropout(dropout)
            
            # add some more layers...

            self.outputlayer = nn.Dense(outputdim)

            # there is a parameter called abc which has the matrix under its 'components_' attribute
            self.thematrix = self.params.get_constant('thematrix', np.random.rand(10,outputdim))


    def hybrid_forward(self, F, input, thematrix):
        x = self.inputlayer(input)
        x = self.batchnorm0(x)
        x = self.dropout0(x)

 
        x = self.outputlayer(x)
        x = F.linalg.gemm2(x, thematrix, transpose_b=True)

        return x

Usage:

ctx = mx.gpu() # or mx.cpu()
net = MyModel_MX(10,0.5)
net.initialize(ctx=ctx)
xx = nd.random.uniform(shape=[10,25],ctx=ctx)
out = net(xx)
2 Likes
3

Thanks, that was what I needed. To clarify (and for anyone else, and for when I come back here in the future wondering how this worked):

  • set the local constants using params.get_constant(), which both finds or creates the necessary structure to save the data
  • importantly!: in your hybrid_forward, provide arguments with the names of your constants and refer to them in the procedure WITHOUT the ‘self’ scope
  • also important: your call to forward() on this class should NOT include arguments for these constants that appear as parameters. Instead, HybridBlock’s forward() will automatically fill in these parameters in the call to your class’s hybrid_forward, and of the correct type according to ‘F’.