I am trying to work with bidirectional GRU. As I am going to work in batches, I will be padding my sequence with 0s in case of sequences with unequal lengths. For that, I need to use
use_sequence_lengths argument in GRU definition. But when I turn this boolean (use_sequence_lengths) to true, defer initialization functionality fails to work for some reason. Below is code to replicate:
from mxnet import gluon import mxnet as mx ctx = [mx.gpu(0)] class TestModel(gluon.nn.HybridBlock): def __init__(self, bidirectional=True): super(TestModel, self).__init__(prefix="TestModel_") with self.name_scope(): self.embed = gluon.nn.Embedding(input_dim=50, output_dim=5) self.rnn = gluon.rnn.GRU(hidden_size=20, bidirectional=bidirectional, use_sequence_length=True) self.dense = gluon.nn.Dense(1) def hybrid_forward(self, F, x, x_len): embed = self.embed(x).transpose((1, 0, 2)) # to make in max_sequence_length, batch_size, other_feature_dims rnn_all = self.rnn(embed, sequence_length=x_len) out = F.SequenceLast(rnn_all, sequence_length=x_len, use_sequence_length=True) out = self.dense(out) return out example_codes = [[1,2,3,4,5], [1,2,3,0,0]] example_len = [5, 3] x_input = mx.nd.array(example_codes).as_in_context(ctx) x_len_input = mx.nd.array(example_len).as_in_context(ctx) mx.random.seed(0) net = TestModel(bidirectional=True) net.initialize(mx.init.Xavier(), ctx=ctx, force_reinit=True) net(x_input, x_len_input)
DeferredInitializationError error. It works in either of the following edits in
self.rnn… line :
# 1. Turn off use_sequence_length self.rnn = gluon.rnn.GRU(hidden_size=20, bidirectional=bidirectional, use_sequence_length=False) # 2. Provide input_size self.rnn = gluon.rnn.GRU(hidden_size=20, bidirectional=bidirectional, input_size=5, use_sequence_length=True)
The above 2 options make the code to work but do not solve my case because 1) I need to use sequence_lengths argument to make sure it doesn’t use anything from padded values, 2) input_size for all batches can vary based on sequence length and I don’t want to pre-fix it.
Can anyone help in debugging this?
There are 2 exceptions I get when I run above code chunk:
DeferredInitializationError: Parameter 'TestModel_gru0_l0_i2h_weight' has not been initialized yet because initialization was deferred. Actual initialization happens during the first forward pass. Please pass one batch of data through the network before accessing Parameters. You can also avoid deferred initialization by specifying in_units, num_features, etc., for network layers. During handling of the above exception, another exception occurred:
> MXNetError: [07:02:38] src/core/symbolic.cc:91: Symbol.ComposeKeyword argument name state_cell not found. > Candidate arguments: > data > parameters > state > sequence_length