Batch_dot one hot vectors with embeddings results in nan

Gluon
1

Hi there,

I have a one hot tensor like this:

31%20PM
hankcs.com 2019-04-13 at 11.30.31 PM.png1732×1606 354 KB

First dim is batch size, second dim is char, third is one hot vector. One batch looks like this:
15%20PM
hankcs.com 2019-04-13 at 11.34.15 PM.png932×1350 54.1 KB

And an embedding tensor like this:

17%20PM
hankcs.com 2019-04-13 at 11.31.17 PM.png1732×1606 365 KB

Each element is in a certain range:
42%20PM
hankcs.com 2019-04-13 at 11.34.42 PM.png938×1344 152 KB

56%20PM
hankcs.com 2019-04-13 at 11.39.56 PM.png1732×1606 260 KB

Then I use char_embed = nd.batch_dot(one_hot, bert_embed) to lookup my embedding, the result contains nan which confuses me for a whole day.

11%20PM
hankcs.com 2019-04-13 at 11.36.11 PM.png844×372 45.7 KB

29%20PM
hankcs.com 2019-04-13 at 11.36.29 PM.png918×1392 101 KB

If there is no nan in bert_embed, why is nan produced when it is batch_dot with a one hot tensor?

2

The bug only occurs on CPU:

src/operator/nn/./fully_connected-inl.h:200: float16 fully connected layer is currentlyonly supported by CuDNN version.

Well, it works fine on GPU. So, not a big deal.