Initializing embeddings with a given array

1

Hey!
Trying to build a recommender system using embeddings computed with MXNet. I’m trying to try initializing the embeddings with SVD. So I compute the SVD and store the factors as numpy arrays. When I try to initialize with those I keep on getting this error:

MXNetError: [18:04:22] src/operator/././operator_common.h:477: Check failed: arr.storage_shape()[0] == arr.shape()[0] SGDUpdate for RowSparse weights is only implemented for RowSparse weights with all rows containing non-zeros. Expects weights.data.shape[0] (748184) == weights.shape[0] (1000000).

I am sure the number of rows and columns in the sparse input data is equal to the number of rows in the U and V factors of the SVD used for initializing. I also tried to add a first row with N,M,1 to the input csv file with row,col,data , but keep on getting the error message above when initializing.
On a synthetic dataset I built I managed getting around the error by making sure each minibatch contains data from the last row and columns. The code seems to raise an error when the data fed to MXNet doesn’t meet the last row and columns I think. Can I relax this hard constraint?

Thanks,

Emile

2

It might be worth to note that there was a warning when training started.

home/ec2-user/mxnet/incubator-mxnet/python/mxnet/ndarray/sparse.py:696: RuntimeWarning: Assigning non-NDArray object to RowSparseNDArray is not efficient
  RuntimeWarning)
3

Hey @Emile @kaiser,
Could you provide a sample code and sample dataset that would reproduce this error so I can attempt to help you?
Thanks!
Thomas

4

Hi Emile,

Can you show how you initialized the weight? Does the weight only have 1 column? I suspect that during the initialization some rows of complete zeros are stripped out when converted to row_sparse format and the optimizer treated this as an anomaly.

5

Are you assigning a numpy object to the weight NDArray? You can probably try this:

instead of

row_sparse_weight[:] = svd_weight_np

try

shape = svd_weight_np.shape
row_sparse_weight[:] = mx.nd.sparse.row_sparse_array((mx.nd.array(svd_weight_np), mx.nd.arange(0, shape[0], dtype='int64')))
1 Like
6

Many thanks! @eric-haibin-lin 's fix worked. @ThomasDelteil is still you want a code example to see the issue for reference I can prepare something for you. Let me know