General purpose Attention Layer?

Hi, All,
Would like to get a general purpose attention layer implementation in MxNet if possible.
Any link/guide is appreciated.

I know sockeye has quite some models but it requires a whole package while I would prefer a very simple layer if possible.


There is a GluonNLP toolkit has been introduced recently. It is a separate package that works using MXNet. It has an AttentionCell as a part of it.

Here is the Github -

Please notice, that it works only with master version of mxnet, so you will have to update to pip install --pre --upgrade mxnet before you can start using NLP Toolkit.

Appreciate this. I was informed once of this existence but didn’t pay attention. It’s really nice to have this in place though it could use more documentation and example :slight_smile: