We tracked the cause of this down to unsafe concurrency in the MXNet Scala library (specifically, Java’s finalizers).
The fix is discussed in this thread. This was eventually also merged into mxnet.
Note that this doesn’t make concurrency safe, but rather gets rid of unsafe concurrency internally. We are still using a dispatcher-object for thread-safe access to the library. We have been running stably (even with more complicated models) at scale since this was fixed.