Error while running the mxnet spark examples and test cases

I am trying to run the mxnet in distributed mode using spark as implemented here : https://github.com/apache/incubator-mxnet/tree/master/scala-package/spark

but I am not able to run the examples and/or tests.

The commented tests in the file : https://github.com/apache/incubator-mxnet/blob/master/scala-package/spark/src/test/scala/org/apache/mxnet/spark/MXNetGeneralSuite.scala keep running into the following error. I get the same error when I try to run the examples in the repo.

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/Seq
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: scala.collection.Seq
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more
Exception in thread "Thread-21" java.lang.IllegalArgumentException: requirement failed: Failed to start ps scheduler process with exit code 1
	at scala.Predef$.require(Predef.scala:224)
	at org.apache.mxnet.spark.MXNet.org$apache$mxnet$spark$MXNet$$startPSSchedulerInner$1(MXNet.scala:159)
	at org.apache.mxnet.spark.MXNet$$anonfun$startPSScheduler$1.apply(MXNet.scala:162)
	at org.apache.mxnet.spark.MXNet$$anonfun$startPSScheduler$1.apply(MXNet.scala:162)
	at org.apache.mxnet.spark.MXNet$MXNetControllingThread.run(MXNet.scala:38)
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/Seq
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: scala.collection.Seq
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more

I have seen this error generally when there’s a mismatch between the scala versions in the api, but in this case I am using the using the pom file that’s in the project and not including any external libraries.
I have also looked at the pom file but I’ve not found any lib that might have a mismatch in this case, all the libraries in pom are 2.11 version

Can you guys help me with it?

Hi,

I can help you here. First of all, how did you use the package. Build from source or grab from Maven?

  • Qing

Thanks @lanking520 ,

I am building it from source, running it with this vm parameter : -

Djava.library.path=/path_to_mxnet_source/incubator-mxnet/scala-package/native/osx-x86_64-cpu/target

to let it find the native library.

Since this looks like it could be a bug I’ve opened a GitHub Issue: https://github.com/apache/incubator-mxnet/issues/13853. @lanking520 should be able to help over there.

1 Like

Your issue for running the examples should be resolved by this PR : #13849 and #13891

There were recent changes to the Maven POM files due to which the examples were temporarily not running. They should be back to normal now. Let me know if you still face issues with running your examples.

And yes. the java library path that you are setting is the correct one.

@piyushghai The error still persists. You can reproduce it by un-commenting the test = run spark with MLP with it’s related methods in the file org/apache/mxnet/spark/MXNetGeneralSuite.scala

Also, the latest version in master doesn’t respect the -Djava.library.path or LD_LIBRARY_PATH_VARIABLES and only picks up the native files from the jar (which doesn’t seem to have these files). So to test this I am running a hacked version to make the libraries get picked up from the path (/incubator-mxnet/scala-package/native/osx-x86_64-cpu/target) I hard code in NativeLibraryLoader.scala

It looks like you are picking up jar files from old version.
In new version, we are no longer output to: /incubator-mxnet/scala-package/native/osx-x86_64-cpu
It’s better clean your workspace and do a fresh build:

cd incubator-mxnet/scala-package/
git clean -dfX .