Any examples - in particular on using RunOnSpark.runOnSpark() ?

[javadba]

I would like to create a small PR with a small Test and/or example. Some suggested code to use for these would be appreciated.

Otherwise I will be attempting to reverse engineer from the API's and put together something.

[jamesjia94]

I created a KMeans learner that uses RunOnSpark under BIDMach_Spark/scripts/KMeansLearner.ssc.

Let me know if you have any questions!

[javadba]

Sounds perfect! Taking a look.

[javadba]

How do I run it? In general : is there a reference about .ssc files? I do not undertand: where is the interpreter for them and how they work.

[javadba]

I can not find any documentation on the web for .ssc files. They contain scala, java, python, and some custom language. Please point some references about them.

[jamesjia94]

From what I understand, the .ssc files are just a custom extension that we use to denote a scala script e.g. you can load them into a scala interpreter using scala -i test.ssc. The spark shell also used to support the -i command, though I don't think it has been working since 1.5.0+. I've been using :load path_to_file.ssc inside the Spark REPL.
I don't know about the .ssc files that aren't in scala, but I imagine it'd be for a similar purpose? @jcanny

[javadba]

So when I do

:load /git/BIDMach_Spark/scripts/KMeansLearner.ssc

the obvious problem is that some of the setup stuff like opts variable that are created via init_bidmach.scala are not there: so it fails. What is the way to run this in REPL?

[jamesjia94]

KMeans.learner() returns a tuple of (learner, opts), so opts should be in scope. Could you paste the specific error that you are receiving?

[javadba]

How are you starting spark-shell? The BIDMat/Mach libraries will need to be present.

Here is what I have done so far:

cala> :require /git/BIDMach/BIDMach.jar
Added '/git/BIDMach/BIDMach.jar' to classpath.

scala> :require /git/BIDMach_Spark/BIDMach_Spark.jar
Cannot load '/git/BIDMach_Spark/BIDMach_Spark.jar'

scala> :require /git/BIDMach_Spark/BIDMatHDFS.jar.jar
Cannot load '/git/BIDMach_Spark/BIDMatHDFS.jar.jar'

scala> :require /git/BIDMach_Spark/BIDMatHDFS.jar
Added '/git/BIDMach_Spark/BIDMatHDFS.jar' to classpath.

scala> :load /git/BIDMach_Spark/scripts/KMeansLearner.ssc
Loading /git/BIDMach_Spark/scripts/KMeansLearner.ssc...
import org.apache.spark.rdd.RDD
import scala.reflect.ClassTag
import BIDMat.MatIO
import BIDMat.SerText
import BIDMach.RunOnSpark._
:30: error: object MockLearner is not a member of package BIDMach
import BIDMach.MockLearner
^

[jamesjia94]

You can delete the MockLearner import or pull from BIDMach_Spark. It's a remnant from an older version of the test.
Also, you'll have to specify the MASTER_DNS as the master DNS address of your spark machine, and also supply your own data in BIDMat matrix forms.
For example, I use the mnist8M dataset which can be downloaded using BIDMach/scripts/getmnist8m.sh. The script automatically saves the matrices into FMats. You can then run BIDMach_Spark/scripts/load_mnist.ssc and BIDMach_Spark/scripts/append_mnist.ssc to load the data onto HDFS.

[javadba]

OK I pulled your update .

How do we load the jars? They are not complementary:

scala> :require /git/BIDMat/BIDMat.jar
Added '/git/BIDMat/BIDMat.jar' to classpath.

scala> :require /git/BIDMach/BIDMach.jar
The path '/git/BIDMach/BIDMach.jar' cannot be loaded, because existing classpath entries conflict.

But if I just do the BIDMach.jar we get this:

error: missing or invalid dependency detected while loading class file 'MatIO.class'.
Could not access type MatIOtrait in package BIDMat,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.)

[jamesjia94]

You can use the --jars option with a comma separated list of jars. Alternatively, you can configure the spark conf via spark.driver.extraClassPath and spark.executor.extraClassPath.

[javadba]

Hi James, yes but that was not my question/issue here. If you review again my last post it was about : "which" jars need to be used - given the apparent incompatibility of the combinations I tried.

Please say explicitly which combination of jars works for you - i.e. that you are using to get this to work.

[jamesjia94]

If you use --jars BIDMach.jar,BIDMat.jar instead of :require, the scala/Spark repl doesn't throw any errors.
That being said, here is my spark conf.

spark.executor.extraClassPath /opt/BIDMach/lib/BIDMatHDFS.jar:/opt/BIDMach/BIDMach.jar:/opt/BIDMach/lib/BIDMat.jar:/opt/BIDMach/lib/ptplot.jar:/opt/BIDMach/lib/ptplotapplication.jar:/opt/BIDMach/lib/jhdf5.jar:/opt/BIDMach/lib/commons-math3-3.2.jar:/opt/BIDMach/lib/lz4-1.3.jar:/opt/BIDMach/lib/json-io-4.1.6.jar:/opt/BIDMach/lib/jfreechart-1.0.19.jar:/opt/BIDMach/lib/jcommon-1.0.23.jar:/opt/BIDMach/lib/jcuda-0.7.0a.jar:/opt/BIDMach/lib/jcublas-0.7.0a.jar:/opt/BIDMach/lib/jcufft-0.7.0a.jar:/opt/BIDMach/lib/jcurand-0.7.0a.jar:/opt/BIDMach/lib/jcusparse-0.7.0a.jar

spark.driver.extraClassPath /opt/BIDMach/lib/BIDMatHDFS.jar:/opt/BIDMach/BIDMach.jar:/opt/BIDMach/lib/BIDMat.jar:/opt/BIDMach/lib/ptplot.jar:/opt/BIDMach/lib/ptplotapplication.jar:/opt/BIDMach/lib/jhdf5.jar:/opt/BIDMach/lib/commons-math3-3.2.jar:/opt/BIDMach/lib/lz4-1.3.jar:/opt/BIDMach/lib/json-io-4.1.6.jar:/opt/BIDMach/lib/jfreechart-1.0.19.jar:/opt/BIDMach/lib/jcommon-1.0.23.jar:/opt/BIDMach/lib/jcuda-0.7.0a.jar:/opt/BIDMach/lib/jcublas-0.7.0a.jar:/opt/BIDMach/lib/jcufft-0.7.0a.jar:/opt/BIDMach/lib/jcurand-0.7.0a.jar:/opt/BIDMach/lib/jcusparse-0.7.0a.jar

spark.driver.extraLibraryPath /opt/BIDMach/lib:/usr/local/cuda/lib64:/usr/local/hadoop/lib/native
spark.executor.extraLibraryPath /opt/BIDMach/lib:/usr/local/cuda/lib64:/usr/local/hadoop/lib/native

[javadba]

Thanks a lot James - I did not realize that ":require" would behave differently than --jars. Trying now.

[javadba]

How did you convert the MNIST files to Sequence files?