I.
dllin is a simple deep learning framework running on Apache Spark. The framework was develped to achieve
The usage is easy for some developers who are already familiar with existing high level deep learning frameworks such as Keras.
import com.lewuathe.dllib.graph.Graph import com.lewuathe.dllib.layer.{AffineLayer, ReLULayer, SoftmaxLayer} import com.lewuathe.dllib.network.Network // Define the network structure as calculation graph. val graph = new Graph(Array( new AffineLayer(100, 784), new ReLULayer(100, 100), new AffineLayer(10, 100), new SoftmaxLayer(10, 10) )) // Model keeps whole network parameters which should be trained. // Default is in-memory model. val model = Model(nn3Graph) val nn3 = Network(model, graph) // MultilayerPerceptron defines the optimization algorithms and hyper parameters. val multilayerPerceptron = new MultiLayerPerceptron("MNIST", nn3) // We can pass Dataset of Spark to the network. val trainedModel = multilayerPerceptron.fit(df) val result = trainedModel.transform(df) result.filter("label = prediction").count()
The imporant components to create network are three.
Layer
Graph
Model
Layer is a component which specifies a layer of neural network. The layer transforms a vector into another vector one by one.
Graph specifies the structure of whole network components. So we can create neural network model by adding
Layers to
Graph. One thing to notice is
Layer and
Graph have no persistent data in themselves. So parameters to be trained should be given other parts. This is
Model. Model keeps whole parameters of each
Layer. So if we want to predict after training, we can give trained
Model to your network. Though currently we have only in-memory model, we’ll implement distributed versions by using parameters server. That must provide high performance with model parallelism.
Since dllib is compatible with Spark ML Dataset based interface, we can easily import existing data into dllib if you already have Spark cluster. Spark ML can be run on local machine as well as distributed mode. You can try with
packages option immediately.
$ ./bin/spark-shell --packages Lewuathe:dllib:0.0.9
dllib is uploaded on Spark packages and maven central.
Last but not least, dllib is under development. We’ll implement CNN, RNN and other network structure. And also documentation should be enhanced. Please give me any feedback or patches if you are interested. It’s my hobby project but I’m very glad if I can cooperate with some developers in the world who are interested in deep learning.
Thank you.Written on January 21st , 2017 by lewuathe