Serving Model Trained with PyTorch
Tensorflow has Tensorflow Serving. I know pytorch is a framework in its early stages, but how do people serve models trained with pytorch. Must it be from Python? I’m specifically looking to serve from C++.
abc
We don’t have a way to serve models from C++ right now, and it’s not a priority for us at this stage. There are many things like distributed training and double backward that we’ll be implementing first. Sorry!
apaszke
Would you say that pytorch was built with serving in mind, e.g. for an API, or more for research purposes?
abc
We’re more research oriented. We’re rather thinking of creating tools to export models to frameworks that are more focused on production usage like Caffe2 and TensorFlow.
apaszke
Also you mentioned double backward. This is the first I’ve heard of it. I found a paper by Yann LeCun on double backpropagation, but was wondering whether it’s common to use such a method.
abc
Hi, I’m playing with a possible solution for serving from C based on TH and THNN. It’ll be limited to statically compilable graphs of course. I should have something to share in the not so distant future.
lantiga
apaszke
lantiga
Sure that sounds cool. It doesn’t seem hacky, it’s just a graph compiler. It’s a very good start, and will likely be capable of producing small binaries. Let us know when there’s going to be any progress or in case you have any trouble. We’ll definitely showcase your solution somewhere.
apaszke
Eugenio_Culurciello
Making progress. As soon as I get the first MNIST example to compile I’ll share what I have.
lantiga
We need to deploy pytorch models to e.g. Android, so we need a method to export a model. This is my starting point. Can you please tell me if I am on the right way or if I am doing something totally stupid? <SCODE>import sys
import torch
from torch import nn
from torchvision import models
from torch.utils.serialization import load_lua

def dump(f):
	s = str(f.__class__)
	sys.stdout.write(s[s.rfind('.')+1:-2]+'(')
	for fa in f.previous_functions:
		if isinstance(fa[0], torch.autograd.Function):
			dump(fa[0])
			sys.stdout.write(',')
		if isinstance(fa[0], torch.nn.parameter.Parameter):
			sys.stdout.write('param,')
		elif isinstance(fa[0], torch.autograd.Variable):
			sys.stdout.write('input,')
	sys.stdout.write(')')


class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=1, bias=False)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(3, 16, kernel_size=1, bias=True)

    def forward(self, x):
        return self.bn1(self.conv1(x))+self.conv2(x)
        

#net = models.alexnet()
#net=load_lua('model.net') #Legacy networks won't work (no support for Variables)
net = MyNet()
input=torch.autograd.Variable(torch.zeros(1,3,128,128))
output=net.forward(input)
dump(output.creator)
print('')
<ECODE> The output for the simple MyNet will be <SCODE>Add(BatchNorm(ConvNd(input,param,),param,param,),ConvNd(input,param,param,),)
<ECODE> Thanks
mvitez
apaszke
Yes, please. I have just sent an invitation request for slack to soumith.
mvitez
I’m not sure how profoundly things will have to be reworked with the upcoming changes in autograd, but it’s fun anyway.
lantiga
lantiga
smth
Great, very nice work, thank you.
mvitez
apaszke
Eugenio_Culurciello