Thanks! Seems simple enough.



 apaszke:

What kind of layer do you want to enlarge/shrink? Is it only for linear modules, or do you also consider things like conv? In general you’ll probably need to fiddle with the weight tensor (just reassign the attr with the new value).


Mostly linear layers. The most important is adding output variables to add classes to a model, e.g. to add a new entity type.



 apaszke:

I’m not sure what’s the exact purpose of this context manager so it’s hard for me to answer that. I’ll need more details.


I’ve just realized I can easily construct this using the to_bytes() and from_bytes() methods. So there’s no real problem here. For the record though, it looks like this:
for epoch in epochs:
    train_epoch(model, optimizer, train_data)
    with model.use_params(optimizer.averages):
        print(model.evaluate(dev_data))

This is used for the “averaging trick”, where you store the EMA of the parameters during training, and use them for inference instead of the final weight values. This helps avoid sharp minima, and in my experience is pretty much always good. The context manager is necessary because you want to print a checkpoint against the weights you’ll actually use for inference, while resuming training from where you left off.



 apaszke:

Although I’d say it’s a bit of a surprising interface (what makes dropout so special?).


It does suck a bit. There are a few of these warts in Thinc at the moment.
The rationale was that I wanted the calling code to control the dropout. This makes it easy to decay or increase the dropout rate without walking the models changing hyper-parameters. I also wanted layers to own how they perform dropout. I was also thinking about allowing variational inference, although I haven’t played with that.