Thanks for reading our proposal.

Do you think you will be able to write jax transformer code with conditional random fields in 3,4 days for the project?

I plan to use FlaxBERT and implement conditional random field on my own.  We want to make it simple as possible. So, if there exists a code snippet for conditional random field, we’d like to use that for this project.

Also we probably don’t need to pretrain the whole model from scratch, but we can just use a pretrained BERT model and adapt it for fast non-autoregressive generation no?

Yes, only fine-tuning pre-trained model could be one simple way to accomplish this project. However, I want to do it from scratch because I didn’t have experience about the pretraining language model due to the lack of computing resource. So, I want to improve my skill for pretraining models through this event. Also, I believe it could be a decent research work if it succeed.
I think building up two milestones for this proposal would be better.

(Required) Reproduce the reference paper through fine-tuning
(Optional) Move the mechanism to pre-trained phase to enhance the language model performance

Thanks for useful advice @patrickvonplaten, I appreciate it!