Use the Funnel Transformer + T5 model from the huggingface hub with some subclassing to convert them into a VAE for text.
The current SOTA VAE is OPTIMUS which still suffers from some posterior collapse.
  

      Microsoft Research
  

  
    

OPTIMUS: Organizing Sentences via Pre-trained Modeling of a Latent Space -... 53

  When trained effectively, the Variational Autoencoder (VAE) (Kingma and Welling, 2013; Bowman et al., 2016) can be both a powerful generative model and an effective representation learning framework for natural language. In this paper, we propose the...


  

  
    
    
  

  


Its training data is all open source.
  

      github.com
  

  
    ChunyuanLI/Optimus/blob/master/download_datasets.md 8


    # Download/Pre-process Datasets

## Wikipedia

Download processed files (11.78G) below, and unzip it (298 files)

https://textae.blob.core.windows.net/optimus/data/datasets/wikipedia_json_64_filtered.zip

Download raw file (11.79G):

https://textae.blob.core.windows.net/optimus/data/datasets/wikipedia.segmented.nltk.txt

Our pre-processing protocal: We split the original wiki text into 298 files, and loop over files in one epoch.

We filter each sentence in wiki based on two constraints: (1) The sentence length is smaller than 64. (2) The tokenized sentence length is smaller than 256 (so that the encoder can take the entire sentence).

To filter the sentence, please change the data folders and run the script:

    sh scripts/scripts_local/run_data_filtering_wiki.sh




  This file has been truncated. show original

  

  
    
    
  

  


From my experiments an MMD-VAE doesn’t suffer form as much posterior collapse in smaller scale models, why not try & scale up?
  
      

      Fraser Greenlee – 13 Aug 20
  

  
    

Transformers as Variational Autoencoders 49

  Avoid posterior collapse even with a huge model.


  

  
    
    
  

  


I’ve actually already got the code working to turn them into a Pytorch MMD-VAE so why not just convert it to JAX/Flax?
  

      github.com
  

  
    Fraser-Greenlee/transformer-vae/blob/master/transformer_vae/model.py 53


    """
    Base transformer-VAE model.
"""
import torch
from torch import nn
from typing import Dict, Any
from transformers.utils import logging
from transformers.modeling_utils import PreTrainedModel
from transformers.modeling_outputs import BaseModelOutput
from transformers.models.funnel.modeling_funnel import upsample
from transformers import AutoModelForSeq2SeqLM, AutoModelForMaskedLM

from transformer_vae.custom_t5 import modify_t5_stack
from transformer_vae.autoencoders import VAE_ENCODER_MODELS, VAE_DECODER_MODELS, EncoderDecoderVAE
from transformer_vae.critic import CRITIC
from transformer_vae.model_outputs import BaseTransformerVAE_Output
from transformer_vae.config import Funnel_T5_VAE_Config


logger = logging.get_logger(__name__)



  This file has been truncated. show original