you could store each output feature of the  VGGish  model with its corresponding target, create a new classification model, and use these output features + targets to train this new classifier.

Cool. Thanks for the tip! Would this seem like reasonable steps to train a new model?

Download audio wav samples from audioset 32  as training data
Send each audio wav sample thru the VGGish to get a corresponding 128-dimension vector (output feature)
Define a Dataset comprising the VGGish output feature as input (x) and the corresponding target (y)
Using nn.module, define a “shallow” model using single layer…say Linear() .
train