They add a CoCa-style caption decoder, where they learn to generate the synthetic caption given the image and the web caption. Their full loss is InfoNCE on random-synthetic-sentence + InfoNCE on web caption + NLL on synthetic tokens.