Training with the vision encoder

[saunair]

Hi @DerrickWang005, did you guys try an experiment with the VIT as the image backbone instead of the Resnet. CLIP says the VIT backbone worked better for the contrastive part. What are your thoughts?