Question about the CLIP model

[Vanint]

Hi, thanks for your great work!

I found that you release several checkpoints, including CLIP ( "clip/image-enc": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/clip_image_enc.pt", "clip/text-enc": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/clip_text_enc.pt").

Are these checkpoints trained with noised images, or are they public CLIP models?

Best wishes,

[woctezuma]

See; https://github.com/openai/glide-text2im/blob/main/model-card.md#datasets

Datasets

GLIDE (filtered) was trained on a filtered version of a dataset comprised of several hundred million text-image pairs
collected from the internet. We constructed a set of filters intended to remove all images of people, violent objects, and some
and hate symbols (see Appendix F of the paper for details). The size of the dataset after filtering was approximately
67M text-image pairs.

Our noised CLIP model which was trained on the dataset described above, augmented with a filtered version of the dataset used to train the original CLIP models. The total size of this augmented dataset is approximately 137M pairs.

Moreover, the original CLIP is from January 2021, while the checkpoints are all from December 2021 (according to the URL), so newly trained (a priori). If the checkpoints were the old CLIP ones, I believe the URL would have been older, because there would be no need to re-host the models at a new location. It is still possible that the new URL is a redirection, but I find it unlikely for the aforementioned reasons.

glide-text2im/glide_text2im/download.py

Lines 10 to 17 in 69b5307

	MODEL_PATHS = {
	"base": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/base.pt",
	"upsample": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/upsample.pt",
	"base-inpaint": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/base_inpaint.pt",
	"upsample-inpaint": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/upsample_inpaint.pt",
	"clip/image-enc": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/clip_image_enc.pt",
	"clip/text-enc": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/clip_text_enc.pt",
	}

[Vanint]

See; https://github.com/openai/glide-text2im/blob/main/model-card.md#datasets

Datasets

GLIDE (filtered) was trained on a filtered version of a dataset comprised of several hundred million text-image pairs
collected from the internet. We constructed a set of filters intended to remove all images of people, violent objects, and some
and hate symbols (see Appendix F of the paper for details). The size of the dataset after filtering was approximately
67M text-image pairs.
Our noised CLIP model which was trained on the dataset described above, augmented with a filtered version of the dataset used to train the original CLIP models. The total size of this augmented dataset is approximately 137M pairs.

Moreover, the original CLIP is from January 2021, while the checkpoints are all from December 2021 (according to the URL), so newly trained (a priori). If the checkpoints were the old CLIP ones, I believe the URL would have been older, because there would be no need to re-host the models at a new URL. It is still possible that the new URL is a redirection, but I find it unlikely for the aforementioned reasons.

glide-text2im/glide_text2im/download.py

Lines 10 to 17 in 69b5307

	MODEL_PATHS = {
	"base": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/base.pt",
	"upsample": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/upsample.pt",
	"base-inpaint": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/base_inpaint.pt",
	"upsample-inpaint": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/upsample_inpaint.pt",
	"clip/image-enc": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/clip_image_enc.pt",
	"clip/text-enc": "https://openaipublic.blob.core.windows.net/diffusion/dec-2021/clip_text_enc.pt",
	}

I agree with you. Thanks~