Train MSVD dataset using VATEX pretrained model? Thanks

[franciszchen]

Hi, I am going to reproduce the reported performance on MSVD dataset with CIDEr of 120.6, but there exists a gap. In my experiment, the first evaluation after the initialization is poor, the initialized CIDEr is almost 0, as
11/02/2022 10:02:18 - INFO - __main__ - evaluation result: {'Bleu_1': 0.0006309148264980254, 'Bleu_2': 2.0612095211839896e-11, 'Bleu_3': 6.744216480610146e-14, 'Bleu_4': 3.9307266631435836e-15, 'METEOR': 0.010075708541163993, 'ROUGE_L': 0.0009159159159159158, 'CIDEr': 4.846693412127264e-06, 'SPICE': 0.0015076134016082418}

But in the provided log (./models/table1/msvd/log/log.txt), the first evaluation reveals a high CIDEs of 50.88.
11/11/2021 21:22:26 - INFO - __main__ - evaluation result: {'Bleu_1': 0.6993166287010636, 'Bleu_2': 0.5518608344658137, 'Bleu_3': 0.43743626324685403, 'Bleu_4': 0.33875510186099217, 'METEOR': 0.36200563653140405, 'ROUGE_L': 0.6269033486705, 'CIDEr': 0.508808824754587, 'SPICE': 0.09748587107161746}

To further check this problem in the log.txt, it seems the training on MSVD is initialized from a model pretrained on the VATEX dataset, by using a specific pretrained_checkpoint ("pretrained_checkpoint": "/xiyin1wu2_maskrcnn/keli/debug_output/videocap_github/vatex/20211023_VidSwinBert_base_224_seq_len50_bsz6_ep15_lr3e-4_mul0.05_featD512_frame32_wkr10_k600_2d0_mlm_mask0.5_45_grad_accu1_sparsemask0.5//checkpoint-15-40605/"), as

11/11/2021 21:21:23 - INFO - __main__ -   Init model from scratch.
11/11/2021 21:21:23 - INFO - __main__ -   Model total parameters: 136106810
11/11/2021 21:21:23 - INFO - __main__ -   video swin (config path): src/modeling/video_swin/swin_base_patch244_window877_kinetics600_22k.py
11/11/2021 21:21:26 - INFO - __main__ -   Loading state dict from checkpoint /xiyin1wu2_maskrcnn/keli/debug_output/videocap_github/vatex/20211023_VidSwinBert_base_224_seq_len50_bsz6_ep15_lr3e-4_mul0.05_featD512_frame32_wkr10_k600_2d0_mlm_mask0.5_45_grad_accu1_sparsemask0.5//checkpoint-15-40605/model.bin

Is it necessary to reproduce the MSVD performance using a VATEX pretrained model? Thanks for your time and attention!

BTW, the TVC log is also trained from a pretrained_checkpoint, but the MSRVTT, YouCook2 and VATEX logs are trained from scratch.

[myccver]

Hi, I faced the same question. The CIDEr is 0 when I train the model using MSVD dataset.