[Self-BLEU] what metric values do you get?

[marco-roberti]

Hello, I'm running the RelGAN code, evaluating diversity using Self-BLEU instead of NLL_gen, that suffers from the already observed issues.
However, I'm getting an unexpected behavior on this metric during training: can anyone please share their own results, just to be sure that I'm not missing anything?

Thanks in advance! :)

[weilinie]

What behavior do you observe with self-bleu during training? Also, do you use the self-bleu implementation in this repo?

[marco-roberti]

It's almost fixed to one for all n-grams (2, 3, 4, 5).
I'm using the Self-BLEU implementation in this repo, simply uncommenting the corresponding flag in coco_relgan.py

[weilinie]

Yes, as far as I know, this behavior is as expected. I have also opened an issue here: geek-ai/Texygen#27 to discuss it.

[weilinie]

Does the above explanation make sense to you? @marco-roberti