Using BlenderBot for other languages

[Parvez2017]

Use this to open other questions or issues, and provide context here.
Is there any work going on to use Blenderbot for dialogue generation for other languages like Korean and Japanese?

I am trying to implement the Blenderbot strategy to train a model on Korean language. Can I retrain Blenderbot from scratch?

Is there any implementation idea for Blenderbot in other language?

Please help me out.

[stephenroller]

I've been working with a MS student who's been doing French. It works okay just fine tuning on French data. Korean probably wouldn't work that well though. You'd need to train a new model from scratch on that or similar.

mBART might be more up your alley for this.

[KingTele10]

Please I'm new and don't know what is going on, can you please update me on anything

[gitsand996]

Hi @stephenroller , what kind of conversational data did you use for your French task? How many data did you use and where did you found them? Did you simulate some kind of multitasking such as bst+convai+wow ?

[Hilal-Urun]

hi @stephenroller :) I want to ask about Turkish language :) I want to fine tune on Turkish data, could you guide like which model is more suitable for this purpose ? Thanks for your time

[KingTele10]

Please I'm new and don't know what is going on, can you please update me on anything

[stephenroller]

My student and I had success with french with the blenderbot 90M model, but that was somewhat chosen for resource constraint reasons. I would suggest giving BlenderBot400 or larger a shot, or otherwise try -m hugging_face/gpt2.

[stephenroller]

-m bart could also work well. We looked into porting mBART but never got around to it. The main thing is we need someone to add SentencePiece tokenization to ParlAI.

[Hilal-Urun]

hi, again @stephenroller . I want to ask that I am trying to add sentencePiece tokenization bur how can I integrate, could yoyu guide me?

[Hilal-Urun]

Okey, Thanks a lot for your response, I got it and I'm gonna try for Turkish with adding turkish tokenizer :) @stephenroller

[Hilal-Urun]

@stephenroller I want to use hugginface tokenizer that has usage like;
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-turkish-cased")
Where can I place this function? Could you guide me please :)
İf I just add a tr_tokenize func

ParlAI/parlai/core/dict.py

Lines 391 to 402 in d191b80

	def nltk_tokenize(self, text, building=False):
	"""
	Tokenize using NLTK PunktTokenizer.
	Uses nltk-trained PunktTokenizer for sentence tokenization and Treebank Word
	Tokenizer for tokenizing words within sentences.
	"""
	return (
	token
	for sent in self.sent_tok.tokenize(text)
	for token in self.word_tok.tokenize(sent)
	)

like that , does it work? For example ;

def tr_tokenize(self,text):
self.tr_tokenizer=tr_tokenizer
return self.tr_tokenizer.tokenize(text)

[lh0x00]

hi @stephenroller, can you suggest some steps need to train a blenderbot with other languages such as vietnamese, japanese,... ? or suggest other agents for this task?

[lh0x00]

Ref, #2830 (comment)
I having a vietnamese dataset and I want to learn about blenderbot.

[Hilal-Urun]

hi again @stephenroller when I tried to fine tune bart model with turkish wikipedia data it gives error like
RuntimeError: CUDA error: device-side assert triggered

[stephenroller]

That usually happens when you have an out-of-bounds error on one of your embedding inputs. Could be either position embeddings (your truncate arguments are too long) or embeddings (something is wrong with the dict)

[Hilal-Urun]

okey I am gonna check it again

[brandaobrandisborges]

Hello, I intend to train a model in Portuguese (Brazil) but I'm facing a few dificulties.
I have followed the documentation to create a dataset and I have trainned it using these arguments¹, but the conversation doesn't seem right - the agent's responses don't seem related to the user input at all.

I suspect this is due to my dataset². It has around 5000 lines - pairs of text, labels and episode_done - the only annotations I understand.
Checking the train.txt from blended_skill_talk, it has much more annotations (e.g free_message, empathetic_dialogues, guided_chosen_suggestion and so on). More on, it has other files such as persona_list, human_annotations and topic_to_persona_list that I don't fully understand.

¹ What I run to train the model:
parlai train_model --task fromfile:parlaiformat --fromfile_datapath ./train.txt -m transformer/generator --multitask-weights 1,3,3,3 --init-model zoo:tutorial_transformer_generator/model --dict-file ./simple.dict --embedding-size 512 --n-layers 8 --ffn-size 2048 --dropout 0.1 --n-heads 16 --learn-positional-embeddings True --n-positions 512 --variant xlm --activation gelu --skip-generation True --fp16 True --text-truncate 512 --label-truncate 128 --dict-tokenizer bpe --dict-lower True -lr 1e-06 --optimizer adamax --lr-scheduler reduceonplateau --gradient-clip 0.1 -veps 0.25 --betas 0.9,0.999 --update-freq 1 --attention-dropout 0.0 --relu-dropout 0.0 --skip-generation True -vp 15 -stim 60 -vme 20000 -bs 16 -vmt ppl -vmm min --save-after-valid True --model-file ./test_train_90M

The simple.dict is generated by running:
parlai build_dict --task fromfile:parlaiformat --fromfile_datapath ./train.txt --dict-file ./simple.dict

²This is a sample from my dataset, train.txt:
text:quiser ficar sozinha esta noite. . . labels: Porquê? Está tudo bem? episode_done:True
text:Sim, apenas cansada do serviço. labels: Que tal eu te contar uma piada para te alegrar? episode_done:True
text:agora não. Vou dormir, tchau. labels: Tudo bem, tchau! episode_done:True

This is what I'm asking for now:

1. Am I on the right path to train a portuguese model by just providing a portuguese dataset or is it more complex?
2. Where can I find a further explanation on the train.txt annotations and other files cited above?
3. From "fine tuning" a model I understand as simply correcting the dataset used for training, am I right or is there a more complex aproach?

Thanks in advance!

[roscazo]

Hi there!

I suppose you won't see this. But I want to know if you got any results from this.
I'm kind of trying the same thing in Spanish and I can't get any good results.

Thanks in advance!

[stephenroller]

My MS student previously got this to work in french reasonably well when we used a fairly large amount of data (maybe 100k examples crawled from the web).

[roscazo]

Thank you for answering! Really appreciate it.

I have a couple more questions if you could help me:

1. Is it necessary to build your own dictionary or you must work with the dictionary provided by the model?
2. Is this parlaiformat task good enough to be used in more complex models such as BlenderBot 400M or even BlenderBot 2? As the original question, would I need a more complex approach?

Thanks again!

[stephenroller]

I would recommend keeping the same dictionary, at least for anything other than the blenderbot 90m model.

Parlai format is fine until your dataset is tens of gigabytes.

[helehh]

Hello! Is anyone here had any success with training (either pre-training from scratch or fine-tuning) BlenderBot for any other language? I am currently working on the same task and would be happy to discuss the approaches and issues with someone :)

[thies1006]

I think you shouldn't start from scratch if you use the same set of characters (like french, italian, .. and not chinese, japanese). What especially helped in my case was continuing to pretrain in the target language.

[helehh]

Hey @thies1006! Thanks for the reply! Did you mean continue pretrain English model in the target language?

[stephenroller]

yeah that would generally be my recommendation. I had this work with a MS student I advised.

[imthebilliejoe]

Hi @stephenroller, with my team we are trying to fine tune blenderbot 90M for italian language with the translated dataset BST+convai2+wow (so the same ones that were used originally in BlenderBot), to check how the model can be adapted to italian.

Do you think we can obtain good result just fine tuning the model on the new dataset (with Recipes command) or do you think it is necessary to change tokenizer and use an italian one?
Do you have any other suggestion to make it work good?

We tried to fine tune the 90M model only on translated BST without changing tokenizer but we didnt obtain good result but we don't know if it was due to the tokenizer or to the fact we used only BST dataset instead of BST+convai2+wow.