SOS 🚨​: Training ASR on Arabic/English data predicts blank token after ~100 steps and model is not learning.

[Houss3m]

Hello team;

I am trying to train an ASR model using NeMo framework, tried many recipies, but all of them are predicting blank for my dataset.

details:

• my dataset has mix of Arabic and English, transcriptions for Arabic are written in Arabic script, while the English are written in Latin scripts.
• When I do train a conformer Tranducer for example, it starts predicting junk transcriptions, after a number of steps, it starts predicting blanks (nothing).
• like the model is not learning and see the blank is the best path and keeps predicting blank only.
• I doubted the dataset, so I created two splits, one for English and one for Arabic.
• I trained on the English data, the model started predicting junk text, then it started to predict blank, then it started to learn after 40% of the first epoch and performed well now.
• I did the same steps for the Arabic split (building tokenizer, tarred the data etc..), and used the same config and parameters but the model starts predicting blank after certain number of steps and it is stuck there.
• the tokenizer works well when I test it on text data.
• I increased the vocab size to 4k and still same issue...
• tried to reduce learning rate and still the same issue.

any thoughts on what could be the issue? like encoding of the json file can be an issue?

example of the outputs:

[NeMo I wer:330] reference:مسبح
[NeMo I wer:331] predicted:ائرائرائرائرائرائرائرائرائرائرلاداهلادلادوةوةنيلي يكنيلي يكن يكن يكن اي يكن يكن يكن يكن يكن 
[NeMo I wer:329] 
    
[NeMo I wer:330] reference:اربع
[NeMo I wer:331] predicted:حيحيحيحيحيحيحيؤولحيحي ده فيده ده ده ده ده ده ده ده دصيرتيصير اف اف ا
.....

# after many steps I get blank and keeps going to epoch 10:

[NeMo I wer:330] reference:وليس هزيمتهم بالامر الذي نتوقعه
[NeMo I wer:331] predicted:

Epoch 7:  37%|███▋      | 10114/27008 [55:21<1:32:27,  3.05it/s, v_num=0, train_step_timing in s=0.362]

samples of the date:

{"audio_filepath": "path0.flac", "duration": 1.17, "text": "تشربو شي حاجة", "shard_id": 0}
{"audio_filepath": "path1.flac", "duration": 1.31, "text": "خمس", "shard_id": 0}
{"audio_filepath": "path2.flac", "duration": 1.32, "text": "شنو واقع", "shard_id": 0}

I build my tokenizer using this command:

python process_asr_text_tokenizer.py \
  --manifest=valid_arabic.jsonl,train_arabic.jsonl \
  --vocab_size=4096 \
  --data_root=arabonly_4096 \
  --tokenizer="spe" \
  --spe_type=bpe \
  --spe_character_coverage=1.0 \
  --spe_max_sentencepiece_length=16 \
  --log

any insights please?

[nithinraok]

Hi,

check tokenizer encode and decode options to validate for arabic you are able to see the text as intended.

Start with transducer instead of hybrid and also start from a pretrained encoder (like parakeet-tdt-0.6b-v3). Would also recommend to use fastconformer rather than conformer configs.