Add substitutions option in ASR evaluation

[avmonte]

Hi there!

I've been experimenting with Armenian ASR and noticed a consistent evaluation issue: the model never outputs the single-character Armenian և (U+0587). Instead it outputs the decomposed form եւ (two characters). As the ground-truth transcripts contain և, this mismatch inflates WER despite the fact that the predicted text is linguistically equivalent.

This PR adds an evaluation-time normalization option to address that type of issues.

• Added text_processing.substitutions for evaluation-time text normalization:
  • Format: SRC~DST;SRC2~DST2;...
• Substitutions are applied to both ground truth and predictions before WER
• Supports literal Unicode and escaped input like \u0587.

Reproduce

Below is the configuration I used to run my evaluation

python NeMo/examples/asr/speech_to_text_eval.py
dataset_manifest=<path_to_manifest.json>
use_cer=False
only_score_manifest=True
text_processing.punctuation_marks=".,?!։՝՜՞՛«»֊—–-…"
text_processing.separate_punctuation=True
text_processing.do_lowercase=True
text_processing.rm_punctuation=True
text_processing.substitutions="\u0587~եւ"

Evaluation results

Base MCV test WER (from the HF model card): 9.90
Normalized MCV test WER: 5.42

[avmonte]

@titu1994, @redoctopus, @jbalam-nv, @okuchaiev

Please take a look :)