Add Transformers benchmarks.

[Vaibhavs10]

Closes: #96

(Preliminary script for now)

[Vaibhavs10]

Maybe I should test for different chunk lengths too?

[Vaibhavs10]

Steps to build the environment:

python -m venv benchmark
source benchmark/bin/activate
pip install --no-cache-dir torch transformers optimum accelerate wheel flash-attn --no-build-isolation

[Vaibhavs10]

Cool I think we're done here! 🚀

You should be able to run this script via python script/benchmark.py

Here's how the output looks like:

Running Model: openai/whisper-large-v3
Flash Attention: True
Batch Size: 1
Total time: 7.989270925521851
Total memory: 3608.0

All the results are saved over a benchmark.json file.

Next step: Let this run on an A100 and a T4 (I'll probably only get to this tomorrow!)

[sanchit-gandhi]

Feel free to have a look at what we used for Distil-Whisper benchmarking as an example: for evaluating over a dataset and computing the WER: https://github.com/huggingface/distil-whisper/blob/914dcdf3919552d5a3826a9d5db99b059ddcc16e/training/flax/run_speed_pt.py#L595

=> it's a bit messy, but largely follows the same structure as the short-form script. You can just pull out the logic we use for pipeline here (rather than original Whisper or model + processor)

[sanchit-gandhi]

The configuration is a bit different for Whisper vs Distil-Whisper:

• Whisper: chunk length 30s, with timestamps
• Distil-Whisper: chunk length 15s, without timestamps

[sanchit-gandhi]

We should also force the language/task for multilingual Whisper/Distil-Whisper checkpoints

[sanchit-gandhi]

As mentioned offline, "relative memory" is probably the best estimate we can give for memory, if we decide to provide one.