Make fp8 work on older GPUs

[yorickvP]

Tested on RTX A5000.

• Use float32 matmul on older GPUs
• Offload based on total GPU memory instead of 'A40' in name
• Tell the FP8 code about offloading and compiling

[daanelson]

@yorickvP this is great, making 8-bit inference work regardless of hardware is useful. That said, when I run cog predict -i prompt=<whatever> on an A40, it takes ~10 minutes to compile and then there's no output. can you take a look? wary of pushing a broken path here

[yorickvP]

I checked again, and the prediction eventually succeeded: https://replicate.com/p/8r81q9z2b5rgp0cjv26977a6kg .
What this changed is that it now attempts to compile fp8 on A40 where it didn't previously, which brings the boot time over 10 minutes.
I'll disable fp8 compile in the offload case!

[daanelson]

@yorickvP fantastic! works great now, merging.