llamacpp: Enable flash attention

[p1-0tr]

No description provided.

[ericcurtin]

Note Georgi suggests doing this for Metal and CUDA:

https://x.com/ggerganov/status/1909657397964292209

So might want to put some conditionals here, like for example, this flag may be better off absent in the case of CPU inferencing

[xenoscopic]

@p1-0tr Is this still worth pushing ahead? I guess we need to shift it to pkg/inference/backends/llamacpp/llamacpp_config.go and condition it on runtime.GOOS == "darwin" || hasCUDA11CapableGPU(...)?

[ericcurtin]

Note they are speaking of an auto-fa flag coming soon upstream:

ggml-org/llama.cpp#15454

this optimization is worth it.

[ericcurtin]

This is not worth adding anymore, there's been a lot of activity upstream, now llama-server will automatically turn on flash attention when appropriate. It will be picked up here whenever llama.cpp is rebased

[xenoscopic]

Perfect. It looks like @p1-0tr has a pending PR to bump llama.cpp with most of the flash attention work you mention. @p1-0tr shall we close this one out?