Load multiple instances of model on 1 GPU

[Cavan1Ed1s0n]

Your code seems not to be able to load multiple instances of model on 1 GPU. Could you solve this issue to enable the system handle concurrent requests

[merrymercy]

1. You can send many concurrent requests to a server. Just use many threads.
2. You can launch multiple instances on a single GPU. Tune --mem-fraction-static