How do I scale Vaex to handle multiple requests

[MHK107]

I'm having some trouble in scaling up my vaex application. I'm hoping I can get some advice here to have a more scalable system.

My Question:
I would like to know if having a single big instance (8-10 cores) would be better for speeding up API response times along with having multiple concurrent requests or having multiple (4-5) smaller 2 core machines would work better?

My goal:
Scaling it up, I was thinking to have multiple such smaller machines to scale up however I read some articles online and according to what I could understand vaex scales up exponentially with the amount of CPU cores available.

Problem:
API's work fine when the requests are sent one at a time (under 1 second)

however when I send just 4-5 requests at the same time ( the CPU utilization goes really high along with response times (8-10 seconds for all API's to respond sometimes even more)

resulting into random crashes the server process sometimes due to high server load. So technically speaking I can barely have 4 users using my machine in parallel

My System and configuration:
I have a dashboard which uses data analysis provided by vaex using
API wrapper build around it with Flask.
I'm using Gunicorn to manage the process (running 12 worker threads).
Currently my application is running on AWS Ec2 2 core CPU.

@maartenbreddels @JovanVeljanoski would love to hear your view on this

[maartenbreddels]

Hi,

interesting question!

what I could understand vaex scales up exponentially with the amount of CPU cores available.

I'm not sure where you read this, but in the ideal case Vaex scales linear with the number of rows, and linear with number of CPUs (modulo issues with the GIL).

It might be useful to take a look at our dash demo:
https://github.com/vaexio/dash-120million-taxi-app

which is also using gunicorn. In our case we run on a 32 core machine (64 with hyperthreading). We control the number of threads each worker is using by setting the envvar VAEX_NUM_THREADS=8, and run with 16 workers (8 will probably be fine as well).
This means that it can handle 16 or 8 concurrent requests, and each request will use 8 cores.

See also how we benchmark it here:
https://github.com/vaexio/dash-120million-taxi-app/tree/master/benchmark

If you have a 2 core system, and all workers are using all two cores, I can imagine performance degradation when handling concurrent requests (lots of context swapping).

If you want 4 concurrent requests, and want to use multithreading, I'd pick a similar layout, say X cores, each worker using X/4 cores, and 4, 8, or 16 workers.

I recommend using this as a guideline, and benchmark to optimize for your use case. I hope this is useful to you.

cheers,

Maarten