running artillery test from aws lambda against apollo graphql router

[seungchanlee]

Hello!
We have our own lambda function and try to run load test from it against apollo graphql router. Our goal is as below
- duration: 60
arrivalRate: 1000
(scenario is just to make a simple post request)

but even with this config -
- duration: 30
arrivalRate: 250

we got results similar to the below .. lots of Errors.
Codes:
200: 5948
Errors:
EMFILE: 1087
ETIMEDOUT: 464
EAI_AGAIN: 1

Do you have any recommendation how to improve our test? What exactly mean EMFILE? It would be very appreciated!

Thanks!

[hassy]

EMFILE is on OS-level error that indicates that the operating system ran out of file descriptors. This is because of arrivalRate: 1000 in your script - each arrival creates a separate TCP connection, each of which uses a file descriptor, and eventually they run out.

You have two options:

1. Reduce the arrivalRate and use a loop in your scenario to send more requests over the same connection. This will let you reach higher RPS without running out of file descriptors. The reason not to do this would be if you actually need that number of open connections because it represents the load profile of your GraphQL service.
2. Scale out horizontally - run more lambdas with a smaller arrivalRate each. If you can do that it's a great option, and will let you scale your tests up as neeed.

Is there a reason you're using your own load generator Lambda instead of using the built-in run-lambda command btw?

[seungchanlee]

Thanks a lot for the quick response! I will try out the 1st option first.

Why not 'run-lambda'? Our org is very strict on using aws resources especially creating roles/permissions using IAM service. I reviewed how run-lambda works briefly, most likely my assumed role doesn't have all permissions required to run run-lambda. However, eventually, we will try and see if there is a way to use it in our org.

[seungchanlee]

I tried the 1st option - during 30s, arrivalRate 50, 5 posts in a scenario. This way EMFILE error gone, but 2/3 of requests TIMEDOUT .. There are 4 routers running .. Hard to understand why many timeout?? Any thoughts? Thank you.

Summary report @ 15:55:32(+0000) 2025-07-17
Scenarios launched: 1500
Scenarios completed: 531
Requests completed: 3927
Mean response/sec: 90.53
Response time (msec):
min: 28
max: 10358
median: 4160
p95: 9147.2
p99: 9842.5
Scenario counts:
0: 1500 (100%)
Codes:
200: 3927
Errors:
ETIMEDOUT: 969

[hassy]

ETIMEDOUT means the response headers were not received within 10s (the default timeout value).

This usually indicates server overload, but sometimes can be caused by CPU exhaustion in Artillery itself (which can happen if the arrival rate is too high, or the test is CPU-intensive, e.g. parsing huge blobs of JSON for every response). Do you have any visibility into the backend, e.g. via an APM or some other way to monitor resource usage?

From the output there it looks like you may be on a really old version of Artillery. Would definitely recommend upgrading to the latest one.

[seungchanlee]

Thanks a lot @hassy for your kind reply!

We are working on upgrading artillery to the latest.

In the meantime I keep testing, but a couple weird things.
if my config is -
duration: 30
arrivalRate: 100
each arrival call a simple post request.

the result is very normal -
Scenarios launched: 3000
Scenarios completed: 3000
Requests completed: 3000
Mean response/sec: 96.71
Response time (msec):
min: 21
max: 1980
median: 142
p95: 921
p99: 1572.5
Scenario counts:
0: 3000 (100%)
Codes:
200: 3000

When I bumped up arrivalRate to 200, there are lots of errors like EMFILE, ETIMEOUT, EBUSY, etc. So I reduced arrivalRate to 40 and scenario called 5 simple POST as you suggested. This config removed EMFILE and EBUSY, and gave me only ETIMEOUT. So I thought the apollo-router capacity is not enough, so I increased apollo-router instances to 2 .. but the number of ETIMEOUT didn't improve. Even I bumped up to 8 instances, ETIMEOUT stays the same (I increased request timeout to 60s for this test). Any thoughts on this? Could be an old artillery issue? Thank you.

duration: 30
arrivalRate: 40
each arrival call 5 simple post request.
8 apollo-router instances running

Scenarios launched: 1200
Scenarios completed: 1200
Requests completed: 6000
Mean response/sec: 89.53
Response time (msec):
min: 28
max: 19390
median: 4148 (this is about 40 times longer than the working case at the top)
p95: 13688.5
p99: 17533.5
Scenario counts:
0: 1200 (100%)
Codes:
200: 6000

[seungchanlee]

we have a difficulty to upgrade artillery because when we tried the latest, the layer exceeds the limit (250M). Are you storing in s3 or efs for run-lambda?

[hassy]

If you're seeing the same number of ETIMEOUTS after scaling Apollo instances, it could be:

1. Artillery running in the Lambda is overloaded and can't process responses quickly enough. You'd want to look at CPU usage to rule that out.
2. There's a bottleneck somewhere else and not in Apollo - perhaps a proxy or load balancer in front of it, or some other service?

[seungchanlee]

Thank you so much!!

[seungchanlee]

artillery just calls router and there is only one graphql service for testing. So it's very hard to say why response time is very long.

One thing really weird is ... if artillery directly points to the graphql, I don't see any timeout or EMFILE etc. I could do even 1000 arrivalRate in 45s duration - this took nearly 15min until tests were done. Still I don't EMFILE or ETIMEOUT.

[seungchanlee]

I wonder if you could share the minimal artillery node dependencies? the decencies property in package.json. It would be very appreciated! (we are having a hard time to make it work using EFS or DataSync, etc)

[hassy]

run-lambda uses a Docker image (this one - https://gallery.ecr.aws/artilleryio/artillery-worker - same image is also used on Fargate), so there are no size-related issues. Appreciate that it can be difficult to get approvals to deploy new stuff into an AWS account, but if it helps: all permissions are documented here - https://www.artillery.io/docs/load-testing-at-scale/aws-lambda + the CloudFormation stack can be inspected, and the source Dockerfile is here: https://github.com/artilleryio/artillery/blob/main/packages/artillery/lib/platform/aws-ecs/worker/Dockerfile. I don't think trying to run an up-to-date version of Artillery in a classic zip-based Lambda is going to work!

[seungchanlee]

OK thanks a lot!! I will definitely look at the docker image options. Much appreciated!!

[seungchanlee]

We created a docker image based lambda and are running artillery from there. Still we got issues with tons of ETIMEOUT. The ecs services are behind ALB and ECS/ALB/etc all generated from our cfn templates which have been around last couple years so I don't think there are problem with them.

What we eventually want to achieve is ... 1000 users per sec during 10 min. Do you think if we could achieve this using artillery/lambda? Currently with the config below, we need to run 36 ecs tasks to remove all ETIMEOUT from artillery.. This doesn't look all right, does it?
arrivalRate: 100
duration: 300
6 lambda executions
(600 concurrent users per sec)

BTW, we just use one very simple POST request in scenario.

Thank you.

[hassy]

Is there a reason you need 1000 users rather than requests per second? since a scenario is only a single POST request you may be better off using loop with a fixed number of users. Remember that each VU sets up and then tears down a whole new TCP connection + SSL negotiation if it's HTTPS. That's a lot of overhead for just one request.

If you need 1000 users per second - you can definitely do it with run-lambda given enough Lambdas. Not sure about your homegrown set up but in theory you should be able to parallelize that too. It's unlikely you'll be able to do 1000 new users per second from one Lambda.

[seungchanlee]

Thank you for the quick response!

We try to simulate 1000 simultaneous users which will happen eventually out there. However, now we use 100 arrivalRate, lambda is kicked off 6 times from aws lambda console (I think this has the same effect as run-lambda --count 6). So you recommend like 10 arrivalRate, 10 requests in scenario to remove overhead? Thank you,

[seungchanlee]

It helped, but still many ETIMEOUT .. I wonder if about 1000 requests per sec from a single lambda would be considered 'artillery overloaded'? Do you have a recommended load for a lambda execution? I understand that depends on the nature of http request, but our case the request is type of '/about'.