Performance of HiGHS locally vs. on AWS Lambda

[jeroenmeijer92]

Hi,

I am running a script that solves a large number of small optimisation problems (a few 100 variables and constraints per problem) sequentially. Per problem, we initialise the solver using Pyomo (6.9.2):
actual_solver = pyo.SolverFactory(solver_properties["name"])

and we solve:
self.results = actual_solver.solve(self._solver, tee=solver_logging)

Locally, this script performs quite well. However, when deploying this same code to AWS Lambda, the runtime increases by a factor 5-10. It seems that instead of solving a single iteration well under a second, the solving time increases to around 5 seconds.

Now I am wondering where those large differences are coming from.

I have already tried:

• Increasing lambda memory to 10GB, but only a small percentage of memory is used.
• Changing the Lambda architecture from x86_64 to ARM64, but not effect.
• Running the same problem on a Docker container with limited CPU and memory, but I get similar results as running locally.

Additional information:

• Python version: 3.10
• highspy -> 1.11.0
• pyomo -> 6.9.2
• numpy -> 2.0.2
• Local machine: Apple M4 Pro

Would anyone have an idea where those large differences are coming from?

Let me know in case I should provide more information

[galabovaa]

It is unclear whether the issue here is in Pyomo interface or HiGHS. Perhaps this is worth commenting on the Pyomo repo, they may know better. We do have some open highspy issues, but I don't think any one of them would obviously explain the behaviour that you are observing.

[odow]

Would anyone have an idea where those large differences are coming from?

Local machine: Apple M4 Pro

Have you considered that the AWS machine might just be 5-10x slower than your local machine?

[jeroenmeijer92]

That could be the case indeed. I was just curious if there could be other possible clarifications for this that are actually solvable.

[sk-surya]

I've been using HiGHS on AWS lambda over a year and I can confirm it is indeed due to those CPUs being slower. And the runtimes are also not stable, probably depends on how packed the physical hardware is at that moment. But this is expected, as lambdas are not designed to run high performance workloads. They are more for burst and lighter loads.

In my case, there are lot of related LPs to solve, which was slower on lambdas and I had to solve them in parallel batches. When you increase to max memory of 10 GB in a lambda, you get 6 vCPUs.

Btw, lambda doesn't support shared memory parallelism, one will have to use Process and Pipes.