Why is the first prewarm container sometimes slower than a pure cold start?

[LucasWang9909]

Hello OpenWhisk maintainers,

I’m observing a curious phenomenon regarding prewarmed containers and cold starts, and I’d appreciate any insight from the community.

Description

I ran two sets of tests, each consisting of 8 invocations of the same action:
• Group A: Always spin up a brand-new prewarm container for each invocation (so effectively never reusing the same container).
• Group B: Always spin up a brand-new container with a typical cold start.

Observed Behavior
1. Group A’s first invocation is slower than Group B’s cold start.
2. However, Group A’s subsequent 7 invocations (still new prewarm containers) are consistently faster than Group B’s cold starts.

Hence, the puzzling result is:

First prewarm container > (longer latency) > cold start

But afterwards, new prewarm containers < (shorter latency) < cold start.

I would have expected a prewarm container to be at least as fast as or faster than a cold start on the first invocation, yet the actual data shows the initial prewarm is slower.

What I Suspect

My current hypothesis is that the prewarm container in OpenWhisk might only partially initialize the runtime (e.g., pulling the base image, starting the container process), but does not complete certain steps (such as /init, code injection, or JIT/loading overheads) until the first actual activation occurs. Thus, the very first time we use a “prewarm” container, we still pay an extra initialization cost, which sometimes ends up being even higher than a straight cold start.

After that, perhaps the system (Docker, cgroup, network, language runtime JIT, etc.) has enough caching or has done some overhead checks, so subsequent prewarm containers spin up quickly and end up faster than a brand-new cold container. This would explain why repeated prewarm containers (from the second to the eighth) become reliably faster.

Questions

Is this expected or documented behavior in OpenWhisk, that the initial prewarm usage might be slower than a pure cold start in some circumstances?

[style95]

I don't think a prewarm start is slower than a cold start, at least not in my experience.
Many factors affect performance, such as the number of inflight actions, the use of the scheduler component, whether the queue is already created (when the scheduler is used), and the performance of the docker, etc.

If you share more about your test setup, I can share the expected performance-impacting points.

[LucasWang9909]

Hi! Thank you for your response!
I did not use the FPCpoolBalancer, so I did not use the scheduler.

Prewarm Duration Discrepancy

Experiment Setup:

I performed five separate experiments.

1. Using ansible to deploy a new openwhisk service every time.
2. Invoke the first action from a Prewarm container.
3. Invoke subsequent actions from the same setup, but using different Prewarm containers.
   

Observations:

1. The first invocation consistently exhibited a higher duration compared to subsequent invocations, even though all were executed from Prewarm containers (not Cold Start or Warm Start).
2. This discrepancy is primarily due to increased initialization time.

Hypotheses

I guess it might have something to do with the File System and Linux Page Cache.
So I deployed Openwhisk one more time and performed five invocations from the same deployment. However, before each invocation, I cleared the system cache using the following command:

sudo sync; echo 3 > /proc/sys/vm/drop_caches

New Observations:

As shown in the figure below , each invocation took similar time ( 1600ms for the ”TN” action), regardless of whether it was the first or a subsequent invocation. This duration matched the first invocation times of the first invocation from the previous experiments.

Prewarm vs. Cold Start

Experiment Setup:

A comparison was made between the Prewarm container’s first invocation duration and the Cold Start duration.

Observations:

Initial observations suggested that Prewarm container initialization could occasionally be slower than Cold Start. However, further experiments revealed that this is not consistently the case. For example, the first Cold Start invocation of the ”TN” action occasionally reached durations of 2000ms or 1800ms, which exceeded the Prewarm container’s first invocation duration. So my previous discovery may only be an accidental phenomenon.

Question

Is my hypothesis correct that the discrepancy in Prewarm container initialization time is caused by the absence of file system caching during the first invocation?

[bdoyle0182]

The first invocation on an invoker will require pulling the runtime image used for the prewarm if don't have tooling to do this before running the invoker and downloading the function / action code from the db. The function code is then cached on the invoker for some period of time / x different functions are attempted to run on it. That is likely what you're seeing.

[style95]

Since he used Ansible to deploy OW, I believe images are already pulled on invoker machines.
When it comes to the difference between two subsequent prewarm starts, along with what Brendan shared above, I also observed that each component needs to create a connection to the other as well.
That's the reason why I added explicit warm-up steps when designing the scheduler.
The warming stage in each component uses a non-existent action, so it is not related to cache, but it does have effect to reduce the initial invocation time.
Since there are less number of components with ShardingPoolBalancer, it might have less impact but I believe network connection would make a difference.

[LucasWang9909]

Hello! Thank you for your attention to this issue and your detailed explanation.

I tried using a new scheduler, like the FPCPoolBalancer and the FunctionPullingContainerProxy/Pool, and conducted several deployments and experiments again. However, I still observed the same issue: the first prewarm invocation is significantly slower, while subsequent invocations from different prewarm containers are faster.

Additionally, I noticed that the duration of subsequent invocations from prewarm containers is slightly longer when using the scheduler compared to when the scheduler is not used.

And I did observe warm-up logs in the system, such as the following:

[2025-01-16T05:12:55.996Z] [INFO] [#tid_sid_warmUp] [FPCInvokerReactive] Warm up scheduler <ByteString@92c1745a size=17 contents="whisk/scheduler/0">.

Does this indicate that the warm-up process is not working as intended?

[style95]

Yes, with the FPCPoolBalancer, a dedicated queue is dynamically created/deleted for each action.
At the very beginning, there would no queue for your action, and the PoolBalancer will try to create a queue. Activations are concurrently sent to the scheduler via Kafka. Once the queue is created, activations will be forwarded to the queue.
Then the scheduler tries to create a container(s) for the given action. The number of containers to be created depends on the number of inflight activations, the average execution time of the action(in the past), etc. The queue(scheduler) considers those factors and periodically decides the number of containers to create(or not).

Once a container is created, it will access the queue to fetch activations and execute them repeatedly.
So the initial invocation is supposed to be slow because there is no a queue yet but subsequent invocation is much faster.
The queue is running for 10 minutes by default even if there is no activation at all. After 10 minutes, it goes to the idle state and after another 10 minutes, it stops. These are configurable and designed to keep them running to avoid too frequent queue creation/deletion.
It's a different aspect from the network-level "warming up".

As you can see, it could theoretically take longer because of the additional network hop, the queue component.
But in the real world, you have many concurrent requests and populating requests into a queue and consuming requests from the queue are working fully asynchronous. FPCPoolBalancer keeps sending requests to a queue while containers keep fetching requests as soon as they finish invocation. With ShardingPoolBalancer, the two parts are combined and performance is severely degraded when multiple actions are invoked. For more information, you can refer to this document and my paper.

Aside from this, how many invokers and prewarm containers did you use?
What is the value of userMemory of your invokers?
How many concurrent activations did you make at a given time?
I think there are still many things that possibly affect your result.

[LucasWang9909]

Thank you once again for your detailed explanation! The diagrams in your document and your paper have indeed given me a deeper understanding of how the scheduler operates.

From your response, may I infer that I should focus more on the system’s concurrency handling capabilities rather than the slower first prewarm invocation? It seems that this initial delay is a necessary trade-off to achieve better concurrency performance, and thus, it is worthwhile in a concurrent context.

In my setup, I used 6 invokers, each with 2 prewarm containers.
The userMemory configuration for each invoker is 512MB.
For testing, I only ran a single activation.