Http Task not Picked soon

[svelango]

Describe the bug
We are executing 100 Workflows per second in 4 Containers pods with 4 core in each pod. We observe that Http Task is not picked immediately. Updated activeWorkerLastPollTimeout to 2 sec where we saw improvements but not great improvement. Wonder is there any other setting we are missing.

In the above example Task2 is picked after 9 seconds and hence the join took more time.

Details
Conductor version: v3.21.5
Persistence implementation: Cassandra, Postgres, MySQL, Dynomite etc Postgres
Queue implementation: Postgres, MySQL, Dynoqueues etc Postgres
Lock: Redis or Zookeeper?
Workflow definition:
Task definition:
Event handler definition:

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
Http Task pulled immediately and better metrics on pending Http Task Qeueu

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
We are comparing this with AWS Step Functions and this issue is holding us to move forward with Conductor