Example for batch is not giving expected results

[mastizada]

I have created a test function to test receiving batch tasks:

@tiger.task(batch=True, queue="test_queue")
def test_task(kwargs):
    print("-------")
    print(kwargs)
    print("--------")

For tiger configuration I have same for both sender and task definition:

tiger = tasktiger.TaskTiger(connection=conn, config={
    'BATCH_QUEUES': {
        'test_queue': 50
    },
})

When executing PYTHONPATH=. tasktiger it prints tasks one-by-one instead of 1 array with the list of args and kwargs.

Also it will be needed to add PYTHONPATH to the supervisor configuration also, which is not mentioned in "Documentation".

Thanks.

[thomasst]

Can you give more detailed steps to reproduce, including the exact output you're seeing?

If there are multiple tasks in the queue at the time you start TaskTiger you should be seeing an array of tasks. This is tested also in tests/test_base.py starting with test_batch_1.

[mrsndmn]

I have the same issue.

tt/batch.py

from tasktiger import TaskTiger
import redis

redis_conn = redis.Redis(db=0, decode_responses=True, host="192.168.48.2")
tiger = TaskTiger(connection=redis_conn, config={
    'BATCH_QUEUES': {
        'batch': 10,
    }
})

@tiger.task(queue='batch', batch=True)
def batch_task(params):
    for task in tiger.current_tasks:
        print("processing task:", task.id)
print("got batch task", params)

Clean redis version=4.0.6 in docker

Starting worker

PYTHONPATH=. tasktiger -h 192.168.48.2 -m tt.batch

And trying to add a lot of tasks

import tt.batch
for i in range(10): tt.batch.tiger.delay(tt.batch.batch_task, args=[1])

But in workers logs all the tasks were processed one by one

{"pid": 14772, "queue": "batch", "attempted": 1, "processed": 1, "event": "processed", "level": "debug", "timestamp": "2019-05-16T21:36:43.222817Z"}                        
{"pid": 14772, "queue": "batch", "task_id": "9406f3e974df447832257e0c285d9b350fdbe495c4f5d462220a640df5a9badc", "event": "done", "level": "info", "timestamp": "2019-05-16T21:36:43.223720Z"}
{"pid": 14772, "queue": "batch", "src_queue": "queued", "dest_queue": "active", "qty": 1, "event": "moved tasks", "level": "debug", "timestamp": "2019-05-16T21:36:43.224228Z"}
{"pid": 14772, "queue": "batch", "child_pid": 14784, "func": "tt.batch:batch_task", "task_id": "39f7c2480da693c676cc5015c18044b2b21e407538b3c2c4ba194d7007ba9aa3", "params":
{"args": [1], "kwargs": {}}, "event": "processing", "level": "info", "timestamp": "2019-05-16T21:36:43.225213Z"}                                                            
processing task: 39f7c2480da693c676cc5015c18044b2b21e407538b3c2c4ba194d7007ba9aa3
got batch task [{'args': [1], 'kwargs': {}}]
{"pid": 14772, "queue": "batch", "attempted": 1, "processed": 1, "event": "processed", "level": "debug", "timestamp": "2019-05-16T21:36:43.227326Z"}                        
{"pid": 14772, "queue": "batch", "task_id": "39f7c2480da693c676cc5015c18044b2b21e407538b3c2c4ba194d7007ba9aa3", "event": "done", "level": "info", "timestamp": "2019-05-16T21:36:43.230271Z"}
{"pid": 14772, "queue": "batch", "src_queue": "queued", "dest_queue": "active", "qty": 1, "event": "moved tasks", "level": "debug", "timestamp": "2019-05-16T21:36:43.232245Z"}
{"pid": 14772, "queue": "batch", "child_pid": 14785, "func": "tt.batch:batch_task", "task_id": "1933ac29beb5ee7013a81e910bcaa38bc0524b8d4d3556c418589e7c09ed8a74", "params":
{"args": [1], "kwargs": {}}, "event": "processing", "level": "info", "timestamp": "2019-05-16T21:36:43.233338Z"}                                                            
processing task: 1933ac29beb5ee7013a81e910bcaa38bc0524b8d4d3556c418589e7c09ed8a74
got batch task [{'args': [1], 'kwargs': {}}]

[mrsndmn]

I should run the worker in this way:

tt.batch.tiger.run_worker()

Now it works)

In my opinion this point was not clear after reading the documentation. Also after I looked through the sources and tests, I found interesting configuration params: "SELECT_TIMEOUT", "SELECT_BATCH_TIMEOUT"

[sschroed99]

I should run the worker in this way:

tt.batch.tiger.run_worker()

Now it works)

In my opinion this point was not clear after reading the documentation. Also after I looked through the sources and tests, I found interesting configuration params: "SELECT_TIMEOUT", "SELECT_BATCH_TIMEOUT"

@mrsndmn I am experimenting with this, you ran tt.batch.tiger.run_worker() from the cli or your script?