active sessions

[lakosm]

Hi,

I have now 2 workers. I initiated 10 multiple chains using self destruction and job_max_instances = 1.
After the 10 chain finished their run, I see 10 active pg_timetable sessions in pg_stat_activity and timetable.active_session contains 39 rows.
When will be the 10 active session idle? Because, I suspest, if I want schedule another 10 chains, may active sessions in pg_stat_activity will grow up to 20, which is not suitable for my 4 cpu db server.

Thanks in advance!

Regards,
Miklos

UPDATE: when I launced the next 10 chains, the active sessions were gone from pg_stat_activity.
But I would like to know, what the rule is regarding this

[pashagolub]

Hi,

assuming when we are speaking about 2 workers we are talking about 2 pg_timetable instances.

Each pg_timetable instance can use max --cron-workers + --interval-workers parallel routines according to the manual.

In the worst scenario (highest load) we need:

• separate connection for each of routines,
• separate connection for Scheduler.retrieveChainsAndRun(),
• separate connection for Scheduler.retrieveIntervalChainsAndRun()
• and another connection for LogHook.send().

That means the max connection (session) number for one pg_timetable instance (worker) would be:

	max_conn = --cron-workers + --interval-workers + 3

In other words, if you want to limit the number of the database sessions you should decrease the --cron-workers and/or --interval-workers start parameters (default value 16 for both).

job_max_instances applies only to the chain. Since you have self-destructed chains it does nothing. This parameter allows to run the same chain in parallel but is limited to the job_max_instances value, e.g. I want to start every minute some chain to process smth but no more than 10 chains at once. When the old chain finishes system will start a new one, so, at any point in time, there will be no more than job_max_instances running chains.

I hope I explained clearly enough. :-)

[lakosm]

Hi,

Wow, thank you for your detailed explaination. Now, I see.
I had issues with the default null job_max_instances (pk violation) so this is why I set it to 1.

[pashagolub]

You are more than welcome.

Strange, timetable.chain.max_instances INTEGER... It can be NULL, which means unlimited. More details...

What version are you running?

We should probably add a statement about the NULL value to make things more clear.

[lakosm]

I am using pg_timetable_4.6.1 and postgresql 13.3
Yesterday I got some "Message": "duplicate key value violates unique constraint errors. After I modified the job_max_instances to 1, the error disappeared.
I regularly check timetable.log table for errors

[DaskalovBG]

Hi,

I am trying to utilize pgtimetable on a semi-loaded vanilla postgres. I have 65 procedures running independent from one another and in addition to this I have only 3 chains running 2 procedures each chain (need to honor some dependency).

My server have some 16 CPUs and 96GB of RAM.

What I observe is as if my procedures keep piling up. I only have around 50 to 60 active sessions when I query the active_session table and whenever I check pg_stat_activity there is only 1 instance of a session form pgtimetable.

My goal is to run all these 65 procs on a minute basis (each proc takes between 5 and 15 seconds to finish). But I seem to wait for workers to free up to pick up the next run.

Max current workers is set to 900, session keep staying between 50 and 60.

Would you mind helping me orientate in the given situation ?

[pashagolub]

Nope, pg_timetable will catch modifications up.
You can check log output or TABLE timetable.active_chain

[DaskalovBG]

Interesting, by the looks of it, I have achieved what I wanted, but for the past hour only 60% of my procedures ran, as if pg_timetable never reached them, while procs which are in these 60% are executed a number of times withing the given period per minute. (I am running 900 workers and 900 interval-workers).

There is 1 procedure which takes around 30 minutes to complete, for that time noone else seems to be working (no other procs are executed). Task is added with job_exclusive => FALSE, and FALSE -- exclusive_execution

Any ideas ?

[pashagolub]

Only one. You might want to restart pg_timetable because of piled up exclusive chains. New will be the normal but previous are still exclusive in queue

[DaskalovBG]

this thing went crazy!

Thank you for the immense support Pasha!

[pashagolub]

You're welcome! :)