Re: RFC: pgAgent Scheduler Design

From: Andreas Pflug <pgadmin(at)pse-consulting(dot)de>
To: Dave Page <dpage(at)vale-housing(dot)co(dot)uk>
Cc: pgadmin-hackers(at)postgresql(dot)org
Subject: Re: RFC: pgAgent Scheduler Design
Date: 2005-03-06 21:43:59
Message-ID: 422B799F.20600@pse-consulting.de
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgadmin-hackers

Dave Page wrote:

>
>-----Original Message-----
>From: Andreas Pflug [mailto:pgadmin(at)pse-consulting(dot)de]
>Sent: Sun 3/6/2005 1:41 PM
>To: Dave Page
>Cc: pgadmin-hackers(at)postgresql(dot)org
>Subject: Re: [pgadmin-hackers] RFC: pgAgent Scheduler Design
>
>
>
>>>Yes, I'm aware of that issue - and even the current design will have
>>>problems in that jobs may stil run late.
>>>
>>>
>>Better late than never.
>>
>>
>
>Yup - but, an exception... say I have a daily job that sends me a simple report via email. If the system is down for a few days, then I don't want it to run all the old instances of the job on restart (cron won't, the MS Task Scheduler won't - in fact, I can't think of any I've used that do would).
>
I don't completely agree. If a job was stuck unexecuted in the queue for
a while, it should run asap, and the next schedule should be calculated
in the future, i.e. a daily job not executed for 5 days shouldn't be
executed 5x, but only once after pgAgent is up again.

> Anyway, I was thinking that when the agent first starts, it should do something like 'update pga_schedule set nextrun = nextrun where jscactive = true and jscrunning = false' to nudge the update trigger to recalculate the next run dates from that point.
>
No, it should run *once* to get in sync again, if it has become due in
the meantime.

>What I'm not so sure about is how to log the failed jobs in that instance. This should be multi-agent safe.
>
>
Actually they don't log. Think of a weekdaily job, pgAgent down since
Sunday, now up on Friday morningagain. The job was due on Monday
evening, and will run on Friday immediately, new schedule Friday night .
After an agent failure, you'll have to check anyway in which state your
jobs are. I don't think something like "monday job ran on friday,
tue/wed/thur was skipped" would help here. The logging "Job due on
Monday at 2200 was run on Friday 830" should be enough.

>
>
>>Threading is nice, but not a guarantee to be exactly on-time for job starts.
>>
>>
>
>Nope, but except on the most overloaded of systems each job should start within a minute of it's schedule. With the current design, one 6 hour job could completely screw things up for other jobs.
>
>
>
>>-a job that is due to run will be running as soon as possible
>>
>>
>
>Yup.
>
>
>
>>- any instance of pgAgent might be configured to run a job threaded
>>- multiple instance may share the pool of due tasks
>>
>>
>
>Eh? No, I'm advocating 1 thread per job. The main thread queries the db, find 5 jobs due and spawns 5 threads to run them. 1 minute later, regardless of the state of the 5 threads, the main thread checks for new tasks to run and spawns more threads if required. As jobs are finished, their threads simply die.
>
>
This doesn't enable multi agents. There should be a limit on threads per
agent to give other instances a chance.

>The use of multiple agents by the vast majority of people seems unlikely to me - especially given the lack of control over what runs on what agent. In particular, the majority of jobs are likely to be SQL jobs, the distribution of which is pretty much irrelevant anyway as all the hard work is done by the server. I'm not convinced that many people will want to run resource hungry batch jobs that may run on random agent machines.
>
>
Binary jobs (shell jobs) need a system qualifier.

>Can we get some third party opinions on what usage models will be useful please?
>
>

Yesplease.

Regards,
Andreas

In response to

Browse pgadmin-hackers by date

  From Date Subject
Next Message Dave Page 2005-03-07 23:02:51 Re: RFC: pgAgent Scheduler Design
Previous Message Dave Page 2005-03-06 18:40:59 Re: RFC: pgAgent Scheduler Design