Adding mail support

[aidanheerdegen]

Chatting with @dkhutch about the changes he made to his fork of payu he used in Sweden

https://github.com/dkhutch/payu/tree/tetralith

he added support for mailing the user on error, or when a job (or sequence of jobs) finished.

I think this is a great idea.

[dkhutch]

Following on from this… I would suggest the following tweak to the current version of payu:
qsub has a flag: -m which sends mail to the user. The one I’m interested in is:
qsub -m a <script>
which sends the user an email when the job terminates.
I would like to edit payu so that:
if n_runs == 1:
<add -m a option to pbs_flags>
That way when a job reaches its last iteration, I will receive an email when the job completes.

[aidanheerdegen]

Yes, did mean to change that to n_runs == 1.

An email on error can be generated by submitting a job whose sole purpose is to convey the information that the calling job has failed. This is clumsy, will have to wait until the queued job runs and has limited ability to pass relevant information to the email sent. In this case it is only the job name that appears in the email, and that has limitations (no spaces or special characters).

The body of such an email might look something like this:

PBS Job Id: 17394310.gadi-pbs
Job Name:   payu_error_jobname
Execution terminated
Exit_status=0
resources_used.cpupercent=0
resources_used.cput=00:00:00
resources_used.mem=6764kb
resources_used.ncpus=1
resources_used.vmem=6764kb
resources_used.walltime=00:00:01

[aidanheerdegen]

There is also the option of using mailx on accessdev like so:

ssh accessdev 'echo "something" | mailx -s "subject" aph502'

but this requires either passwordless ssh keys or forwarding an authentication agent to the job. Seems complicated.

[marshallward]

This mail query would only be done for PBS, right?

[dkhutch]

In response to Aidan's suggestion to create a payu_error_jobname script. It might seem clumsy, but I would certainly find it useful. I would simply use the original payu job name as an identifier. When the user receives this email, they will then have to log on and check the run to see what actually happened. But that's what you need... a prompt to log on and check that particular run that has crashed.

[aidanheerdegen]

Yeah the idea would be to make it something like payu_error_$jobname, e.g. payu_error_run_1deg_iaf.

It is good to know that even this limited functionality would be useful.

[aidanheerdegen]

ping @marshallward in case he didn't see this discussion

[aidanheerdegen]

In fact David's fork was also modified to use slurm, and why he wanted to add this to the system payu.

It is also useful to check the slurm scheduler support.

[dkhutch]

The slurm system I used in Sweden had a mail option called 'FAIL', which is pretty handy. Unfortunately the PBS system doesn't seem to have an equivalent.

[aidanheerdegen]

Yeah, and that is what is annoying/difficult about supporting multiple schedulers, allowing for the overlap, and lack thereof.

[marshallward]

raise NotImplementedError in PBS?

[aidanheerdegen]

Will require good documentation, love good docs, hate writing them.