We have spent the past few days tracking down an issue in which LSF-DRMAA wasn't working on our cluster. In our cluster, bsub -N and bsub -B are disabled, and so trying to submit a job with SUB_NOTIFY_BEGIN or SUB_NOTIFY_END options set results in a failure.
We have finally tracked the issue down to lsf-drmaa. The hint was that we are able to submit jobs as long as output_path is not set. If, however, output_path is set (i.e. by calling setOutputPath()), then the job submissions fail.
The code at fault appears to be: https://github.com/PlatformLSF/lsf-drmaa/blob/master/lsf_drmaa/job.c#L833-841
I'm not sure what the intention of this code block is, but the end result it that if the conditions are met, it sets the SUB_NOTIFY_END option (even if DRMAA_BLOCK_EMAIL is set and resulted in the block before (https://github.com/PlatformLSF/lsf-drmaa/blob/master/lsf_drmaa/job.c#L803-831) making sure that SUB_NOTIFY_END is not set!
The if condition basically says if the session's prepand_report_to_output boolean is false, and the SUB_NOTIFY_END option is not set, and the output_path is not null:
if( !((lsfdrmaa_session_t*)session)->prepand_report_to_output
&& (req->options & SUB_NOTIFY_END) == 0
&& output_path != NULL )The session variable prepand_report_to_output is set from the configuration file in session.c and it defaults to false. It is set from lsf_drmaa.conf by the (undocumented) prepand_report configuration directive.
We are able to work around our issue by setting:
prepand_report: 1 in lsf_drmaa.conf
I don't know what prepand_report is meant to do (is that a misspelling of "prepend" or is it meant to be "prep_and_report"?) and cannot find any documentation on it, but the default behaviour of lsf-drmaa in which SUB_NOTIFY_END is forced to be set even when email is specifically blocked doesn't seem right.
We have spent the past few days tracking down an issue in which LSF-DRMAA wasn't working on our cluster. In our cluster,
bsub -Nandbsub -Bare disabled, and so trying to submit a job withSUB_NOTIFY_BEGINorSUB_NOTIFY_ENDoptions set results in a failure.We have finally tracked the issue down to lsf-drmaa. The hint was that we are able to submit jobs as long as
output_pathis not set. If, however,output_pathis set (i.e. by callingsetOutputPath()), then the job submissions fail.The code at fault appears to be: https://github.com/PlatformLSF/lsf-drmaa/blob/master/lsf_drmaa/job.c#L833-841
I'm not sure what the intention of this code block is, but the end result it that if the conditions are met, it sets the
SUB_NOTIFY_ENDoption (even ifDRMAA_BLOCK_EMAILis set and resulted in the block before (https://github.com/PlatformLSF/lsf-drmaa/blob/master/lsf_drmaa/job.c#L803-831) making sure thatSUB_NOTIFY_ENDis not set!The if condition basically says if the session's
prepand_report_to_outputboolean is false, and theSUB_NOTIFY_ENDoption is not set, and theoutput_pathis not null:The session variable
prepand_report_to_outputis set from the configuration file insession.cand it defaults tofalse. It is set fromlsf_drmaa.confby the (undocumented)prepand_reportconfiguration directive.We are able to work around our issue by setting:
prepand_report: 1inlsf_drmaa.confI don't know what
prepand_reportis meant to do (is that a misspelling of "prepend" or is it meant to be "prep_and_report"?) and cannot find any documentation on it, but the default behaviour of lsf-drmaa in whichSUB_NOTIFY_ENDis forced to be set even when email is specifically blocked doesn't seem right.