parasweep.dispatchers module

Dispatchers for running parallel jobs.

class parasweep.dispatchers.Dispatcher

Bases: abc.ABC

Abstract base class for dispatchers.

Subclasses should implement the initialize_session, dispatch, and wait_all methods.

initialize_session()

Must be called before dispatching jobs.

dispatch(command, wait)

Dispatch a command using the dispatcher.

Parameters:

• command (str) – The command to dispatch.
• wait (bool) – Whether to wait for the process to finish before returning.

wait_all()

Wait for the running/scheduled processes to finish, then return.

class parasweep.dispatchers.SubprocessDispatcher(max_procs=None)

Bases: parasweep.dispatchers.Dispatcher

Dispatcher using subprocesses.

Parameters:max_procs (int, optional) – The maximum number of processes to run simultaneously. By default, uses the number of processors on the machine (this is a good choice for CPU-bound work).

initialize_session()

Must be called before dispatching jobs.

dispatch(command, wait)

Dispatch a command using the dispatcher.

Parameters:

• command (str) – The command to dispatch.
• wait (bool) – Whether to wait for the process to finish before returning.

wait_all()

Wait for the running/scheduled processes to finish, then return.

class parasweep.dispatchers.DRMAADispatcher(job_template=None)

Bases: parasweep.dispatchers.Dispatcher

Dispatcher for DRMAA.

Parameters:job_template (drmaa.JobTemplate instance, optional) – A job template containing settings for running the jobs with the job scheduler. Documentation for the different options is available in the Python drmaa package. Some options specific to each job scheduler, called the native specification, may have to be set using the job_template.nativeSpecification attribute, the options for which can be found in the job scheduler’s DRMAA interface (e.g., slurm-drmaa for Slurm and pbs-drmaa for PBS).

Examples

>>> import drmaa
>>> jt = drmaa.JobTemplate(hardWallclockTimeLimit=60)
>>> dispatcher = DRMAADispatcher(jt)

session = None

initialize_session()

Must be called before dispatching jobs.

dispatch(command, wait)

Dispatch a command using the dispatcher.

Parameters:

• command (str) – The command to dispatch.
• wait (bool) – Whether to wait for the process to finish before returning.

wait_all()

Wait for the running/scheduled processes to finish, then return.