Schedulers
Bioluigi extend the scope of Luigi by interfacing external schedulers such as Slurm to dispatch the execution of external programs.
The simplest way of specifying a scheduler is via the configuration so that it become the default for any scheduled external programs.
[bioluigi]
scheduler=slurm
scheduler_extra_args=
The second option is to set it explicitly when invoking a task.
bcftools.Annotate(scheduler='slurm')
Local
The local scheduler performs the execution within the context of a Luigi
worker and consumes two kind of resources: cpus
and memory
. These
resources must be specified in luigi.cfg
.
[bioluigi]
cpus=16
memory=32
This scheduler is used by default.
Slurm
Unlike the local scheduler, no resource allocation is performed via Luigi
resource mechanism. Instead, two resources are consumed: slurm_jobs
and
slurm_cpus
to respectively control how many jobs and CPUs can be allocated.
[bioluigi]
slurm_jobs=32
slurm_cpus=256
Note
Each scheduled external program consume a mostly idle Luigi worker and for
concurrency to be achieved, many of them have to be specified with the
--workers
flag.
luigi --module tasks --workers 32 <task> <task_args>