API reference

Executors

class submitit.AutoExecutor(folder, cluster=None, **kwargs)

Bases: Executor

Executor submitting to the best available cluster: Slurm when srun is available, the local executor otherwise. Prefer it to the other executors, so that the same code runs on several clusters.

Parameters:

  • folder (Path | str) – folder for storing job submission/output and logs, %j is replaced by the job id.

  • cluster (str) – executor name: “slurm”, “local” (even on a Slurm cluster), “debug” or a plugin’s (default: detected, see executor.cluster)

  • kwargs (Any) – executor constructor arguments, prefixed by the executor name, e.g. slurm_python for the python argument of SlurmExecutor.

Update parameters:

  • name (str) – job name

  • timeout_min (int) – time limit in minutes (defaults: see each executor)

  • mem_gb (float) – memory per node in GB

  • nodes (int) – number of nodes

  • tasks_per_node (int) – number of tasks per node, each running the submitted function

  • cpus_per_task (int) – number of CPUs per task

  • gpus_per_node (int) – number of GPUs per node

  • stderr_to_stdout (bool) – writes stderr to the stdout log

  • <executor>_<parameter> – parameter of one executor, e.g. slurm_partition (see SlurmExecutor and LocalExecutor), ignored by the other executors. A prefixed generic parameter, e.g. slurm_timeout_min, overrides the generic one on that executor only.

class submitit.Executor(folder, parameters=None)

Base job executor.

Parameters:

folder (Path | str) – folder for storing job submission/output and logs.

batch(allow_implicit_submissions=False)

Creates a context within which all submissions are packed into a job array, submitted when the context exits

Parameters:

allow_implicit_submissions (bool) – submits the current batch whenever a job attribute is accessed instead of raising an exception

Raises:

AttributeError – if trying to access a job instance attribute while the batch is not exited, and intermediate submissions are not allowed.

Example

>>> jobs = []
>>> with executor.batch():
...     for k in range(12):
...         jobs.append(executor.submit(pow, 2, k))
submit(fn, *args, **kwargs)

Submits fn(*args, **kwargs) as a job

map_array(fn, *iterable)

A distributed equivalent of the map() built-in function

Parameters:

  • fn (callable) – function to compute

  • *iterable (Iterable) – lists of arguments that are passed as arguments to fn.

Returns:

A list of Job instances.

Return type:

list[Job]

Example

>>> a = [1, 2, 3]
>>> b = [10, 20, 30]
>>> executor.map_array(add, a, b)  # computes 1 + 10, 2 + 20, 3 + 30
update_parameters(**kwargs)

Updates the parameters of the next submissions, see AutoExecutor, SlurmExecutor and LocalExecutor for the valid ones. An invalid parameter raises an error listing the valid ones.

class submitit.SlurmExecutor(folder, max_num_timeout=3, max_pickle_size_gb=1.0, python=None)

Bases: PicklingExecutor

Executor submitting jobs to Slurm with sbatch.

Parameters:

  • folder (Path | str) – folder for storing job submission/output and logs.

  • max_num_timeout (int) – number of times a callable with a __submitit_checkpoint__ method is requeued after a timeout

  • max_pickle_size_gb (float) – maximum size of the pickled function and arguments of a submission, as large pickles overload the shared filesystem

  • python (str | None) – shell command starting Python in the job, e.g. to run it in a container (default: the interpreter of the submitting process)

Update parameters:

  • job_name (str) – --job-name (generic: name, default: “submitit”)

  • time (int) – --time, in minutes (generic: timeout_min, default: 5)

  • nodes (int) – --nodes (default: 1)

  • ntasks_per_node (int) – --ntasks-per-node (generic: tasks_per_node)

  • cpus_per_task (int) – --cpus-per-task

  • gpus_per_node (int) – --gpus-per-node. On clusters rejecting it (Invalid generic resource (gres) specification), request GPUs through gres instead, e.g. “gpu:2”

  • mem (str) – --mem, e.g. “16GB” (generic: mem_gb)

  • partition, account, qos, constraint, exclude, nodelist, dependency, comment (str) – sbatch option of the same name, with _ replaced by -

  • gres, gpus_per_task, cpus_per_gpu, mem_per_gpu, mem_per_cpu, exclusive, mail_type, mail_user (str) – same as above; cpus_per_gpu requires gpus_per_task rather than gpus_per_node

  • wckey (str) – --wckey (default: “submitit”), set it to “” to remove it if the cluster rejects it (Invalid wckey specification)

  • array_parallelism (int) – maximum number of jobs of a job array running at the same time (default: 256)

  • signal_delay_s (int) – seconds before the time limit at which the job is signaled, for checkpointing (default: 90)

  • setup (list of str) – shell commands run in the sbatch script before the job

  • teardown (list of str) – shell commands run in the sbatch script after the job

  • srun_args (list of str) – extra arguments of the srun call running the job

  • use_srun (bool) – runs the job through srun in the sbatch script, or directly if False (default: True)

  • stderr_to_stdout (bool) – writes stderr to the stdout log

  • additional_parameters (dict) – sbatch options not supported above, e.g. {"reservation": "my_reservation"}, overriding any other value

  • num_gpus (int) – deprecated, use gpus_per_node

class submitit.LocalExecutor(folder, max_num_timeout=3, max_pickle_size_gb=1.0, python=None)

Bases: PicklingExecutor

Executor running jobs as subprocesses of the current machine, one per task, with a timeout.

Parameters:

  • folder (Path | str) – folder for storing job submission/output and logs.

  • max_num_timeout (int) – see SlurmExecutor

  • max_pickle_size_gb (float) – see SlurmExecutor

  • python (str | None) – see SlurmExecutor

Update parameters:

  • timeout_min (int) – time limit in minutes (default: 2)

  • tasks_per_node (int) – number of tasks, each running the submitted function (default: 1)

  • gpus_per_node (int) – sets CUDA_VISIBLE_DEVICES to the first gpus_per_node GPUs (of visible_gpus if provided): GPUs are not shared out between concurrent jobs

  • visible_gpus (Sequence[int]) – ids of the GPUs that jobs can use

  • nodes (int) – must be 1

  • signal_delay_s (int) – seconds before the time limit at which the job is signaled, for checkpointing (default: 30)

  • stderr_to_stdout (bool) – writes stderr to the stdout log

  • setup (list of str) – shell commands run before the job

Note

Through AutoExecutor, only the generic parameters and local_setup are accepted.

class submitit.DebugExecutor(folder)

Bases: Executor

Executor running jobs in the current process, when their result is requested. Breakpoints work, and a failure opens a post-mortem debugger. Parameters are ignored.

Jobs

class submitit.Job(folder, job_id, tasks=(0,))

Access to a cluster job information and result.

Parameters:

  • folder (Path | str) – the folder of the job

  • job_id (str | int) – the id of the cluster job

  • tasks (list[int]) – The ids of the tasks associated to this job. By default, the job has only one task (with id = 0)

property job_id: str
property paths: JobPaths

Paths of the submission, result and log files of the job

property num_tasks: int

Returns the number of tasks in the Job

submission()

Returns the submitted object, with attributes function, args and kwargs

task(task_id)

Returns a given sub-Job (task).

Parameters:

task_id (int) – The id of the task, between 0 and num_tasks - 1

Returns:

The sub_job. You can call all Job methods on it (done, stdout, …) If the job doesn’t have sub jobs, return the job itself.

Return type:

Job

cancel(check=True)

Cancels the job

Parameters:

check (bool) – raises if the cancel command fails

result()

Waits for and outputs the result of the submitted function, for jobs with one task

results()

Waits for and outputs the result of the submitted function

Returns:

the output of the submitted function. If the job has several tasks, it will return the output of every tasks in a List

Return type:

output

Raises:

FailedJobError, UncompletedJobError – see exception()

exception()

Waits for completion and returns (not raise) the exception containing the error log of the job

Returns:

the exception if any was raised during the job. If the job has several tasks, it returns the exception of the task with smallest id that failed: FailedJobError if the function raised, UncompletedJobError if the job ended without result.

Return type:

Exception | None

wait()

Waits until the job is finished, see done()

done(force_check=False)

Checks whether the job is finished. This is done by checking if the result file is present, or checking the job state regularly (up to every 10 minutes on Slurm) If the job has several tasks, the job is done once all tasks are done.

Parameters:

force_check (bool) – Forces the slurm state update

Returns:

whether the job is finished or not

Return type:

bool

Note

A job which failed without writing its result file may be reported as unfinished until the next state check (up to 10 minutes on Slurm), use force_check=True to check the state now.

property state: str

State of the job (does not force an update)

stdout()

Returns a string with the content of the print log file or None if the file does not exist yet

stderr()

Returns a string with the content of the error log file or None if the file does not exist yet

awaitable()

Returns a proxy object that provides asyncio methods for this Job.

class submitit.SlurmJob(folder, job_id, tasks=(0,))

Bases: Job[R]

Job submitted to Slurm, see Job

class submitit.core.utils.JobPaths(folder, job_id=None, task_id=None)

Creates paths related to the slurm job and its submission

property folder: Path
property submission_file: Path
property submitted_pickle: Path
property result_pickle: Path
property stderr: Path
property stdout: Path
class submitit.JobEnvironment(*args)

Describe the environment inside which the job is running. This includes the job id and the rank of the task.

This class can only be instantiated from a running submitit job.

property paths: JobPaths

Provides the paths used by submitit, including stdout, stderr, submitted_pickle and folder.

property hostname: str
property hostnames: Sequence[str]
property job_id: str
property num_tasks: int

Total number of tasks for the job

property num_nodes: int

Total number of nodes for the job

property node: int

Id of the current node

property global_rank: int

Global rank of the task

property local_rank: int

Local rank of the task, ie on the current node.

Helpers

class submitit.helpers.CommandFunction(command, verbose=True, cwd=None, env=None)

Wraps a command as a function returning everything sent to stdout, and raising a FailedJobError with the stderr if the command fails. The output is buffered in memory: for large outputs, use subprocess.run instead.

Parameters:

  • command (list) – command to run, as a list

  • verbose (bool) – prints the command and stdout at runtime

  • cwd (Path | str) – path to the location where the command must run from

  • env (dict) – environment of the command, replacing the inherited one

Returns:

Everything that has been sent to stdout

Return type:

str

class submitit.helpers.DelayedSubmission(function, *args, **kwargs)

The call function(*args, **kwargs), to run later, e.g. returned by __submitit_checkpoint__.

class submitit.helpers.Checkpointable(*args, **kwargs)

Derived callable classes are requeued after preemption or timeout with their current state dumped at checkpoint.

__call__ method must be implemented to make your class a callable.

Note

The following implementation of the __submitit_checkpoint__ method resubmits the full current state of the callable (self) with the initial argument. You may want to replace the method to curate the state (dump a neural network to a standard format and remove it from the state so that it is not pickled) and change/remove the initial parameters.

__submitit_checkpoint__(*args, **kwargs)

Resubmits the same callable with the same arguments

class submitit.helpers.FunctionSequence(*args, **kwargs)

Runs several calls in sequence in one job, and returns their outputs. Finished calls are not run again after requeuing.

Parameters:

verbose (bool) – prints how many functions were already computed when starting

Example

func = FunctionSequence()
func.add(my_function1, arg1, kwarg1=value_kwarg1)
func.add(my_function2, arg1, arg2)
result1, result2 = func()

Note

This function is checkpointable because:

  • it derives from Checkpointable

  • it keeps DelayedSubmission objects as attribute, which in turn store the results of the computation in memory once they are computed. So at checkpoint time, those results will be saved, and only the non-computed results will be computed once the job restarts.

add(func, *args, **kwargs)
submitit.helpers.as_completed(jobs, timeout=None, poll_frequency=10)

Yields jobs as they complete (finished, failed or were cancelled). Raises a TimeoutError if the result isn’t available after timeout seconds. timeout can be an int or float. If timeout is not specified or None, there is no limit to the wait time.

Parameters:

  • jobs (list) – Jobs instances

  • timeout (int | float) – Maximum time (in sec) to wait for jobs completion

  • poll_frequency (float) – seconds between two checks of the job states

Yields:

Job – The next completed job

submitit.helpers.monitor_jobs(jobs, poll_frequency=30, test_mode=False, custom_logging=_default_custom_logging)

Continuously monitors given jobs until they are all done or failed.

Parameters:

  • jobs (list[Job]) – A list of jobs to monitor

  • poll_frequency (int) – The time (in seconds) between two refreshes of the monitoring, at least 30.

  • test_mode (bool) – If in test mode, we do not check the length of poll_frequency

submitit.helpers.clean_env(extra_names=())

Removes slurm and submitit related environment variables so as to avoid interferences when submitting a new job from a job.

Parameters:

extra_names (Sequence[str]) – Additional environment variables to hide inside the context, e.g. TRITON_CACHE_DIR and TORCHINDUCTOR_CACHE_DIR when using torch.compile.

Note

A slurm job submitted from within a slurm job inherits some of its attributes, which can cause gres or PyTorch distributed errors. Submitting within this context should prevent this. SLURM_CONF (cluster configuration location) is kept unless listed in extra_names.

Example

with submitit.helpers.clean_env():
    executor.submit(...)
class submitit.helpers.RsyncSnapshot(snapshot_dir, root_dir=None, with_submodules=False, exclude=(), include=())

Context manager taking a snapshot of the git repository of the current directory (or root_dir), and moving to it: jobs submitted inside run from the snapshot, with the code from when they are submitted, not from when they start or restart.

Requires git and rsync.

Parameters:

  • snapshot_dir (Path) – A path to where the snapshot should be created

  • root_dir (Path) – root of the repository to snapshot (default: the repository of the current directory)

  • with_submodules (bool) – Whether or not submodules should be included in the snapshot

  • exclude (Sequence[str]) – An optional list of patterns to exclude from the snapshot

  • include (Sequence[str]) – A list of relative file names to include in the snapshot. Useful for .so or other build artifacts that are generally not tracked by git.

Note

Only files that are checked in to the repository are included in the snapshot. If you have experimental code that you would like to include in the snapshot, you’ll need to git add the file first for it to be included, or use include.

Example

with submitit.helpers.RsyncSnapshot(snapshot_dir=Path("snapshot")):
    job = executor.submit(train)
class submitit.helpers.TorchDistributedEnvironment

Holds the parameters required to properly setup PyTorch distributed (with the default env:// initialization method).

Examples

>>> dist_env = TorchDistributedEnvironment().export()
>>> torch.distributed.init_process_group(backend="nccl")
>>> print(f"master: {dist_env.master_addr}:{dist_env.master_port}")
export(set_cuda_visible_devices=True, overwrite=False)

Export all the environment variables required to properly setup PyTorch distributed (with the default env:// initialization method) i.e. MASTER_ADDR, MASTER_PORT, RANK, WORLD_SIZE (to which LOCAL_RANK and LOCAL_WORLD_SIZE are added).

Parameters:

  • set_cuda_visible_devices (bool) – if True, updates CUDA_VISIBLE_DEVICES to use only the device matching the local rank.

  • overwrite (bool) – if True, overwrites the environment variables if they exist; this can be useful when launching a job from another job.

Returns:

the current instance

Return type:

TorchDistributedEnvironment

Errors

exception submitit.FailedJobError

The submitted function raised, the job reached its time limit without checkpointing, or the submission was rejected by the cluster

exception submitit.UncompletedJobError

Job ended without result (e.g. out of memory, cancelled, node failure), parent of FailedJobError

exception submitit.core.utils.FailedSubmissionError

The submission command succeeded but its output contained no job id