pyworkflow.protocol.executor module

This module have the classes for execution of protocol steps. The basic one will run steps, one by one, after completion. There is one based on threads to execute steps in parallel using different threads and the last one with MPI processes.

class pyworkflow.protocol.executor.QueueStepExecutor(hostConfig, submitDict, nThreads, **kwargs)

Bases: ThreadStepExecutor

getThreadJobId(stepId)

Returns the job id extension assigned to each thread/step

renameGpuIds()

Reorganize the gpus ids starting from 0 since the queue engine is the one assigning them. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars

runJob(log, programName, params, numberOfMpi=1, numberOfThreads=1, env=None, cwd=None, executable=None)

This function is a wrapper around runJob, providing the host configuration.

class pyworkflow.protocol.executor.StepExecutor(hostConfig, **kwargs)

Bases: object

Run a list of Protocol steps.

getGpuList()

Return the GPU list assigned to current thread.

runJob(log, programName, params, numberOfMpi=1, numberOfThreads=1, env=None, cwd=None, executable=None)

This function is a wrapper around runJob, providing the host configuration.

runSteps(steps, stepStartedCallback, stepFinishedCallback, stepsCheckCallback, stepsCheckSecs=3)

setProtocol(protocol)

Set protocol to append active jobs to its jobIds.

class pyworkflow.protocol.executor.StepThread(step, lock)

Bases: Thread

Thread to run Steps in parallel.

This constructor should always be called with keyword arguments. Arguments are:

group should be None; reserved for future extension when a ThreadGroup class is implemented.

target is the callable object to be invoked by the run() method. Defaults to None, meaning nothing is called.

name is the thread name. By default, a unique name is constructed of the form “Thread-N” where N is a small decimal number.

args is the argument tuple for the target invocation. Defaults to ().

kwargs is a dictionary of keyword arguments for the target invocation. Defaults to {}.

If a subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.__init__()) before doing anything else to the thread.

run()

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

class pyworkflow.protocol.executor.ThreadStepExecutor(hostConfig, nThreads, **kwargs)

Bases: StepExecutor

Run steps in parallel using threads.

cleanVoidGPUs(gpuList)

freeGpusSlot(node)

getFreeGpuSlot(stepId=None)

Returns a free gpu slot available or None. If node is passed it also reserves it for that node

Parameters

node – node to make the reserve of Gpus

getGpuList()

Return the GPU list assigned to current thread or empty list if not using GPUs.

runSteps(steps, stepStartedCallback, stepFinishedCallback, stepsCheckCallback, stepsCheckSecs=5)

Creates threads and synchronize the steps execution.

Parameters

• steps – list of steps to run

• stepStartedCallback – callback to be called before starting any step

• stepFinishedCallback – callback to be run after all steps are done

• stepsCheckCallback – callback to check if there are new steps to add (streaming)

• stepsCheckSecs – seconds between stepsCheckCallback calls