pyworkflow.protocol.protocol module

This modules contains classes required for the workflow execution and tracking like: Step and Protocol

class pyworkflow.protocol.protocol.FunctionStep(func=None, funcName=None, *funcArgs, wait=False, interactive=False, needsGPU=True)

Bases: Step

This is a Step wrapper around a normal function This class will ease the insertion of Protocol function steps through the function _insertFunctionStep

Params:

func: the function that will be executed. funcName: the name assigned to that function (will be stored) *funcArgs: argument list passed to the function (serialized and stored) **kwargs: extra parameters.

class pyworkflow.protocol.protocol.LegacyProtocol(**kwargs)

Bases: Protocol

Special subclass of Protocol to be used when a protocol class is not found. It means that have been removed or it is in another development branch. In such, we will use the LegacyProtocol to simply store the parameters and inputs/outputs.

classmethod getClassDomain()

Return the Domain class where this Protocol class is defined.

class pyworkflow.protocol.protocol.ProtImportBase(**kwargs)

Bases: Protocol

Base Import protocol

class pyworkflow.protocol.protocol.ProtStreamingBase(**kwargs)

Bases: Protocol

Base protocol to implement streaming protocols. stepsGeneratorStep should be implemented (see its description) and output should be created at the end of the processing Steps created by the stepsGeneratorStep. To avoid concurrency error, when creating the output, do it in a with self._lock: block. Minimum number of threads is 3 and should run in parallel mode.

resumableStepGeneratorStep(ts)

This allow to resume protocols. ts is the time stamp so this stap is alway different form previous exceution

stepsExecutionMode = 1

stepsGeneratorStep()

This step should be implemented by any streaming protocol. It should check its input and when ready conditions are met call the self._insertFunctionStep method.

Returns

None

class pyworkflow.protocol.protocol.Protocol(**kwargs)

Bases: Step

The Protocol is a higher type of Step. It also have the inputs, outputs and other Steps properties, but contains a list of steps that are executed

addSummaryWarning(warningDescription)

Appends the warningDescription param to the list of summaryWarnings. Will be printed in the protocol summary.

allowsDelete(obj)

allowsGpu()

Returns True if this protocol allows GPU computation.

appendJobId(jobId)

Append active jobs to the list

checkSummaryWarnings()

Checks for warnings that we want to tell the user about by adding a warning sign to the run box and a description to the run summary. List of warnings checked: 1. If the folder for this protocol run exists.

citations()

Return a citation message to provide some information to users.

cleanExecutionAttributes()

Clean all the executions attributes

cleanTmp()

Delete all files and subdirectories under Tmp folder.

cleanWorkingDir()

Delete all files and subdirectories related with the protocol

closeMappers()

Close the mappers of all output Sets.

continueFromInteractive()

TODO: REMOVE this function. Check if there is an interactive step and set as finished, this is used now mainly in picking, but we should remove this since is weird for users.

copy(other, copyId=True, excludeInputs=False)

Copies its attributes into the passed protocol

Parameters

• other – protocol instance to copt the attributes to

• copyId – True (default) copies the identifier

• excludeInputs – False (default). If true input attributes are excluded

copyDefinitionAttributes(other)

Copy definition attributes to other protocol.

property cpuTime

Return the sum of all durations of the finished steps

debug(message)

deleteOutput(output)

error(message, redirectStandard=True)

evalExpertLevel(paramName)

Return the expert level evaluation for a param with the given name.

evalParamCondition(paramName)

Eval if the condition of paramName in _definition is satisfied with the current values of the protocol attributes.

evalParamExpertLevel(param)

Return True if the param has an expert level is less than the one for the whole protocol.

findAttributeName(attr2Find)

getCitations(bibTexOutput=False)

classmethod getClassDomain()

Return the Domain class where this Protocol class is defined.

classmethod getClassLabel(prependPackageName=True)

Return a more readable string representing the protocol class

classmethod getClassPackage()

Return the package module to which this protocol belongs. This function will only work, if for the given Domain, the method Domain.getProtocols() has been called once. After calling this method the protocol classes are registered with it Plugin and Domain info.

classmethod getClassPackageName()

classmethod getClassPlugin()

getDbPath()

getDefaultRunName()

getDefinition()

Access the protocol definition.

getDefinitionDict()

Similar to getObjDict, but only for those params that are in the form. This function is used for export protocols as json text file.

getEnumText(paramName)

This function will retrieve the text value of an enum parameter in the definition, taking the actual value in the protocol.

Parameters

paramName – the name of the enum param.

Returns

the string value corresponding to the enum choice.

getFileTag(fn)

getFiles()

getGpuList()

classmethod getHelpText()

Get help text to show in the protocol help button

getHostConfig()

Return the configuration host.

getHostFullName()

Return the full machine name where the protocol is running.

getHostName()

Get the execution host name. This value is only the key of the host in the configuration file.

getInputStatus()

Returns if any input pointer is not ready yet and if there is any pointer to an open set

getJobIds()

Return an iterable list of jobs Ids associated to a running protocol.

getLogPaths()

getLogsAsStrings()

getLogsLastLines(lastLines=None, logFile=0)

Get the last(lastLines) lines of a log file.

:param lastLines, if None, will try ‘PROT_LOGS_LAST_LINES’ env variable, otherwise 20 :param logFile: Log file to take the lines from, default = 0 (std.out). 1 for stdErr.

getMapper()

getNextOutputName(outputPrefix)

Return the name to be used for a new output.

getObjectTag(objName)

getOutputFiles()

Return the output files produced by this protocol. This can be used in web to download results back.

getOutputSuffix(outputPrefix)

Return the suffix to be used for a new output. For example: output3DCoordinates7. It should take into account previous outputs and number with a higher value.

getOutputsSize()

getPackageCitations(bibTexOutput=False)

getParam(paramName)

Return a _definition param give its name.

getParsedMethods()

Get the _methods results and parse possible cites.

getPath(*paths)

Same as _getPath but without underscore.

getPid()

classmethod getPlugin()

classmethod getPluginLogoPath()

getPossibleOutputs()

getProject()

getProtocolsToUpdate()

This function returns a list of protocols ids that need to update their database to launch this protocol (this method is only used when a WORKFLOW is restarted or continued). Actions done here are:

1. Iterate over the main input Pointer of this protocol

   (here, 3 different cases are analyzed):

   A #. When the pointer points to a protocol

   B #. When the pointer points to another object (INDIRECTLY).

   The pointer has an _extended value (new parameters configuration in the protocol)

   C #. When the pointer points to another object (DIRECTLY).

   • The pointer has not an _extended value (old parameters configuration in the protocol)

2. The PROTOCOL to which the pointer points is determined and saved in

   the list

3. If this pointer points to a set (case B and C):

• Iterate over the main attributes of the set - if attribute is a pointer then we add the pointed protocol to the ids list

getQueueParams()

getRelations()

Return the relations created by this protocol.

getRunMode()

Return the mode of execution, either: MODE_RESTART or MODE_RESUME.

getRunName()

getScheduleLog()

getSize()

Returns the size of the folder corresponding to this protocol

getStatusMessage()

Return the status string and if running the steps done.

getStderrLog()

getStdoutLog()

getSteps()

Return the steps.sqlite file under logs directory.

getStepsFile()

Return the steps.sqlite file under logs directory.

getStepsGraph(refresh=True)

Build a graph taking into account the dependencies between steps. In streaming we might find first the createOutputStep (e.g 24) depending on 25

getSubmitDict()

Return a dictionary with the necessary keys to launch the job to a queue system.

classmethod getUrl()

getWorkingDir()

static hasDefinition(cls)

Check if the protocol has some definition. This can help to detect “abstract” protocol that only serve as base for other, not to be instantiated.

hasExpert()

This function checks if the protocol has any expert parameter

hasQueueParams()

hasSummaryWarnings()

info(message, extra=None)

classmethod isBase()

Return True if this Protocol is a base class. Base classes should be marked with _label = None.

classmethod isBeta()

isChild()

Return true if this protocol was invoked from a workflow (another protocol)

isContinued()

Return if running in continue mode (MODE_RESUME).

classmethod isDisabled()

Return True if this Protocol is disabled. Disabled protocols will not be offered in the available protocols.

isInStreaming()

classmethod isInstalled()

classmethod isNewDev()

classmethod isUpdated()

iterDefinitionAttributes()

Iterate over all the attributes from definition.

iterDefinitionSections()

Iterate over all the section of the definition.

iterInputAttributes()

Iterate over the main input parameters of this protocol. Now the input are assumed to be these attribute which are pointers and have no condition.

iterInputPointers()

This function is similar to iterInputAttributes, but it yields all input Pointers, independently if they have value or not.

iterOutputAttributes(outputClass=None, includePossible=False)

Iterate over the outputs produced by this protocol.

legacyCheck()

Hook defined to run some compatibility checks before display the protocol.

loadMappers()

Open mapper connections from previous closed outputs.

loadSteps()

Load the Steps stored in the steps.sqlite file.

makePathsAndClean()

Create the necessary path or clean if in RESTART mode.

makeWorkingDir()

methods()

Return a description about methods about current protocol execution.

modeParallel()

Returns true if steps are run in parallel

modeSerial()

Returns true if steps are run one after another

property numberOfSteps

processImportDict(importDict, importDir)

This function is used when we import a workflow from a json to process or adjust the json data for reproducibility purposes e.g. resolve relative paths Params: importDict: Dict of the protocol that we got from the json importDir: dir of the json we’re importing

removeJobId(jobId)

Remove inactive jobs from the list

requiresGpu()

Return True if this protocol can only be executed in GPU.

run()

Before calling this method, the working dir for the protocol to run should exist.

runJob(program, arguments, **kwargs)

runProtocol(protocol)

Setup another protocol to be run from a workflow.

setAborted()

Abort the protocol, finalize the steps and close all open sets

setFailed(msg)

Set the run failed and close all open sets.

setHostConfig(config)

setHostFullName(hostFullName)

setHostName(hostName)

Set the execution host name (the host key in the config file)

setJobId(jobId)

Reset this list to have the first active job

setJobIds(jobIds)

Reset this list to have a list of active jobs

setMapper(mapper)

Set a new mapper for the protocol to persist state.

setPid(pid)

setProject(project)

setQueueParams(queueParams)

setRunning()

Do not reset the init time in RESUME_MODE

setStepsExecutor(executor=None)

setWorkingDir(path)

property stepsDone

Return the number of steps executed.

stepsExecutionMode = 0

summary()

Return a summary message to provide some information to users.

updateSteps()

After the steps list is modified, this methods will update steps information. It will save the steps list and also the number of steps.

useQueue()

Return True if the protocol should be launched through a queue.

useQueueForProtocol()

This function will return True if the protocol has been set to be launched through a queue

useQueueForSteps()

This function will return True if the protocol has been set to be launched through a queue by steps

usesGpu()

validate()

Check that input parameters are correct. Return a list with errors, if the list is empty, all was ok.

classmethod validateInstallation()

Check if the installation of this protocol is correct. By default, we will check if the protocols’ package provide a validateInstallation function and use it. Returning an empty list means that the installation is correct and there are not errors. If some errors are found, a list with the error messages will be returned.

classmethod validatePackageVersion(varName, errors)

Function to validate the package version specified in configuration file ~/.config/scipion/scipion.conf is among the available options and it is properly installed.

Parameters

• package – the package object (ej: eman2 or relion). Package should contain the following methods: getVersion(), getSupportedVersions()

• varName – the expected environment var containing the path (and version)

• errors – list of strings to add errors if found

warning(message, redirectStandard=True)

warnings()

Return some message warnings that can be errors. User should approve to execute a protocol with warnings.

classmethod worksInStreaming()

class pyworkflow.protocol.protocol.RunJobStep(runJobFunc=None, programName=None, arguments=None, resultFiles=[], **kwargs)

Bases: FunctionStep

This Step will wrapper the commonly used function runJob for launching specific programs with some parameters. The runJob function should be provided by the protocol when inserting a new RunJobStep

Params:

func: the function that will be executed. funcName: the name assigned to that function (will be stored) *funcArgs: argument list passed to the function (serialized and stored) **kwargs: extra parameters.

class pyworkflow.protocol.protocol.Step(interactive=False, needsGPU=True, **kwargs)

Bases: Object

Basic execution unit. It should define its Input, Output and define a run method.

addPrerequisites(*newPrerequisites)

getElapsedTime(default=datetime.timedelta(0))

Return the time that took to run (or the actual running time if still is running )

getError()

getErrorMessage()

getIndex()

getPrerequisites()

getStatus()

isAborted()

isActive()

isFailed()

isFinished()

isInteractive()

isLaunched()

isNew()

isRunning()

isSaved()

isScheduled()

isWaiting()

needsGPU() → bool

run()

Do the job of this step

setAborted()

Set the status to aborted and updates the endTime.

setFailed(msg)

Set the run failed and store an error message.

setFinished()

Set the status to finish updates the end time

setIndex(newIndex)

setInteractive(value)

setPrerequisites(*newPrerequisites)

setRunning()

The the state as STATE_RUNNING and set the init and end times.

setSaved()

Set the status to saved and updated the endTime.

setStatus(value)

class pyworkflow.protocol.protocol.StepSet(filename=None, prefix='', mapperClass=None, **kwargs)

Bases: Set

Special type of Set for storing steps.

pyworkflow.protocol.protocol.getProtocolFromDb(projectPath, protDbPath, protId, chdir=False)

Retrieve the Protocol object from a given .sqlite file and the protocol id.

pyworkflow.protocol.protocol.getUpdatedProtocol(protocol)

Retrieve the updated protocol and close db connections

pyworkflow.protocol.protocol.isProtocolUpToDate(protocol)

Check timestamps between protocol lastModificationDate and the corresponding runs.db timestamp

pyworkflow.protocol.protocol.runProtocolMain(projectPath, protDbPath, protId)

Main entry point when a protocol will be executed. This function should be called when:

scipion runprotocol ...

Parameters

• projectPath – the absolute path to the project directory.

• protDbPath – path to protocol db relative to projectPath

• protId – id of the protocol object in db.