pyworkflow.protocol.protocol module

class pyworkflow.protocol.protocol.FunctionStep(func=None, funcName=None, *funcArgs, **kwargs)

Bases: pyworkflow.protocol.protocol.Step

This is a Step wrapper around a normal function This class will ease the insertion of Protocol function steps through the function _insertFunctionStep

class pyworkflow.protocol.protocol.LegacyProtocol(**kwargs)

Bases: pyworkflow.protocol.protocol.Protocol

Special subclass of Protocol to be used when a protocol class is not found. It means that have been removed or it is in another development branch. In such, we will use the LegacyProtocol to simply store the parameters and inputs/outputs.

class pyworkflow.protocol.protocol.Protocol(**kwargs)

Bases: pyworkflow.protocol.protocol.Step

The Protocol is a higher type of Step. It also have the inputs, outputs and other Steps properties, but contains a list of steps that are executed

addSummaryWarning(warningDescription)

Appends the warningDescription param to the list of summaryWarnings. Will be printed in the protocol summary.

allowsDelete(obj)

allowsGpu()

Returns True if this protocol allows GPU computation.

checkSummaryWarnings()

Checks for warnings that we want to tell the user about by adding a warning sign to the run box and a description to the run summary. List of warnings checked: 1. If the folder for this protocol run exists.

citations()

Return a citation message to provide some information to users.

cleanTmp()

Delete all files and subdirectories under Tmp folder.

closeMappers()

Close the mappers of all output Sets.

continueFromInteractive()

TODO: REMOVE this function. Check if there is an interactive step and set as finished, this is used now mainly in picking, but we should remove this since is weird for users.

copy(other, copyId=True, excludeInputs=False)

Copy all attributes values from one object to the other. The attributes will be created if needed with the corresponding type. Params:

other: the other object from which to make the copy. copyId: if true, the _objId will be also copied. ignoreAttrs: pass a list with attributes names to ignore.

copyDefinitionAttributes(other)

Copy definition attributes to other protocol.

debug(message)

deleteOutput(output)

error(message, redirectStandard=True)

evalExpertLevel(paramName)

Return the expert level evaluation for a param with the given name.

evalParamCondition(paramName)

Eval if the condition of paramName in _definition is satified with the current values of the protocol attributes.

evalParamExpertLevel(param)

Return True if the param has an expert level is less than the one for the whole protocol.

findAttributeName(attr)

getCitations(bibTexOutput=False)

classmethod getClassLabel(prependPackageName=True)

Return a more readable string representing the protocol class

classmethod getClassPackage()

Return the package module to which this protocol belongs

classmethod getClassPackageName()

classmethod getClassPlugin()

getDbPath()

getDefaultRunName()

getDefinition()

Access the protocol definition.

getDefinitionDict()

Similar to getObjDict, but only for those params that are in the form. This function is used for export protocols as json text file.

getEnumText(paramName)

This function will retrieve the text value of an enum parameter in the definition, taking the actual value in the protocol. Params:

paramName: the name of the enum param.

Returns:

the string value corresponding to the enum choice.

getFileTag(fn)

getFiles()

getGpuList()

classmethod getHelpText()

Get help text to show in the protocol help button

getHostConfig()

Return the configuration host.

getHostFullName()

Return the full machine name where the protocol is running.

getHostName()

Get the execution host name. This value is only the key of the host in the configuration file.

getJobId()

Return the jobId associated to a running protocol.

classmethod getLastUpdateVersion()

getLogPaths()

getLogsAsStrings()

getLogsLastLines(lastLines=None)

Get the log last(lastLines) lines

getMapper()

getObjectTag(objName)

getOutputFiles()

Return the output files produced by this protocol. This can be used in web to download or in remote executions to copy results back.

getOutputsSize()

getPackageCitations(bibTexOutput=False)

getParam(paramName)

Return a _definition param give its name.

getParsedMethods()

Get the _methods results and parse possible cites.

getPid()

classmethod getPluginLogoPath()

getProject()

getQueueParams()

getRelations()

Return the relations created by this protocol.

getRunMode()

Return the mode of execution, either: MODE_RESTART or MODE_RESUME.

getRunName()

getStatusMessage()

Return the status string and if running the steps done.

getSteps()

Return the steps.sqlite file under logs directory.

getStepsFile()

Return the steps.sqlite file under logs directory.

getStepsGraph(refresh=True)

Build a graph taking into account the dependencies between steps. In streaming we might find first the createOutputStep (e.g 24) depending on 25

getSubmitDict()

Return a dictionary with the necessary keys to launch the job to a queue system.

getWorkingDir()

static hasDefinition(cls)

Check if the protocol has some definition. This can help to detect “abstract” protocol that only serve as base for other, not to be instantiated.

hasExpert()

This function checks if the protocol has any expert parameter

hasLinkedInputs()

Return if True if some of the input pointers are referring to an output that is not ready yet.

hasSummaryWarnings()

info(message, redirectStandard=True)

inputProtocolDict()

This function returns a dictionary of protocols that need to update their database to launch this protocol (this method is only used when a WORKFLOW is restarted or continued). Actions done here are: 1. Iterate over the main input Pointer of this protocol

(here, 3 different cases are analyzed)

1. When the pointer points to a protocol

2. When the pointer points to another object (INDIRECTLY). - The pointer has an _extended value (new parameters configuration

   in the protocol)

3. When the pointer points to another object (DIRECTLY). - The pointer has not an _extended value (old parameters

   configuration in the protocol)

2. The PROTOCOL to which the pointer points is determined and saved in the dictionary

3. If this pointer points to another object (case B and C): - Iterate over the main attributes of this pointer - if any attribute is a pointer, then we move to PROTOCOL and repeat

   this procedure from step 1

classmethod isBase()

Return True if this Protocol is a base class. Base classes should be marked with _label = None.

isChild()

Return true if this protocol was invoked from a workflow (another protocol)

isContinued()

Return if running in continue mode (MODE_RESUME).

classmethod isDisabled()

Return True if this Protocol is disabled. Disabled protocols will not be offered in the available protocols.

isInStreaming()

classmethod isInstalled()

classmethod isNew()

iterDefinitionAttributes()

Iterate over all the attributes from definition.

iterDefinitionSections()

Iterate over all the section of the definition.

iterInputAttributes()

Iterate over the main input parameters of this protocol. Now the input are assumed to be these attribute which are pointers and have no condition.

iterInputPointers()

This function is similar to iterInputAttributes, but it yields all input Pointers, independently if they have value or not.

iterOutputAttributes(outputClass=None)

Iterate over the outputs produced by this protocol.

legacyCheck()

Hook defined to run some compatibility checks before display the protocol.

loadMappers()

Open mapper connections from previous closed outputs.

loadSteps()

Load the Steps stored in the steps.sqlite file.

makePathsAndClean()

Create the necessary path or clean if in RESTART mode.

methods()

Return a description about methods about current protocol execution.

numberOfSteps

processImportDict(importDict, importDir)

This function is used when we import a workflow from a json to process or adjust the json data for reproducibility purposes e.g. resolve relative paths Params: importDict: Dict of the protocol that we got from the json importDir: dir of the json we’re importing

requiresGpu()

Return True if this protocol can only be executed in GPU.

run()

Before calling this method, the working dir for the protocol to run should exists.

runJob(program, arguments, **kwargs)

runProtocol(protocol)

Setup another protocol to be run from a workflow.

setHostConfig(config)

setHostFullName(hostFullName)

setHostName(hostName)

Set the execution host name (the host key in the config file)

setJobId(jobId)

setMapper(mapper)

Set a new mapper for the protocol to persist state.

setPid(pid)

setProject(project)

setQueueParams(queueParams)

setStepsExecutor(executor=None)

setWorkingDir(path)

stepsDone

Return the number of steps executed.

summary()

Return a summary message to provide some information to users.

updateSteps()

After the steps list is modified, this methods will update steps information. It will save the steps list and also the number of steps.

useQueue()

Return True if the protocol should be launched through a queue.

useQueueForSteps()

This function will return True if the protocol has been set to be launched thorugh a queue by steps

usesGpu()

validate()

Check that input parameters are correct. Return a list with errors, if the list is empty, all was ok.

classmethod validateInstallation()

Check if the installation of this protocol is correct. By default, we will check if the protocols’ package provide a validateInstallation function and use it. Returning an empty list means that the installation is correct and there are not errors. If some errors are found, a list with the error messages will be returned.

classmethod validatePackageVersion(varName, errors)

Function to validate the the package version specified in configuration file ~/.config/scipion/scipion.conf is among the available options and it is properly installed. Params:

package: the package object (ej: eman2 or relion). Package should contain the

following methods: getVersion(), getSupportedVersions()

varName: the expected environment var containing the path (and version) errors: list to added error if found

warning(message, redirectStandard=True)

warnings()

Return some message warnings that can be errors. User should approve to execute a protocol with warnings.

classmethod worksInStreaming()

class pyworkflow.protocol.protocol.RunJobStep(runJobFunc=None, programName=None, arguments=None, resultFiles=[], **kwargs)

Bases: pyworkflow.protocol.protocol.FunctionStep

This Step will wrapper the commonly used function runJob for launching specific programs with some parameters. The runJob function should be provided by the protocol when inserting a new RunJobStep

class pyworkflow.protocol.protocol.Step(**kwargs)

Bases: pyworkflow.object.OrderedObject

Basic execution unit. It should defines its Input, Output and define a run method.

addPrerequisites(*newPrerequisites)

getElapsedTime(default=datetime.timedelta(0))

Return the time that took to run (or the actual running time if still is running )

getError()

getErrorMessage()

getIndex()

getPrerequisites()

getStatus()

isAborted()

isActive()

isFailed()

isFinished()

isInteractive()

isLaunched()

isRunning()

isSaved()

isScheduled()

isWaiting()

run()

Do the job of this step

setAborted()

Set the status to aborted and updated the endTime.

setFailed(msg)

Set the run failed and store an error message.

setIndex(newIndex)

setInteractive(value)

setPrerequisites(*newPrerequisites)

setRunning()

The the state as STATE_RUNNING and set the init and end times.

setStatus(value)

class pyworkflow.protocol.protocol.StepSet(filename=None, prefix='', mapperClass=None, **kwargs)

Bases: pyworkflow.object.Set

Special type of Set for storing steps.

exception pyworkflow.protocol.protocol.ValidationException

Bases: exceptions.Exception

pyworkflow.protocol.protocol.getProtocolFromDb(projectPath, protDbPath, protId, chdir=False)

Retrieve the Protocol object from a given .sqlite file and the protocol id.

pyworkflow.protocol.protocol.getUpdatedProtocol(protocol)

Retrieve the updated protocol and close db connections

pyworkflow.protocol.protocol.isProtocolUpToDate(protocol)

Check timestamps between protocol lastModificationDate and the corresponding runs.db timestamp

pyworkflow.protocol.protocol.runProtocolMain(projectPath, protDbPath, protId)

Main entry point when a protocol will be executed. This function should be called when: scipion runprotocol … Params:

projectPath: the absolute path to the project directory. protDbPath: path to protocol db relative to projectPath protId: id of the protocol object in db.

pyworkflow.protocol.protocol.runProtocolMainMPI(projectPath, protDbPath, protId, mpiComm)

This function only should be called after enter in runProtocolMain and the proper MPI scripts have been started…so no validations will be made.