scipion.install.funcs module

class scipion.install.funcs.Command(env, cmd, targets=None, **kwargs)[source]

Bases: object

execute()[source]

class scipion.install.funcs.CommandDef(cmd: str, targets: list = [])[source]

Bases: object

Basic command class to hold the command string and the targets

Constructor

e.g.: Command(“git clone …/myrepo”, “myrepo”)

Parameters

cmd – String with the command/s to run.
targets – Optional, a list or a string with file/s or folder/s that should exist as a consequence of the commands.

addTarget(targets: list)[source]: Centralized internal method to add targets. They could be a list of string commands or a single command

append(newCmd: str, targets=None, sep='&&')[source]

Appends an extra command to the existing one.

Parameters

newCmd – New command to append
targets – Optional, additional targets in case this command produce them
sep – Optional, separator used between the existing command and this new added one. (&&)

:return itself Command

cd(folder)[source]

Appends a cd command to the existing one

Parameters: folder – folder to changes director to

getCommands() → list[source]: Returns the commands

isEmpty()[source]

new(cmd='', targets=None)[source]: Creates a new command element becoming the current command to do appends on it

touch(fileName, isTarget=True)[source]

Appends a touch command and its target based on the fileName

Parameters: fileName – file to touch. Should be created in the binary home folder. Use ../ in case of a previous cd command
Returns: CondaCommandDef (self)

class scipion.install.funcs.CondaCommandDef(envName, condaActivationCmd='')[source]

Bases: CommandDef

Extends CommandDef with some conda specific methods

Constructor

e.g.: Command(“git clone …/myrepo”, “myrepo”)

Parameters

cmd – String with the command/s to run.
targets – Optional, a list or a string with file/s or folder/s that should exist as a consequence of the commands.

ENV_CREATED = 'env-created.txt'

activate(appendCondaActivation=False)[source]

Activates the conda environment

Parameters: appendCondaActivation – Pass true to prepend the conda activation command

condaInstall(packages)[source]: Appends conda install to the existing command adding packages

create(extraCmds='', yml=None)[source]

Creates a conda environment with extra commands if passed

Parameters: extraCmds – additional commands (string) after the conda create -n envName
Returns: CondaCommandDef (self)

pipInstall(packages)[source]: Appends pip install to the existing command adding packages

class scipion.install.funcs.Environment(**kwargs)[source]

Bases: object

addLibrary(name, **kwargs)[source]

Add library <name> to the construction process.

Checks that the needed programs are in PATH, needed libraries can be found, downloads the given url, untars the resulting tar file, configures the library with the given flags, compiles it (in the given buildDir) and installs it.

If default=False, the library will not be built.

Returns the final targets, the ones that Make will create.

addPackage(name, **kwargs)[source]

Download a package tgz, untar it and create a link in software/em. Params in kwargs:

Parameters

tar – the package tar file, by default the name + .tgz. Pass None or VOID_TGZ if there is no tar file.
commands – a list with actions to be executed to install the package
buildDir – Optional folder where build/extraction happens. If not passed will be inferred from tgz’s name
neededProgs – Optional, list of programs needed. E.g: make, cmake,…
version – Optional, version of the package.
libChecks – Optional, a list of the libraries needed. E.g: libjpeg62, gsl (GSL - GNU Scientific Library)

addPipModule(name, version='', pipCmd=None, target=None, default=True, deps=[])[source]

Add a new module to our built Python. Params in kwargs:

Parameters

name – pip module name
version – module version - must be specified to prevent undesired updates.
default – Optional. True if the module has to be installed right after the installation/update of the plugin.

:returns target containing the pip module definition

addTarget(name, *commands, **kwargs)[source]

addTargetAlias(name, alias)[source]: Add an alias to an existing target. This function will be used for installing the last version of each package.

execute()[source]

static getBin(name)[source]

static getBinFolder(*paths)[source]

static getEm(name)[source]

static getEmFolder()[source]

static getIncludeFolder()[source]

getLib(name)[source]

static getLibFolder(*paths)[source]

getLibSuffix()[source]

static getLogFolder(*path)[source]

getPackage(name)[source]

getPackages()[source]: Return all plugin packages

getProcessors()[source]

static getPython()[source]

static getPythonPackagesFolder()[source]

static getSoftware(*paths)[source]

getTarget(name)[source]

getTargetList()[source]

getTargets()[source]

static getTmpFolder()[source]

hasPackage(name)[source]: Returns true if it has the package

hasTarget(name)[source]

printHelp()[source]

setDefault(default)[source]: Set default values of all packages to the passed parameter

updateCudaEnviron(package)[source]: Update the environment adding CUDA_LIB and/or CUDA_BIN to support packages that uses CUDA. package: package that needs CUDA to compile.

class scipion.install.funcs.InstallHelper(packageName: str, packageHome: Optional[str] = None, packageVersion: str = '1.0')[source]

Bases: object

### This class is intended to be used to ease the plugin installation process.

#### Usage: InstallHelper class needs to be instanciated before it can be used. After that, commands can be chained together to run them in the defined order. The last command always needs to be addPackage().

#### Example: installer = InstallHelper() # Instanciating class

installer.getCloneCommand(‘test-package’, ‘/home/user/myCustomPath’, ‘github.com/myRepo’) # Cloning GitHub repository

installer.getCondaenvCommand(‘test-package’) # Creating conda enviroment

installer.addPackage(env, ‘test-package’) # Install package

#### It can also be done in a single line: installer.getCloneCommand(‘test-package’, ‘/home/user/myCustomPath’, ‘github.com/myRepo’).getCondaenvCommand(‘test-package’).addPackage(env, ‘test-package’)

#### If you want to check the command strings you are producing, use the function getCommandList() instead of addPackage() and assign it to a variable so you can print it.

### Constructor for the InstallHelper class.

#### Parameters: packageName (str): Name of the package. packageHome (str): Optional. Path to the package. It can be absolute or relative to current directory. packageVersion (str): Optional. Package version.

DEFAULT_VERSION = '1.0'

addCommand(command: str, targetName: str = '', workDir: str = '')[source]

### This function adds the given command with target to the command list. ### The target file needs to be located inside packageHome’s directory so Scipion can detect it.

#### Parameters: command (str): Command to be added. targetName (str): Optional. Name of the target file to be produced after commands are completed successfully. workDir (str): Optional. Directory where the command will be executed from.

#### Usage: installer.addCommand(‘python3 myScript.py’, targetName=’MYSCRIPT_COMPLETED’, workDir=’/home/user/Documents/otherDirectory’)

#### This function call will generate the following commands: cd /home/user/Documents/otherDirectory && python3 myScript.py && touch /home/user/scipion/software/em/test-package-1.0/MYSCRIPT_COMPLETED

addCommands(commandList: List[str], binaryName: Optional[str] = None, workDir: str = '', targetNames: List[str] = [])[source]

### This function adds the given commands with targets to the command list.

#### Parameters: commandList (list[str]): List containing the commands to add. binaryName (str): Optional. Name of the binary. Default is package name. workDir (str): Optional. Directory where the commands will be executed from. targetNames (list[str]): Optional. List containing the name of the target files for this commands.

#### Usage: installer.addCommands([‘python3 myScript.py’, ‘ls’], binaryName=’myBinary’, workDir=’/home/user/Documents/otherDirectory’,

targetNames=[‘MYSCRIPT_COMPLETED’, ‘DIRECTORY_LISTED’])

#### This function call will generate the following commands: cd /home/user/Documents/otherDirectory && python3 myScript.py && touch /home/user/scipion/software/em/test-package-1.0/MYSCRIPT_COMPLETED

cd /home/user/Documents/otherDirectory && ls && touch /home/user/scipion/software/em/test-package-1.0/DIRECTORY_LISTED

addCondaPackages(packages: List[str], binaryName: Optional[str] = None, binaryVersion: Optional[str] = None, channel: Optional[str] = None, targetName: Optional[str] = None)[source]

### This function returns the command used for installing extra packages in a conda enviroment.

#### Parameters: binaryName (str): Name of the binary. Default is package name. packages (list[str]): List of conda packages to install. binaryVersion (str): Optional. Binary’s version. Default is package version. channel (str): Optional. Channel to download the package from. targetName (str): Optional. Name of the target file for this command.

#### Usage: installer.addCondaPackages(packages=[‘pytorch==1.1.0’, ‘cudatoolkit=10.0’], binaryName=’myBinary’,

binaryVersion=‘1.5’, channel=’conda-forge’, targetName=’CONDA_PACKAGES_INSTALLED’)

#### This function call will generate the following command: eval “$(/home/user/miniconda/bin/conda shell.bash hook)”&& conda activate myBinary-1.5 && conda install -y pytorch==1.1.0 cudatoolkit=10.0 -c conda-forge && touch CONDA_PACKAGES_INSTALLED #### The path in the first command (eval …) might vary, depending on the value of CONDA_ACTIVATION_CMD in your scipion.conf file.

addPackage(env, dependencies: List[str] = [], default: bool = True, **kwargs)[source]

### This function adds the given package to scipion installation with some provided parameters.

#### Parameters: env: Scipion enviroment. dependencies (list[str]): Optional. List of dependencies the package has. default (bool): Optional. Defines if this package version is automatically installed with the plugin. **kwargs: Optional. Other possible keyword parameters that will be directly passed to env.addPackage. Intended for cases where multiple versions of the same package coexist in the same plugin.

#### Usage: installer.addPackage(env, dependencies=[‘wget’, ‘conda’], default=True)

getCloneCommand(url: str, binaryFolderName: str = '', targeName: Optional[str] = None)[source]

### This function creates the neccessary command to clone a repository from Github.

#### Parameters: url (str): URL to the git repository. binaryFolderName (str): Optional. Name of the binary directory. targetName (str): Optional. Name of the target file for this command.

#### Usage: installer.getCloneCommand(‘https://github.com/myRepo.git’, binaryFolderName=’myCustomBinary’, targeName=’BINARY_CLONED’)

#### This function call will generate the following command: cd /home/user/scipion/software/em/test-package-1.0 && git clone https://github.com/myRepo.git myCustomBinary && touch BINARY_CLONED

getCommandList() → List[Tuple[str, str]][source]

### This function returns the list of commands with targets for debugging purposes or to export into another install helper.

#### Returns: (list[tuple[str, str]]): Command list with target files.

#### Usage: commandList = installer.getCommandList()

getCondaEnvCommand(binaryName: Optional[str] = None, binaryPath: Optional[str] = None, binaryVersion: Optional[str] = None, pythonVersion: Optional[str] = None, requirementsFile: bool = False, requirementFileName: str = 'requirements.txt', requirementList: List[str] = [], extraCommands: List[str] = [], targetName: Optional[str] = None)[source]

### This function creates the command string for creating a Conda enviroment and installing required dependencies for a given binary inside a package.

#### Parameters: binaryName (str): Optional. Name of the binary. Default is package name. binaryPath (str): Optional. Path to the binary. It can be absolute or relative to current directory. binaryVersion (str): Optional. Binary’s version. Default is package version. pythonVersion (str): Optional. Python version needed for the package. requirementsFile (bool): Optional. Defines if a Python requirements file exists. requirementFileName (bool): Optional. Name of the Python requirements file. requirementList (list[str]): Optional. List of Python packages to be installed. Can be used together with requirements file, but packages cannot be repeated. extraCommands (list[str]): Optional. List of extra conda-related commands to execute within the conda enviroment. targetName (str): Optional. Name of the target file for this command.

#### Usage: installer.getCondaEnvCommand(binaryName=’myBinary’, binaryPath=’/home/user/scipion/software/em/test-package-1.0/myBinary’, binaryVersion=‘1.5’, pythonVersion=‘3.11’,

requirementsFile=True, requirementFileName=’requirements.txt’, requirementList=[‘torch==1.2.0’, ‘numpy’], extraCommands=[‘conda info –envs’], targetName=’CONDA_ENV_CREATED’)

#### This function call will generate the following command: eval “$(/home/user/miniconda/bin/conda shell.bash hook)”&& conda create -y -n myBinary-1.5 python=3.11 && conda activate myBinary-1.5 && cd /home/user/scipion/software/em/test-package-1.0/myBinary && conda install pip -y && $CONDA_PREFIX/bin/pip install -r requirements.txt && $CONDA_PREFIX/bin/pip install torch==1.2.0 numpyconda info –envs && cd /home/user/scipion/software/em/test-package-1.0 && touch CONDA_ENV_CREATED #### The path in the first command (eval …) might vary, depending on the value of CONDA_ACTIVATION_CMD in your scipion.conf file.

getExtraFile(url: str, targetName: str = '', location: str = '.', workDir: str = '', fileName: Optional[str] = None)[source]

### This function creates the command to download with wget the file in the given link into the given path. ### The downloaded file will overwrite a local one if they have the same name. ### This is done to overwrite potential corrupt files whose download was not fully completed.

#### Parameters: url (str): URL of the resource to download. targetName (str): Optional. Name of the target file for this command. location (str): Optional. Location where the file will be downloaded. It can be absolute or relative to current directory. workDir (str): Optional. Directory where the file will be downloaded from. fileName (str): Optional. Name of the file after the download. Use intended for cases when expected name differs from url name.

#### Usage: installer.getExtraFile(‘https://site.com/myfile.tar’, targetName=’FILE_DOWNLOADED’, location=’/home/user/scipion/software/em/test-package-1.0/subdirectory’, workDir=’/home/user’, fileName=’test.tar’)

#### This function call will generate the following command: cd /home/user && mkdir -p /home/user/scipion/software/em/test-package-1.0/subdirectory && wget -O /home/user/scipion/software/em/test-package-1.0/subdirectory/test.tar https://site.com/myfile.tar && touch /home/user/scipion/software/em/test-package-1.0/FILE_DOWNLOADED

getExtraFiles(fileList: List[Dict[str, str]], binaryName: Optional[str] = None, workDir: str = '', targetNames: Optional[List[str]] = None)[source]

### This function creates the command to download with wget the file in the given link into the given path. ### The downloaded file will overwrite a local one if they have the same name. ### This is done to overwrite potential corrupt files whose download was not fully completed.

#### Parameters: fileList (list[dict[str, str, str]]): List containing files to be downloaded. Example: [{‘url’: url1, ‘path’: path1, ‘name’: ‘test.tar’}, {‘url’: url2, ‘path’: path2, ‘name’: ‘test2.tar’}] binaryName (str): Optional. Name of the binary. Each file is a list contaning url and location to download it. Paths can be an empty string for default location. workDir (str): Optional. Directory where the files will be downloaded from. targetNames (list[str]): Optional. List containing the name of the target files for this commands.

#### Usage: installer.getExtraFiles(

[
{‘url’: ‘https://site.com/myfile.tar’, ‘path’: ‘/home/user/scipion/software/em/test-package-1.0/subdirectory1’, ‘name’: ‘test.tar’}, {‘url’: ‘https://site.com/myfile.tar2’, ‘path’: ‘/home/user/scipion/software/em/test-package-1.0/subdirectory2’, ‘name’: ‘test2.tar2’}

], binaryName=’myBinary’, workDir=’/home/user’, targetNames=[‘DOWNLOADED_FILE_1’, ‘DOWNLOADED_FILE_2’])

#### This function call will generate the following commands: cd /home/user && mkdir -p /home/user/scipion/software/em/test-package-1.0/subdirectory1 && wget -O /home/user/scipion/software/em/test-package-1.0/subdirectory1/test.tar https://site.com/myfile.tar && touch /home/user/scipion/software/em/test-package-1.0/DOWNLOADED_FILE_1

cd /home/user && mkdir -p /home/user/scipion/software/em/test-package-1.0/subdirectory2 && wget -O /home/user/scipion/software/em/test-package-1.0/subdirectory2/test2.tar2 https://site.com/myfile.tar2 && touch /home/user/scipion/software/em/test-package-1.0/DOWNLOADED_FILE_2

getFileDict(url: str, path: str = '.', fileName: Optional[str] = None) → Dict[str, str][source]

### This function generates the dictionary for a downloadable file.

#### Parameters: url (str): Url of the file to download. path (str): Optional. Relative or absolute path to download the file to. fileName (str): Optional. Local file name intented for that file after the download.

#### Returns: (dict[str, str]): Dictionary prepared for the download of one file for function getExtraFiles.

#### Usage: getFileDict(‘https://www.mywebsite.com/downloads/file.tar.gz’, path=’/path/to/myfile’, fileName=’newFile.tar.gz’)

importCommandList(commandList: List[Tuple[str, str]])[source]

### This function inserts the given formatted commands from another install helper into the current one.

#### Parameters: commandList (list[tuple[str, str]]): List of commands generated by an install helper.

#### Usage: installer1 = InstallHelper(‘package1’, packageHome=’/home/user/package2’, packageVersion=‘1.0’) installer1.addCommand(‘cd /home’, ‘CHANGED_DIRECTORY’) installer2 = InstallHelper(‘package2’, packageHome=’/home/user/package2’, packageVersion=‘1.0’) installer2.importCommandList(installer1.getCommandList())

#### Note: Argument ‘packageHome’ of the first installer must be the same as second installer.

class scipion.install.funcs.Link(packageLink, packageFolder)[source]

Bases: object

createPackageLink(packageLink, packageFolder)[source]: Create a link to packageFolder in packageLink, validate that packageFolder exists and if packageLink exists it is a link. This function is supposed to be executed in software/em folder.

class scipion.install.funcs.Target(env, name, *commands, **kwargs)[source]