Understanding the Plugin class

Table of contents

• Understanding the Plugin class

  • Associated resources

  • Starting with scipion-em-template

    • Exercise 1 (easy)

    • Exercise 2 (easy)

    • Exercise 3 (easy)

    • Exercise 4 (intermediate)

    • Exercise 5 (hard)

  • Summary

Associated resources

Here you can find resources associated with this content, like videos or presentations used in courses:

Course presentation

Detailed documentation

Starting with scipion-em-template

Let’s go to our template plugin folder and setup everything ready for the practice session:

cd scipion-em-template
git checkout course5_exBase

You might notice a new file protocol_run_myprogram.py. Have a look inside. You will see that this simple protocol only has one step function runStep that executes a program via runJob. If you are using PyCharm you will see that Plugin.getProgram() function is undefined.

Open Scipion and create a new project. Open it and search for my program protocol (use Ctrl+F to search for the protocols). As it does not have any input variables, just execute it. It should fail with the following error message:

Let’s start working on the Plugin class from scratch.

Exercise 1 (easy)

First, create myplugin/constants.py file and define the following constants:

MYPROGRAM_HOME = 'MYPROGRAM_HOME'
MYPROGRAM = 'MYPROGRAM'
V1_0 = '1.0'

Pay attention that for the first two vars we defined the “name” of the constant (as a string) and not it’s value. Usually it’s a good idea if you want to re-use the constant names throughout the plugin code.

Now switch to __init__.py file and start populating the Plugin class with homeVar, pathVars, list of supported versions and url (you can put any web address you like). Don’t forget that you need to import constants from the constants.py file! Then add _defineVariables class method and define MYPROGRAM_HOME as myprogram-1.0 and MYPROGRAM as example_script.sh. Make sure MYPROGRAM_HOME points to the path inside software/em folder (where usually all packages are installed).

---

Exercise 2 (easy)

We still need to define plugin binaries (and actually create them), so:

cd scipion3/software/em
mkdir myprogram-1.0
cd myprogram-1.0
touch example_script.sh
chmod a+x example_script.sh

Tip

It’s good to follow the name-version (myprogram-1.0) convention when defining the software folder names. This way Scipion can easily detect the version number.

Open example_script.sh in any editor and paste the following code:

#!/bin/bash
echo "Starting myprogram..."
sleep 5
echo "This script has finished running!"

Save the script. As you can see it will simply print some text after waiting for 5 seconds. Add defineBinaries class method to the Plugin class. You can provide any value for tar argument (e.g. tar=’myprogram_v1.0.tgz’) as we have already “installed” the binaries.

Now, remember that we are still missing getProgram function for our protocol? Create it inside the Plugin class and make it return the full path to MYPROGRAM (Hint: you will need to use cls.getHome())

---

Exercise 3 (easy)

Now let’s test the plugin binaries.

cd scipion3
./scipion3 installb -h

If you see something like

myprogram                1.0     [X]

congratulations! You’ve got the binaries properly defined and installed. Reopen your Scipion project and restart our protocol. Once it is finished, check the output log. You should see something similar to:

Now, imagine a user who wants to use a different binary version. At the moment we have the default version defined in _defineVariables as 1.0. For simplicity, let’s close the Scipion project and rename our package folder:

cd scipion3/software/em
mv myprogram-1.0 myprogram-1.1

Reopen the project (run scipion last) and try to restart the finished protocol. You should see an error message that is raised by validateInstallation function from pyworkflow/plugin.py:

This means that the user have to redefine MYPROGRAM_HOME to point to version 1.1. So, redefine it in your scipion3/config/scipion.conf file (if you do not have it, execute ./scipion3 config first to generate the file) and then re-run the protocol:

MYPROGRAM_HOME = software/em/myprogram-1.1

Tip

You can also define such variables in your shell environment if you like. The priority goes as following: environment > config > default value.

---

Exercise 4 (intermediate)

You might have noticed that right now we are using the full path to the binary (example_script.sh) in getProgram function. Another possibility would be to use getEnviron function to add the package folder to the PATH. Have a look at the presentation slides and find an example of this function. You can also look at the Environ class inside pyworkflow/utils.py. Can you now define getEnviron function inside the Plugin class so that the myprogram-1.1 path is added to the beginning of the PATH var?

Tip

You would need to use update function of the Environ class and position=BEGIN.

If you have given up, checkout the resulting code in the following branch (exercises 1-4 completed):

cd scipion-em-template
git checkout course5_ex1-4

Once the PATH contains the path to our myprogram-1.1, we don’t need anymore to define the full path in the getProgram function. Change the function and re-run the protocol. In the output log you should see now only the binary name instead of the full path to the executable. This was a very simple case, but in principle getEnviron function can be used to modify things like LD_LIBRARY_PATH or define specific CUDA libraries / MPI versions etc.

The getEnviron function gets automatically recognised by pyworklow/protocol.py when executing the plugin protocols, however you can also explicitly define the environment inside runJob function:

self.runJob(program, params, env=Plugin.getEnviron())

Exercise 5 (hard)

Let’s explore conda-based installation. A lot of new cryo-em software comes with its own conda environment so this seems like an appropriate use case.

First, you need to make sure that you have conda installed (we recommend miniconda3). If you have used conda for Scipion installation then there should be no problem. You also need to know how to activate it, this depends on the system and the SHELL you are using. Below is an example for bash:

. /path/to/miniconda3/etc/profile.d/conda.sh

We need to define the following constants in your scipion3/config/scipion.conf file:

CONDA_ACTIVATION_CMD = . /path/to/miniconda3/etc/profile.d/conda.sh
MYPROG_ENV_ACTIVATION = conda activate myprogenv-1.0

Add the following functions to the Plugin class:

@classmethod
def getDependencies(cls):
    # try to get CONDA activation command
    condaActivationCmd = cls.getCondaActivationCmd()
    neededProgs = []
    if not condaActivationCmd:
        neededProgs.append('conda')

    return neededProgs

@classmethod
def getMyProgEnvActivation(cls):
    """ Remove the scipion home and activate the conda environment. """
    activation = cls.getVar(MYPROG_ENV_ACTIVATION)
    scipionHome = Config.SCIPION_HOME + os.path.sep

    return activation.replace(scipionHome, "", 1)

The first one is used to make sure the “conda” command is available (either after activation or from PATH), while the second one is required to activate the specific conda environment.

You also need to import Config from pyworkflow.utils at the top of the file and define MYPROG_ENV_ACTIVATION both in defineVariables and also the constants.py file.

Modify the getProgram function so that it concatenates 3 commands: cls.getCondaActivationCmd(), cls.getMyProgEnvActivation() and the program variable (example_script.py).

Now change addPackage function inside defineBinaries: you need to set tar=void.tgz add commands argument that will create and activate a new conda environment. e.g:

conda create -y -n myprogenv-1.0 python=3;
conda activate myprogenv-1.0;
touch IS_INSTALLED;

Here we did not install any packages inside the new conda env since our simple script is not a python package.

Last, let’s modify getEnviron function and remove PYTHONPATH key from Environ dict - this is required for the virtual environment to work properly within Scipion.

Now we are ready to install myprogram-1.0 with conda environment (remember, we have renamed to myprogram-1.1, so now our binaries are missing if you run ./scipion3 installb -h) Execute the following:

./scipion3 installb myprogram

Here we did not provide the version, so the version with default=True flag is installed. If the installation have completed successfully, you are in luck! Look into software/em/myprogram-1.0, you should see IS_INSTALLED empty file. Remove MYPROGRAM_HOME from the scipion.conf file and move our little script back:

cd scipion3/software/em
mv myprogram-1.1/example_script.sh myprogram-1.0/

We are finally ready to run our protocol. If everything went well, you should see something like this:

Results for this exercise can be found in the following git branch:

cd scipion-em-template
git checkout course5_ex5

Summary

In this tutorial we have explored the Plugin class and its main methods. Once you get familiar with them you should be able to create __init__.py file for your plugin easily. If you are brave enough, you can have a look at the Sphire plugin code that provides a more complex conda-based installation, use of void.tgz among other things:

https://github.com/scipion-em/scipion-em-sphire/blob/master/sphire/__init__.py