scipion logo

Writing Tests

Writing tests for Scipion is much like writing tests for other python code. Tests need to be thorough, fast, isolated, consistently repeatable, and as simple as possible. We try to have tests both for normal behaviour and for error conditions. Tests live in the <module>/tests directory, where every file that includes tests has a test_ prefix.

When you are adding tests to an existing test file, it is also recommended that you study the other existing tests; it will teach you which precautions you have to take to make your tests robust and portable.

BaseTest Class

BaseTest is a class that inherits from the Python’s standard library called unittest.TestCase which contains tools for testing your code. Unit testing checks if all specific parts of your function’s behavior are correct, which will make integrating them together with other parts much easier.

In order to write BaseTest tests, you must:

  • Write your tests as methods within classes.
  • Use a series of built-in assertion methods.

Passing a test

Here’s a typical scenario for writing tests. We’ll use cryosparc 2D classification test as example which from a set of particles classifies them into a set of classes.

First you need to create a test file (we’ll call our example test_protocols_cryosparc2.py). Then the BaseTest class from which our class will inherit will be imported. On the other hand, we will import the Dataset class which is in charge of handling the DataSets, and lastly, write a series of methods to test all the cases of your function’s behavior.

BaseTest and DataSet classes are located in pyworkflow module. It also contains a set of methods that are useful for managing the project generated by Scipion.

There’s a line by line explanation below the following code:

from pyworkflow.tests import BaseTest, DataSet, setupTestProject
  • setupTestProject: Method to create and setup a Project for a give Test class.

    Info:

    If this method is invoked, the project will be created at the path assigned to the SCIPION_USER_DATA variable in the Scipion configuration file. Otherwise, the project will be created in the path assigned to the SCIPION_TESTS_OUTPUT variable. These variables can be modified.

Then we’ll create a class, for example TestCryosparcClassify2D, with a method setUpClass which hook for setting up class fixture before running tests in the class. This class inherits from the BaseTest class.

Remember that every method that starts with “test” within a class that derives from BaseTest will be run automatically when you run a Scipion test.
class TestCryosparcClassify2D(BaseTest):
    @classmethod
    def setUpClass(cls):
        setupTestProject(cls)
        dataSetName = 'xmipp_tutorial'
        dataset = DataSet.getDataSet(dataSetName)
        self.partFolderPath = dataset.getFile('particles')
        self.pattern = 'BPV_*.stk'
  • DataSet.getDataSet(dataSetName): this method is called every time the dataset want to be retrieved.
  • dataset.getFile: method that returns the path where the files are located.

Once method setUpClass has been created, each of the tests is written.

In our example:

def testCryosparc2D(self):

Inside, we will create a Scipion workflow invoking and executing each of the necessaries protocols to our test.

First we’ll import a set of particles using ProtImportParticles protocol in order to be classify. For that, we’ll create a new protocol instance (newProtocol(ProtocolClass) method) through the project and return a newly created protocol of the given class. After that, we will proceed to execute it and then we will check if the output has been correct.

  • newProtocol: method to create new protocols instances through the project and return a newly created protocol of the given class. This method accept kwargs that represent the protocol parameters.
# Define import particles protocol
objLabel = 'Import from file (particles)'
protImportPart = cls.newProtocol(ProtImportParticles,
                                 objLabel=objLabel,
                                 filesPath=self.partFolderPath,
                                 filesPattern=self.pattern,
                                 samplingRate=samplingRate,
                                 importFrom=ProtImportParticles.IMPORT_FROM_FILES)

# Lunching the import particle protocol
cls.launchProtocol(protImportPart)
# Check that input images have been imported
self.assertSetSize(protImportPart.outputParticles,
                    msg='Import of images: %s, failed. outputParticles is
                     'None.' % self.partPattern)
  • launchProtocol: method to launch a given protocol

Once the particles have been imported, an instance of the Cryosparc 2D classification protocol (ProtCryo2D) will be created which will have as input the particles imported by the ProtImportParticles protocol.

# Define cryosparc 2D classification protocol
prot2D = self.newProtocol(ProtCryo2D,
                          doCTF=False, maskDiameterA=340,
                          numberOfMpi=4, numberOfThreads=1)
prot2D.inputParticles.set(protImportPart.outputParticles)
prot2D.numberOfClasses.set(5)
prot2D.numberOnlineEMIterator.set(40)
prot2D.compute_use_ssd.set(False)
prot2D.setObjLabel(label)
self.launchProtocol(prot2D)

# Check if 2D Classification protocol finish successfully
self.assertSetSize(cryosparcProt.outputClasses,
                   msg="There was a problem with Cryosparc 2D classify")

# Check if the classes has 2D alignment
for class2D in cryosparcProt.outputClasses:
    self.assertTrue(class2D.hasAlignment2D())

As can be seen, all parameters of a protocol can be modified using the set method.

The following code shows the complete implementation of the test:

from pyworkflow.tests import BaseTest, DataSet, setupTestProject
from pwem.protocols import ProtImportParticles

class TestCryosparcClassify2D(BaseTest):
    @classmethod
    def setUpClass(cls):
        setupTestProject(cls)
        dataSetName = 'xmipp_tutorial'
        dataset = DataSet.getDataSet(dataSetName)
        self.partFolderPath = dataset.getFile('particles')
        self.pattern = 'BPV_*.stk'

    def testCryosparc2D(self):
        def _runCryosparcClassify2D(label=''):

            # Define import particles protocol
            objLabel = 'Import from file (particles)'
            protImportPart = cls.newProtocol(ProtImportParticles,
                                             objLabel=objLabel,
                                             filesPath=self.partFolderPath,
                                             filesPattern=self.pattern,
                                             samplingRate=samplingRate,
                                             importFrom=ProtImportParticles.IMPORT_FROM_FILES)

            # Lunching the import particle protocol
            cls.launchProtocol(protImportPart)
            # Check that input images have been imported
            self.assertSetSize(protImportPart.outputParticles,
                               msg='Import of images: %s, failed. outputParticles is
                               'None.' % self.partPattern)

            # Define cryosparc 2D classification protocol
            prot2D = self.newProtocol(ProtCryo2D,
                                      doCTF=False, maskDiameterA=340,
                                      numberOfMpi=4, numberOfThreads=1)
            prot2D.inputParticles.set(protImportPart.outputParticles)
            prot2D.numberOfClasses.set(5)
            prot2D.numberOnlineEMIterator.set(40)
            prot2D.compute_use_ssd.set(False)
            prot2D.setObjLabel(label)
            self.launchProtocol(prot2D)
            return prot2D

        def _checkAsserts(cryosparcProt):
            self.assertSetSize(cryosparcProt.outputClasses,
                               msg="There was a problem with Cryosparc 2D classify")

            for class2D in cryosparcProt.outputClasses:
                self.assertTrue(class2D.hasAlignment2D())

        cryosparcProtGpu = _runCryosparcClassify2D(label="Cryosparc classify2D GPU")
        _checkAsserts(cryosparcProtGpu)

How to Write Assertions

The last step of writing a test is to validate the output against a known response. We use one of the assert*() methods provided by the BaseTest class. If the test fails, an exception will be raised with an explanatory message, and BaseTest will identify the test case as a failure. Any other exceptions will be treated as errors.

There are some general best practices around how to write assertions:

  • Make sure tests are repeatable and run your test multiple times to make sure it gives the same result every time.
  • Try and assert results that relate to your input data, such as verifying that a set of particles has been imported correctly or that they have been classified into a set of classes.

Note that BaseTest by inheriting from de unittest, comes with lots of methods to assert on the values, types, and existence of variables. Here are some of the most commonly used methods:

Method Equivalent to
.assertEqual(a, b) a == b
.assertTrue(x) a == b
.assertFalse(x) a == b
.assertIs(a, b) a == b
.assertIsNone(x) a == b
.assertIn(a, b) a == b

Running our tests

Once the test is created, we would only have to run it. [clic here] for more information on how to run the tests.