.. figure:: /docs/images/scipion_logo.gif :width: 250 :alt: scipion logo .. _writing-tests: ============= Writing Tests ============= Writing tests for Scipion is much like writing tests for other python code. Tests need to be thorough, fast, isolated, consistently repeatable, and as simple as possible. We try to have tests both for normal behaviour and for error conditions. Tests live in the `/tests` directory, where every file that includes tests has a **test\_** prefix. When you are adding tests to an existing test file, it is also recommended that you study the other existing tests; it will teach you which precautions you have to take to make your tests robust and portable. --------------- BaseTest Class --------------- **BaseTest** is a class that inherits from the Python’s standard library called *unittest.TestCase* which contains tools for testing your code. Unit testing checks if all specific parts of your function’s behavior are correct, which will make integrating them together with other parts much easier. In order to write BaseTest tests, you must: - Write your tests as methods within classes. - Use a series of built-in assertion methods. ------------------ Passing a test ------------------ Here’s a typical scenario for writing tests. We'll use cryosparc 2D classification test as example which from a set of particles classifies them into a set of classes. First you need to create a test file (we'll call our example test_protocols_cryosparc2.py). Then the BaseTest class from which our class will inherit will be imported. On the other hand, we will import the Dataset class which is in charge of handling the `DataSets `_, and lastly, write a series of methods to test all the cases of your function’s behavior. BaseTest and DataSet classes are located in **pyworkflow** module. It also contains a set of methods that are useful for managing the project generated by Scipion. There’s a line by line explanation below the following code: .. code-block:: python from pyworkflow.tests import BaseTest, DataSet, setupTestProject - **setupTestProject**: Method to create and setup a Project for a give Test class. **Info:** If this method is invoked, the project will be created at the path assigned to the SCIPION_USER_DATA variable in the Scipion configuration file. Otherwise, the project will be created in the path assigned to the SCIPION_TESTS_OUTPUT variable. These variables can be modified. Then we’ll create a class, for example TestCryosparcClassify2D, with a method `setUpClass` which hook for setting up class fixture before running tests in the class. This class inherits from the BaseTest class. Remember that every method that starts with “test” within a class that derives from BaseTest will be run automatically when you run a Scipion test. .. code-block:: python class TestCryosparcClassify2D(BaseTest): @classmethod def setUpClass(cls): setupTestProject(cls) dataSetName = 'xmipp_tutorial' dataset = DataSet.getDataSet(dataSetName) self.partFolderPath = dataset.getFile('particles') self.pattern = 'BPV_*.stk' * **DataSet.getDataSet(dataSetName)**: this method is called every time the dataset want to be retrieved. * **dataset.getFile**: method that returns the path where the files are located. Once method setUpClass has been created, each of the tests is written. In our example: .. code-block:: python def testCryosparc2D(self): Inside, we will create a Scipion workflow invoking and executing each of the necessaries protocols to our test. First we'll import a set of particles using `ProtImportParticles` protocol in order to be classify. For that, we'll create a new protocol instance (`newProtocol(ProtocolClass) method`) through the project and return a newly created protocol of the given class. After that, we will proceed to execute it and then we will check if the output has been correct. * **newProtocol**: method to create new protocols instances through the project and return a newly created protocol of the given class. This method accept *kwargs* that represent the protocol parameters. .. code-block:: python # Define import particles protocol objLabel = 'Import from file (particles)' protImportPart = cls.newProtocol(ProtImportParticles, objLabel=objLabel, filesPath=self.partFolderPath, filesPattern=self.pattern, samplingRate=samplingRate, importFrom=ProtImportParticles.IMPORT_FROM_FILES) # Lunching the import particle protocol cls.launchProtocol(protImportPart) # Check that input images have been imported self.assertSetSize(protImportPart.outputParticles, msg='Import of images: %s, failed. outputParticles is 'None.' % self.partPattern) * **launchProtocol**: method to launch a given protocol Once the particles have been imported, an instance of the Cryosparc 2D classification protocol (`ProtCryo2D`) will be created which will have as input the particles imported by the ProtImportParticles protocol. .. code-block:: python # Define cryosparc 2D classification protocol prot2D = self.newProtocol(ProtCryo2D, doCTF=False, maskDiameterA=340, numberOfMpi=4, numberOfThreads=1) prot2D.inputParticles.set(protImportPart.outputParticles) prot2D.numberOfClasses.set(5) prot2D.numberOnlineEMIterator.set(40) prot2D.compute_use_ssd.set(False) prot2D.setObjLabel(label) self.launchProtocol(prot2D) # Check if 2D Classification protocol finish successfully self.assertSetSize(cryosparcProt.outputClasses, msg="There was a problem with Cryosparc 2D classify") # Check if the classes has 2D alignment for class2D in cryosparcProt.outputClasses: self.assertTrue(class2D.hasAlignment2D()) As can be seen, all parameters of a protocol can be modified using the *set* method. The following code shows the complete implementation of the test: .. code-block:: python from pyworkflow.tests import BaseTest, DataSet, setupTestProject from pwem.protocols import ProtImportParticles class TestCryosparcClassify2D(BaseTest): @classmethod def setUpClass(cls): setupTestProject(cls) dataSetName = 'xmipp_tutorial' dataset = DataSet.getDataSet(dataSetName) self.partFolderPath = dataset.getFile('particles') self.pattern = 'BPV_*.stk' def testCryosparc2D(self): def _runCryosparcClassify2D(label=''): # Define import particles protocol objLabel = 'Import from file (particles)' protImportPart = cls.newProtocol(ProtImportParticles, objLabel=objLabel, filesPath=self.partFolderPath, filesPattern=self.pattern, samplingRate=samplingRate, importFrom=ProtImportParticles.IMPORT_FROM_FILES) # Lunching the import particle protocol cls.launchProtocol(protImportPart) # Check that input images have been imported self.assertSetSize(protImportPart.outputParticles, msg='Import of images: %s, failed. outputParticles is 'None.' % self.partPattern) # Define cryosparc 2D classification protocol prot2D = self.newProtocol(ProtCryo2D, doCTF=False, maskDiameterA=340, numberOfMpi=4, numberOfThreads=1) prot2D.inputParticles.set(protImportPart.outputParticles) prot2D.numberOfClasses.set(5) prot2D.numberOnlineEMIterator.set(40) prot2D.compute_use_ssd.set(False) prot2D.setObjLabel(label) self.launchProtocol(prot2D) return prot2D def _checkAsserts(cryosparcProt): self.assertSetSize(cryosparcProt.outputClasses, msg="There was a problem with Cryosparc 2D classify") for class2D in cryosparcProt.outputClasses: self.assertTrue(class2D.hasAlignment2D()) cryosparcProtGpu = _runCryosparcClassify2D(label="Cryosparc classify2D GPU") _checkAsserts(cryosparcProtGpu) How to Write Assertions ----------------------- The last step of writing a test is to validate the output against a known response. We use one of the `assert*()` methods provided by the `BaseTest` class. If the test fails, an exception will be raised with an explanatory message, and `BaseTest` will identify the test case as a failure. Any other exceptions will be treated as errors. There are some general best practices around how to write assertions: * Make sure tests are repeatable and run your test multiple times to make sure it gives the same result every time. * Try and assert results that relate to your input data, such as verifying that a set of particles has been imported correctly or that they have been classified into a set of classes. Note that **BaseTest** by inheriting from de `unittest`, comes with lots of methods to assert on the values, types, and existence of variables. Here are some of the most commonly used methods: +--------------------+-----------------+ | Method | Equivalent to | +--------------------+-----------------+ | .assertEqual(a, b) | a == b | +--------------------+-----------------+ | .assertTrue(x) | a == b | +--------------------+-----------------+ | .assertFalse(x) | a == b | +--------------------+-----------------+ | .assertIs(a, b) | a == b | +--------------------+-----------------+ | .assertIsNone(x) | a == b | +--------------------+-----------------+ | .assertIn(a, b) | a == b | +--------------------+-----------------+ Running our tests ----------------- Once the test is created, we would only have to run it. `[click here] `_ for more information on how to run the tests.