Table of Contents
Since Scipion3 is released in smaller components it is more convenient to make release notes per each of the components. Main component’s release notes can be found at:
For each of the plugins you should go to each plugins URL where you’ll likely find the same CHANGES.txt file.
Fixed GUI errors: - fixing the workflow collapse/expand nodes - resetting nodes with manual subset looses the parent - reporting the protocol wall time - finished protocol can’t be modified
Fixed: execute a protocol directly that is not streaming and that has an open input must be scheduled
All sets in streaming are closed when stopping a protocol
fixing an error closing some viewers(the action of closing some viewers closed Scipion completely)
fixing an error drawing the protocols step tree
fixing an error deleting a protocol using the “delete” key
logging DEBUG mode will be activated when toggling debug mode
A dramatic plugin import error will be tolerated. Plugin will not be loaded but scipion will open
Possible outputs definition is now possible. Protocols can define possible outputs to allow designing a workflow in advance without executing them.
New variable added: SCIPION_STEPS_CHECK_SEC. Defaults to 3 (seconds). It represents how often new input is checked in streamified protocols. Increasing this value will relax the stress in the system, but will feel less “alive”.
Workflow templates can now be imported into an existing project
Possible Outputs allowed: dict or enum definition possible, see Protocol._possibleOutput
Allow readonly parameters in protocols. Useful in combination with wizards or just informative parameters.
Added the variable SCIPION_PRIORITY_PACKAGE_LIST that is used in the protocols tree view in order to load first the plugins that contains the main sections. This variable is defined as string with the packages names separated by a space. e.g: SCIPION_PRIORITY_PACKAGE_LIST = “pwem tomo cistem”
New protocol: numeric classes extractor: Extracts items from a SetOfClasses based on number of items assigned to the classes
Filter ser protocol can filter by ranking: ascending or descending getting N items or a percentage.
Filebrowser preview for images limited to 500MB (default). MAX_PREVIEW_FILE_SIZE (in MB) can be used to change this value.
Outputs predefined for: import masks, import micrographs, import movies, movie alignment methods, ctf estimators, particle pickers, particle extractors.
We are pleased to announce the new major version release of Scipion: 3.0. There is no need to make any manual download anymore, it’s available from your terminal. Please follow installation instructions
100% of Scipion is updatable! The core is updatable just by typing
scipion3 update while
the rest, as with the previous version, through the plugin manager or command line.
There is a set of plugins for tomography that are available but, IMPORTANT, this is still in beta and therefore not stable yet. Wait for the announcement!
We added several new features, made improvements and fixed bugs.
We are also very happy to see Scipion developers community growing as well as the number of plugins increased.
Workflow templates: Plugins can define their own templates and Scipion will detect them and offer them in the workflow template manager through
scipion3 template. You can also pass protocols options as command-line arguments!
- Several GUI improvements:
Wizards usability revisited
Search boxes added in some dialogs (project window) and search itself improved in the logs panel
scipion.conf, hosts.conf, protocols.conf are now optional
restart/continue workflow improved
reset part of a workflow removing results
All plugins and the core are migrated to Python3
Improved plugin structure for developers
- Improved installation:
use system libraries if possible
can use Python virtual environments
can use Conda environments
plugins and core are installed as standard Python modules
Queue params are editable in the form before execution
“Wait for” is now exported to/imported from json
Relion: Integrates relion3.1 methods
Eman: uses 2.31
Facilities plugin: new plugin with facilities monitors and all streaming tools and reporting to grafana as an option
Grigoriefflab is deprecated and replaced by cistem plugin
Gctf-1.18 and Gautomatch-0.56 have new binaries for CUDA 10.1
Cryosparc updated to use latest versions and more protocols
New plugins for SPA: cryoDRGN, sidesplitter, cryoassess
Local resolution viewers offer control of the scale.
Sphire (Cryolo & JANNI) updated
protocol to invert the handedness of a map
Chimera:Updated to chimerax-1.1. Chimera viewers have been updated.
Phenix: 1.18 version
new protocol EMValidation
scipion-em-tomo: core plugin for tomography, providing the model, generic import protocols and some utilities as extract from subtomograms, assign alignments and relate tomograms with set of subtomograms
imod: cross-correlation tilt-series pre alignment, fiducial tilt-series alignment, tilt-series normalization, tomogram reconstruction, tomogram normalization, etomo interactive, CTF estimation, CTF correction
novactf: 3D CTF corrected tomogram reconstruction
reliontomo contains: 3dctf, tomo reconstruction, classify, refine.
eman contains: tomogram reconstruction, manual picking, template matching, subtomogram extraction, initial model, refinement
plugin for Jose Jesus Fernandez Scripts: tomogram denoising, tilt series alignment and reconstruction
gctf ctf estimation
cistem ctf estimation
pyseg (import starfile only so far)
dynamo: subtomogram averaging (PCA and MRA), model manager, vectorial picking and extraction, import subtomograms from Dynamo (with table and catalogue)
xmipp: mapback, apply 3D alignment, connected components, subtomogram projection, subtomogram phantom creation, definition of ROIs in imageJ
xmipp2: plugin for xmipp version 2.4 to run MLtomo (subtomogram classification and alignment)
deepfinder: deepfinder workflow implemented in scipion.
cryoCARE: tomogram CARE denoiser
FlexAlign - a new program for movie alignment (GPU accelerated)
Align Significant - a new (GPU accelerated) program replacing reconstruct_significant, which is used in e.g. Highres protocol
Faster PSD computation
GPU accelerated Halves restoration
Micrograph Cleaner - Deep learning for micrograph carbon, ice, contaminants… removal.
Deep consensus - Deep learning for particle picking consensus
MPI version of nma_alignment
Multiple bug fixes
Multiple performance optimizations
Directory layout changed, installation refactored
DeepRes - Local resolution
FSC-Q - Map-to-model Analysis
DeepEMhancer - Deep learning for map post-processing
We are pleased to announce the new major release of version Scipion 2.0. It is available for download on Linux here
Scipion is now a plugin framework. This will speed up the release of new packages or new updates without the need of releasing a new Scipion version.
We added several new features, made improvements and fixed bugs. We are also very happy to see the Scipion development becoming more distributed. The main developers team is in Madrid, but now core developers are also in Stockholm, Montreal and Cambridge. We will be really happy to have more people on-board, come and join us!
Pluginization: each EM package is now a plugin developed and updated independently from Scipion core. This is a first step towards making Scipion even more modular and easy to use and install for both users and developers. The new repository hosting official plugins is now here
For your convenience we have developed a Plugin Manager which provides an easy GUI to manage plugins and associated binaries. We already have over 30 plugins including several new ones, such as 3DFSC, CryoEF, Cryolo, EMPIAR depositor and others. For more information on each plugin consult its corresponding repo and/ or developers.
Almost all existing EM packages were updated, such as RELION3, EMAN2.21, Gctf 1.18, SPIDER 25.02, Motioncor2-1.2.1 and others.
Model building tools from CCP4, Chimera, Phenix and powerfit have been added.
Several improvements for streaming: multi-GPU jobs distribution, more protocols supporting batch processing, jobs scheduling, new XMIPP streaming protocols (movie maxshift, ctf consensus, GL2D classification on GPU etc.)
Multiple workflow usability and GUI improvements as well as bug fixes for several tasks.
For developers and facility managers we now have a slack workspace where you can easily reach other developers and facility staff to get a quick feedback: https://scipion.slack.com/. Contact us at email@example.com for an invitation.
New protocol from Xmipp to trigger data (movies, micrographs, particles…) only when a certain among of that data is reached, in three modes: in batches, full streaming, from streaming to static.
Xmipp Movie Alignment is updated to work on GPU and to do local alignment by means of splines fititings.
New protocol to select/discard movies on-the-fly depending on the shift jumps between frames or/and the total travel drift.
Batch support for CTF estimation protocols.
CTF selection and CTF discrepancy from Xmipp are merged on the CTF consensus in order to select/discard CTFs on-the-fly depending on three assertions types: common parameters (defocus range, resolution, astigmatism…), Xmipp criteria and discrepancy with an auxiliar CTF estimation.
New protocol from Xmipp to automatically estimate the particle size.
CRYOLO automatic picking from Sphire now is available.
Now the protocol extract coordinates is ready to follow streaming processing.
New protocol to eliminate/select empty particles/classes in streaming processing.
New GL2D protocol from Xmipp for 2D classification in streaming mode in two modes: static (when a particle is assigned to a class, it remains there), full streaming (continuously updating the classes and re-assigning particles).
More streaming methods available…
Cryolo picker from SPHIRE.
EMAN2 updated to 2.21, new protocols added: new boxer (interactive and autopicker), ctf_auto, refine 2d and ref2d bispectra, tilt validation.
Relion 3.0 and associated protocols.
SPIDER 25.02: projection matching refinement without defocus groups (“gold-standard”).
Grigoriefflab: ctftilt program added.
XMIPP 3.19.04: align volume and particles, center particles, compare angles, GL2D (streaming and static), consensus classes 3D, 2D kmeans clustering, CTF consensus, deep consensus picking, deep denoising, directional ResDir, eliminate empty classes/particles, extract unit cell, generate reprojections, local MonoTomo, metaprotocol heterogeneity, movie maxshift, particle boxsize, screen deep learning, significant heterogeneity, swarm consensus intial volumes, split volume, trigger data…
Motioncor2: updated to version 1.2.1, added gain rotation/flip options. Motioncor/dosefgpu is deprecated.
Locscale: computes contrast-enhanced cryo-EM maps by local amplitude scaling using a reference model
New model building module, including several protocols from different packages: extract unit cell protocol (XMIPP) to isolate the smallest asymmetrical subunit of the map; Chimera model from template, to get the initial structure from a sequence based on Modeler web server and sequence homology, and other Chimera-derived protocols to handle structures and perform intermediate operations, such as Chimera operate and Chimera restore-session, as well as the operator protocol of Atomstructutils; Chimera rigid fit and Powerfit protocols allow to accomplish rigid fitting of structures in maps; Coot refinement and Refmac protocols, from CCP4, and real space refinement protocol, from Phenix, implement the process of flexible fitting and refinement; EMRinger and MolProbity Phenix protocols have been added to validate the final structure generated; the analysis of this structure is simplified with superpose pdbs protocol, from Phenix, and Chimera contacts protocol, that computes interactions among structure chains; the Scipion protocol export to EMDB has been modified to facilitate the submission of map and its derived structure.
Protocol tree is now auto-generated from protocols.conf of each plugin.
Added functions to restart/continue project workflow.
Scheduling has been improved dealing better with exceptions and non streaming protocols.
Protocols output refactored: now the can output/input scalar objects. Discovering outputs have been sped up (#1810).
Added QueueStepExecutor: an alternative way to execute jobs in a queue system that sends only the actual package command (e.g. relion_refine) instead of the whole protocol run (#1807).
More versatile way to blacklist files during import: by regular expressions, by date, set exclusion or just a plain black list (#1702).
Gctf refinement protocol is now split into multiple steps (#1748).
Deprecation of motioncor1, igbmc gEMpicker, cryoem, ctffind3 (in progress, #1813).
Libtiff updated to version 4 to support files over 4Gb from SerialEM (#1837).
MRC 4-bit support (#1401).
Add run ID to input list of objects (#928).
Gain reference files can be used in dm4 format directly (#1000).
And many more minor features and bug fixes! ;)
We are pleased to announce another release of Scipion, v1.2.1 is now available! We added several new features, made improvements and fixed bugs, specially for on-the-fly data processing.
We are also very happy to see the Scipion development becoming more distributed. The main developers team is in Madrid, but now core developers are also in Stockholm, Montreal and Cambridge. We will be really happy to have more people on-board, come and join us!
For the next release, we are going for a more strong “pluginization”, to make the whole platform more easy to maintain and the development more agile. Stay tuned and keep an eye!
Allow protocols to run in multiple GPUs (e.g Gctf and Gautomatch)
Protocols can now wait for other protocols to finish before starting (internal scheduling)
Allow some protocols to “wait” and work in “batch” mode to reduce the IO operations in fast protocols (while checking for updates in the stream).
Schedule batch of 2D classification jobs one after each other (protocol ‘2d streamer’)
Added cryomethods package added from McGill developers.
New volume selector protocol: produces and selects the best initial map automatically.
Relion protocol to center averages (center of mass in relion_image_handler)
Protocol to export particles in Relion format (both .star file and stacks)
Monores Xmipp protocol to estimate local resolution has been accelerated.
Allow to merge sets with different attributes
Use double-click to select in Dialogs (more intuitive)
Allow to quickly rename a protocol label (right-click -> Rename option)
CTFModel now allows to have phaseShift information as part of the model
Picking wizard now allows to pick all micrographs at once (more efficient in some programs)
Bug fixed in Relion particles-extraction when using not integer scale
Fixed bug in Relion autopick protocol when downscaling in streaming
Allow to provide references for 2D and 3D classification
Allow to pass the calibrated pixel size in Relion postprocess protocol
Relion auto-pick can use batch steps
Relion extract-particles re-factored, now in with batch mode and unified with non-streaming.
Mask 3D protocol was updated (labels and help) and test added
Re-factoring Gautomatch to use bad coords in streaming
Update and test Motioncor2-1.1.0 (mainly update help for new options)
We are very pleased to announce the release of a new version of Scipion. We have put our efforts in improving the Streaming functionality to work better in facilities. We have also updated some EM packages versions and done some bug-fixing and enhancements.
We have pushed our streaming capabilities until particle extraction, this means that all pickings (not manual one of course) can run in streaming mode and the particle extraction can be done also on the fly. Yep…we are getting closer to have 2D classification and rough initial model on the fly.
We have revisited our HTML report and have polished it. We’ve added a table with details and images per micrograph/ctf, a defocus coverage chart and a resolution histogram. Check the latest version [here]
We have added phase shift, max. resolution and CTF fit quality as global parameters to our Scipion CTF model. Old package-specific parameters are still kept for compatibility with previous versions.
Added Relion v2.1.0 support: several new options in refinement & classification protocols, new local resolution, initial model and symmetry expansion protocols, refactored particle polishing protocol
Added dark/gain reference correction to Unblur protocol
Motioncor2 package updated to version 1.0.5. Also, now input tif files are read directly by the program without any conversion. Moreover, we have fixed the bug causing wrong drift plots.
New protocol (xmipp3 - ctf selection) to make a selection of meaningful CTFs based on the defocus values, the astigmatism, and the resolution
New protocol (scipion - average frames) for computing raw frame averages (for simple diagnosis, nothing else)
New protocol (scipion - picking difference) to compute the difference between a reference SetOfPartices and another set (usually a negative reference).
Further improvements of streaming protocols:
Do not import files that are already imported (when continuing or restarting a stopped/failed streaming protocol)
Possibility to schedule jobs that link to previous unfinished ones (still in development, now available only when you import a Scipion workflow - json file)
Improved the performance during input set selection, especially when a lot of SetOfClasses2D/3D where generated by several runs of Relion
Python version updated to 2.7.14
create_project.py script to create a project from a Scipion workflow file (json),
schedule_project.py to schedule all protocols given a project name (already existing project)
edit_workflow.py allows to edit a json workflow using the same project GUI
File browser now has shortcuts as well as Search function and keyboard navigation
Shorting protocol names: now when you copy a protocol, the new name will be oldName (copy N), where N is a number
We are very pleased to announce the release of a new version of Scipion. It’s been over a year since the previous and first version and we have been working on 3 main goals for this release:
Consolidation: We put and will always put our best effort into making Scipion a robust and reliable software. We have improved performance, usability and fixed multiple bugs.
EM packages integration: We have updated several EM packages to their latest versions (relion 2.0.4) and added new ones (motioncor2, gctf, gautomatch, …). Single movie alignment protocol (as in Scipion 1.0) has been split into several ones for each program.
Streaming capabilities: To speed up first preprocessing steps we have enabled Scipion to work in “streaming mode”, allowing users to compute aligned movies and estimate CTF as soon as a movie or micrograph comes out of the microscope PC.
Xmipp has also been greatly improved and many new methods have been added. Please see link: xmipp release notes for details.
Ethan picker Automated detection of spherical particles from electron micrographs.
gAutomatch GPU-accelerated particle picking program developed by K. Zhang allows template-based and “gaussian-blob” (no references) picking. All advanced parameters (exclusive picking, filtering etc.) are available.
gCTF GPU-accelerated program for CTF determination, refinement and evaluation. At this moment movie options, CTF refinement for particles and tilt refinement options are not supported yet.
Imagic We have added Imagic MSA classification method. Further information is available from our link:https://github.com/I2PC/scipion/wiki/ImagicProtMSA[wiki].
Localized reconstruction A general method for the localized three-dimensional reconstruction of substructures bound to a larger particle. After determination of the particle orientations via conventional methods, local areas corresponding to the subunits (‘subparticles’) can be extracted and treated as single particles.
magDistortion This program from Grigorieff’s lab allows to estimate and correct magnification distortions in electron micrographs. Correction is also available for particle coordinates. Hint: results of this protocol can be used for motion correction with motioncor2!
Motioncor2 Completely re-written (after motioncorr/dosefgpu) software from D. Agard lab allows anisotropic beam-induced motion correction at single pixel level across the whole frame using GPUs. Options for dose filtering, correction of magnification distortion and saving movie stacks are available.
Relion 2.0 We have updated Relion to its latest available version (2.0.4). You can benefit from the substantial reduction of computing time using Relion with your GPUs. But do not worry if you don’t have GPU, you can also use Relion on CPUs. We have also added new relion extract particle protocol and refactored the picking protocol. Moreover, new protocols for particle sorting, 3D mask creation, projection subtraction and CTF export to STAR file were added.
Streaming processing allows to overlap data acquisition with the first steps of digital image processing. Protocols adapted for streaming can be executed even if the algorithm that produces the data that is going to be used as input has only produced part of the output dataset.
Movies can now be imported in “streaming” mode. Also, individual frames can be used as input and stacked on-the-fly, making the movies available for later protocols immediately.
All movie alignment protocols are able to use movies in streaming (motioncorr, motioncor2, xmipp-opticalflow, xmipp-crosscorrelation, unblur).
We have designed monitors to follow the status of several running protocols within a Scipion project. They can track and plot values for “system status” (cpu, memory, swap), ctf values (defocusU, defocusV). Alerts can be setup to email the microscopist/user when certain values rise above/below a custom threshold.
The summary monitor generates an HTML report that summarizes the status of the system and the data being processed, plotting the defoci and system data.
This wizard creates a folder structure for your project and can be customized to reflect different microscope or camera setups. You can choose what preprocessing steps you would like to do during movie streaming.
We have also done a lot of work to consolidate Scipion, improving the usability and adding small features to make Scipion a better and more robust software:
Extract coordinates protocol can now apply shifts to the particle coordinates.
Scipion install script now provides a possibility to choose package version for installation.
Export bibliographic references into bibtext file.
Notebook: write project notes in your favourite text editor.
Labels: any protocol can now be labelled with a name and/or color. This helps to orient within a protocol tree in large projects. (Use Ctrl+T to loop through modes)
Age mode: Coloring the boxes by “age”, the younger the bluest. (Use Ctrl+T to loop through modes)
Zooming and panning a project’s protocol tree (Use “Shift + mouse wheel”)
Linear picking mode, eraser size modification are now available in xmipp particle picker.
Project import: besides import/export of workflows, now it is also possible to import whole projects, e.g. from another computer.
Collect statistics: we collect and analyze usage statistics information to better understand the usage of the different protocols and prioritize maintenance and support. You can choose to enable or disable the collection of information at any time.
New scripts: create movie stacks, mirror directory, scipion box wizard.
Sort objects in browse windows: now all objects can be sorted by name, info or creation date. Useful when you have a lot of different object sets.
Highlight direct connections of selected protocols.
Several performance improvements to decrease project loading time.
Improved movie model: added initial dose (pre-exposure) and dose per frame.
Several protocol fixes:
Fixed bug when creating the output for Frealign (in some cases some information from input particles was not properly propagated)
Fixed some bugs in movie alignment protocols (summovie and unblur) and tests added
Some minor bugs fixed in Relion protocols
Bugs fixed in Resmap protocol when using two half volumes
Fixed several bugs in Spider protocols:
converting input particles with alignment
wrong regular expression for replacing some variables in script template
parsing of the resulting dendrogram
some additional validations and removed unused code
Bugfixes and inprovements in Xmipp protocols:
Protocols screen-classes merged into one: compare-reprojections
Complete refactoring of operate-particles and operate-volumes protocols (previously called 2D and 3D calculator). Tests added
Picking and Viewer:
Warning if particles are picked in a temporary folder and the SetOfParticles was not created
Improved implementation of assign-tiltpairs protocol in Xmipp and some refactoring of picking methods
Fixed bug that caused GUI to freeze sometimes
Some bugs fixed when displaying and exporting particles
Sorting arrows displayed after sorting by a column. Hourglass displayed while sorting.
Some bug fixed when creating subset from classes
Other fixes or improvements:
ImageHandler’s methods convert and writeStack now accepts alignment parameters
Fixed bug when displaying Movies summary (sqlite files were not closed)
Fixed bug when spawning Eman process to write particles
Added REMOTE_MESA_LIB environment var for using OpenGL in remote desktops
Created a LegacyProtocol class to read deprecated protocols
Cleanup in some tests and added new ones for core classes or functions
Allows to combine several EM software packages (~ 100 protocols):
All protocols from Xmipp
Most of protocols from Relion
MDA protocols from Spider
Some protocols from Eman2/Sparx
From Grigorieff lab: CTFFIND, FREALIGN, unblur and summovie.
A few tools from Bsoft
ResMap, gEMpicker, dogpicker, motioncorr
Full tracking and reproducibility:
Display runs as a list or a tree.
Inspect the parameters of a previous run
Repeat one or several runs
Export/Import a workflow template
Data analysis: * Visualization and operation with Sets. (Particles, Micrographs, CTFs, etc) * Visualization of Volumes * Resolution and angular distribution plots