scipion logo

On-the-fly processing

For facilities

Currently, scipion is being used every day in several facilities in Europe, US, Canada, Israel, Australia… If you are running a Cryo EM facility and want more info, please contact us. We will be happy to help you run Scipion there. Also, we have a Slack framework to maintain a direct communication channel.

facilities map

Facilities map

Streaming processing

Scipion is able to process data in streaming, i.e, at the same time movies (or micrographs) are coming from the microscope PC. It can also be called on-the-fly or on-line processing. This allows to overlap computing time with the acquisition. This is interesting for two main reasons: (i) to save processing time and provide users with more than movies and (ii) to monitor the acquisition in order to be able to take decisions in very early stages. This idea is implemented in different labs mainly by using custom-made scripts, but also it can be implemented using templates in a very easy way. The advantage of Scipion solution is that you have the usual flexibility to choose what operations to do and the traceability to re-do some of the steps later. It is basically the same Scipion interface with one key change: the output is produced as soon as the first element is available, and it is later updated with new output elements. This allows concatenating several operations before the first one is completed, resulting in a whole workflow running at once.

Learn how to create, import, export and launch streaming workflows.

Scipion v.3.0 - Diocletian updates/introduce a big number of streaming tools (CHECK LINK). We review and analyze in deep the use and the combination of these on-the-fly tools and protocols for facilities in the D. Maluenda et.al. ‘Flexible workflows for on-the-fly electron microscopy single particle image processing using Scipion’ Acta Cryst. (2019). D75, 882-894 paper.

On top of that, we have paid special attention to data flow management and acquisition monitoring, for that Scipion includes a Trigger data protocol, the Movie max shift protocol, the CTF consensus protocol, the Extract coordinates protocol in streaming and 2D classification in streaming (see also GL2D-static and GL2D-streaming). But also we have included general streaming tools, such as enabling to Continue and Restart workflows from a certain protocol, providing more stability in streaming protocols keeping processing new data even if a certain bunch fails for some reason, etc.

In addition, we have work more in the concept of consensus protocols, for merging and combining different approaches (from different EM softwares) to do the same tasks, in order to get an improved result based on that different results. The consensus protocols family are for CTF estimation, Picking, Initial Volume and 3D classes, so far.

Moreover, we are also working in the monitor protocols, those special protocols that constantly check how the execution of other protocols is going on. We have developed several GUIs that are refreshed periodically and produce a graphical summary (e.g, CTF estimated values, system load, etc). Examples of monitors reports generated in HTML format can be seen at http://scipion.cnb.csic.es/scipionbox/lastHTMLReport, while you can see a Grafana based report in the URL http://scipion.cnb.csic.es:3000/lastHTMLReport. (Unfortunately, at present, Grafana server can only be accessed from within our local area network). Learn how to customize the reports according to your facilities’ needs.