scipion logo

Streaming Processing

For facilities

Currently, scipion is being used every day in several facilities in Europe, US, Canada, Israel and Australia. If you are running a Cryo EM facility and want more info, please contatct us. We will be happy to help you run Scipion there. Also, we have a Slack framework to maintain a direct communication channel.

facilities map

Facilities map

Streaming processing

Scipion is able to process data in streaming, i.e, at the same time movies (or micrographs) are coming from the microscope PC. It can also be called on-the-fly processing. This allows to overlap computing time with the acquisition (reducing computational needs) and also to detect problems at early stages. This idea is implemented in different labs mainly by using custom-made scripts, but also it can be implemented using templates in a very easy way. The advantage of our Scipion solution is that you have the usual flexibility to choose what operations to do and the traceability to re-do some of the steps later. It is basically the same Scipion interface with one key change: the output is produced as soon as the first element is available, and it is later updated with new output elements. This allows concatenating several operations before the first one is completed.

Learn how to create, import, export and launch streaming workflows.

Scipion v.2.0 - Diocletian updates/introduce a big number of streaming tools. We review and analyze in deep the use and the combination of these on-the-fly tools and protocols for facilities in the D. Maluenda et.al. ‘Flexible workflows for on-the-fly electron microscopy single particle image processing using Scipion’ Acta Cryst. (2019). Dxx, yy-zz paper (not published, yet. Please contact us to ask a preprint).

On top of that, we have paid special attention to the data flow management including the Trigger data protocol, the Movie max shift protocol, the CTF consensus protocol, the Extract coordinates protocol in streaming and 2D classification protocol in streaming (GL2D-static and GL2D-streaming). But also we have included general streaming tools, such as enabling to Continue and Restart workflows from a certain protocol, providing more stability in streaming protocols keeping processing new data even if a certain bunch fails for some reason, etc.

In addition, we have work more in the concept of consensus protocols, for merging and combining different approach (from different EM softwares) to do the same tasks, in order to get an improved result based on that different results. The consensus protocols family are for CTF estimation, Picking, Initial Volume and 3D classes, so far.

Moreover, we are also working in the monitor protocols, those special protocols that constantly check how the execution of other protocols is going. We have developed several GUIs that are refreshed periodically and produce a graphical summary (e.g, CTF estimated values, system load, etc). This summary is generated in HTML format to be easily copied to a public website. Learn how to customize the HTML report according to your facilities’ needs.

Finally, this tutorial focused on streaming processing simulates a cryo-EM acquisition in order to sum up all the available facilities tools.