.. figure:: /docs/images/scipion_logo.gif
   :width: 250
   :alt: scipion logo

.. _facilities:

=====================
On-the-fly processing
=====================

For facilities
---------------
Currently, scipion is being used every day in several facilities in Europe, US,
Canada, Israel, Australia... If you are running a Cryo EM facility and want more
info, please :ref:`contact us <contact-us>`. We will be happy to help you run
Scipion there. Also, we have a `Slack <https://scipion.slack.com>`_ framework to
maintain a direct communication channel.

.. figure:: /docs/images/facilities_map.png
   :align: center
   :width: 900
   :alt: facilities map

`Facilities map <https://www.google.com/maps/d/viewer?
mid=1MHEnnhBsUarOGJnlo0BapQrrGtA&ll=23.859083678630366%2C-5.749884867547308&z=3>`_

Streaming processing
--------------------

Scipion is able to process data in streaming, i.e, at the same time movies
(or micrographs) are coming from the microscope PC. It can also be called
on-the-fly or on-line processing. This allows to overlap computing time with the
acquisition. This is interesting for two main reasons:
(i) to save processing time and provide users with more than movies and
(ii) to monitor the acquisition in order to be able to take decisions in very
early stages.
This idea is implemented in different labs mainly by using
custom-made scripts, but also it can be implemented using templates in a very
easy way. The advantage of Scipion solution is that you have
the usual flexibility to choose what operations to do and the traceability to
re-do some of the steps later. It is basically the same Scipion interface with
one key change: the output is produced as soon as the first element is
available, and it is later updated with new output elements. This allows
concatenating several operations before the first one is completed,
resulting in a whole workflow running at once.

Learn `how to create, import, export and launch streaming workflows <facilities-workflows>`_.

Scipion v.3.0 - Diocletian `updates/introduce a big number of streaming tools
</docs/docs/developer/release-notes#v3-0-2020-?-?-??>`_ (`CHECK LINK`).
We review and analyze in deep the use and the combination of these on-the-fly
tools and protocols for facilities in the `D. Maluenda et.al. 'Flexible
workflows for on-the-fly electron microscopy single particle image processing
using Scipion' Acta Cryst. (2019). D75, 882-894 <https://doi.org/10.1107/S2059798319011860>`_ paper.

On top of that, we have paid special attention to data flow management and
acquisition monitoring, for that Scipion includes
a `Trigger data <https://github.com/I2PC/scipion-em-xmipp/blob/648ebe3a4f8dc2f3022332c080fb3d300d273bd7/
xmipp3/protocols/protocol_trigger_data.py#L41-L53>`_ protocol, the `Movie max shift
<https://github.com/I2PC/scipion-em-xmipp/blob/648ebe3a4f8dc2f3022332c080fb3d300d273bd7/
xmipp3/protocols/protocol_movie_max_shift.py#L43-L53>`_ protocol, the
`CTF consensus <https://github.com/I2PC/scipion-em-xmipp/blob/648ebe3a4f8dc2f3022332c080fb3d300d273bd7/
xmipp3/protocols/protocol_ctf_consensus.py#L49-L51>`_ protocol, the
`Extract coordinates <https://github.com/I2PC/scipion/blob/d1a60f69960d1079bbbecde5bf3f5f4017b36927/
pyworkflow/em/protocol/protocol_extract_coordinates.py#L44-L49>`_ protocol in
streaming and `2D classification in
streaming <http://workflows.scipion.i2pc.es/workflow_detail/56/2d_class_streaming_gl2d_assignation/>`_
(see also `GL2D-static <https://github.com/I2PC/scipion-em-xmipp/blob/648ebe3a4f8dc2f3022332c080fb3d300d273bd7/
xmipp3/protocols/protocol_classification_gpuCorr_semi.py#L68-L70>`_ and
`GL2D-streaming <https://github.com/I2PC/scipion-em-xmipp/blob/648ebe3a4f8dc2f3022332c080fb3d300d273bd7/
xmipp3/protocols/protocol_classification_gpuCorr_full.py#L68-L70>`_).
But also we have included general streaming tools, such as enabling to
*Continue* and *Restart* workflows from a certain protocol,
providing more stability in streaming protocols keeping processing new data
even if a certain bunch fails for some reason, etc.

In addition, we have work more in the concept of consensus protocols, for
merging and combining different approaches (from different EM software)
to do the same tasks, in order to get an improved result based on that
different results.
The consensus protocols family are for *CTF estimation*, *Picking*,
*Initial Volume* and *3D classes*, so far.

Moreover, we are also working in the monitor protocols, those special protocols
that constantly check how the execution of other protocols is going on. We have
developed several GUIs that are refreshed periodically and produce a graphical
summary (e.g, CTF estimated values, system load, etc).
Examples of monitors reports generated in *HTML* format can be seen at
`http://scipion.cnb.csic.es/scipionbox/lastHTMLReport <http://scipion.cnb.csic.es/scipionbox/lastHTMLReport>`_,
while you can see a Grafana based report in the URL
`http://scipion.cnb.csic.es:3000/lastHTMLReport <http://scipion.cnb.csic.es:3000/lastHTMLReport>`_.
(Unfortunately, at present, Grafana server can only be accessed from within our local area network).
Learn `how to customize the reports <customize-report>`_ according to your facilities' needs.