xmipp3.protocols.protocol_screen_deepConsensus module

Deep Consensus picking protocol

class xmipp3.protocols.protocol_screen_deepConsensus.XmippProtDeepConsSubSet(**args)

Bases: ProtUserSubSet

Create subsets from the GUI for the Deep Consensus protocol. This protocol will be executed mainly calling the script ‘pw_create_image_subsets.py’ from the ShowJ gui. The enabled/disabled changes will be stored in a temporary sqlite file that will be read to create the new subset.

class xmipp3.protocols.protocol_screen_deepConsensus.XmippProtScreenDeepConsensus(**args)

Bases: ProtParticlePicking, XmippProtocol

Protocol to compute a smart consensus between different particle picking algorithms. The protocol takes several Sets of Coordinates calculated by different programs and/or different parameter settings. Let’s say: we consider N independent pickings. Then, a neural network is trained using different subset of picked and not picked cooridantes. Finally, a coordinate is considered to be a correct particle according to the neural network predictions. In streaming, the network is trained and used to predict in batches. The network is trained until the number of particles set is reached, meanwhile, a preliminary output is generated. Once the threshold is reached, the final output is produced by batches.

ADD_DATA_TRAIN_CUST = 2

ADD_DATA_TRAIN_CUSTOM_OPT = ['Particles', 'Coordinates']

ADD_DATA_TRAIN_CUSTOM_OPT_COORS = 1

ADD_DATA_TRAIN_CUSTOM_OPT_PARTS = 0

ADD_DATA_TRAIN_NONE = 0

ADD_DATA_TRAIN_PRECOMP = 1

ADD_DATA_TRAIN_TYPES = ['None', 'Precompiled', 'Custom']

ADD_MODEL_TRAIN_NEW = 0

ADD_MODEL_TRAIN_PRETRAIN = 1

ADD_MODEL_TRAIN_PREVRUN = 2

ADD_MODEL_TRAIN_TYPES = ['New', 'Pretrained', 'PreviousRun']

CONSENSUS_COOR_PATH_TEMPLATE = 'consensus_coords_%s'

CONSENSUS_PARTS_PATH_TEMPLATE = 'consensus_parts_%s'

ENDED = False

EXTRACTING = {'AND': False, 'NOISE': False, 'OR': False}

LAST_ROUND = False

NET_TEMPLATE = 'nnetData{}'

PARTICLES_TEMPLATE = 'particles{}.xmd'

PREDICTING = False

PREDICT_BATCH_MAX = 20

PREPROCESSING = False

PREPROCESS_BATCH_MAX = 200

PRE_PROC_MICs_PATH = 'preProcMics'

TO_EXTRACT_MICFNS = {'ADDITIONAL_COORDS_FALSE': [], 'ADDITIONAL_COORDS_TRUE': [], 'AND': [], 'NOISE': [], 'OR': []}

TO_TRAIN_MICFNS = []

TRAINED_PARAMS_PATH = 'trainedParams.pickle'

TRAINING = False

TRAIN_BATCH_MAX = 20

USING_INPUT_COORDS = False

USING_INPUT_MICS = False

allFree()

Kind of “traficlight” that specifies if there is not extraction, training or prediction going on, which would alterate the states of the protocol

calculateCoorConsensusStep(outCoordsDataPath, mode)

Calculates the consensus coordinates from micrographs whose particles haven’t been extracted yet in “mode”

checkIfNewMics(mode='')

Check if the are new micrographs ready for extracting particles

checkIfParentsFinished()

Check the streamState of the coordinates input to check if the parent protocols are finsihed

cnnFree()

counter = 0

createFinalOutput(closeStream=False)

createOutputStep(closeStream=False)

createPreliminarOutput(trPass)

doTraining()

Prepares the positive (AND) and negative (NOISE) coordinates for the training and executes it

endPredictingStep()

endProtocolStep()

Finish the protocol with a final prediction using the final CNN

endTrainingStep()

extractParticles(mode)

Extract the particles from a set of micrographs with their corresponding coordinates

getAllCoordsInputMicrographs(shared=False)

Returns a dic {micFn: mic} with the input micrographs present associated with all the input coordinates sets. If shared, the list contains only those micrographs present in all input coordinates sets, else the list contains all microgrpah present in any set (Intersection vs Union) Do not create a set, because of concurrency in the database

getCoordinatesOutput()

getExtractedMicFns(mode)

Return the list of extracted micrograph filenames (micrographs where particles of type “mode” have been extracted)

getInpCoordsFns(mode, extractedSetOfCoordsFns)

getInputMicsFns(shared)

Returns the input micrographs filenames

getMicrographFnsWithCoordinates(shared=True)

Return a list with the filenames of those microgrpahs which already have coordinates associated in the input sets. If shared, it must be in all the sets, if not shared, at least in one

getMicsIds(filterOutNoCoords=False)

Returns the input micrographs Ids

getParticlesOutput(partSet)

getPreCoordinatesOutput()

getPreParticlesOutput(partSet)

getPreProcParamsFromForm()

getPredictedMicFns()

Return the list of microgrpahs whose particles have been used for prediction

getPreprocessedMicFns()

Return the list of preprocessed micrograph filenames

getTrainedMicFns()

Return the list of microgrpahs whose particles have been used for training

initializeStep()

Create paths where data will be saved

insertCaculateConsensusSteps(mode, prerequisites)

Insert the steps neccessary for calculating the consensus coordinates of type “mode”

insertExtractPartSteps(mode, prerequisites)

Inserts the steps necessary for extracting the particles from the micrographs

joinSetOfParticlesStep(mode, micFns='', trainingPass='', clean=False)

Stores the particles extracted from a set of micrographs in a images.xmd metadata file

justPredict()

lastRoundStep()

Starts the last round of training and predictions with the remainign microgrpahs when all the inputs have arrived

loadCoords(posCoorsPath, mode, micSet=[])

loadMeanAccuracy()

loadMicSetFromFns(inputDir, micFns)

Returns a set of Micrographs from their filenames

loadTrainedParams()

Load the dictionary stored in pickle format which stores the trained parameters. Creates a initial one if it does not exist yet

networkReadyToPredict()

Returns true if the CNN is trained or the user specified it does not need to be trained

pickNoise()

Find noise coordinates from micrographs in order to use them as negatives in the training process

predictCNN()

Predict the particles from the micrographs and calificates the consensus coordinates

predictionsOn()

Return a boolean for whether to perform a prediction. True if there must be a preliminar prediction or if the training process has finished (trainingPass==’‘)

preprocessMicsStep()

Step which preprocesses the input micrographs

prunePaths(paths)

readyPreliminarPrediction()

Return a boolean for whether to perform a preliminar predition. True if the user set it and the current trained network has not been used yet

readyToExtractMicFns(mode)

Return the list of micrograph filenames which are ready to be extracted (preprocessed and common for all inputs) and have not or are not being extracted yet

readyToPredictMicFns()

Return the list of micrograph filenames which are ready to be used for prediction and have not or are not being predicted yet

readyToPreprocessMics(shared)

Return the list of micrograph filenames which are ready to be preprocessed and have not been preprocessed yet

readyToTrainMicFns()

Return the list of micrograph filenames which are ready to be used for training and have not or are not being trained yet

retrievePreviousPassModel(trPass, lastTrPass='')

Retrieves a previous CNN model and copies its folders to used the network in a new location

retrievePreviousRunModel(prevProt, trPass='')

Retrieves a CNN model from other protocol and copies its folders to used the network in a new location

retrieveTrainSets()

Retrieve, link and return a setOfParticles corresponding to the NegativeTrain DeepConsensus trainning set with certain extraction conditions (phaseFlip/invContrast)

saveTrainedParams(params)

Save the trained parameters dictionary

trainCNN(toTrainMicFns)

Trains the CNN with the particles from the ready to train micrographs

trainingOn()

Return a boolean for whether to perform training. True if the training must not be skipped and if the finish training criteria has not been reached yet

updateOutput(closeStream=False)

updatePreOutput(closeStream=False)

uploadTrainedParam(keyParam, newValue)

Upload the value of a parameter from the trained parameters

waitFreeInputCoords()

waitFreeInputMics()