pyworkflow.utils.file_transfer module

Created on Apr 8, 2013

@author: antonio

class pyworkflow.utils.file_transfer.FileTransfer

Bases: object

checkFiles(filePaths, hostsPasswords, gatewayHosts=None, numberTrials=1, forceOperation=False, operationId=1)

Check if file paths exists. filepaths – List of file paths to check with this format: “userName@hostName:absolute_file_path” gatewayHosts – Gateway hosts dictionary with this format: “userName1@hostName1:userName2@hostName2”:”userName@hostName” hostsPasswords – Passwords needed to connect to involved hosts with this format: “userName@hostName”:”hostPassword” numberTrials – Number of trials in error cases. forceOperation – Flag to indicate if, when an error happens and number of trials is exceeded, the operation must continue with the rest of files. operationId – Operation identifier. returns – List of not located file paths.

checkOneHostFiles(filePaths, hostName, userName, hostsPassword, gatewayHosts=None, numberTrials=1, forceOperation=False, operationId=1)

Check if file paths exists. filepaths – List of file paths to check with this format: [“absolute_file_path1”,”absolute_file_path2”] gatewayHosts – Gateway hosts dictionary with this format: “userName1@hostName1:userName2@hostName2”:”userName@hostName” hostsPasswords – Passwords needed to connect to involved hosts with this format: “userName@hostName”:”hostPassword” numberTrials – Number of trials in error cases. forceOperation – Flag to indicate if, when an error happens and number of trials is exceeded, the operation must continue with the rest of files. operationId – Operation identifier. returns – List of not located file paths.

copyFiles(filePaths, numberTrials=1, forceOperation=False, operationId=1)

filePaths – Files dictionary with this format: “source_file_path”: “target_file_path” forceOperation – Flag to indicate if, when an error happens and number of trials is exceeded, the operation must continue with the rest of files. operationId – Operation identifier.

deleteDirectories(directoryPaths, hostsPasswords, gatewayHosts=None, numberTrials=1, forceOperation=False, operationId=1)

Delete a list of directory paths. directoryPaths – List of file paths to remove with this format: “userName@hostName:absolute_directory_path” gatewayHosts – Gateway hosts dictionary with this format: “userName1@hostName1:userName2@hostName2”:”userName@hostName” hostsPasswords – Passwords needed to connect to involved hosts with this format: “userName@hostName”:”hostPassword” numberTrials – Number of trials in error cases. forceOperation – Flag to indicate if, when an error happens and number of trials is exceeded, the operation must continue with the rest of files. operationId – Operation identifier.

deleteFiles(filePaths, hostsPasswords, gatewayHosts=None, numberTrials=1, forceOperation=False, operationId=1)

Delete a list of file paths. filepaths – List of file paths to remove with this format: “userName@hostName:absolute_file_path” gatewayHosts – Gateway hosts dictionary with this format: “userName1@hostName1:userName2@hostName2”:”userName@hostName” hostsPasswords – Passwords needed to connect to involved hosts with this format: “userName@hostName”:”hostPassword” numberTrials – Number of trials in error cases. forceOperation – Flag to indicate if, when an error happens and number of trials is exceeded, the operation must continue with the rest of files. operationId – Operation identifier.

sftp = None

ssh = None

transferFiles(filePaths, hostsPasswords, gatewayHosts=None, numberTrials=1, forceOperation=False, operationId=1)

filePaths – Files dictionary with this format: “userName@hostName:absolute_file_path”: [“userName1@hostName1:absolute_file_path1”, “userName2@hostName2:absolute_file_path2”] Key is the source file path and value the target file paths. gatewayHosts – Gateway hosts dictionary with this format: “{userName1@hostName1:userName2@hostName2”:[“userName3@hostName3”,”userName4@hostName4”]} hostsPasswords – Passwords needed to connect to involved hosts with this format: “userName@hostName”:”hostPassword” numberTrials – Number of trials in error cases. forceOperation – Flag to indicate if, when an error happens and number of trials is exceeded, the operation must continue with the rest of files. operationId – Operation identifier.

transferFilesFrom(filePaths, hostName, userName, hostPassword, gatewayHosts=None, numberTrials=1, forceOperation=False, operationId=1)

filePaths – Files dictionary with this format: “source_file_path”: “target_file_path” hostName – Remote host to transfer files. username – User name for remote host. hostsPassword – Passwords needed to connect to involved host. gatewayHosts – Gateway hosts List with this format: [“userName1@hostName1”,”userName2@hostName2”] numberTrials – Number of trials in error cases. forceOperation – Flag to indicate if, when an error happens and number of trials is exceeded, the operation must continue with the rest of files. operationId – Operation identifier.

transferFilesTo(filePaths, hostName, userName, hostPassword, gatewayHosts=None, numberTrials=1, forceOperation=False, operationId=1)

filePaths – Files dictionary with this format: “source_file_path”: “target_file_path” hostName – Remote host to transfer files. username – User name for remote host. hostsPassword – Passwords needed to connect to involved host. gatewayHosts – Gateway hosts List with this format: [“userName1@hostName1”,”userName2@hostName2”] numberTrials – Number of trials in error cases. forceOperation – Flag to indicate if, when an error happens and number of trials is exceeded, the operation must continue with the rest of files. operationId – Operation identifier.

pyworkflow.utils.file_transfer.getFilePathList(filePaths)

Get target file paths list from target file path dictionary. filePaths – Files dictionary with this format: “userName@hostName:absolute_file_path”: [“userName1@hostName1:absolute_file_path1”, “userName2@hostName2:absolute_file_path2”] returns – List of target files with this format: “useName@hostName:/filePath”.

pyworkflow.utils.file_transfer.getRemoteFiles(sftp, folderPath, recursive=True)

Recover all files in the given folder. Params:

sftp: Sftp session. folderPath: Folder to get files. recursive: if True go recursively inside other subfolders.

Returns: List of files.

pyworkflow.utils.file_transfer.getRemoteFolderFiles(hostName, userName, password, folderPath, recursive=True)

Recover all files in the given folder. Params:

hostName: Remote host name. userName: User name. password: Password. folderPath: Folder to get files. recursive: if True go recursively inside other subfolders.

Returns: List of files.

pyworkflow.utils.file_transfer.isRemoteDir(sftp, path)

Check if one remote directory exists Params:

sftp: Sftp session. path: Path to check.

Returns: True if the given path is a directory, false if it is not a directory.

pyworkflow.utils.file_transfer.removeRemoteFolder(hostName, userName, password, folderPath)

Removes a remote folder and all it content. Params:

hostName: Remote host name. userName: User name. password: Password. folderPath: Folder to delete.

Returns:

Tuple with standard input, standard output and error output.

pyworkflow.utils.file_transfer.removeTree(ssh, folderPath)

Removes a remote folder and all it content. Params:

ssh: Ssh session. folderPath: Folder to delete.

Returns:

Tuple with standard input, standard output and error output.