EchoStorage

Echo module based on the GFAL2_StorageBase class.

class DIRAC.Resources.Storage.EchoStorage.EchoStorage(storageName, parameters)

Bases: DIRAC.Resources.Storage.GFAL2_StorageBase.GFAL2_StorageBase

class EchoStorage

Interface to the Echo storage.

This plugin will work with both gsiftp and root protocol. According to the RAL admins, Echo best use is done:

using gsiftp for all WAN transfers, and LAN write.
using root from LAN read.

Note that there are still a few issues to be sorted out with xroot

This would translate in a configuration such as:

RAL-ECHO
{
  BackendType = Echo
  SEType = T0D1
  AccessProtocols = gsiftp,root
  WriteProtocols = gsiftp
  XRootConfig
  {
    Host = xrootd.echo.stfc.ac.uk
    PluginName = Echo
    Protocol = root
    Path = lhcb:user
    Access = remote
  }
  GidFTPConfig
  {
    Host = gridftp.echo.stfc.ac.uk
    PluginName = Echo
    Protocol = gsiftp
    Path = lhcb:user
    Access = remote
  }
}
Operations
{
  Defaults
  {
    DataManagement
    {
      ThirdPartyProtocols=srm,gsiftp,dips
      RegistrationProtocols=srm,gsiftp,dips
    }
  }
}

DYNAMIC_OPTIONS = {}

PROTOCOL_PARAMETERS = ['Protocol', 'Host', 'Path', 'Port', 'SpaceToken', 'WSUrl']

__init__(storageName, parameters): c’tor

changeDirectory(directory): Change the directory to the supplied directory

constructURLFromLFN(lfn, withWSUrl=False)

Construct URL from the given LFN according to the VO convention for the primary protocol of the storage plagin

Parameters

lfn (str) – file LFN
withWSUrl (boolean) – flag to include the web service part into the resulting URL

Return result

result[‘Value’] - resulting URL

createDirectory(path) → Union[DIRAC.Core.Utilities.ReturnValues.DOKReturnType[DIRAC.Core.Utilities.ReturnValues.T], DIRAC.Core.Utilities.ReturnValues.DErrorReturnType]

Create directory on the storage

Parameters: path (str) – path to be created on the storage (pfn : srm://…)
Returns: Successful dict {path : True } Failed dict {path : error message } S_ERROR in case of argument problems

exists(path) → Union[DIRAC.Core.Utilities.ReturnValues.DOKReturnType[DIRAC.Core.Utilities.ReturnValues.T], DIRAC.Core.Utilities.ReturnValues.DErrorReturnType]

Check if the path exists on the storage

Parameters: path (str) – path or list of paths to be checked
Returns: Failed dictionary: {pfn : error message} Successful dictionary: {pfn : bool} S_ERROR in case of argument problems

getCurrentDirectory(): Get the current directory

getCurrentURL(fileName)

Obtain the current file URL from the current working directory and the filename

Parameters

self – self reference
fileName (str) – path on storage

getDirectory(path, localPath=False)

Not available on Echo

Returns: S_ERROR

getDirectoryMetadata(path)

Not available on Echo

Returns: S_ERROR

getDirectorySize(path)

Not available on Echo

Returns: S_ERROR

getEndpoint()

This will get endpoint of the storage. It basically is the same as getURLBase(): but without the basePath

Returns: ‘proto://hostname<:port>’

getFile(path, localPath=False) → Union[DIRAC.Core.Utilities.ReturnValues.DOKReturnType[DIRAC.Core.Utilities.ReturnValues.T], DIRAC.Core.Utilities.ReturnValues.DErrorReturnType]

Make a local copy of storage :path:

Parameters

path (str) – path (or list of paths) on storage (srm://…)
localPath – destination folder. Default is from current directory

Returns

Successful dict: {path : size} Failed dict: {path : errorMessage} S_ERROR in case of argument problems

getFileMetadata(path) → Union[DIRAC.Core.Utilities.ReturnValues.DOKReturnType[DIRAC.Core.Utilities.ReturnValues.T], DIRAC.Core.Utilities.ReturnValues.DErrorReturnType]

Get metadata associated to the file(s)

Parameters: path (str) – path (or list of paths) on the storage (srm://…)
Returns: successful dict { path : metadata } failed dict { path : error message } S_ERROR in case of argument problems

getFileSize(path) → Union[DIRAC.Core.Utilities.ReturnValues.DOKReturnType[DIRAC.Core.Utilities.ReturnValues.T], DIRAC.Core.Utilities.ReturnValues.DErrorReturnType]

Get the physical size of the given file

Parameters: path – path (or list of path) on storage (pfn : srm://…)
Returns: Successful dict {path : size} Failed dict {path : error message } S_ERROR in case of argument problems

getName(): The name with which the storage was instantiated

getOccupancy(**kwargs)

Get the StorageElement occupancy info in MB.

This generic implementation download a json file supposed to contain the necessary info.

Parameters: occupancyLFN – (mandatory named argument) LFN of the json file.
Returns: S_OK/S_ERROR dictionary. The S_OK value should contain a dictionary with Total and Free space in MB

getParameters(): Get the parameters with which the storage was instantiated

getTransportURL(pathDict, protocols)

Get a transport URL for a given URL. For a simple storage plugin it is just returning input URL if the plugin protocol is one of the requested protocols

Parameters

pathDict (dict) – URL obtained from File Catalog or constructed according to convention
protocols (python:list) – a list of acceptable transport protocols in priority order

getURLBase(withWSUrl=False)

This will get the URL base. This is then appended with the LFN in DIRAC convention.

Parameters

self – self reference
withWSUrl (bool) – flag to include Web Service part of the url

Returns

URL

isDirectory(path)

Not available on Echo

Returns: S_ERROR

isFile(path) → Union[DIRAC.Core.Utilities.ReturnValues.DOKReturnType[DIRAC.Core.Utilities.ReturnValues.T], DIRAC.Core.Utilities.ReturnValues.DErrorReturnType]

Check if the path provided is a file or not

Parameters: str – path or list of paths to be checked ( ‘srm://…’)
Returns: Failed dict: {path : error message} Successful dict: {path : bool} S_ERROR in case of argument problems

isNativeURL(url)

Check if URL :url: is valid for :self.protocol:

Parameters

self – self reference
url (str) – URL

isURL(path)

Guess if the path looks like a URL

Parameters

self – self reference
path (string) – input file LFN or URL

Returns boolean

True if URL, False otherwise

listDirectory(path)

Not available on Echo

Returns: S_ERROR

prestageFile(path, lifetime: int = 86400) → Union[DIRAC.Core.Utilities.ReturnValues.DOKReturnType[DIRAC.Core.Utilities.ReturnValues.T], DIRAC.Core.Utilities.ReturnValues.DErrorReturnType]

Issue prestage request for file(s)

Parameters

path (str) – path or list of paths to be prestaged
lifetime (int) – prestage lifetime in seconds (default 24h)

Returns

succesful dict { url : token } failed dict { url : message } S_ERROR in case of argument problems

prestageFileStatus(path) → Union[DIRAC.Core.Utilities.ReturnValues.DOKReturnType[DIRAC.Core.Utilities.ReturnValues.T], DIRAC.Core.Utilities.ReturnValues.DErrorReturnType]

Checking the staging status of file(s) on the storage

Parameters: path (dict) – dict { url : token }
Returns: succesful dict { url : bool } failed dict { url : message } S_ERROR in case of argument problems

putDirectory(path)

Not available on Echo

Returns: S_ERROR

putFile(path, sourceSize: int = 0) → Union[DIRAC.Core.Utilities.ReturnValues.DOKReturnType[DIRAC.Core.Utilities.ReturnValues.T], DIRAC.Core.Utilities.ReturnValues.DErrorReturnType]

Put a copy of a local file or a file on another srm storage to a directory on the physical storage.

Parameters

path – dictionary { lfn (srm://…) : localFile }
sourceSize – size of the file in byte. Mandatory for third party copy (WHY ???) Also, this parameter makes it essentially a non bulk operation for third party copy, unless all files have the same size…

Returns

Successful dict: { path : size } Failed dict: { path : error message } S_ERROR in case of argument problems

releaseFile(path) → Union[DIRAC.Core.Utilities.ReturnValues.DOKReturnType[DIRAC.Core.Utilities.ReturnValues.T], DIRAC.Core.Utilities.ReturnValues.DErrorReturnType]

Release a pinned file

Parameters: path (str) – PFN path { pfn : token } - pfn can be an empty string, then all files that have that same token get released. Just as you can pass an empty token string and a directory as pfn which then releases all the files in the directory an its subdirectories
Returns: successful dict {url : token}, failed dict {url : message} S_ERROR in case of argument problems

removeDirectory(path, recursive=False)

Not available on Echo

Returns: S_ERROR

removeFile(path) → Union[DIRAC.Core.Utilities.ReturnValues.DOKReturnType[DIRAC.Core.Utilities.ReturnValues.T], DIRAC.Core.Utilities.ReturnValues.DErrorReturnType]

Physically remove the file specified by path

A non existing file will be considered as successfully removed

Parameters: path (str) – path (or list of paths) on storage (srm://…)
Returns: Successful dict {path : True} Failed dict {path : error message} S_ERROR in case of argument problems

resetCurrentDirectory(): Reset the working directory to the base dir

setParameters(parameterDict): Set standard parameters, method can be overriden in subclasses to process specific parameters

setStorageElement(se)

updateURL(url, withWSUrl=False): Update the URL according to the current SE parameters