datafs.managers package

Submodules

datafs.managers.manager module

class datafs.managers.manager.BaseDataManager(table_name)

Bases: object

Base class for DataManager metadata store objects

Should be subclassed. Not intended to be used directly.

TimestampFormat = '%Y%m%d-%H%M%S'

add_tags(archive_name, tags)

Add tags to an archive

Parameters:

• archive_name (s tr) – Name of archive
• tags (list or tuple of strings) – tags to add to the archive

batch_get_archive(archive_names)

Batched version of _get_archive_listing()

Returns a list of full archive listings from an iterable of archive names

Note

Invalid archive names will simply not be returned, so the response may not be the same length as the supplied archive_names.

Parameters:archive_names (list) – List of archive names Returns:archive_listings – List of archive listings Return type:list

create_archive(archive_name, authority_name, archive_path, versioned, raise_on_err=True, metadata=None, user_config=None, tags=None, helper=False)

Create a new data archive

Returns:archive – new DataArchive object Return type:object

create_archive_table(table_name, raise_on_err=True)

Parameters:

• table_name (str) –
• a table to store archives for your project (Creates) –
• creates and populates a table with basic spec for user and (Also) –
• config (metadata) –

Returns:

Return type:

None

classmethod create_timestamp()

Utility function for formatting timestamps

Overload this function to change timestamp formats

delete_archive_record(archive_name)

Deletes an archive from the database

Parameters:archive_name (str) – name of the archive to delete

delete_table(table_name=None, raise_on_err=True)

delete_tags(archive_name, tags)

Delete tags from an archive

Parameters:

• archive_name (s tr) – Name of archive
• tags (list or tuple of strings) – tags to delete from the archive

get_archive(archive_name)

Get a data archive given an archive name

Returns:archive_specification – archive_name: name of the archive to be retrieved authority: name of the archive’s authority archive_path: service path of archive Return type:dict

get_latest_hash(archive_name)

Retrieve the file hash for a given archive

Parameters:archive_name (str) – name of the archive for which to retrieve the hash Returns:hashval – hash value for the latest version of archive_name Return type:str

get_metadata(archive_name)

Retrieve the metadata for a given archive

Parameters:archive_name (str) – name of the archive to be retrieved Returns:metadata – current archive metadata Return type:dict

get_tags(archive_name)

Returns the list of tags associated with an archive

get_version_history(archive_name)

required_archive_metadata

required_archive_patterns

required_user_config

search(search_terms, begins_with=None)

Parameters:search_terms (str) –

strings of terms to search for

If called as api.manager.search(), search_terms should be a list or a tuple of strings

set_required_archive_metadata(metadata_config)

Sets required archive metatdata for all users

Parameters:metadata_config (dict) –

Dictionary of required archive metada and metadata field descriptions. All archives created on this manager table will be required to have these keys in their archive’s metadata.

If the archive metadata does not contain these keys, an error will be raised with the description in the value associated with the key.

set_required_archive_patterns(required_archive_patterns)

Sets archive_name regex patterns for the enforcement of naming conventions on archive creation

Parameters:required_archive_patterns (strings of args) –

set_required_user_config(user_config)

Sets required user metadata for all users

Parameters:user_config (dict) –

Dictionary of required user metadata and metadata field descriptions. All archive creation and update actions will be required to have these keys in the user_config metadata.

If the archive or version metadata does not contain these keys, an error will be raised with the descrtiption in the value associated with the key.

table_names

update(archive_name, version_metadata)

Register a new version for archive archive_name

Note

need to implement hash checking to prevent duplicate writes

update_metadata(archive_name, archive_metadata)

Update metadata for archive archive_name

update_spec_config(document_name, spec)

Set the contents of a specification document by name

This method should not be used directly. Instead, use set_required_user_config() or set_required_archive_metadata().

Parameters: document_name : str

Name of a specification document’s key

spec : dict

Dictionary metadata specification

datafs.managers.manager_dynamo module

class datafs.managers.manager_dynamo.DynamoDBManager(table_name, session_args=None, resource_args=None)

Bases: datafs.managers.manager.BaseDataManager

Parameters:

• table_name (str) – Name of the data archive table
• session_args (dict) – Keyword arguments used in initializing a boto3.Session object
• resource_args (dict) – Keyword arguments used in initializing a dynamodb ServiceResource object

config

datafs.managers.manager_mongo module

class datafs.managers.manager_mongo.MongoDBManager(database_name, table_name, client_kwargs=None)

Bases: datafs.managers.manager.BaseDataManager

Parameters:

• database_name (str) – Name of the database containing the DataFS tables
• table_name (str) – Name of the data archive table
• client_kwargs (dict) – Keyword arguments used in initializing a pymongo.MongoClient object

collection

config

database_name

db

spec_collection

table_name

Module contents