Test Data Manager

To simplify and organize handling of data to be used by BDD and system tests a test data manager module has been added to allow the tester to:

upload test data
download test data
list the existing test data under a directory

These files will be hosted and managed on the artefact repository (using nexus at the moment) in a directory tree. Each listing of files under a sub-directory will be a mapping to a component. This is captured under the named argument artefact_directory in all three functions. For e.g. if the test data file will be used for subarray tests, then the destination may be like test/subarray/.

Note the positions of the slashes in the artefact directory in each usage case.

Install

Install the ska-ser-skallop Python package.

pip3 install -U ska-ser-skallop --extra-index-url https://artefact.skao.int/repository/pypi-internal/simple

Once installed the data-manager script will be available in your Python bin directory.

Usage

Upload Test Data

To upload test data:

from skallop.bdd_test_data_manager.data_manager import upload_test_data

def main():
    outcome = upload_test_data("/path/to/test_data", "destination/directory/")
    if outcome:
        # upload was successful

The command-line utility works as follows:

$ data-manager upload --help

usage: data-manager upload [-h] [-f FILE_PATH] [-a ARTEFACT_DIRECTORY]

optional arguments:
  -h, --help            show this help message and exit
  -f FILE_PATH, --file-path FILE_PATH
                        The absolute path of the file to be uploaded
  -a ARTEFACT_DIRECTORY, --artefact-directory ARTEFACT_DIRECTORY
                        e.g. path/to/files/

Download Test Data

To download test data:

from skallop.bdd_test_data_manager.data_manager import download_test_data

def main():
    path_to_downloaded_file = download_test_data("test_data", "destination/directory/")
    with open(path_to_downloaded_file) as test_data:
        # do something

The command-line utility works as follows:

$ data-manager download --help

usage: data-manager download [-h] [-f FILE_NAME] [-a ARTEFACT_DIRECTORY]

optional arguments:
  -h, --help            show this help message and exit
  -f FILE_NAME, --file-name FILE_NAME
                        The name of the file on the repository
  -a ARTEFACT_DIRECTORY, --artefact-directory ARTEFACT_DIRECTORY
                        e.g. path/to/files/

Search for Test Data

To list test data in a directory:

from skallop.bdd_test_data_manager.data_manager import search_for_file

def main():
    data_files = search_for_file("*", "/destination/directory")
    if "destination/directory/some_specific_file" in data_files:
        # do something

$ data-manager download --help

usage: data-manager search [-h] [-f FILTER] [-a ARTEFACT_DIRECTORY]

optional arguments:
  -h, --help            show this help message and exit
  -f FILTER, --filter FILTER
                        The name of the file on the repository. It can also be
                        a wildcard which will have all files listed in that
                        directory
  -a ARTEFACT_DIRECTORY, --artefact-directory ARTEFACT_DIRECTORY
                        e.g. /path/to/files