How to Use Skallop Fixtures

When to use it

When the System Under Test (SUT) needs to be set up and torn down in a broad and robust manner separating code managing the SUT from code testing the SUT.

More specifically if you need to run tests requiring the telescope to be automatically maintained ON/OFF, and/or subarrays to be in an IDLE/READY state; and for which the configuration settings necessary to get to that state is secondary to testing the behaviour in that state.

A large SUT with many moving “asynchronous” parts can create a large set of failure conditions requiring extensive code dealing with “dirty” states which need to be cleared during tearing down.

The skallop fixtures are in essence there to create a set of “context aware” objects, placed in a specific state and returned to that same state when the test has finished (tear down). Before doing so it checks the readiness of the system to be taken to that state and attempt to handle an unready SUT through various controls and checks.

Thus the first question to answer when selecting a skallop fixture is to determine what the given state of the SUT must be in, before the test should exercise the SUT. The table below lists the main set of fixtures and the corresponding state they are responsible for.

Fixture Name	State	Dependency	Object
telescope_context	Telescope OFF/ON	None (base)	`TelescopeContext`
running_telescope	Telescope ON	telescope_context	`TelescopeContext`
standby_telescope	Telescope OFF	telescope_context	`TelescopeContext`
allocated_subarray	Subarray IDLE	running_telescope	`SubarrayContext`
configured_subarray	Subarray READY	allocated_subarray	`SubarrayContext`

What you need before starting

Skallop package installed. The package will automatically install these fixtures.

Setting up your environment

When the skallop fixtures performs “readiness” checks and failure handling, it does so based on knowing the specific ska telescope environment the SUT is a part of.

Configuring the telescope type

The skallop package automatically selects the correct type of telescope based on user defined configuration. To set this from a host env variable, either one of the following variable names must be used:

Var Name	Value meaning SKA Low	Value meaning SKA Mid
SKA_TELESCOPE	SKA-low	SKA-Mid
TEL	low	mid

Note that the env var TEL will be set when you setup remote connections using the setenv.sh command (see Setting ENV variables using script. )

Todo

Section describing how to change scope of readiness checks once merge for feature into master done.

Using skallop fixtures

Environment Values

By default, the fixtures dealing with the overall state of the telescope will maintain the telescope in an “operational” state (e.g. ON) for the duration of the entire test session. This saves time during a test session in which each test requires an operational telescope, but may lead to time wasted when the tests themselves are about setting the telescope parts in an operational state.

Therefore this behaviour can be “opt outed” by setting the environment value DISABLE_MAINTAIN_ON to True.

Implicit Usage

The simplest way to use fixtures is to just have the necessary fixture referenced by pytest:

@pytest.mark.usefixtures('configured_subarray')
def test_scan():
    my_own_testing_code_to_run_a_scan()
    my_own_testing_code_to_check_results()

Not only will the configured_subarray be used to set up and tear down the SUT for you implicitly, the dependent fixtures needed to realise a configured subarray will also perform their set up and tear downs. Each dependency on a skallop fixture will cause the setting up and tearing down to be placed on a stack as illustrated on the diagram below:

However with each layer setting up your SUT behind the scenes, more and more configuration assumptions will be made, creating potential hidden dependencies on your test working correctly. The following key configuration settings are fixed for corresponding fixture:

Fixture	Settings
running_telescope	maintained on after test (unless overridden)
standby_telescope	switched back on after test (unless overridden)
allocated_subarray	subarray id = 1; receptor ids = 1 and 2 (for ska mid); resource configuration = `STANDARD`
configured_subarray	subarray id = 1; receptor ids = 1 and 2 (for ska mid); resource configuration = `STANDARD` scan configuration = `STANDARD`

Explicit Usage - factory_functions

Sometimes you want to have more direct control over the input parameters used for creating the given fixture. In order to use you can substitute the actual fixture with the factory function used to create the fixture. Just prepend the suffix factory_ in front of the needed fixture. This factory function is itself a fixture which either uses the default input arguments for creating the fixture or those injected by the user.

for example: .. code-block:: python

@given(“An allocated subarray with id {subarray_id}”, target_fixture=allocated_subarray) def an_allocated_subarray(

subarray_allocation_spec: fxt_types.subarray_allocation_spec, factory_allocated_subarray: fxt_types.factory_allocated_subarray

):
subarray_allocation_spec.subarray_id = subarray_id # we inject a manipulated subarray_allocation_spec as input argument return factory_allocated_subarray(subarray_allocation_spec=subarray_allocation_spec)

Explicit Usage - StackableContext

In order to have more direct control over the setup and teardown configuration the corresponding fixture objects can be used directly in your test code.

In essence your code takes responsibility for the configuration and control parts that was usually done by the fixtures themselves. In order to understand how to use these objects the concept of a StackableContext needs to be understood first.

The fixture objects enable a user to load predefined tear down and setup code as a contextmanager arguments. For example when running on pytest the following my_test() function…

@contextmanager
def my_cm():
  setup()
  yield
  teardown()

def my_test(context: StackableContext):
  context.push_context_onto_test(my_cm())

… pytest will cause the setup() function to be called immediately and the teardown() only when the test is finished. Pushing a context manager will result in tear downs loaded in a stackable queue that will be called in a FILO order. This ensures your state will be removed in a layered fashion in the same reverse order in which yuu have affected it.

The objects provided to you from the fixtures makes use of this mechanism when you are making a setup call ensuring the correct tear down always goes together with its corresponding setup. The user does therefore not have to concern the testing code with teardown when calling setup commands on the fixture object.

In essence, using fixtures explicitly requires two steps:

Determine and define Configuration: settings determining how the setup should be conducted. These variables will be used in subsequent setup call as parameters.
Setup: configures or changes SUT state with correct tearing down loaded onto pytest

Each fixture object has the corresponding setup call and input arguments as defined by the method call’s signature.

Fixture	Object	Setup call
running_telescope	`TelescopeContext`	`set_up_a_telescope()`
standby_telescope	`TelescopeContext`	`set_down_a_telescope()`
allocated_subarray	`TelescopeContext`	`allocate_a_subarray()`
configured_subarray	`SubarrayContext`	`configure()`

Note

The explicit use of fixtures requires you to ‘override’ the existing fixture; i.e. the dependency listed by the function gets a different object injected than would have been the case without explicit fixture manipulation. It is therefore important that the names of your fixtures are correct.

The example below shows how a tester sets up a configured subarray by explicitly calling the subarray setup commands (telescope fixtures are still implicit). Note the use of the namespaced object fxt_types to explicitly get the correct fixture type resulting from injection.

# overrides allocated_subarray
# note the return statement as the result of the
# allocate command return a subarray_context object
@pytest.fixture(name="allocated_subarray")
def fxt_allocated_subarray(
    running_telescope: fxt_types.running_telescope, exec_settings
) -> fxt_types.allocated_subarray:
    subarray_configuration = get_my_subarray_configuration(id=1)
    return running_telescope.allocate_a_subarray(
        subarray_configuration.id,
        subarray_configuration.receptors,
        subarray_configuration.sb_config,
        exec_settings,
        subarray_configuration.composition,
    )

# overrides configured_subarray
# note the return statement as the result of the configure
# command returns the same object after being affected
@pytest.fixture(name="configured_subarray")
def fxt_configured_subarray(
    allocated_subarray: fxt_types.allocated_subarray, exec_settings
) -> fxt_types.configured_subarray:
    subarray_configuration = get_my_subarray_scan_configuration(id=1)
    return allocated_subarray.configure(
        subarray_configuration.configuration,
        subarray_configuration.duration,
        exec_settings,
    )


@pytest.mark.usefixtures("configured_subarray")
def test(configured_subarray: fxt_types.configured_subarray):
    set_up_my_test()
    exercise_my_test()
    check_my_test()

Using Fixture objects automatic teardown

Often the exercising of a test has a known change in state and therefore a standard teardown to go with it. The tester can therefore make use of the fixture object’s api to set specific tear downs at points just before the test will be exercised.

The list of api methods to use for setting up your tear down for the corresponding fixture object is shown in the table below:

Type of Test	Fixture	Object	tear down
switch on telescope	standby_telescope	`TelescopeContext`	`switch_off_after_test()`
allocate a subarray	running_telescope	`TelescopeContext`	`release_subarray_when_finished()`
configure a subarray	allocated_subarray	`SubarrayContext`	`clear_configuration_when_finished()`
run a scan on a subarray	configured_subarray	`SubarrayContext`	`check_configuration_when_finished()`

The examples below shows how each test sets a specific teardown to go with it. Note the tester is only required to set the SUT back to the state it received the fixture in. The fixture itself will take care of tearing down itself afterwards.

def test_set_to_running(standby_telescope: fxt_types.standby_telescope, exec_settings):
    set_up_my_test_for_setting_it_to_running()
    # sets a teardown to switch telescope off at the end
    standby_telescope.switch_off_after_test(exec_settings)
    exercise_my_test_startup()
    check_my_test()
    # switch off will happen automatically


def test_allocate_subarray(
    running_telescope: fxt_types.running_telescope, exec_settings
):
    configuration = set_up_my_test_for_allocating_a_subarray()
    # sets a teardown to release subarray
    running_telescope.release_subarray_when_finished(
        configuration.subarray_id, configuration.receptors, exec_settings
    )
    exercise_my_test_allocate(configuration)
    check_my_test()
    # subarray release will happen automatically


def test_configure_subarray(
    allocated_subarray: fxt_types.allocated_subarray, exec_settings
):
    configuration = set_up_my_test_for_configuring_a_subarray()
    # sets a teardown to clear subarray configuration (take to state IDLE)
    allocated_subarray.clear_configuration_when_finished(exec_settings)
    exercise_my_test_configure(configuration)
    check_my_test()
    # subarray release will happen automatically


def test_scan_subarray(
    configured_subarray: fxt_types.configured_subarray, exec_settings
):
    configuration = set_up_my_test_for_configuring_a_subarray()
    # sets a teardown to check configuration (no waiting)
    configured_subarray.check_configuration_when_finished(exec_settings)
    exercise_my_test_scan(configuration)
    # assumes waiting for scan to complete happens
    check_my_test()
    # subarray will automatically check scanning completed correctly