[uss_qualifier] Add filter option when running tests

[the-glu]

When developing / debugging the application, we usually want to run a specific test (or subset of tests), but in the context of existing configuration set.

I usually used the approach of 'commenting unwanted tests' in yml configuration files, but that can be tedious, especially since they reference them self.

This PR propose to add a new option, --filter, that, if used, will apply a regex on the test scenario ran.

Eg. to run ConflictEqualPriorityNotPermitted, --filter ConflictEqualPriorityNotPermitted can be used. Or to run the two conflicts tests, --filter ConflictEqualPriorityNotPermitted\|ConflictHigherPriority (or just --filter Conflict in that case).

[BenjaminPelletier]

A key feature of uss_qualifier is that the test configuration fully determines the test behavior. This is important because we generate test reports with identifiers describing what happened in the test, and those identifiers are computed as hashes of the test configuration. This option would change test behavior but not change those identifiers which would be problematic.

Instead, portions of a test configuration can be selectively executed using execution control; for example, in uspace_f3548.

[the-glu]

A key feature of uss_qualifier is that the test configuration fully determines the test behavior. This is important because we generate test reports with identifiers describing what happened in the test, and those identifiers are computed as hashes of the test configuration. This option would change test behavior but not change those identifiers which would be problematic. Instead, portions of a test configuration can be selectively executed using execution control; for example, in uspace_f3548.

The goal here it mostly to debug test configuration / tests, rather than having to maintain copy/references of tests in specific tests config, and only for those cases. I see then as a different case that having a different configuration.

If the usage of the option would change identifiers, similar to the 'dirty' flag when there are uncommitted git changes , would that be acceptable or should we close that PR? I still think the option can ease development, because currently I mostly do mass comment/uncomment in configuration files :)

[BenjaminPelletier]

The goal here it mostly to debug test configuration / tests, rather than having to maintain copy/references of tests in specific tests config, and only for those cases.

This is the intended objective of the execution control functionality -- instead of temporarily adding a command line flag (presumably run_locally.sh is still being used because it provides a lot of necessary command line options for the actual docker run/uv run command), the developer would temporarily add something like the below to the configuration:

execution:
  include_action_when:
    - is_action_generator: {}
    - is_test_suite: {}
    - is_test_scenario:
        types: [scenarios.astm.utm.nominal_planning.conflict_higher_priority.ConflictHigherPriority]

I think it would be totally reasonable to enable a particular use case with less friction, with something like:

execution:
  exclude_action_when:
    - is_test_scenario: {}
      regex_does_not_match_type: Conflict

or

execution:
  exclude_action_when:
    - is_test_scenario
      except_when:
        regex_matches_type: Conflict

or

execution:
  include_action_when:
    - enables_running_scenario:
        regex_matches_type: Conflict

uspace_f3548 is defined as a persistent separate configuration because running just the F3548 parts of the U-space configuration is a common use case and it is useful to have a demonstration of the execution control functionality.

I still think the option can ease development, because currently I mostly do mass comment/uncomment in configuration files :)

Could you elaborate on why you mass comment/uncomment rather than using the execution control functionality?

[BenjaminPelletier]

Per Tuesday discussion, I think I'm expecting adjustments which:

1. Ensure that test identifiers (especially baseline identifier & environment identifier) change when an execution-control argument is used
2. Ensure that the basic idea of how test identifiers are generated remains relatively transparent (e.g., "it's just a hash of the codebase version + relevant config content")
3. Keep added complexity to a minimum

I think those are the only major constraints I have in mind, and the current PR content already meets 2 & 3.

[the-glu]

I updated the PR with the filter now stored in the ExecutionConfiguration. This have the positive side effects of changing the baseline identifier (¹) and saving the value in the report.

¹ Remark / question: the environment identified is not modified, but if I understand correctly how signatures are computed, the environment is not linked to things executed, correct?