Allow use of frost without requiring a local clone of frost

[hwine]

Originally posted by @ajvb in #401 (comment)

Currently, while frost can be installed from pypi, that isn't very meaningful, as you still have to execute frost from the top of a clone of the frost repository. (Due to customized conftest.py, etc.)

Allowing frost to be run against an arbitrary set of tests would be useful, but it's unclear if the packaging can be munged around to do that cleanly.

Desired Behavior

When a user installs frost from pypi, it comes bundled with the current test suites. (Enhancement, package each service's test suite as a separate extension.) frost will execute only the tests bundled with the installation. Both frost and the test suites will be updated by a normal update from pypi.

This implies that when a user is developing tests, they are effectively developing frost. To do so, they would need to be working on a checked out version of the frost repository, and ensure they run the frost in the checkout via appropriate isolation techniques (virtualenv, docker, etc.)

Scheduling

This enhancement is unlikely to be considered until a Phase 2 or Phase 3 implementation time frame.

[g-k]

I think we could:

1. move top level tests and directories into the frost/ directory and update the packaging manifest (refs: https://github.com/mozilla/frost/blob/master/frost/cli.py#L16)
2. change frost commands to not cd to a package directory with switch_to_frost_parent_directory
3. change frost test to include the first-party tests path by default (with a flag to disable it), but run from the directory it's invoked in to include third party / local / dev tests

[hwine]

I think there is also the challenge of keeping pytest happy with it's perception of the world at frost execution time.

Normally, pytest runs from the root of the test suite, and all conftest.py and other files are only in that directory tree. We'll be asking it to consider some disjoint directory (wherever the frost package was installed) as the "top root", and then the normal test root. I.e. we need pytest to include the frost conftest.py file as part of the chain of magic conftest imports it does.

@g-k I think I understand #3, and that wasn't a case I had considered. But we should define the behavior we support.

I think we're conflating "using" with "developing". With most python packages, if I'm only going to use the package, I just install it via pipx or some such. If I also want to develop on it, I clone the repo, create a venv, and install in "develop" mode.

We can make a case for there being a third mode for frost "install system wide, but use my own test suite", but I'd be concerned that the costs of supporting that "unusual mode" might be high with all the magic already happening.

Let me define some behavior in original comment -- we can debate & update that, so we have clarity going forward.

[hwine]

@g-k @ajvb - I proposed a behavior specification in the opening comment, just to get a discussion going in a visible place (i.e. we'll update that when we get agreement). Have at the comments!

[g-k]

I think it'd be nice to support developing and running custom tests that:

• use frost resources e.g. EC2 instances
• run from a pypi installed package / do not require a full clone and venv of frost

[hwine]

I agree it would be nice. However, I don't think supporting that use case is as urgent as some other items.

@g-k Let me ask this: do you see specifying the "traditional package dev model" as preventing an eventual move to support this use case?

If part of your concern is the cost of a dev setup for someone who isn't a python user, I think we could do a docker image and/or add vscode support to make it "one click" for non-devs.

[g-k]

I agree it would be nice. However, I don't think supporting that use case is as urgent as some other items.

Absolutely, I know you and AJ have a roadmap.

@g-k Let me ask this: do you see specifying the "traditional package dev model" as preventing an eventual move to support this use case?

I don't. We might not want to support it anyway, since it'd probably result in confusing end user behavior and drawing a stricter distinction between running and developing might be good especially if we went another direction like allowing users to provide tests via plugin.

I mainly wanted to mention items 1 and 2 from #411 (comment) since it's tech debt that impacts how frost is packaged and run.

[hwine]

[implementation note: pytest-pythonpath may help with "pointing to the right place". See/listen/read to segment 2 of episode 220 of Python Bytes podcast for a better description of a use case that may be similar to frost.]