tests pass/fail randomly

[DavidAnderegg]

The following snipped sometimes fails and sometimes passes

import unittest

class DummyVariable:
    def __init__(self) -> None:
        self.flag = False

    def activate_flag(self):
        if self.flag:
            raise ValueError('Already flagged')
        self.flag = True


variable = DummyVariable()

class TestVariable(unittest.TestCase):

    def test_1(self):
        variable.activate_flag()

    def test_2(self):
        variable.activate_flag()

This is due to the consistent state of variable. As far as I understand, testflo executes these tests in different subprocesses based on randomization. If both tests happen to be run in the same process, the second test fails. If both tests happen to be run on different processes, the second test passes.

If I specify the -i, --isolated flag which forces testflo to run each test in its own subprocess, both tests pass all the time.

If I run this tests with python -m unittest, the second test fails all the time.

You could argue the snipped above is bad test design and I would agree but I think it is very dangerous to have such random behavior in a test setup. I think it should either fail or pass in a consistent way.

[naylor-b]

I agree with you that we should change the default behavior of testflo to be safer. Right now, it uses the 'big boy rules' philosophy where it just tries to run the tests as fast as it can and assumes that you've made your tests so they can run properly under concurrent execution. We could make testflo run in isolated mode by default and just provide an option to turn that off. The other thing that often causes 'random' test failures is when multiple tests write output files with the same names to the same directory. Maybe we could also by default run each test in a temp directory while providing a flag to turn that off globally, and maybe some other way to turn it off for individual tests or test cases. This would slow things down but would eliminate most of the 'random' failures.