Flaky cache tests

[dickwolff]

The cache tests are flaky and don't always succeed. This is also a problem when running locally and needs to be addressed.

[dickwolff]

Not picking this up yet until the backlog of issues is cleared, but putting it on here to not forget it.