Tests are incredibly slow after 8.23.0 (9.x included)

[AJenbo]

We use RefreshDatabase in our test to avoid running the migration from scratch on each test and instead use savepoints/transactions. It appears that since 8.23.0 migrations are now being forced to run for each test similar to how DatabaseMigrations is intended to work.

My guess is that this is a workaround that was made for some people who incorrectly using RefreshDatabase in combination with SQLite, for Laravel 10 and older this was not supported and would fall back to the DatabaseMigrations behavior, but in Laravel 11 RefreshDatabase now works correctly for SQLite. But if some project was actually incompatible with this behavior and expecting behavior similar to DatabaseMigrations then they might have experienced things working anymore using RefreshDatabase.

Our test now takes 33 minutes to run in GitHub Action, before that it was 31 seconds, are there any settings that can be made to avoid this behavior?

[AJenbo]

This looks to be the change that breaks things: 413d10b

I can understand that some users might have complained that there migrations didn't work but they should have switched away from RefreshDatabase rather then breaking it.

https://github.com/orchestral/testbench-core/releases/tag/v9.0.9 this goes against the whole point of LazilyRefreshDatabase, and why only force it for SQLite :memory:, this really appears to be trying to make it mimic the missing feature in Laravel 10 and not something that makes general sens (forcing it for all database types).

[crynobone]

The tests would only reset the database when all following conditions are met:

1. Uses in-memory SQLite
2. Uses RefreshDatabase or LazilyRefreshDatabase
3. Uses Testbench migration helpers such as loadMigrationsFrom.

Alongside 3, you can expect it to work just like Laravel Application tests where migrations and seeders are fixed for the whole testsuite.

Alternatively, You can also use persistent database connection since you are using already using savepoint and transaction.

[AJenbo]

Thank you for the quick response.

The tests would only reset the database when all following conditions are met.

Why does it do this at all for RefreshDatabase or LazilyRefreshDatabase?

• We use in-memory SQLite for testing, less dependencies better performance.
• We use LazilyRefreshDatabase to avoid rerunning the migration multiple times.
• In our service provider we have $this->loadMigrationsFrom(__DIR__ . '/../../database/migrations'); are you suggesting we load migrations in a different way?

Alternatively, You can also use persistent database connection since you are using already using savepoint and transaction.

Well that is what it does normally in Laravel 11 when using LazilyRefreshDatabase, what extra steps are needed for when using orchestral/testbench?

We use our package in two Laravel projects and there migrations do not force reset, it's only when running the package's own test though orchestral/testbench that it forces the migrations to run from scratch for every single test despite using LazilyRefreshDatabase. If I wanted to run all migrations for every test I would use DatabaseMigrations instead as per the documentation: https://laravel.com/docs/11.x/database-testing#resetting-the-database-after-each-test

[crynobone]

We use in-memory SQLite for testing, less dependencies better performance.

Persistent database already works best for your scenario, Packages developers tend to use In-memory SQLite because it doesn't requires database cleanup between tests. I'm not changing this.

Well that is what it does normally in Laravel 11 when using LazilyRefreshDatabase, what extra steps are needed for when using orchestral/testbench?

Just setup a persistent database connection and you should be good to go. Laravel Framework GitHub Actions has a lot of example for this: https://github.com/laravel/framework/blob/11.x/.github/workflows/databases.yml

[AJenbo]

Persistent database already works best for your scenario, Packages developers tend to use In-memory SQLite because it doesn't requires database cleanup between tests. I'm not changing this.

I don't think you understand, we are working on a package. We only have this issue with package development.

Just setup a persistent database connection and you should be good to go.

That's not helpful for developers running things locally, it's not just affecting GitHub action, but it felt like a good reference point to show how much slower things where because of this change that has been made to how LazilyRefreshDatabase behaves.

Can you please explain why you feel it should be forced to behave like DatabaseMigrations?

[crynobone]

Can you please explain why you feel it should be forced to behave like DatabaseMigrations?

Because most packages developers would and can have different sets of database migrations between tests, this way packages developers can tests the package usage on different behaviours unlike Laravel apps where there's only a single ready to use state. You don't test the application where for example with and without Jetstream install etc

[AJenbo]

Because most packages developers would and can have different sets of database migrations between tests, this way packages developers can tests the package usage on different behaviours

Then they should use DatabaseMigrations then.

Forcing LazilyRefreshDatabase to behave like DatabaseMigrations means we no longer have an option to choose the one that fits best with what we are doing.

And what if they want to test different migrations on MySQL, since you only force it for SQLite they would have to change anyway to make things work then.

[crynobone]

And what if they want to test different migrations on MySQL, since you only force it for SQLite they would have to change anyway to make things work then.

This was forced to Testbench when Laravel 11 introduced In-memory SQLite improvements between tests when that shouldn't have happened in the first place. If you want a persistent database connection in your tests and gain benefit from it use a persistent database.

This is my last response here, I don't intend to dump everything I learn on developing and maintaining Testbench to explain every single reason why everything the way it is. You are free to use above suggestion or override methods used by Testbench and more on.

[AJenbo]

If you want a persistent database connection in your tests and gain benefit from it use a persistent database.

Switching to a different database setup would take away other things from us, this is not a solution. We run tests in parallel, so using a single database instance would likely cause even more issues.

This is my last response here, I don't intend to dump everything I learn on developing and maintaining Testbench to explain every single reason why everything the way it is.

I’m not asking you to drop everything. I’m just trying to understand if there’s a solution that works for everyone. If the behavior doesn’t align with the documentation, it’s challenging to suggest alternatives. It’s also difficult to grasp the full picture since many commits are labeled as “wip.”

So far, it seems like the only explanation is that you disagree with upstream and aren’t considering my use case. It would be helpful to have a setting to address this since overwriting internal methods can be unreliable. I’m willing to help with that, but I’d like to know if this is something you would consider before investing time in it.