Make database tests run in parallel and not fail because of other failed tests

[dullbananas]

• Create a new build_db_pool_for_tests function which:
  • Builds the pool in a LazyCell which runs migrations and maybe inserts stuff in the initialization function
  • Gets a pooled connection, calls the AsyncConnection::begin_test_transaction method, and returns it by wrapping it in DbPool::Conn
• Remove the serial attribute from tests, since concurrent transactions don't see each other's effects

[dessalines]

Could be faster, but also probably not worth the effort, as the tests don't take that long. Federation tests are much slower anyway.

[dullbananas]

Test failures causing some subsequent tests to fail is a big enough inconvenience in my opinion

[Nutomic]

Here are two blog posts related to Postgres testing. The first one also mentions some problems that happen when using migrations for tests.

https://blog.dogac.dev/pg-test-table-track/

https://calpaterson.com/against-database-teardown.html

And some discussion about these posts: https://lobste.rs/s/84ysx5/start_with_clean_slate_integration

Also this tool: https://github.com/boustrophedon/pgtemp

[Nutomic]

A full CI run on main branch currently takes 49 minutes. Of that 21 min are unit tests, 11 min are clippy, federation tests only 7 min.

I actually wrote something like this to run tests in parallel for Ibis. It uses AtomicI32 to allow a limited number of parallel tests. And it generates a new temporary db for each test case. This code can be adapted for Lemmy, with multiple temp dbs which get reused between tests.

https://github.com/Nutomic/ibis/blob/master/crates/backend/tests/common.rs

[dessalines]

pgtemp looks like the best option to me, along with removing serial for as many of these as possible.

It'd still need to run the migrations for each test (or within each file), but at least everything could be done in parallel, esp with cargo test --test-threads=X

[dullbananas]

Running all migrations again for each test is not what's currently done, and it would probably be way too slow. I think the begin_test_transaction solution is best. Per-test databases like in Ibis can be used for the very small number of tests that need it (the only one I know of is the migrations test).

[Nutomic]

Yes it needs some adjustments but you can see the general idea. For Lemmy it should work like this:

• Create single cluster with pg_ctl init
• Create multiple databases with CREATE DATABASE name; and run migrations
• Put these into deadpool
• In test-context setup method, retrieve a database from the pool
• Now run the test (having exclusive access to the current db)
• In teardown method release db back to pool

[dessalines]

If you can get is working with transactions, then that'd be best. Then you don't have to deal with multiple DBs and teardowns.

[Nutomic]

The post I linked above describes why transactions dont work well for tests. Having multiple DBs is not complicated, and we are already doing teardowns now.

[momentary-lapse]

Will look at the issue this week

[momentary-lapse]

Found a crate which basically implements an approach mentioned by @Nutomic, and here's an example of usage with diesel. Do you see any pitfalls of using it in lemmy code?

[dessalines]

Nice, looks legit. Its worth a try.

There's also https://github.com/boustrophedon/pgtemp if that doesn't work.