Tests fail: "Too many open files"

[onehr]

System Information

• cargo 1.36.0-nightly (b6581d383 2019-04-16)
• rustc 1.36.0-nightly (6d599337f 2019-04-22)
• Linux 4.18.0-18-generic 18.04.1-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

description

Command:

cargo test

Test Message (Ignored the un-related part before):

Got several FAILED tests, and the test seems just stunned.

thread 'tokio-runtime-worker-5' panicked at 'not yet implemented', noria-server/src/controller/migrate/materialization/mod.rs:674:29
test controller::sql::tests::it_incorporates_join_projecting_join_columns ... FAILED (allowed)
thread 'tokio-runtime-worker-7' panicked at 'called `Result::unwrap()` on an `Err` value: Error { message: "IO error: While open a file for appending: /tmp/.tmpMfRmmZ/it_incorporates_explicit_multi_join-votes-1.db/MANIFEST-000005: Too many open files" }', src/libcore/result.rs:999:5
test controller::sql::tests::it_incorporates_compound_selection ... ok
thread 'tokio-runtime-worker-12' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Other, message: "Too many open files" }', src/libcore/result.rs:999:5
thread 'tokio-runtime-worker-5' panicked at 'ErrorMessage { msg: "could not add new domain SendError(\"...\")" }', noria-server/src/worker/mod.rs:181:22
test controller::sql::tests::it_incorporates_literal_projection ... ok
test controller::sql::tests::it_incorporates_self_join ... ok
test controller::sql::tests::it_incorporates_arithmetic_projection ... ok
test controller::sql::tests::it_incorporates_aggregation_no_group_by ... ok
test controller::sql::tests::it_queries_over_aliased_view ... ignored
test controller::sql::tests::it_distinguishes_predicates ... ok
test controller::sql::tests::it_incorporates_aggregation_count_star ... ok
test controller::sql::tests::it_incorporates_aggregation ... ok
test controller::sql::tests::it_does_not_reuse_if_disabled ... ok
test controller::sql::tests::it_incorporates_join_with_nested_query ... ok
test controller::sql::tests::it_incorporates_simple_selection ... ok
test controller::sql::tests::it_orders_parameter_columns ... ok
test controller::sql::tests::it_parses ... ok
test handle::tests::limit_mutator_creation ... ok
test controller::sql::tests::it_reuses_identical_query ... ok
test integration::correct_nested_view_schema ... ok
test integration::add_columns ... ok
test controller::sql::tests::it_incorporates_implicit_multi_join ... ok
test controller::sql::tests::it_reuses_by_extending_existing_query ... ok
test controller::sql::tests::it_reuses_with_different_parameter ... ok
thread 'tokio-runtime-worker-9' panicked at 'called `Result::unwrap()` on an `Err` value: Error { message: "IO error: While open a file for appending: /tmp/.tmpeILAWl/crossing_migration-b-0.db/000006.dbtmp: Too many open files" }', src/libcore/result.rs:999:5
thread 'io-worker-1' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Other, message: "Too many open files" }', src/libcore/result.rs:999:5
test integration::albums ... FAILED
thread 'tokio-runtime-worker-3' panicked at 'called `Result::unwrap()` on an `Err` value: Error { message: "IO error: While open directory: /tmp/.tmpxqLtwd/double_shuffle-Car-1.db: Too many open files" }', src/libcore/result.rs:999:5
thread 'tokio-runtime-worker-10' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Other, message: "Too many open files" }', src/libcore/result.rs:999:5
thread 'tokio-runtime-worker-13' panicked at 'called `Result::unwrap()` on an `Err` value: Error { message: "IO error: While open directory: /tmp/.tmpeKlgDV/forced_shuffle_despite_same_shard-Car-0.db: Too many open files" }', src/libcore/result.rs:999:5
test integration::cascading_replays_with_sharding ... FAILED
test integration::base_mutation ... ok
thread 'tokio-runtime-worker-4' panicked at 'called `Result::unwrap()` on an `Err` value: Error { message: "IO error: While open directory: /tmp/.tmpITov0z/do_full_vote_migration_false-article-0.db: Too many open files" }', src/libcore/result.rs:999:5
test integration::domain_amend_migration ... FAILED
test integration::empty_migration ... ok
thread 'tokio-runtime-worker-3' panicked at 'called `Result::unwrap()` on an `Err` value: Error { message: "IO error: While open a file for appending: /tmp/.tmpf1SJKJ/it_recovers_persisted_bases-Car-1.db/MANIFEST-000005: Too many open files" }', src/libcore/result.rs:999:5
thread 'tokio-runtime-worker-4' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Other, message: "Too many open files" }', src/libcore/result.rs:999:5
test integration::full_vote_migration_new_and_old_unsharded ... FAILED
thread 'tokio-runtime-worker-9' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Other, message: "Too many open files" }', src/libcore/result.rs:999:5
thread 'tokio-runtime-worker-5' panicked at 'SendError("...")', noria-server/src/startup.rs:155:30
test integration::independent_domain_migration ... ok
test integration::it_works_basic ... ok
thread 'tokio-runtime-worker-9' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Other, message: "Too many open files" }', src/libcore/result.rs:999:5
test integration::full_vote_migration_new_and_old ... FAILED
test integration::it_works_w_mat ... ok
test integration::it_works_deletion ... ok
test integration::it_works_w_partial_mat ... ok
test integration::it_works_w_partial_mat_below_empty ... ok
test integration::it_works_with_arithmetic_aliases ... ok
test integration::it_works_with_double_query_through ... ok
thread 'tokio-runtime-worker-12' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Other, message: "Too many open files" }', src/libcore/result.rs:999:5
thread 'tokio-runtime-worker-11' panicked at 'SendError("...")', noria-server/src/startup.rs:155:30
test integration::it_works_with_function_arithmetic ... ok
test integration::it_works_with_multiple_arithmetic_expressions ... ok
test integration::it_works_with_reads_before_writes ... ok
test integration::it_works_with_simple_arithmetic ... ok
test integration::it_works_with_sql_recipe ... ok
test integration::it_works_with_vote ... ok
test integration::key_on_added ... ok
test integration::live_writes ... ok
thread 'tokio-runtime-worker-3' panicked at 'called `Result::unwrap()` on an `Err` value: Error { message: "IO error: While open directory: /tmp/.tmpQcZ9JE/lobsters-hats-0.db: Too many open files" }', src/libcore/result.rs:999:5
test controller::sql::tests::it_incorporates_explicit_multi_join ... test controller::sql::tests::it_incorporates_explicit_multi_join has been running for over 60 seconds
test controller::sql::tests::it_incorporates_simple_join ... test controller::sql::tests::it_incorporates_simple_join has been running for over 60 seconds
test integration::crossing_migration ... test integration::crossing_migration has been running for over 60 seconds
test integration::double_shuffle ... test integration::double_shuffle has been running for over 60 seconds
test integration::finkelstein1982_queries ... test integration::finkelstein1982_queries has been running for over 60 seconds
test integration::forced_shuffle_despite_same_shard ... test integration::forced_shuffle_despite_same_shard has been running for over 60 seconds
test integration::full_aggregation_with_bogokey ... test integration::full_aggregation_with_bogokey has been running for over 60 seconds
test integration::full_vote_migration_only_new ... test integration::full_vote_migration_only_new has been running for over 60 seconds
test integration::it_recovers_persisted_bases ... test integration::it_recovers_persisted_bases has been running for over 60 seconds
test integration::it_recovers_persisted_bases_w_multiple_nodes ... test integration::it_recovers_persisted_bases_w_multiple_nodes has been running for over 60 seconds
test integration::it_works_with_join_arithmetic ... test integration::it_works_with_join_arithmetic has been running for over 60 seconds
test integration::lobsters ... test integration::lobsters has been running for over 60 seconds

[ms705]

What does ulimit -n return? The Linux default of 1024 open FDs per process is likely too low when running the Noria tests on ten threads in parallel (Rust default).

Workarounds: either increase the FD limit, or run with RUST_TEST_THREADS=1 set, so that only one test runs at a time.

[onehr]

Seems this is the solution, FD limit is too small for many threads. Thanks.

[ms705]

@jonhoo Can we somehow set a default RUST_TEST_THREADS=1 for cargo to pick up? I seem to remember that it's possible to set workspace-specific environment variables for cargo. It'd be good if cargo test worked out of the box on other vanilla Linux machines.

[onehr]

@ms705 Would it be better to make some parts serial (which causes the fails) and keep others parallel? For performance.

[jonhoo]

@ms705 there sadly isn't really a way to do this in a good way. Maybe the closest approximation is using serial_test. There's rust-lang/rust#43155, but I don't know that anything has come of that.