This document covers some of the generic tests available using Archipelago's unit testing system, as well as some basic steps on how to write your own.
Some generic tests are run on every World to ensure basic functionality with default options. These basic tests can be found in the general test directory.
In order to run tests from your world, you will need to create a test package within your world package. This can be
done by creating a test directory with a file named __init__.py inside it inside your world. By convention, a base
for your world tests can be created in this file that you can then import into other modules.
In order to test basic functionality of varying options, as well as to test specific edge cases or that certain interactions in the world interact as expected, you will want to use the WorldTestBase. This class comes with the basics for test setup as well as a few preloaded tests that most worlds might want to check on varying options combinations.
Example /worlds/<my_game>/test/__init__.py:
from test.bases import WorldTestBase
class MyGameTestBase(WorldTestBase):
game = "My Game"The basic tests that WorldTestBase comes with include test_all_state_can_reach_everything,
test_empty_state_can_reach_something, and test_fill. These test that with all collected items everything is
reachable, with no collected items at least something is reachable, and that a valid multiworld can be completed with
all steps being called, respectively.
Adding runs for the basic tests for a different option combination is as easy as making a new module in the test
package, creating a class that inherits from your game's TestBase, and defining the options in a dict as a field on the
class. The new module should be named test_<something>.py and have at least one class inheriting from the base, or
define its own testing methods. Newly defined test methods should follow standard PEP8 snake_case format and also start
with test_.
Example /worlds/<my_game>/test/test_chest_access.py:
from . import MyGameTestBase
class TestChestAccess(MyGameTestBase):
options = {
"difficulty": "easy",
"final_boss_hp": 4000,
}
def test_sword_chests(self) -> None:
"""Test locations that require a sword"""
locations = ["Chest1", "Chest2"]
items = [["Sword"]]
# This tests that the provided locations aren't accessible without the provided items, but can be accessed once
# the items are obtained.
# This will also check that any locations not provided don't have the same dependency requirement.
# Optionally, passing only_check_listed=True to the method will only check the locations provided.
self.assertAccessDependency(locations, items)When tests are run, this class will create a multiworld with a single player having the provided options, and run the
generic tests, as well as the new custom test. Each test method definition will create its own separate solo multiworld
that will be cleaned up after. If you don't want to run the generic tests on a base, run_default_tests can be
overridden. For more information on what methods are available to your class, check the
WorldTestBase definition.
Unit tests can also be created using TestBase or unittest.TestCase depending on your use case. These may be useful for generating a multiworld under very specific constraints without using the generic world setup, or for testing portions of your code that can be tested without relying on a multiworld to be created first.
When defining a test that needs to cover a range of inputs it is useful to parameterize (to run the same test for multiple inputs) the base test. Some important things to consider when attempting to parametrize your test are:
-
Subtests can be used to have parametrized assertions that show up similar to individual tests but without the overhead of needing to instantiate multiple tests; however, subtests can not be multithreaded and do not have individual timing data, so they are not suitable for slow tests.
-
Archipelago's tests are test-runner-agnostic. That means tests are not allowed to use e.g.
@pytest.mark.parametrize. Instead, we define our own parametrization helpers in test.param. -
Classes inheriting from
WorldTestBase, including those created by the helpers intest.param, will run all base tests by default, make sure the produced tests actually do what you aim for and do not waste a lot of extra CPU time. Consider usingTestBaseorunittest.TestCasedirectly or settingWorldTestBase.run_default_teststo False.
Archipelago is big enough that the runtime of unittests can have an impact on productivity.
Individual tests should take less than a second, so they can be properly multithreaded.
Ideally, thorough tests are directed at actual code/functionality. Do not just create and/or fill a ton of individual Multiworlds that spend most of the test time outside what you actually want to test.
Consider generating/validating "random" games as part of your APWorld release workflow rather than having that be part of continuous integration, and add minimal reproducers to the "normal" tests for problems that were found. You can use @unittest.skipIf with an environment variable to keep all the benefits of the test framework while not running the marked tests by default.
In PyCharm, running all tests can be done by right-clicking the root test directory and selecting Run 'Archipelago Unittests'. Unless you configured PyCharm to use pytest as a test runner, you may get import failures. To solve this, edit the run configuration, and set the working directory to the Archipelago directory which contains all the project files.
If you only want to run your world's defined tests, repeat the steps for the test directory within your world. Your working directory should be the directory of your world in the worlds directory and the script should be the tests folder within your world.
You can also find the 'Archipelago Unittests' as an option in the dropdown at the top of the window next to the run and debug buttons.
Run pip install pytest pytest-subtests, then use your IDE to run tests or run pytest from the source folder.
pytest can run multiple test runners in parallel with the pytest-xdist extension.
Install with pip install pytest-xdist.
Run with pytest -n12 to spawn 12 process that each run 1/12th of the tests.