How to run async tests seperated from eachother so that each has it's own database?

[wilcoschoneveld]

We are using django-channels and have written some tests for our websocket features. We have looked at https://channels.readthedocs.io/en/stable/topics/testing.html and have setup async tests, specifically:

@pytest.mark.django_db(transaction=True)
@pytest.mark.asyncio
async def test_one():
    ...  # test code here
    ...
    ... # writes and reads to database

Running the tests one at a time succeeds as expected.

However, running the entire test suite fails.
My assumption is that this happens because the tests are executed concurrently, and while one test is reading the database, another test is writing, causing errors in assertions.

Is my assumption correct? How to resolve this issue?

Thanks a lot!

[carltongibson]

Separate transactions should be isolated from each other. So this is a bit mysterious TBH.

[BramKasteel]

Could this be related to: link ?

[carltongibson]

@BramKasteel — yes, that looks it.

So... 🤔 All ORM code will be run inside sync_to_async(thread_sensitive=True) so it should in essence be single threaded at that point — but if one coroutine writes some data and then lets another have access, then data will be visible.

Until the ORM is async-capable I'm not sure we have anything to say here.

From the linked issue:

I've changed my tests to use django.test.TransactionTestCase with serialized_rollback = True so that database transactions are not used, and while it's slow it does work.

[wilcoschoneveld]

So... 🤔 All ORM code will be run inside sync_to_async(thread_sensitive=True) so it should in essence be single threaded at that point — but if one coroutine writes some data and then lets another have access, then data will be visible.

Yes, that is what I think is happening here. Do you know if it's possible to run the tests one after the other with pytest, instead of concurrently?

[carltongibson]

Not off-hand no... I'm sure there will be some option but...

[wilcoschoneveld]

Mmm... I did some more investigation and it looks like pytest actually does run the tests one after the other by default, that makes this an even bigger mystery.

[BramKasteel]

Note: I am currently working with OP on this problem. The test setup we have is as follows:

class Consumer(JsonWebsocketConsumer):
    groups = ['main']  # Name of the group in which group communication occurs

    # Receive message from main group
    def main_message(self, content):
        message = content['message']
        self.send_json(content=message)

These two tests succeed when run individually, but fail when run together:

@pytest.mark.asyncio
@pytest.mark.django_db(transaction=True)
async def test_consumer_receives_group_messages(mock_db):
    message = {'type': 'open_registration', 'reg_id': 1, 'user_id': 1}
    content = {
        'type': 'main_message',
        'message': message,
    }
    channel_layer = get_channel_layer()

    # connect to consumer
    communicator = WebsocketCommunicator(Consumer(), "/ws/")
    connected, _ = await communicator.connect()
    assert connected

    await channel_layer.group_send("main", content)

    response = await communicator.receive_json_from()
    assert response == message

    # Close
    await communicator.disconnect()


@pytest.mark.django_db(transaction=True)
@pytest.mark.asyncio
async def test_setup_channels_and_groups(mock_db):
    channel_layer = get_channel_layer()
    # Create two communicators
    communicator1 = WebsocketCommunicator(Consumer(), "/ws/")
    communicator2 = WebsocketCommunicator(Consumer(), "/ws/")

    assert len(channel_layer.channels) == 0   # <<-- Fails when running multiple tests

    # Connect
    connected1, _ = await communicator1.connect()
    connected2, _ = await communicator2.connect()
    assert connected1 and connected2

    # After connection, a group 'main' should be created, plus a channel per connection
    assert 'main' in channel_layer.groups
    assert len(channel_layer.channels) == 2

    assert list(channel_layer.channels.keys())[0] in channel_layer.groups['main']
    assert list(channel_layer.channels.keys())[1] in channel_layer.groups['main']

    # Close
    await communicator1.disconnect()
    await communicator2.disconnect()

In the original problem statement it was theorised that this could have something to do with database transactions. The problem now seems reduced to something more general: tests interfering with each other on a more basic level.
How can we ensure that tests are run seperate from each other?

[wilcoschoneveld]

We have figured out that disabling await communicator.disconnect() in the first test makes both tests pass 🤨