Why does a mesh without initializing make the process hang?

[lrtfm]

The code below will hang when run in parallel. Just create a mesh, del it, and then plot one figure, then the process will hang. Can someone explain this?

from firedrake import RectangleMesh, COMM_WORLD
from firedrake.petsc import PETSc
from matplotlib import pyplot as plt

def syncPrint(msg):
    rank, size = COMM_WORLD.rank, COMM_WORLD.size
    PETSc.Sys.syncPrint(f'[{rank}/{size}] {msg}')
    PETSc.Sys.syncFlush()

def test():
    syncPrint(f'Enter test')
    m = RectangleMesh(5, 5, 1, 1)
    # m.init()
    del m
    syncPrint(f'Middle test')
    if COMM_WORLD.rank == 0:
        plt.figure()
        plt.close()
    syncPrint(f'Exit test')
    
if __name__ == '__main__':
    test()

[dham]

This looks like an incarnation of the known issue with parallel object destruction. Basically, the cyclic garbage collector in Firedrake destroys objects in a non-deterministic way, while PETSc requires that all objects are collectively destroyed by all ranks at once. The combination of these means that Firedrake object destruction can cause deadlocks, especially if different ranks execute different code paths. This is the case here, and it doesn't help that the different code path contains no Firedrake code.

@JDBetteridge is close to a fix for this one, which involves a quite non-trivial change to the garbage collection in PETSc4py.