A nuance of intent: Do we want to think of nodegroup messages as a within-node load balancing mechanism also? If used for that purpose, it should be low priority. "Finish all your local work assigned to you and then pay attention to nodegroup messages" makes sense for this scenario. However we do have task queue for that purpose. And nodegroup messages have another purpose: to quickly process data arriving from outside the node. (Basically saying: any pe, please execute this, it is urgent so I don't want to wait for any particular PE to become available). The common use case appears to be the latter, so its worth fixing.

I have tested a fix in my copy, which works, but can sometimes slow things down. (I think because checking the nodeq, which is accessed via lock/fence, has a cost.. But also because changing order in which things happen can have unexpected consequences). But mostly it does what we expect. I will work on making a good branch with that fix and push a PR. (The idea is to check nodegroup messages with highest priority once in every K iterations of the scheduler.)

If you want a nodegroup entry method to be executed urgently upon arrival, isn't that what [immediate] is for?

Immediate will execute on commthread, which we don't want in this case. (comm thread will become a bottleneck). Also, we do want to parallelize/distribute the (not insignificant) work of handling incoming messages to all PEs... not for each message, but across them.