Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add specific communicator for neighborhood communication #1588

Open
wants to merge 12 commits into
base: index-map-pgm
Choose a base branch
from

Conversation

MarcelKoch
Copy link
Member

@MarcelKoch MarcelKoch commented Apr 4, 2024

This PR adds a communicator that only handles neighborhood all-to-all communication. It implements the new interface collective_communicator, which provides different implementations for a selected set of collective mpi routines. Currently, this only includes the non-blocking all-to-all.

The communication uses a fixed pattern, i.e. the send/recv sizes are fixed when the neighborhood communicator is constructed. I would have liked to decouple that, but this would require some knowledge of how the sizes are stored at the interface level. If someone has an idea for that, please let me know.

This is the first part of splitting up #1546.

The neighborhood all-to-all has a bug in OpenMPI < v4.1.0, which makes it necessary to disable the neighborhood communicator in this case. As replacement, there is also a dense all-to-all communicator.

Todo:

  • documentation

PR Stack:

@MarcelKoch MarcelKoch self-assigned this Apr 4, 2024
@ginkgo-bot ginkgo-bot added reg:build This is related to the build system. reg:testing This is related to testing. mod:core This is related to the core module. labels Apr 4, 2024
@MarcelKoch MarcelKoch requested a review from pratikvn April 4, 2024 10:41
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from 6acf7c4 to 8aa6ab9 Compare April 4, 2024 11:00
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from b42ab92 to 8f104fd Compare April 4, 2024 11:00
@MarcelKoch MarcelKoch modified the milestone: Ginkgo 1.8.0 Apr 5, 2024
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from 8aa6ab9 to 77398bd Compare April 17, 2024 16:28
@MarcelKoch MarcelKoch requested a review from upsj April 19, 2024 09:19
@MarcelKoch MarcelKoch mentioned this pull request Apr 19, 2024
7 tasks
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from 77398bd to d278cad Compare April 19, 2024 14:39
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from 8f104fd to a0824a8 Compare April 19, 2024 14:39
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from d278cad to d6112ef Compare April 22, 2024 11:11
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from a0824a8 to 8ad3f2f Compare April 22, 2024 11:11
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from d6112ef to 1582673 Compare April 25, 2024 07:16
@MarcelKoch MarcelKoch mentioned this pull request Apr 30, 2024
6 tasks
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from 1582673 to db9b48a Compare April 30, 2024 13:41
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from 8ad3f2f to 26678b3 Compare April 30, 2024 13:41
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from db9b48a to 72eafff Compare April 30, 2024 15:20
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from 26678b3 to 006d67d Compare April 30, 2024 15:20
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from 72eafff to 3c70106 Compare May 2, 2024 10:04
@MarcelKoch MarcelKoch force-pushed the read-distributed-with-index-map branch from 006d67d to b295b11 Compare May 2, 2024 10:04
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from 3c70106 to a1567b8 Compare May 2, 2024 10:06
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from db7f6ed to 0ad4ee8 Compare August 9, 2024 13:40
@MarcelKoch MarcelKoch requested a review from pratikvn August 9, 2024 13:41
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from 0ad4ee8 to 1f49b91 Compare August 16, 2024 15:21
@MarcelKoch MarcelKoch requested review from upsj and removed request for upsj August 27, 2024 12:05
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from 1f49b91 to 4db050c Compare October 7, 2024 13:06
Copy link
Member

@upsj upsj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass, mostly interface and implementation

  • What is the moved-from state of a Communicator? Should it match that of an MPI communicator wrapper, and be MPI_NULL, or preserve the MPI communicator?

core/distributed/dense_communicator.cpp Outdated Show resolved Hide resolved
core/distributed/dense_communicator.cpp Outdated Show resolved Hide resolved
core/distributed/neighborhood_communicator.cpp Outdated Show resolved Hide resolved
@MarcelKoch
Copy link
Member Author

@upsj I think the moved-from that should be using MPI_COMM_NULL as communicator. Then trying to launch a communication will fail quite badly.

@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from 4db050c to 1ebe59f Compare October 23, 2024 13:32
@MarcelKoch MarcelKoch force-pushed the index-map-pgm branch 2 times, most recently from 604a6e9 to ba0982e Compare October 23, 2024 15:13
MarcelKoch and others added 11 commits October 30, 2024 15:38
- fix include guards
- update docs
- implement copy/move constructors/assignment with tests
- add equality test for collective communicators (needed for testing)
- always enable neighborhood comm, just throw if openmpi is too old
- define moved-from state as MPI_COMM_NULL + empty sizes/offsets
- remove unnecessary namespace
- make virtual function protected

Co-authored-by: Pratik Nayak <[email protected]>
Co-authored-by: Tobias Ribizel <[email protected]>
@MarcelKoch MarcelKoch force-pushed the neighborhood-communicator branch from 1ebe59f to e7d32a1 Compare October 30, 2024 15:10
@MarcelKoch MarcelKoch modified the milestones: Ginkgo 1.9.0, Ginkgo 1.10.0 Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1:ST:ready-for-review This PR is ready for review mod:core This is related to the core module. reg:build This is related to the build system. reg:testing This is related to testing.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants