Add ConfigeratorStats to store sharding plan in config store #3327
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D81185992 |
aliafzal
force-pushed
the
export-D81185992
branch
from
1471188 to
d87ff92
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Aug 28, 2025
…#3327) Summary: Context: This change is part of the effort in improving planners overall UX and reliability. This Diff: 1. Add ConfigeratorStats to upload sharding plan to config store. Differential Revision: D81185992
|
This pull request was exported from Phabricator. Differential Revision: D81185992 |
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Aug 28, 2025
…#3327) Summary: Context: This change is part of the effort in improving planners overall UX and reliability. This Diff: 1. Add ConfigeratorStats to upload sharding plan to config store. Differential Revision: D81185992
aliafzal
force-pushed
the
export-D81185992
branch
from
d87ff92 to
199cf87
Compare
|
This pull request was exported from Phabricator. Differential Revision: D81185992 |
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Aug 28, 2025
…#3327) Summary: Pull Request resolved: pytorch#3327 Context: This change is part of the effort in improving planners overall UX and reliability. This Diff: 1. Add ConfigeratorStats to upload sharding plan to config store. Differential Revision: D81185992
aliafzal
force-pushed
the
export-D81185992
branch
2 times, most recently
from
24c8dfd to
3a87aca
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Aug 28, 2025
…#3327) Summary: Context: This change is part of the effort in improving planners overall UX and reliability. This Diff: 1. Add ConfigeratorStats to upload sharding plan to config store. Differential Revision: D81185992
|
This pull request was exported from Phabricator. Differential Revision: D81185992 |
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Aug 29, 2025
…#3327) Summary: internal Context: This change is part of the effort in improving planners overall UX and reliability. This Diff: 1. Add ConfigeratorStats to upload sharding plan to config store. Differential Revision: D81185992
aliafzal
force-pushed
the
export-D81185992
branch
from
3a87aca to
2ca9b6f
Compare
|
This pull request was exported from Phabricator. Differential Revision: D81185992 |
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Aug 29, 2025
…#3327) Summary: internal Context: This change is part of the effort in improving planners overall UX and reliability. This Diff: 1. Add ConfigeratorStats to upload sharding plan to config store. Differential Revision: D81185992
aliafzal
force-pushed
the
export-D81185992
branch
from
2ca9b6f to
4a1b042
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 2, 2025
…#3327) Summary: internal Context: This change is part of the effort in improving planners overall UX and reliability. This Diff: 1. Add ConfigeratorStats to upload sharding plan to config store. **How is a sharding plan stored in Configerator?** The Thrift definition of a sharding plan includes two fields: Topology and Dict[int, ShardingOption]. 1. Topology: The Topology field contains the information mentioned in this diff D79142495. 2. Dict[int, ShardingOption]: This field represents a dictionary where each key is a 64-bit hash of a sharding option, and the value is the corresponding Thrift-converted sharding option. The hash is calculated using the storage_hash function within the ShardingOption object, which takes into account factors such as the fqn, sharding type, and compute kernel. **How can a loaded plan be merged with an enumerated search space?** **Background:** When a plan is preserved during the logging stage, a hash is generated to ensure that the same plan can be loaded and validated later. The [hash is calculated](https://www.internalfb.com/code/fbsource/[fdf90ff2be9041f867bc6c9e4aec6ee94862fa11]/fbcode/torchrec/distributed/planner/types.py?lines=1010-1026) using input fields such as topology, batch size, constraints, storage reservation, and storage reservation policy, as well as fields from the sharding options like fqn, sharding type, kernel type, shards, and cache parameters. Once the plan is loaded and validated, we can safely assume that all loaded sharding options are a 1:1 map of enumerated sharded options. During the loading process, we traverse the enumerated search space, calculate the storage hash for each sharding option, look up the corresponding sharding option from the loaded plan, and replace the Shards of the enumerated sharding option with those of the loaded sharding option. This approach enables us to generate precise sharding options that can be seamlessly converted into a sharing plan as done by the planner and this also ensures consistent logging while also facilitating plan replay. Differential Revision: D81185992
|
This pull request was exported from Phabricator. Differential Revision: D81185992 |
…#3327) Summary: internal Context: This change is part of the effort in improving planners overall UX and reliability. This Diff: 1. Add ConfigeratorStats to upload sharding plan to config store. **How is a sharding plan stored in Configerator?** The Thrift definition of a sharding plan includes two fields: Topology and Dict[int, ShardingOption]. 1. Topology: The Topology field contains the information mentioned in this diff D79142495. 2. Dict[int, ShardingOption]: This field represents a dictionary where each key is a 64-bit hash of a sharding option, and the value is the corresponding Thrift-converted sharding option. The hash is calculated using the storage_hash function within the ShardingOption object, which takes into account factors such as the fqn, sharding type, and compute kernel. **How can a loaded plan be merged with an enumerated search space?** **Background:** When a plan is preserved during the logging stage, a hash is generated to ensure that the same plan can be loaded and validated later. The [hash is calculated](https://www.internalfb.com/code/fbsource/[fdf90ff2be9041f867bc6c9e4aec6ee94862fa11]/fbcode/torchrec/distributed/planner/types.py?lines=1010-1026) using input fields such as topology, batch size, constraints, storage reservation, and storage reservation policy, as well as fields from the sharding options like fqn, sharding type, kernel type, shards, and cache parameters. Once the plan is loaded and validated, we can safely assume that all loaded sharding options are a 1:1 map of enumerated sharded options. During the loading process, we traverse the enumerated search space, calculate the storage hash for each sharding option, look up the corresponding sharding option from the loaded plan, and replace the Shards of the enumerated sharding option with those of the loaded sharding option. This approach enables us to generate precise sharding options that can be seamlessly converted into a sharing plan as done by the planner and this also ensures consistent logging while also facilitating plan replay. Differential Revision: D81185992
aliafzal
force-pushed
the
export-D81185992
branch
from
4a1b042 to
b21ed1b
Compare
|
This pull request was exported from Phabricator. Differential Revision: D81185992 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Context: This change is part of the effort in improving planners overall UX and reliability.
This Diff:
Differential Revision: D81185992