Completion generation for RMA completions

[traxys]

There a few unclear points to me for RMA completions:

• If FI_RMA_EVENT is not provided, is the provider still required to generate completions in case of fi_writedata (or flags & FI_REMOTE_CQ_DATA)
• If FI_RMA_EVENT is provided and the application provides a data with FI_REMOTE_CQ_DATA, does the provider generate one completion with both the FI_REMOTE_CQ_DATA and FI_REMOTE_WRITE flags, or does it generate two completions, each with one of the flags

[shefty]

The man page is not clear. FI_RMA_EVENT only applies to completion counters. This is slightly clearer if you read the fi_endpoint man page for fi_ep_bind. (It's not clear, just clearer, like some of the mud has settled in the water.)

fi_writedata will generate a single completion at the target. This is independent of FI_RMA_EVENT. If the app supports remote_cq_data, it must support writing a CQ entry when a write with cq data is received.

If FI_RMA_EVENT is enabled on the endpoint, then an fi_writedata will generate a CQ entry, plus increment a completion counter associated with remote writes.

[swelch]

@shefty - In the case of the cxi provider, we have applications that use one-sided operations that execute completely without generating events at the target. If FI_RMA_EVENT is specified then remote counters can be updated, still without generating events at the target, but likely target progress is made because the counter is of interest.
We plan to implement fi_writedata()/fi_inject_writedata() near term and for our use case a receive buffer is not required/consumed for the writedata event generation, and the CQ event can be written without receive buffers posted (so we don't set FI_RX_CQ_DATA). However, this requires that hardware events now be generated for RMA at the target so that the immediate data can be written to the libfabric CQ. We only want to enable these hardware events at the target if we know they will be needed and I had planned to use FI_RMA_EVENT to indicate that. It sounds like what we want is exactly what you are describing, i.e. extend FI_RMA_EVENT in a generic fashion and move to have FI_REMOTE_WRITE as valid in the EP to CQ binding (don't have a read use case). In that case, if FI_RMA_EVENT cap is set, then the hardware events will be enabled and the application must progress the target EP even if writedata is never used. Does this sound correct?

[shefty]

If the app sends remote CQ data, then the expectation is that the target will write a CQ entry with that data. You have control over this through the cq_data_size (or whatever the field is called). If that's set to 0, then remote CQ data is not required or supported.

[swelch]

@shefty - I should have read your response a bit closer, I missed the part of writing the CQ entry without remote CQ data :-( so my question didn't really make sense, sorry.

I believe currently the hints domain_attr->cq_data_size is validated to only ensure the requested remote CQ data size is not larger than the CQ data size supported by the provider. Are you suggesting that the application should be able to request zero cq_data_size and the provider should alter the info hints overriding the default so it will know it can avoid having to worry about progression at the target, or that the provider should give two fi_info options (one with cq_data_size=0 and one with our now cq_data_size=8) and let the application take the one it wants?

Originally, the belief was that FI_RMA_EVENT was used for both remote counters and completion data, which simplified the determination if hardware events should be enabled. Trying to avoid yet another environment variable that restricts cq_data_size advertised in the fi_info to zero, but that is another option if we must determine event enablement from cq_data_size and the above are not correct solutions.

[shefty]

In your case, it sounds like you could set the cq_data_size > 0 if the app requests a non-zero size or FI_RMA_EVENT. If you're responding to a no-hint fi_getinfo(), returning 2 fi_info's is reasonable. (Or pick one as the default and only return that.) If the app provides hints, sets cq_data_size = 0, and doesn't set FI_RMA_EVENT, you could disable cq data and report an fi_info with that information.

[swelch]

This seems to align with my expectation on allowing the application hints to override and should be sufficient. Thanks for the information, I think we can meet the needs of applications this way.