Launch policies for algorithms

[devreal]

At the moment, accesses to global memory of DASH containers through subscript operators are serialized using blocking put operations (dart_put_blocking) in GlobRef, potentially resulting in poor performance for naive users. I understand that this is useful to avoid synchronization problems in the general case, but allowing asynchronous access through subscript operators would provide an easy way of improving performance of remote access to multiple elements while using the same (simple to use) interface.

Hence, we could introduce asynchronous epochs, which have an explicit begin and end, e.g.,

dash::Array<int> arr(100);
arr.start_epoch();
if (dash::myid() == 0) {
  for (int i = 0; i < arr.size(); ++i) {
    arr[i] = i; // not immediately visible
  }
}
arr.end_epoch(); 
// all changes visible now

Within an epoch, GlobRef will issue non-blocking put (+get?) operations and data transfers are guaranteed to complete only after the end of the epoch.

Note: I am aware of dash::put_value_async in OneSided but I think it's not intuitively usable and quite wordy, e.g.,

dash::Array<int> arr(100);
if (dash::myid() == 0) {
  for (int i = 0; i < arr.size(); ++i) {
    dash::put_value_async(i, arr[i]); // is this correct?
  }
}
dash::fence(arr[0]); // sufficient to fence on the first element?
// all changes visible now

The documentation is not clear on the use of these methods and there are no tests/examples either. I also cannot efficiently mix dash::put_value_async and operator[] because the latter causes a flush. The idea of asynchronous epochs closely resembles the model employed by MPI-RMA (MPI_Put and MPI_Flush), just with a nicer interface :)

Please let me know what you think. Maybe there have been discussions about it earlier?

[fuerlinger]

Something like that is definitely needed and there was previous discussion about it. Some parts of the functionality described below are in DASH already but I think its not 100% working at the moment.

The idea to provide similar functionality as you describe with an .async modifier, like so:

for (int i = 0; i < arr.size(); ++i) {
  arr.async[i] = i; // not immediately visible
}
arr.flush(); // or arr.fence();

While arr[i] returns a dash::GlobRef arr.async[i] returns a dash::AsyncGlobRef and that would cause put and get operations without flush.

[rkowalewski]

We have GlobAsyncRef which should satisfy all the requirements

[fuchsto]

I introduced the GlobAsync* types quite some time ago and they do exactly that. We don't want to use the low-level function interface dash::put_value*, of course.
I'm about to define launch policies, though, so these won't be needed much longer. Otherwise we would need Glob x (Local*, Async* × Deferred* ...) x (Ptr, Iter, Ref).
But the container concept persists:

array.async[i] = value

and

value = array.async[i]; value.wait()

Another related concept is dash::Future.

[fuchsto]

... and the same launch policies will also be used for algorithms.
So the current interface

dash::copy_async(first, final)

will be changed to

dash::copy(dash::launch::async, first, final)

There's not much to it, just mimicking the C++11 concepts.
I already presented the new style at HPCC'16, thinking of it ... will care about this during the semester break.

[devreal]

Thanks for the clarifications, I wasn't aware that there were already activities in that direction in the DASH part. Looking forward to seeing this happen and leaving this ticket open to track progress.

[fuchsto]

For the latest proposal on execution and launch policies, see my comment in #300

#300 (comment)