[WIP] Finish Global Dynamic Memory and Allocator Concepts #310

fuchsto · 2017-03-02T02:00:01Z

Brief (as possible) summary of the concept definitions that emerged from my GDM evaluation for discussion:

Memory Space and Allocator Concepts

Memory Space

The principle task of memory spaces is to maintain a global address space.
This consequently implies the following aspects:

Pointer Semantics

A memory space provides a dependent pointer type that realizes random
access semantics within its address range. Memory space pointers have
two principle concerns:
- Random access iteration (conventional pointer arithmetics) in
  the address range of their memory space
- Access to values in the memory space (dereference)
- Conversion to compatible pointers, e.g. between local and
  global pointers
Pointers depend on a memory space instance to provide these operations.
For global address space, pointer types must be defined as
specializations of dash::GlobPtr and specified in
dash::memory_space_traits as dependent type:
```
using MemSpaceType<T, ...>::pointer = GlobPtr<Tc, MemSpaceType<T, ...>>`
```
The compatible operations for pointer types of two different memory
spaces is specified by the pointer's implementation.
Resource Interface for Allocators

Allocators depend on memory spaces to attach local memory in the global
address space. In this, memory space types might support access from
multiple allocator instances on the same memory space.
The memory space concept therefore must provide methods to query total
and available capacity.
For example, a static memory space is initialized with a fixed capacity
which might be shared by multiple allocators. Dynamic memory spaces
represent a global heap that automatically grows and shrinks on
allocation and de-allocation.
Synchronization of Distributed Memory Subspaces

Changes on a memory segment usually are not visible at remote locations
immediately. Dynamic memory spaces may, among other aspects, differ in
the communication of memory segment sizes between units (collective epoch
synchronization, deferred point-to-point synchronization, ...) or
data modifications.
For example, a memory space could issue asynchronous pointers that
perform an implicit flush operation (= future) when dereferenced at a
remote address.

Allocator

Allocation Balance

The number of elements allocated at the single units depends on the
allocator, not the memory space type.
More specific, allocator semantics define whether AllocType.allocate(n)
will reserve n elements at every unit (balanced) or n elements at
the active unit (unbalanced).
An allocator's memory space might support irregular (unbalanced) address
ranges but a balanced allocator may still perform balanced allocation
on it.
Unit Participation in Allocation

Allocator semantics define whether a call of AllocType.allocate is
a one-sided, collective, or partially collective operation.
Allocators are not responsible for communicating allocated memory
regions across units, however.
For example, a call of AllocType.allocate(l) could be unbalanced
(local allocation with l varying between units) and collective.
The propagation of the local sizes l between units relies to the
memory space.

Traits

Not all combinations of allocator- and memory space types are compatible,
of course. Compatibility can be checked at compile time by matching allocator
traits and memory space traits, just like already implemented in pattern traits.

The same principles used to deduce pattern types from container- and algorithm
constraints can applied to deduce allocator types for memory spaces, or vice
versa, or depending on algorithm execution- and launch policies, etc.

The aspects listed above represent the primary dimensions (= categories) of allocator
and memory space traits.

Interaction

From application perspective, memory can only be acquired from an allocator
instance that is initialized with an underlying memory space.
Memory spaces can again utilize allocators for memory management of their
sub-spaces. A general call dependency is:

AllocOuterT
  .allocate(n)
    -> MemSpaceOuterT
         .allocate(n)
           - mso_add_capacity = n - free_capacity
           - mso_bucket =
               grow(mso_add_capacity) 
               -> AllocInnerT.allocate(mso_add_capacity)
                  // ... arbitrarly nested
               <- msi_pointer
           - attach(msi_pointer)
           - capacity += mso_add_capacity
             (might commit attached memory here)
         <- mso_bucket
    <- mso_pointer = mso_bucket.begin()
  <- mso_pointer

…303-glob-dyn-alloc

…skip ci]

…sign

…at-umap-redesign Conflicts: dash/include/dash/Memory.h dash/include/dash/allocator/DynamicAllocator.h dash/include/dash/memory/MemorySpace.h

…at-umap-redesign

…-303-glob-dyn-alloc Conflicts: dash/include/dash/GlobDynamicMem.h dash/include/dash/GlobMem.h dash/include/dash/Memory.h dash/include/dash/memory/GlobDynamicMem.h dash/include/dash/memory/GlobHeapMem.h dash/include/dash/memory/GlobMem.h dash/include/dash/memory/GlobStaticMem.h dash/include/libdash.h

…h into feat-303-glob-dyn-alloc

Conflicts: dash/include/dash/GlobPtr.h dash/include/dash/map/UnorderedMap.h dash/include/dash/memory/MemorySpace.h dash/test/memory/GlobHeapMemTest.cc

…dyn-alloc Conflicts: dart-if/include/dash/dart/if/dart_globmem.h dash/include/dash/allocator/EpochSynchronizedAllocator.h

devreal

Since I am on my way to the airport, I don't have time to fully grasp all the details. Comments inline, good otherwise 👍

devreal · 2017-04-01T14:31:33Z

dart-if/include/dash/dart/if/dart_globmem.h

+  {
+    return !(*this == rhs);
+  }
+#endif


I'm not happy with having additional C++ code in DART. Maybe implement these operators somewhere in DASH, potentially in a header wrapping dart_globmem.h .

devreal · 2017-04-01T14:43:51Z

dash/include/dash/allocator/internal/Types.h

+  }
+
+  // Move Constructor
+  memory_block(memory_block &&x) noexcept { *this = std::move(x); }


Shouldn't this .reset() x?

This looks quite strange. If this works, better use memory_block(memory_block &&) noexcept = default;

devreal · 2017-04-01T14:51:34Z

dash/include/dash/map/UnorderedMap.h

+    const_iterator hint,
+    const value_type & value)
+  {
+    Timer::timestamp_t ts_enter = Timer::Now(), ts_insert, ts_find, d_insert, d_find;


The timestamps are only needed for debugging. They should not be used in builds without debug output.

This was actually a commit by accident. Will remove this either. It is something which does not really belong to this pull request.

devreal · 2017-04-01T14:52:43Z

dash/include/dash/map/UnorderedMap.h

+    /* rkowalewski:
+     * Why do we increment local size and _local_cumul_size for the corresponding unit at the same time?
+     * This seems strange to me!!
+    */


Is this still an issue?

no it is not. will remote these comments. :)

devreal · 2017-04-01T14:56:11Z

dash/include/dash/memory/GlobHeapMem.h

+
+#ifdef DASH_ENABLE_TRACE_LOGGING
+    std::for_each(std::begin(_num_attach_buckets),
+                  std::end(_num_attach_buckets),


Iterating over num_unattached_buckets may be more efficient.

devreal · 2017-04-01T15:06:51Z

dash/include/dash/memory/MemorySpace.h

+ * considerations and to optimize communication.
+ * If the memory space concept complies to the STL memory_resource
+ * interface, only DART_TYPE_BYTE instead of the actual value type
+ * is available. This would therefore harm stability and performance.


Note: the type argument in DART allocation functions is not used and you can safely pass DART_TYPE_BYTE to them. It do not affect alignment, which will be taken care of later (DART memory allocation overhaul). In fact, passing the type to the underlying MPI implementation only affects the indexing in access functions (which is in bytes for us, for reasons).

devreal · 2017-04-01T15:16:16Z

dash/test/allocator/EpochSynchronizedAllocatorTest.cc

+  OtherDynAllocTraits::allocator_type otherDynAlloc{dash::Team::All()};
+  OtherDynAllocTraits::pointer gp2 = OtherDynAllocTraits::allocate(otherDynAlloc, n);
+  OtherDynAllocTraits::deallocate(otherDynAlloc, gp2, n);
+}


Missing newline.

devreal · 2017-04-01T15:16:46Z

dash/test/allocator/EpochSynchronizedAllocatorTest.h

+class EpochSynchronizedAllocatorTest : public dash::test::TestBase {
+};
+
+#endif


No newline.

devreal · 2017-04-01T15:18:50Z

dash/test/memory/SimpleMemoryPoolTest.h

+
+/**
+ * Test fixture for class DASH unit id types.
+ */


That comment seems strange ;)

fuchsto · 2017-04-01T23:30:00Z

dash/include/dash/map/UnorderedMap.h

+      // Unit mapped to the new element's key by the hash function:
+      DASH_LOG_TRACE("UnorderedMap.insert", "target unit:", unit);
+      // No element with specified key exists, insert new value.
+      ts_insert = Timer::Now();


Benchmarking / debugging leftovers? If these changes are intended to improve performance, we should add a benchmark example.

fmoessbauer · 2017-04-06T14:32:15Z

Update: Not necessary anymore, as already fixed by @devreal

While implementing this, please also fix the GlobRef<T>.member() accessor. This is broken due to the recent changes in GlobPtr. I will make an issue for this after I am back again.

fmoessbauer · 2017-05-09T19:03:41Z

dash/include/dash/allocator/internal/Types.h

+    return length != 0 && ptr != nullptr;
+  }
+
+  void *ptr;


should be encapsulated in std::unique_ptr, as otherwise the ptr is not set to nullptr on default move.

…dyn-alloc

Roger Kowalewski and others added 2 commits February 26, 2017 12:47

introducted local memory spaces

b686a7f

Introducing allocator traits [no ci]

95dd2eb

fuchsto self-assigned this Mar 2, 2017

Merge branch 'development' of github.com:dash-project/dash into feat-…

4b0314c

…303-glob-dyn-alloc

fuchsto mentioned this pull request Mar 4, 2017

Implement Allocator Traits #303

Closed

fuchsto and others added 17 commits March 5, 2017 00:49

Documentation of allocator- and memory space concepts, intermediate […

6095f1c

…skip ci]

Documentation of allocator- and memory space concepts, intermediate […

286fa93

…skip ci]

Merge branch 'development' into feat-303-glob-dyn-alloc

1858ddb

Merge remote-tracking branch 'origin/development' into feat-umap-rede…

48e6d01

…sign

added Hash Policy for UnorderedMap

d663220

added a simple memory pool for efficient allocation and deallocation

8e99f4a

fixed includes after moving GlobDynamicMem

d6978c6

added a small unit test for SimpleMemoryPool

9badabc

Merge remote-tracking branch 'origin/feat-303-glob-dyn-alloc' into fe…

fc7bb81

…at-umap-redesign Conflicts: dash/include/dash/Memory.h dash/include/dash/allocator/DynamicAllocator.h dash/include/dash/memory/MemorySpace.h

Merge branch 'development' into feat-303-glob-dyn-alloc

3bdc29f

added documentation for SimpleMemoryPool

be9fc50

Merge remote-tracking branch 'origin/feat-303-glob-dyn-alloc' into fe…

a5560e0

…at-umap-redesign

introducing scoped local and global memory allocators

78740c5

temporarily disable unit tests

1f555c1

integrate Memory Spaces Concept in Dynmic Allocator

77759eb

Merge branch 'development' into feat-303-glob-dyn-alloc

35849ee

integrated allocator traits and memory spaces

ad1930b

rkowalewski self-assigned this Mar 17, 2017

Merge remote-tracking branch 'origin/feat-303-glob-dyn-alloc' into fe…

eb7f671

…at-umap-redesign

fuchsto force-pushed the feat-303-glob-dyn-alloc branch from 47192d3 to 286fa93 Compare March 18, 2017 21:19

fuchsto and others added 5 commits March 18, 2017 22:20

Merge branch 'development' into feat-303-glob-dyn-alloc

e7b9d14

Merge branch 'feat-303-glob-dyn-alloc' of github.com:dash-project/das…

207dd98

…h into feat-303-glob-dyn-alloc

STL-compliant definition of MemorySpace, added documentation and notes

7d1de17

Make Allocators aware of the underlying memory spaces

8c1fc03

fuchsto and others added 9 commits March 24, 2017 18:10

Merge branch 'development' into feat-303-glob-dyn-alloc

c893c2c

Reduce communication overhead in GlobHeapMem

379eb3c

Merge branch 'feat-globptr' into feat-303-glob-dyn-globptr-alloc

673c8a1

Conflicts: dash/include/dash/GlobPtr.h dash/include/dash/map/UnorderedMap.h dash/include/dash/memory/MemorySpace.h dash/test/memory/GlobHeapMemTest.cc

Minor fixes

f83955c

minor fixes in the allocators

57ab07b

Merge remote-tracking branch 'origin/development' into feat-303-glob-…

c611916

…dyn-alloc Conflicts: dart-if/include/dash/dart/if/dart_globmem.h dash/include/dash/allocator/EpochSynchronizedAllocator.h

fixed resize of attached memory chunks

6cd0184

fixed GlobalHeapMemTest

14f809d

minor changes

1ffd812

fuchsto requested review from rkowalewski, fmoessbauer and devreal March 31, 2017 19:39

devreal requested changes Apr 1, 2017

View reviewed changes

Merge branch 'development' into feat-303-glob-dyn-alloc

a3b6dd0

fuchsto commented Apr 1, 2017

View reviewed changes

Roger Kowalewski and others added 4 commits April 11, 2017 10:06

addresses review comments

c48b280

Merge branch 'development' into feat-303-glob-dyn-alloc

833e114

Merge branch 'development' into feat-303-glob-dyn-alloc

2006dab

Merge branch 'development' into feat-303-glob-dyn-alloc

6c08616

fmoessbauer reviewed May 9, 2017

View reviewed changes

fmoessbauer mentioned this pull request May 11, 2017

Copy / Assignment of dash::Atomic #289

Open

fuchsto and others added 5 commits May 24, 2017 01:27

Merge branch 'development' into feat-303-glob-dyn-alloc

31dac7d

Merge remote-tracking branch 'origin/development' into feat-303-glob-…

29c12bf

…dyn-alloc

Merge remote-tracking branch 'origin/development' into feat-303-glob-…

0ff1cbe

…dyn-alloc

Memory concept: use only local memory domain

9f79019

symmtric allocator: use dynamic windows

2afc898

devreal modified the milestone: dash-0.4.0 Sep 20, 2018

devreal merged commit 2afc898 into development Nov 8, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Finish Global Dynamic Memory and Allocator Concepts #310

[WIP] Finish Global Dynamic Memory and Allocator Concepts #310

fuchsto commented Mar 2, 2017 •

edited

Loading

devreal left a comment

devreal Apr 1, 2017

devreal Apr 1, 2017

fmoessbauer May 9, 2017 •

edited

Loading

devreal Apr 1, 2017

rkowalewski Apr 1, 2017

devreal Apr 1, 2017

rkowalewski Apr 1, 2017

devreal Apr 1, 2017

devreal Apr 1, 2017

devreal Apr 1, 2017

devreal Apr 1, 2017

devreal Apr 1, 2017

fuchsto Apr 1, 2017 •

edited

Loading

fmoessbauer commented Apr 6, 2017 •

edited

Loading

fmoessbauer May 9, 2017

[WIP] Finish Global Dynamic Memory and Allocator Concepts #310

[WIP] Finish Global Dynamic Memory and Allocator Concepts #310

Conversation

fuchsto commented Mar 2, 2017 • edited Loading

Memory Space and Allocator Concepts

Memory Space

Allocator

Traits

Interaction

devreal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fmoessbauer May 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fuchsto Apr 1, 2017 • edited Loading

Choose a reason for hiding this comment

fmoessbauer commented Apr 6, 2017 • edited Loading

Choose a reason for hiding this comment

fuchsto commented Mar 2, 2017 •

edited

Loading

fmoessbauer May 9, 2017 •

edited

Loading

fuchsto Apr 1, 2017 •

edited

Loading

fmoessbauer commented Apr 6, 2017 •

edited

Loading