use cases: initial commit of hybrid programming models

Chap_Use_Cases.tex

@@ -8,4 +8,145 @@ \chapter{Use-Cases}
 
 Each use case is structured to provide background information about the high-level use case as well as specific details about how the PMIx interfaces are used within the use case.  Some use cases even provide code snippets.  These code snippets are apart of larger code examples located within the standard's source code repository, and each complete code example is fully compilable and runnable. The related interfaces and attributes collected at the bottom of each use case are mainly for conveinence and link to the full standardized definitions.
 
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\section{Hybrid Programming Models}
+\label{chap:hybrid_programming_models}
+
+\subsection{Use Case Summary}
+
+Hybrid applications (i.e., applications that utilize more than one lprogramming model, such as an MPI application that also uses OpenMP or PGAS) are growing in popularity, especially as chips with increasingly large numbers of cores and processors proliferate. Unfortunately, the various models currently operate under the assumption that they alone control execution. This leads to conflicts in hybrid applications. Deadlock of parallel applications can occur when one model prevents the other from making progress due to lack of coordination between the multiple programming models~\cite{2016:Hamidouche}.  Sub-optimal performance can also occur due to uncoordinated division of hardware resources between the programming models~\cite{2018:Vallee,ompix-moc}. This use-case offers potential solutions to the problem by providing a pathway for programming models to coordinate their actions.
+
+\subsection{Use Case Details}
+
+\subsubsection{Identifying Active Programming Models}
+
+The current state-of-the-practice for programming models to detect one another is via set environment variables.  For example, OpenMP looks for environment variables to indicate that MPI is active.  Unfortunately, this technique is not completely reliable as environment variables change over time and with new software versions.  Also, the fact that an environment variable is present doesn't guarantee that a particular programming is in active use since Resource Managers routinely set environment variables "just in case" the application needs them. PMIx provides a reliable mechanism by which each library can determine that another library is in operation.
+
+When initializing PMIx, programming models can register themselves, including their name, version, and threading model.  This information is then cached locally and can then be read asynchronously by other programming models using PMIx's Event Notification system (see next section for more details).
+
+This initialization mechanism also allows libraries to share knowledge of each other's resources and intended resource utilization. For example, if OpenMP knows which hardware threads that MPI is using it could potentially avoid processor and cache contention.
+
+\littleheader{Code Example}
+
+\pmixCodeImportC[]{sources/hybrid-prog-model/declare_model.c}
+
+\littleheader{Related Interfaces}
+
+{\large \refapi{PMIx_Init}}
+\pasteSignature{PMIx_Init}
+
+\littleheader{Related Attributes}
+
+\pasteAttributeItem{PMIX_PROGRAMMING_MODEL}
+\pasteAttributeItem{PMIX_MODEL_LIBRARY_NAME}
+\pasteAttributeItem{PMIX_MODEL_LIBRARY_VERSION}
+\pasteAttributeItem{PMIX_THREADING_MODEL}
+\pasteAttributeItem{PMIX_MODEL_NUM_THREADS}
+\pasteAttributeItem{PMIX_MODEL_NUM_CPUS}
+\pasteAttributeItem{PMIX_MODEL_CPU_TYPE}
+\pasteAttributeItem{PMIX_MODEL_PHASE_NAME}
+\pasteAttributeItem{PMIX_MODEL_PHASE_TYPE}
+\pasteAttributeItem{PMIX_MODEL_AFFINITY_POLICY}
+
+\subsubsection{Coordinating at Runtime}
+
+The PMIx Event Notification system provides a mechanism by which the resource manager can communicate system events to applications, thus providing applications with an opportunity to generate an appropriate response. Hybrid applications can leverage these events for cross-library coordination.
+
+Programming models can access the information provided by other programming models during their initialization using the event notification system.  In this case, programming models should register a callback for the \refconst{PMIX_MODEL_DECLARED} event.
+
+Programming models can also use the PMIx event notification system to communicate dynamic information, such as entering a new application phase (\refconst{PMIX_MODEL_PHASE_NAME}) or a change in resources used (\refconst{PMIX_MODEL_RESOURCES}).  This dynamic information can be broadcast to other programming models using the \refapi{PMIx_Notify_event} function.  Other programming models can register callback functions to run when these events occur (i.e., callback functions) using \refapi{PMIx_Register_event_handler}.
+
+\littleheader{Code Example}
+
+Registering a callback to run when another programming model initializes:
+\pmixCodeJoinStart{}%
+\pmixCodeImportC[]{sources/hybrid-prog-model/declare_model_cb.c}
+\pmixCodeJoin{}%
+\pmixCodeImportC[]{sources/hybrid-prog-model/register_declare_model_cb.c}
+\pmixCodeJoinEnd{}%
+
+
+Notifying an event:
+\pmixCodeImportC[]{sources/hybrid-prog-model/notify_event.c}
+
+\littleheader{Related Interfaces and their Attributes}
+
+{\large \refapi{PMIx_Notify_event}}
+\pasteSignature{PMIx_Notify_event}
+
+\refconst{PMIX_MODEL_DECLARED} \\
+\refconst{PMIX_MODEL_RESOURCES} \\
+\refconst{PMIX_OPENMP_PARALLEL_ENTERED} \\
+\refconst{PMIX_OPENMP_PARALLEL_EXITED} \\
+
+{\large \refapi{PMIx_Register_event_handler}}
+\pasteSignature{PMIx_Register_event_handler}
+
+%\pastePRIAttributeItem{PMIX_RANGE}
+
+{\large \refapi{pmix_event_notification_cbfunc_fn_t}}
+\pasteSignature{pmix_event_notification_cbfunc_fn_t}
+
+\refconst{PMIX_EVENT_ACTION_COMPLETE}
+
+
+
+\subsubsection{Coordinating at Runtime with Multiple Event Handlers}
+
+Coordinating with a threading library such as OpenMP creates the need for separate event handlers for threads of the same process.  For example in an MPI+OpenMP hybrid application, the MPI thread and the main OpenMP thread may both want to be notified anytime an OpenMP worker thread enters a parallel region.  This requiring support for multiple threads to potentially register different event handlers against the same status code.
+
+Multiple event handlers registered against the same event are processed in a chain-like manner based on the order in which they were registered, as modified by directive. Registrations against specific event codes are processed first, followed by registrations against multiple event codes and then any default registrations. At each point in the chain, an event handler is called by the PMIx progress thread and given a function to call when that handler has completed its operation. The handler callback notifies PMIx that the handler is done, returning a status code to indicate the result of its work. The results are appended to the array of prior results, with the returned values combined into an array within a single pmix_info_t as follows:
+\begin{itemize}
+\item \texttt{array[0]}: the event handler name provided at registration (may be an empty field if a string name was not given) will be in the key, with the pmix_status_t value returned by the handler
+\item \texttt{array[*]}: the array of results returned by the handler, if any.
+\end{itemize}
+
+The current PMIx standard does not actually specify a default ordering for event handlers as they are being registered. However, it does include an inherent ordering for invocation. Specifically, PMIx stipulates that handlers be called in the following categorical order:
+
+\begin{itemize}
+\item single status event handlers - i.e., handlers that were registered against a single specific status.
+\item multi status event handlers - those registered against more than one specific status
+\item default event handlers - those registered against no specific status
+\end{itemize}
+
+\littleheader{Code Example}
+
+From the OpenMP master thread:
+
+\pmixCodeJoinStart{}%
+\pmixCodeImportC[]{sources/hybrid-prog-model/parallel_omp_cb.c}%
+\pmixCodeJoin{}%
+\pmixCodeImportC[]{sources/hybrid-prog-model/omp_thread.c}
+\pmixCodeJoinEnd{}%
+
+From the MPI thread:
+
+\pmixCodeJoinStart{}%
+\pmixCodeImportC[]{sources/hybrid-prog-model/parallel_mpi_cb.c}
+\pmixCodeJoin{}%
+\pmixCodeImportC[]{sources/hybrid-prog-model/mpi_thread.c}
+\pmixCodeJoinEnd{}%
+
+\littleheader{Related Interfaces and their Attributes}
+
+{\large \refapi{PMIx_Register_event_handler}}
+\pasteSignature{PMIx_Register_event_handler}
+
+\pasteAttributeItem{PMIX_EVENT_HDLR_NAME}
+\pasteAttributeItem{PMIX_EVENT_HDLR_FIRST}
+\pasteAttributeItem{PMIX_EVENT_HDLR_LAST}
+\pasteAttributeItem{PMIX_EVENT_HDLR_FIRST_IN_CATEGORY}
+\pasteAttributeItem{PMIX_EVENT_HDLR_LAST_IN_CATEGORY}
+\pasteAttributeItem{PMIX_EVENT_HDLR_BEFORE}
+\pasteAttributeItem{PMIX_EVENT_HDLR_AFTER}
+\pasteAttributeItem{PMIX_EVENT_HDLR_APPEND}
+
+{\large \refapi{pmix_event_notification_cbfunc_fn_t}}
+\pasteSignature{pmix_event_notification_cbfunc_fn_t}
+
+\refconst{PMIX_EVENT_NO_ACTION_TAKEN} \\
+\refconst{PMIX_EVENT_PARTIAL_ACTION_TAKEN} \\
+\refconst{PMIX_EVENT_ACTION_DEFERRED} \\
+
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

pmix-standard.bib

@@ -60,3 +60,11 @@ @misc{event2
     Title = {{RFC0018: Extend the Event Notification RFC}},
     Year = {2017}}
 
+@misc{ompix-moc,
+}
+
+@inproceedings{2016:Hamidouche,
+}
+
+@inproceedings{2018:Vallee,
+}

sources/hybrid-prog-model.c

@@ -0,0 +1,28 @@
+#include <pmix.h>
+
+#include "hybrid-prog-model/declare_model_cb.c"
+#include "hybrid-prog-model/parallel_omp_cb.c"
+#include "hybrid-prog-model/parallel_mpi_cb.c"
+
+static int openmp_thread () {
+    pmix_info_t *info;
+#include "hybrid-prog-model/omp_thread.c"
+}
+
+static int mpi_thread () {
+    pmix_info_t *info;
+#include "hybrid-prog-model/mpi_thread.c"
+}
+
+int main() {
+#include "hybrid-prog-model/declare_model.c"
+#include "hybrid-prog-model/register_declare_model_cb.c"
+
+    if () {
+      openmp_thread();
+    } else {
+      mpi_thread();
+    }
+
+#include "hybrid-prog-model/notify_event.c"
+}

sources/hybrid-prog-model/declare_model.c

@@ -0,0 +1,15 @@
+// in `main` function
+pmix_proc_t myproc;
+pmix_info_t *info;
+
+PMIX_INFO_CREATE(info, 4);
+PMIX_INFO_LOAD(&info[0], PMIX_PROGRAMMING_MODEL,
+               "MPI", PMIX_STRING);
+PMIX_INFO_LOAD(&info[1], PMIX_MODEL_LIBRARY_NAME,
+               "FooMPI", PMIX_STRING);
+PMIX_INFO_LOAD(&info[2], PMIX_MODEL_LIBRARY_VERSION,
+               "1.0.0", PMIX_STRING);
+PMIX_INFO_LOAD(&info[3], PMIX_THREADING_MODEL,
+               "pthread", PMIX_STRING);
+pmix_status_t rc = PMIx_Init(&myproc, info, 4);
+PMIX_INFO_FREE(info, 4);

sources/hybrid-prog-model/declare_model_cb.c

@@ -0,0 +1,22 @@
+static void
+model_declared_cb(size_t evhdlr_registration_id, pmix_status_t status,
+                  const pmix_proc_t *source, pmix_info_t info[],
+                  size_t ninfo, pmix_info_t results[], size_t nresults,
+                  pmix_event_notification_cbfunc_fn_t cbfunc,
+                  void *cbdata)
+{
+    for (n = 0; n < ninfo; n++)  {
+        if  (PMIX_CHECK_KEY(info[n], PMIX_PROGRAMMING_MODEL) &&
+             strcmp(info[n].value.data.string, "MPI") == 0) {
+            /* ignore our own declaration */
+            break;
+        } else {
+            /* actions to perform when another model registers */
+        }
+    }
+    if (NULL != cbfunc) {
+        /* tell the event handler that we are only a partial step */
+        cbfunc(PMIX_EVENT_PARTIAL_ACTION_TAKEN, NULL, 0, NULL, NULL,
+               cbdata);
+    }
+}

sources/hybrid-prog-model/mpi_thread.c

@@ -0,0 +1,10 @@
+pmix_status_t code = PMIX_OPENMP_PARALLEL_ENTERED;
+PMIX_INFO_CREATE(info, 2);
+PMIX_INFO_LOAD(&info[0], PMIX_EVENT_HDLR_NAME,
+               "MPI-Thread", PMIX_STRING);
+PMIX_INFO_LOAD(&info[1], PMIX_EVENT_HDLR_AFTER,
+               "OpenMP-Master", PMIX_STRING);
+rc = PMIx_Register_event_handler(&code, 1, info, 1,
+                                 parallel_region_entered_cb,
+                                 NULL, NULL);
+PMIX_INFO_FREE(info, 2);

sources/hybrid-prog-model/notify_event.c

@@ -0,0 +1,3 @@
+// in `main` function
+PMIx_Notify_event(PMIX_OPENMP_PARALLEL_ENTERED, &myproc, myproc.nspace,
+                  NULL, 0, NULL, NULL);

sources/hybrid-prog-model/omp_thread.c

@@ -0,0 +1,10 @@
+pmix_status_t code = PMIX_OPENMP_PARALLEL_ENTERED;
+PMIX_INFO_CREATE(info, 2);
+PMIX_INFO_LOAD(&info[0], PMIX_EVENT_HDLR_NAME,
+               "OpenMP-Master", PMIX_STRING);
+PMIX_INFO_LOAD(&info[1], PMIX_EVENT_HDLR_FIRST,
+               true, PMIX_BOOL);
+rc = PMIx_Register_event_handler(&code, 1, info, 1,
+                                 parallel_region_entered_cb,
+                                 NULL, NULL);
+PMIX_INFO_FREE(info, 2);

sources/hybrid-prog-model/parallel_mpi_cb.c

@@ -0,0 +1,16 @@
+static void
+parallel_region_MPI_cb(size_t evhdlr_registration_id,
+                       pmix_status_t status,
+                       const pmix_proc_t *source,
+                       pmix_info_t info[], size_t ninfo,
+                       pmix_info_t results[], size_t nresults,
+                       pmix_event_notification_cbfunc_fn_t cbfunc,
+                       void *cbdata)
+{
+    /* do what we need MPI to do on entering a parallel region */
+    if (NULL != cbfunc) {
+        /* tell the event handler that we are only a partial step */
+        cbfunc(PMIX_EVENT_ACTION_COMPLETE, NULL, 0, NULL, NULL,
+               cbdata);
+    }
+}

sources/hybrid-prog-model/parallel_omp_cb.c

@@ -0,0 +1,16 @@
+static void
+parallel_region_OMP_cb(size_t evhdlr_registration_id,
+                       pmix_status_t status,
+                       const pmix_proc_t *source,
+                       pmix_info_t info[], size_t ninfo,
+                       pmix_info_t results[], size_t nresults,
+                       pmix_event_notification_cbfunc_fn_t cbfunc,
+                       void *cbdata)
+{
+    /* do what we need OpenMP to do on entering a parallel region */
+    if (NULL != cbfunc) {
+        /* tell the event handler that we are only a partial step */
+        cbfunc(PMIX_EVENT_PARTIAL_ACTION_TAKEN, NULL, 0, NULL, NULL,
+               cbdata);
+    }
+}

sources/hybrid-prog-model/register_declare_model_cb.c

@@ -0,0 +1,4 @@
+// in `main` function
+pmix_status_t code = PMIX_MODEL_DECLARED;
+rc = PMIx_Register_event_handler(&code, 1, NULL, 0, model_declared_cb,
+                                 NULL, NULL);