Skip to content

Commit 3653cf7

Browse files
committed
Add HPC WaaS first description
1 parent f84c39d commit 3653cf7

File tree

12 files changed

+133
-5
lines changed

12 files changed

+133
-5
lines changed

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
build
2+
.vscode

doc/source/Sections/03_HPCWaaS_Methodology.rst

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,14 @@
22
HPCWaaS Methodology
33
======================
44

5-
The eFlows4HPC proposes the HPC Workflow as a Service (HPCWaaS) methodology which tries to apply the usage model of the Functions as a Service in Cloud environments to the workflows for HPC systems. In this model to main roles are identified. From one side, the function developer which is in charge of developing and registering the fuction in the FaaS platform, which transparently deploys de function in the cloud infrastructure. On the other side, the final user executes the deployed function using a REST API. In the case of running workflows in HPC systems, we can find a similar roles. First, we can find the workflow developer, which is charge of developing and deploying the workflow in the computing infrastructure, and the users' comunities which are usually scientist which just want to execute the workflow and collect their results to avance in their scientific goals.
5+
The eFlows4HPC proposes the HPC Workflow as a Service (HPCWaaS) methodology which tries to apply the usage model of the Functions
6+
as a Service in Cloud environments to the workflows for HPC systems. In this model tow main roles are identified.
7+
From one side, the function developer which is in charge of developing and registering the function in the FaaS platform,
8+
which transparently deploys de function in the cloud infrastructure. On the other side, the final user executes the deployed
9+
function using a REST API. In the case of running workflows in HPC systems, we can find a similar roles. First, we can find
10+
the workflow developer, which is charge of developing and deploying the workflow in the computing infrastructure, and the
11+
users' communities which are usually scientist which just want to execute the workflow and collect their results to advance
12+
in their scientific goals.
613

714
.. _fig_hpcwaas:
815
.. figure:: 03_HPCWaaS_Methodology/Figures/HPCWaaS.png
@@ -13,8 +20,10 @@ The eFlows4HPC proposes the HPC Workflow as a Service (HPCWaaS) methodology whic
1320
.center{HPC Workfow as a Service overview}
1421

1522

16-
:numref:`fig_hpcwaas` shows how these two roles interacts with the proposed HPCWaaS methodology. Worklfow developers implement and describe the workflow in a way that allow the eFlows4HPC Gateway services to automatically deploy and orchestrate the workflow execution. This is done interacting with the Developement Interface offered by the Alien4Cloud tool to describe workflows as a TOSCA application.
17-
Once the workflow is deployed users' communities can invoke this workflow using the Execution API.
23+
:numref:`fig_hpcwaas` shows how these two roles interacts with the proposed HPCWaaS methodology. Workflow developers implement
24+
and describe the workflow in a way that allow the eFlows4HPC Gateway services to automatically deploy and orchestrate the workflow
25+
execution. This is done interacting with the Development Interface offered by the Alien4Cloud tool to describe workflows as a
26+
TOSCA application. Once the workflow is deployed users' communities can invoke this workflow using the Execution API.
1827

1928
Next sections provide more details about these interfaces. A simple workflow example about can be found `here <04_Usage_Example.rst>`_.
2029

Lines changed: 94 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,98 @@
11
Development Interface
22
=====================
33

4+
Setup
5+
-----
46

5-
TODO: (Loïc) How to use Alien4Cloud developing and Deploying TOSCA Workflow
7+
Alien4Cloud & Yorc
8+
##################
9+
10+
Please refer to the documentation of the Alien4Cloud & Yorc project for more information.
11+
12+
A instance of Alien4Cloud and Yorc is available on Juelich cloud, ask to the project to have an access
13+
14+
Importing required components into Alien4Cloud
15+
##############################################
16+
17+
Some TOSCA components and topologies templates need to be imported into Alien4Cloud.
18+
If you are using the instance on Juelich cloud, this is already done and you can move to the next paragraph.
19+
20+
You should first move to the ``Catalog`` tab and then the ``Manage archives`` tab, finally click on ``Git import`` to add components.
21+
22+
.. figure:: Figures/catalog.png
23+
:figwidth: 50 %
24+
:alt: Alien4Cloud catalog
25+
:align: center
26+
27+
Click on ``Git import`` to add components
28+
29+
You should have at least the three repositories defined as shown in :numref:`fig_alien4cloud_git_catalog`:
30+
31+
.. _fig_alien4cloud_git_catalog:
32+
.. figure:: Figures/git-catalog.png
33+
:figwidth: 50 %
34+
:alt: Alien4Cloud git catalog
35+
:align: center
36+
37+
Click on ``Git location`` to define imports from a git repository
38+
39+
40+
Click on ``Git location`` to define imports from a git repository as shown in :numref:`fig_alien4cloud_git_import_setup`
41+
42+
.. _fig_alien4cloud_git_import_setup:
43+
.. figure:: Figures/git-import-setup.png
44+
:figwidth: 50 %
45+
:alt: Alien4Cloud setup a catalog git repository
46+
:align: center
47+
48+
Alien4Cloud setup a catalog git repository
49+
50+
Once done you can click on ``Import all``.
51+
52+
Creating an application based on the minimal workflow example
53+
-------------------------------------------------------------
54+
55+
Move to the ``Applications`` tab and click on ``New application``.
56+
57+
.. figure:: Figures/applications.png
58+
:figwidth: 50 %
59+
:alt: Alien4Cloud applications list
60+
:align: center
61+
62+
Manage applications in Alien4Cloud
63+
64+
Then create a new application based on the minimal workflow template as shown in :numref:`fig_alien4cloud_app_create`
65+
66+
.. _fig_alien4cloud_app_create:
67+
.. figure:: Figures/new-application.png
68+
:figwidth: 50 %
69+
:alt: Alien4Cloud create a new application
70+
:align: center
71+
72+
Alien4Cloud create a template based application
73+
74+
Edit the topology to fit your needs:
75+
76+
.. figure:: Figures/topology.png
77+
:figwidth: 50 %
78+
:alt: Alien4Cloud minimal workflow topology
79+
:align: center
80+
81+
Alien4Cloud minimal workflow topology
82+
83+
Then click on ``Deploy`` to deploy the application.
84+
85+
.. figure:: Figures/deploy.png
86+
:figwidth: 50 %
87+
:alt: Alien4Cloud deploy an application
88+
:align: center
89+
90+
Alien4Cloud deploy an application
91+
92+
Make your workflow available to end-users using the WaaS API
93+
------------------------------------------------------------
94+
95+
In order for the WaaS API to know which workflow to allow users to use, you should add a specific tag to your Alien4Cloud application.
96+
Move to your application main panel and under the ``Tags`` section add a tag named ``hpcwaas-workflows``.
97+
The tag value should be a coma-separated list of workflow names that could be called through the HPC WaaS API.
98+
In the minimal workflow example, this tag value should be ``exec_job``.
Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,28 @@
11
Execution API
22
=============
3-
TODO: (Loïc) Execution API
3+
4+
The execution API is still under active development and is subject to changes.
5+
Please refer to the repository documentation_ for a detailed description of the current status of the different
6+
endpoints of this API.
7+
8+
A CLI is planned to interact with the API, but as this is not available yet, the API can be called using a tool like ``curl``.
9+
10+
There is a running instance on Juelich cloud, ask to the team for access to the API.
11+
12+
Basic usage
13+
-----------
14+
15+
First you need to setup your SSH credentials using the `Create an SSH Key Pair for a given user endpoint <https://github.com/eflows4hpc/hpcwaas-api/blob/main/docs/rest-api.md#create-an-ssh-key-pair-for-a-given-user>`.
16+
By calling this endpoint the API will create a new SSH key pair and store it into a vault you will receive in return of this call
17+
the public key. You will never get or even see the private key.
18+
Add this public key as an authorized key for your HPC user account in order to let transfer data to your user account and run
19+
PyCOMPS jobs for you in an automated way.
20+
21+
Then you can use the `list available workflows endpoint <https://github.com/eflows4hpc/hpcwaas-api/blob/main/docs/rest-api.md#list-available-workflows>`
22+
to get the list of endpoints you can access.
23+
24+
You can then `trigger a workflow execution <https://github.com/eflows4hpc/hpcwaas-api/blob/main/docs/rest-api.md#trigger-a-workflow-execution>`.
25+
26+
And finally `monitor the workflow execution <https://github.com/eflows4hpc/hpcwaas-api/blob/main/docs/rest-api.md#monitor-a-workflow-execution>`.
27+
28+
.. _documentation: https://github.com/eflows4hpc/hpcwaas-api/blob/main/docs/rest-api.md
112 KB
Loading
23.1 KB
Loading
25.4 KB
Loading
70 KB
Loading
53.8 KB
Loading
34.9 KB
Loading

0 commit comments

Comments
 (0)