|
5 | 5 | .. contents:: |
6 | 6 | :backlinks: none |
7 | 7 |
|
8 | | -Installation |
9 | | -============ |
| 8 | +This document describes how you can run a local instance of the CERN Open Data |
| 9 | +portal in order to work with the content records and associated documentation. |
| 10 | + |
| 11 | +Prerequisites |
| 12 | +============= |
| 13 | + |
| 14 | +You will need to clone two repositories: |
10 | 15 |
|
11 | | -This module contains the content of the CERN Open Data instance. It does not contain the code of the portal itself. |
12 | | -For local development, it is recommended to have an installation of the CERN Open Data Portal. For detail instructions |
13 | | -on how to install the portal, please follow `these instructions <https://github.com/cernopendata/cernopendata-portal/blob/main/DEVELOPING.rst>`_. |
| 16 | +- `opendata.cern.ch <https://github.com/cernopendata/opendata.cern.ch>`_ which |
| 17 | + contains the open data content; |
| 18 | + |
| 19 | +- `cernopendata-portal <https://github.com/cernopendata/cernopendata-portal>`_ |
| 20 | + which contains the portal infrastructure code. |
| 21 | + |
| 22 | +Please make sure to also install `Docker |
| 23 | +<https://docs.docker.com/get-started/get-docker/>`_ and `Docker Compose |
| 24 | +<https://docs.docker.com/compose/install/>`_ v2 that is used for local |
| 25 | +developments. |
14 | 26 |
|
15 | 27 | Quick start |
16 | | -------------------- |
17 | | -For a quickstart guide, do the following: |
| 28 | +=========== |
| 29 | + |
| 30 | +In order to create a local CERN Open Data portal instance, please proceed as |
| 31 | +follows: |
18 | 32 |
|
19 | 33 | .. code-block:: console |
20 | 34 |
|
21 | | - $ # Checkout this repository |
22 | | - $ git clone https://github.com/cernopendata/opendata.cern.ch.git |
23 | | - $ # Checkout the module with the portal |
24 | | - $ git clone https://github.com/cernopendata/cernopendata-portal.git |
25 | | - $ # Move to the directory of the content |
| 35 | + $ git clone https://github.com/johndoe/opendata.cern.ch |
| 36 | + $ git clone https://github.com/johndoe/cernopendata-portal |
26 | 37 | $ cd opendata.cern.ch |
27 | | - $ # Make sure that the latest images are available |
28 | 38 | $ docker compose pull |
29 | | - $ # Start the services |
30 | 39 | $ docker compose up -d |
31 | | - $ # Give enough time to the containers to start properly. Note that there are some dependencies among them, |
32 | | - $ # and the web container starts by setting up the development environment |
33 | | - $ sleep 120 |
34 | | - $ # Create the basic structure |
35 | | - $ docker exec -i -t opendatacernch-web-1 /code/scripts/populate-instance.sh --skip-records --skip-docs |
36 | | - $ docker exec -i -t opendatacernch-web-1 cernopendata fixtures records \ |
37 | | - --mode insert-or-replace \ |
38 | | - -f /content/data/records/cms-primary-datasets.json |
39 | | -.. |
| 40 | + $ sleep 120 # give enough time for the containers to start properly |
| 41 | + $ docker exec -i -t opendatacernch-web-1 /code/scripts/populate-instance.sh \ |
| 42 | + --skip-records --skip-docs |
| 43 | +
|
| 44 | +This will create a running instance of the CERN Open Data portal with a |
| 45 | +relatively empty content. The portal will be accessible locally at |
| 46 | +`http://127.0.0.1:500 <http://127.0.0.1:5000>`_. |
40 | 47 |
|
| 48 | +If you would like to stop and delete your local instance, you can do: |
41 | 49 |
|
42 | | -At this point, all the services should be up and running. If you go to a web browser to http://0.0.0.0:5000/, you should |
43 | | -see the web portal, with the vocabularies and some documents about the portal itself. |
| 50 | +.. code-block:: console |
44 | 51 |
|
45 | | -From this moment on, |
| 52 | + $ docker compose down -v |
46 | 53 |
|
47 | | -Defining new entries |
| 54 | +Working with records |
48 | 55 | ==================== |
49 | 56 |
|
50 | | -This repository has the following data structure: |
| 57 | +If you would like to work with certain data records and test your edits on your |
| 58 | +local instance, you can proceed as follows. |
| 59 | + |
| 60 | +Edit the record file, such as CMS 2012 collision dataset records: |
| 61 | + |
| 62 | + .. code-block:: console |
51 | 63 |
|
52 | | -* data: |
53 | | - * records: Put here the entries that should be inserted as records |
54 | | - * docs: This folder will be for the documents |
55 | | - * images: And this is for static images that might be needed |
56 | | -* scripts: Directory with shell scripts to help the development |
| 64 | + $ vim data/records/cms-primary-datasets.json |
| 65 | + |
| 66 | +Upload the local file into your instance: |
| 67 | + |
| 68 | +.. code-block:: console |
| 69 | +
|
| 70 | + $ docker exec -i -t opendatacernch-web-1 cernopendata fixtures records \ |
| 71 | + --mode insert-or-replace \ |
| 72 | + -f /content/data/records/cms-primary-datasets.json |
| 73 | +
|
| 74 | +You can then check your changes at `http://127.0.0.1:500 |
| 75 | +<http://127.0.0.1:5000>`_. |
| 76 | + |
| 77 | +Note that you can take advantage of shell scripting if you would like to upload |
| 78 | +all experiment records locally, for example for ATLAS: |
| 79 | + |
| 80 | +.. code-block:: console |
| 81 | +
|
| 82 | + $ for file in data/records/atlas-*; do \ |
| 83 | + docker exec -i -t opendatacernch-web-1 cernopendata fixtures records \ |
| 84 | + --mode insert-or-replace -f $file; \ |
| 85 | + done |
| 86 | +
|
| 87 | +Understanding metadata fields |
| 88 | +============================= |
| 89 | + |
| 90 | +When working with data records, there are several fields such as |
| 91 | +`collision_energy` that you can use to store the content. The list of all |
| 92 | +available record fields, together with their semantic meaning, is described in |
| 93 | +the JSON Schema files. You can find the `record schema |
| 94 | +<https://github.com/cernopendata/cernopendata-portal/blob/main/cernopendata/jsonschemas/records/record-v1.0.0.json>`_ |
| 95 | +in the portal infrastructure repository. |
| 96 | + |
| 97 | +If you would like to modify the JSON schema, for example to add a new field, |
| 98 | +this would require working with the `cernopendata-portal` sister repository. |
| 99 | +Please see its own `documenation |
| 100 | +<https://github.com/cernopendata/cernopendata-portal/>`_ about how to add new |
| 101 | +metadata fields. We would be happy to assist with the process. |
| 102 | + |
| 103 | +Understanding output templates |
| 104 | +============================== |
| 105 | + |
| 106 | +If you would like to change the way how the data records are displayed on the |
| 107 | +web, for example to introduce new section displaying newly added field, this is |
| 108 | +something that is governed by `Jinja templating language |
| 109 | +<https://jinja.palletsprojects.com/en/2.10.x/templates/>`_ in the |
| 110 | +`cernopendata-portal` sister repository. Please see its own `documenation |
| 111 | +<https://github.com/cernopendata/cernopendata-portal/>`_ about how to amend |
| 112 | +look and feel of the record metadata. We would be happy to assist with the |
| 113 | +process. |
| 114 | + |
| 115 | +Verifying metadata conformance |
| 116 | +============================== |
| 117 | + |
| 118 | +You can use the provided helper script `check_fixtures.py` to check the |
| 119 | +conformance of record files to the required minimal standard: |
| 120 | + |
| 121 | +.. code-block:: console |
57 | 122 |
|
58 | | -If you want to modify the json schema, mappings or templates, you will find these folders in the |
59 | | -`cernopendata portal <https://github.com/cernopendata/cernopendata-portal/>`_ repository |
| 123 | + ./scripts/check_fixtures.py |
60 | 124 |
|
61 | 125 | Working with docs/records |
62 | 126 | ------------------------- |
63 | 127 |
|
64 | 128 | The recommended development process is the following: |
65 | 129 |
|
66 | | -1. Create the entries under data/(records/docs) |
67 | | -2. Validate that the yaml syntax is correct |
| 130 | +1. Create the entries under data/(records/docs) 2. Validate that the yaml |
| 131 | + syntax is correct |
68 | 132 |
|
69 | 133 | .. code-block:: console |
70 | 134 |
|
|
0 commit comments