Skip to content

Commit

Permalink
docs: adapt documentation after infrastructure code split
Browse files Browse the repository at this point in the history 
Adapts documentation for data curators following up the split of the
portal's infrastructure code from the content in two repositories.

Implements comments from the #3590 pull request review.

Closes #3678.
  • Loading branch information
@tiborsimko
tiborsimko committed Sep 20, 2024
1 parent d86c6f8 commit 1f506cb
Show file tree
Hide file tree
Showing 3 changed files with 134 additions and 75 deletions.
45 changes: 19 additions & 26 deletions CONTRIBUTING.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,33 +19,26 @@ portal developments, you can `watch ongoing discussions
`become part of the team
<https://github.com/orgs/cernopendata/teams>`_.

Code contributions
------------------
Contributions
-------------

We follow typical `GitHub flow
<https://guides.github.com/introduction/flow/index.html>`_.
<https://docs.github.com/en/get-started/using-github/github-flow>`_.

1. Fork this repository into your personal space.
2. Start a new topical branch for any contribution. Name it sensibly,
say ``fix-event-display-icons``.
3. Test your branch on a local site. If everything works as expected,
please `sign your commits
<https://invenio.readthedocs.io/en/latest/technology/git.html#r2-remarks-on-commit-log-messages>`_
to indicate its quality.
4. Create `logically separate commits for logically separate things
<https://invenio.readthedocs.io/en/latest/technology/git.html#r1-remarks-on-commit-history>`_.
Check out our usual `development practices
<https://invenio-github.readthedocs.io/en/latest/contributing.html>`_.
5. Please add any ``(closes #123)`` directives in your commit log
message if your pull request closes an open issue.
6. Issue a pull request. If the branch is not quite ready yet, please
indicate ``WIP`` (=work in progress) in the pull request title.

For more information on how we work with branches, see our `developing
guide <DEVELOPING.rst>`_.

Chatroom
--------

Our chatroom is `on gitter
<https://gitter.im/cernopendata/opendata.cern.ch>`_.
2. Start a new topical branch for any contribution. Name it descriptively, for
example ``fix-cms-2012-collision-energy``.
3. Ideally, test your branch on a local development site to see if everything
works correctly.
4. Open a pull request against this repository. Verify that all the continuous
integration checks are passing. If not, please amend the pull request
accordingly.

For more information on the process, please see our `developing guide
<DEVELOPING.rst>`_.

Support
-------

You can also get in touch via our `Mattermost
<https://mattermost.web.cern.ch/it-dep/channels/opendata>`_ chat room.
138 changes: 101 additions & 37 deletions DEVELOPING.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,66 +5,130 @@
.. contents::
:backlinks: none

Installation
============
This document describes how you can run a local instance of the CERN Open Data
portal in order to work with the content records and associated documentation.

Prerequisites
=============

You will need to clone two repositories:

This module contains the content of the CERN Open Data instance. It does not contain the code of the portal itself.
For local development, it is recommended to have an installation of the CERN Open Data Portal. For detail instructions
on how to install the portal, please follow `these instructions <https://github.com/cernopendata/cernopendata-portal/blob/main/DEVELOPING.rst>`_.
- `opendata.cern.ch <https://github.com/cernopendata/opendata.cern.ch>`_ which
contains the open data content;

- `cernopendata-portal <https://github.com/cernopendata/cernopendata-portal>`_
which contains the portal infrastructure code.

Please make sure to also install `Docker
<https://docs.docker.com/get-started/get-docker/>`_ and `Docker Compose
<https://docs.docker.com/compose/install/>`_ v2 that is used for local
developments.

Quick start
-------------------
For a quickstart guide, do the following:
===========

In order to create a local CERN Open Data portal instance, please proceed as
follows:

.. code-block:: console
$ # Checkout this repository
$ git clone https://github.com/cernopendata/opendata.cern.ch.git
$ # Checkout the module with the portal
$ git clone https://github.com/cernopendata/cernopendata-portal.git
$ # Move to the directory of the content
$ git clone https://github.com/johndoe/opendata.cern.ch
$ git clone https://github.com/johndoe/cernopendata-portal
$ cd opendata.cern.ch
$ # Make sure that the latest images are available
$ docker compose pull
$ # Start the services
$ docker compose up -d
$ # Give enough time to the containers to start properly. Note that there are some dependencies among them,
$ # and the web container starts by setting up the development environment
$ sleep 120
$ # Create the basic structure
$ docker exec -i -t opendatacernch-web-1 /code/scripts/populate-instance.sh --skip-records --skip-docs
$ docker exec -i -t opendatacernch-web-1 cernopendata fixtures records \
--mode insert-or-replace \
-f /content/data/records/cms-primary-datasets.json
..
$ sleep 120 # give enough time for the containers to start properly
$ docker exec -i -t opendatacernch-web-1 /code/scripts/populate-instance.sh \
--skip-records --skip-docs
This will create a running instance of the CERN Open Data portal with a
relatively empty content. The portal will be accessible locally at
`http://127.0.0.1:500 <http://127.0.0.1:5000>`_.

If you would like to stop and delete your local instance, you can do:

At this point, all the services should be up and running. If you go to a web browser to http://0.0.0.0:5000/, you should
see the web portal, with the vocabularies and some documents about the portal itself.
.. code-block:: console
From this moment on,
$ docker compose down -v
Defining new entries
Working with records
====================

This repository has the following data structure:
If you would like to work with certain data records and test your edits on your
local instance, you can proceed as follows.

Edit the record file, such as CMS 2012 collision dataset records:

.. code-block:: console
* data:
* records: Put here the entries that should be inserted as records
* docs: This folder will be for the documents
* images: And this is for static images that might be needed
* scripts: Directory with shell scripts to help the development
$ vim data/records/cms-primary-datasets.json

Upload the local file into your instance:

.. code-block:: console
$ docker exec -i -t opendatacernch-web-1 cernopendata fixtures records \
--mode insert-or-replace \
-f /content/data/records/cms-primary-datasets.json
You can then check your changes at `http://127.0.0.1:500
<http://127.0.0.1:5000>`_.

Note that you can take advantage of shell scripting if you would like to upload
all experiment records locally, for example for ATLAS:

.. code-block:: console
$ for file in data/records/atlas-*; do \
docker exec -i -t opendatacernch-web-1 cernopendata fixtures records \
--mode insert-or-replace -f $file; \
done
Understanding metadata fields
=============================

When working with data records, there are several fields such as
`collision_energy` that you can use to store the content. The list of all
available record fields, together with their semantic meaning, is described in
the JSON Schema files. You can find the `record schema
<https://github.com/cernopendata/cernopendata-portal/blob/main/cernopendata/jsonschemas/records/record-v1.0.0.json>`_
in the portal infrastructure repository.

If you would like to modify the JSON schema, for example to add a new field,
this would require working with the `cernopendata-portal` sister repository.
Please see its own `documenation
<https://github.com/cernopendata/cernopendata-portal/>`_ about how to add new
metadata fields. We would be happy to assist with the process.

Understanding output templates
==============================

If you would like to change the way how the data records are displayed on the
web, for example to introduce new section displaying newly added field, this is
something that is governed by `Jinja templating language
<https://jinja.palletsprojects.com/en/2.10.x/templates/>`_ in the
`cernopendata-portal` sister repository. Please see its own `documenation
<https://github.com/cernopendata/cernopendata-portal/>`_ about how to amend
look and feel of the record metadata. We would be happy to assist with the
process.

Verifying metadata conformance
==============================

You can use the provided helper script `check_fixtures.py` to check the
conformance of record files to the required minimal standard:

.. code-block:: console
If you want to modify the json schema, mappings or templates, you will find these folders in the
`cernopendata portal <https://github.com/cernopendata/cernopendata-portal/>`_ repository
./scripts/check_fixtures.py
Working with docs/records
-------------------------

The recommended development process is the following:

1. Create the entries under data/(records/docs)
2. Validate that the yaml syntax is correct
1. Create the entries under data/(records/docs) 2. Validate that the yaml
syntax is correct

.. code-block:: console
Expand Down
26 changes: 14 additions & 12 deletions README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,34 +8,36 @@
.. image:: https://img.shields.io/badge/licence-GPL_2-green.svg?style=flat
:target: https://raw.githubusercontent.com/cernopendata/opendata.cern.ch/master/LICENSE

.. image:: https://badges.gitter.im/Join%20Chat.svg
:target: https://gitter.im/cernopendata/opendata.cern.ch?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge

About
-----

This is the source code behind CERN Open Data portal. You can access the portal
at `http://opendata.cern.ch/ <http://opendata.cern.ch>`_. The source code uses
`Invenio <https://inveniosoftware.org/>`_ digital repository framework.
This repository is part of for the `CERN Open Data portal
<https://opendata.cern.ch>`_, hosting the source versions of the content,
notably data records and associated documentation pages. The CERN Open Data
portal's infrastructure code is living in the sister `cernopendata-portal
<https://github.com/cernopendata/cernopendata-portal>`_ repository. The CERN
Open Data portal is built upon the `Invenio <https://inveniosoftware.org/>`_
digital repository framework.

Developing
----------

If you'd like to install a demo site locally for personal developments, please
see `developing guide <DEVELOPING.rst>`_ for more information.
If you'd like to install a demo site of the CERN Open Data portal locally for
personal developments, please see the `developing guide <DEVELOPING.rst>`_ for
more information.

Contributing
------------

Bug reports, feature requests and code contributions are encouraged and
welcome! Please see `contributing guide <CONTRIBUTING.rst>`_ for more
information.
Bug reports, feature requests and contributions are encouraged and welcome!
Please see the `contributing guide <CONTRIBUTING.rst>`_ for more information.

Support
-------

You can ask questions at our `Forum <https://opendata-forum.cern.ch/>`_ or get
in touch via our `Chatroom <https://gitter.im/cernopendata/opendata.cern.ch>`_.
in touch via our `Mattermost
<https://mattermost.web.cern.ch/it-dep/channels/opendata>`_ chat room.

Authors
-------
Expand Down

0 comments on commit 1f506cb

Please sign in to comment.