fff_dqmtools

fff_dqmtools aka DQM^2.

Page with more information here.

Building a package

Note

The makerpm.sh script will only work on RHEL-like OSes (e.g. an RHEL8 OpenStack VM). To run this script you will need to install rpmdevtools and rpmlint.

• Create an RPM by executing ./utils/makerpm.sh in the repository's root folder.

Installation

All machines at once

Installation on P5 machines is done on the cmsdropboxcc8.cms machine, using the dropbox2 command (see more information about the dropbox here) and an .rpm archive, created using the ./utils/makerpm.sh script.

This procedure applies the update to all of the DQM machines at once. You will need to be in the appropriate *_librarian group. Open a ticket to CMSONS to be added to the group.

Procedure:

1. Copy the .rpm you generated to a directory inside a dedicatated directory inside your home directory of the P5 cluster (e.g., /nfshome0/<your username>/fff_dqmtools).
2. Then:

  ssh <your P5 username>cmsdropboxcc8.cms
  sudo dropbox2 -z cms -o cc8 -s dqm -u /nfshome0/<your username>/fff_dqmtools/

Note

Once the operation completes, you will receive a report of what machines were updated.

Note

The fff_dqmtools service is restarted after the rpm is reinstalled.

Manually (and remotely)

For the manual remote installation at P5, copy the source code and the built package and then execute: ./utils/install.py --remote <machine_name> in the repository's root folder. This installation will be undone by puppet days/months later.

Developing

Openstack allows for great flexibility and access to CMSSW tools.

cmssw_deploy

1. Create an OpenStack machine (currently suggested image: RHEL8).
2. Install locmap (see here). You now have CVMFS.
3. Install required tools (yum install -y git gcc python38). You might need to ln -s /usr/bin/python3 ~/bin/python.
4. Switch to your CERN user (su <username).
5. Clone fff_dqmtools.
6. source /cvmfs/cms.cern.ch/cmsset_default.sh # You now have scram
7. The openstack machine must have the SSH keys you use to authenticate with github (so that scram can do whatever needed to clone CMSSW).
8. Configure git (git config --global user.name, as well as user.email).

You should now be ready to run cmssw_deploy.

DQM^2 workflows

To update DQM^2 DB with client info

Create JSON report files in:

fff_dqmtools/applets/analyze_files.py

Line 100 in a62a1a3

def make_report(self, backlog=5):

Then read JSONs in folder:

fff_dqmtools/applets/fff_filemonitor.py

Line 181 in a62a1a3

def process_dir(self):

And upload with http:

fff_dqmtools/applets/fff_filemonitor.py

Line 82 in a62a1a3

def http_upload(lst_gen, port, log=None, test_webserver=False):

To be catched with web client:

fff_dqmtools/applets/fff_web.py

Line 462 in a62a1a3

def upload():

And saved to DB:

fff_dqmtools/applets/fff_web.py

Line 148 in a62a1a3

def direct_transactional_upload(self, bodydoc_generator):