Skip to content

Functions for data dictionary validation and summaries

Notifications You must be signed in to change notification settings

NIH-NCPI/abacus

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Abacus

A data validation tool.

Overview

The abacus repository includes scripts and tools that facilitate various forms of validation between datasets and their data dictionaries(data expectiations).

Installation

  1. Create and activate a virtual environment (recommended):
    Here for more on virtual environments.

    # Step 1: cd into the directory to store the venv
    
    # Step 2: run this code. It will create the virtual env named abacus_venv in the current directory.
    python3 -m venv abacus_venv
    
    # Step 3: run this code. It will activate the abacus_venv environment
    source abacus_venv/bin/activate # On Windows: venv\Scripts\activate
    
    # You are ready for installations! 
    # If you want to deactivate the venv run:
    deactivate
  2. Install the package and dependencies:

    pip install git+https://github.com/NIH-NCPI/abacus.git
  3. Run a command/action :diz:

    Available actions:

    Commands

    validate_csv

    validate_csv runs cerberus validation on a datadictionary/dataset pair and returns results of the validation in the terminal.
    See data expectations here

    validate_csv -dd {path/to/datadictionary.csv} -dt {path/to/dataset.csv} -m {Format of missing values in the dataset choose one (i.e. NA, na, null, ...)}   
    
    # example
    validate_csv -dd data/input/data_dictionary.csv -dt data/input/dataset.csv -m NA 
    

    summarize_csv

    summarize_csv returns aggregates and attributes of the provided dataset which is exported as a yaml file.
    See data expectations here

    summarize_csv -dd {path/to/datadictionary.csv} -dt {path/to/dataset.csv} -m {Format of missing values in the dataset choose one (i.e. NA, na, null, ...)} -e {export/filepath/summary.yaml}
    
    # example 
    summarize_csv -dd data/input/data_dictionary.csv -dt data/input/dataset.csv -m NA -e data/output/summary.yaml
    

    validate_linkml

    validate_linkml runs linkml validation on a datadictionary/dataset pair and returns results of the validation in the terminal from the directory that contains the datafiles. (datadictionary, dataset, AND iIMPORTS-adjoining datadictionaries)
    See data expectations here

    validate_linkml -dd {path/to/datadictionary.csv} -dt {path/to/dataset.csv} -dc {data class - linkml tree_root}
    
    # example 
    validate_linkml -dd data/input/assay.yaml -dt data/input/assay_data.yaml -dc Assay
    

    Data Expectations

    csv - validation(cerberus) and summary

    data dictionary format:

    Visit this link for more indepth specs

    dataset format:

    Datasets should be csvs, follow the format described by the data dictionary, and have consitant notation of missing data [NULL, NA, etc.].

    yaml/json - validation(linkml)

    data dictionary format:

    Data dictionaries should be a yaml file formatted for linkml, and contain all dataset expectations for validation. Validation requires all data dictionaries referenced in the imports section present in the same file location. Imports beginning with linkml: can be ignored
    Example seen below.

    id: https://w3id.org/include/assay
    imports:
    - linkml:types
    - include_core
    - include_participant
    - include_study
    

    dataset format:

    Datasets should be yaml, json or csv file formatted for linkml, follow the format described by the data dictionary, and have consitant notation of missing data [NULL, NA, etc.].

    If the dataset is a csv, multivalue fields should have pipe separators
    See examples below.

    # Yaml file representation
    # Instances of Biospecimen class
    - studyCode: "Study1"
      participantGlobalId: "PID123"
      ...
      ...
      ...
    - studyCode: "Study1"
      participantGlobalId: "PID123"
    

    CSV representation

    studyCode,studyTitle,program
    study_code,Study of Cancer,program1|program2
    

    Working on a branch?

    If working on a new feature it is possible to install a package version within the remote or local branch

    # remote
    pip install git+https://github.com/NIH-NCPI/abacus.git@{branch_name}
    
    # local
    pip install -e .
    

About

Functions for data dictionary validation and summaries

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages