Merge tag '2.1.0' into develop

Major update to underlying algs, performance and approach to calling. 2.1.0 should be used over any other 2.0 branch.

Author: keiranmraine

.dockerignore

@@ -2,5 +2,6 @@
 /examples/
 /images/
 /git-hooks/
-/*.*
+/CHANGES.md
+/README.md
 /LICENSE

.travis.yml

@@ -44,4 +44,3 @@ script:
  - docker run -t --rm dockstore-cgpwgs alleleCounter --version
  - docker run -t --rm dockstore-cgpwgs brass.pl -version
  - docker run -t --rm dockstore-cgpwgs ascat.pl -version
- - docker run -t --rm dockstore-cgpwgs battenberg.pl -version

CHANGES.md

@@ -1,5 +1,25 @@
 # CHANGES
 
+## 2.1.0
+
+* Update base image to dockstore-cgpwxs:3.1.6 (from 3.0.3)
+  * Includes, memory footprint reduction for Caveman flagging
+  * Significant speedup in inde input generation
+  * Parallel flagging for SNV
+  * Fragment based SNV/Indel calling
+* Upgrade allele counter and brass
+* Reduce number of layers in final image.
+* Remove some legacy env vars from wrappers
+* Battenberg removed
+  * Was only included to do base allele counts not complete analysis
+  * Underlying data files are not supported for new builds or other species
+  * Contact [original authors](https://github.com/Wedge-Oxford/battenberg) for ongoing support
+* Wrapper script didn't explicitly specify sex SNP loci file (ASCAT)
+  * no impact for GRCh37 other species/build would have used wrong sex chromosome SNPs
+* CaVEMan step wasn't being passed normal contamination value, scientific impact:
+  * previously `caveman.pl` default of `0.1` was applied
+  * this will now honor the value from ASCAT (`1-$rho`)
+
 ## 2.0.2
 
 * Correct skipqc flag

Dockerfile

@@ -1,4 +1,4 @@
-FROM  quay.io/wtsicgp/dockstore-cgpwxs:3.0.3 as builder
+FROM  quay.io/wtsicgp/dockstore-cgpwxs:3.1.6 as builder
 
 USER  root
 
@@ -58,39 +58,42 @@ MAINTAINER  [email protected]
 
 LABEL vendor="Cancer Genome Project, Wellcome Trust Sanger Institute"
 LABEL uk.ac.sanger.cgp.description="CGP WGS pipeline for dockstore.org"
-LABEL uk.ac.sanger.cgp.version="2.0.2"
-
-RUN bash -c 'apt-get update -yq >& this.log || (cat this.log 1>&2 && exit 1)'
-RUN bash -c 'apt-get install -qy --no-install-recommends lsb-release >& this.log || (cat this.log 1>&2 && exit 1)'
-
-RUN echo "deb http://cran.rstudio.com/bin/linux/ubuntu `lsb_release -cs`/" >> /etc/apt/sources.list
-RUN gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9
-RUN gpg -a --export E084DAB9 | apt-key add -
-
-RUN bash -c 'apt-get update -yq >& this.log || (cat this.log 1>&2 && exit 1)'
-RUN bash -c 'apt-get install -yq --no-install-recommends\
-  apt-transport-https\
-  locales\
-  curl\
-  ca-certificates\
-  libperlio-gzip-perl\
-  libssh2-1\
-  bzip2\
-  psmisc\
-  time\
-  zlib1g\
-  liblzma5\
-  libncurses5\
-  libcairo2\
-  gfortran\
-  r-base\
-  exonerate\
-  libboost-iostreams-dev\
-  p11-kit\
-     >& this.log || (cat this.log 1>&2 && exit 1)'
-
-RUN bash -c 'locale-gen en_US.UTF-8 >& this.log || (cat this.log 1>&2 && exit 1)'
-RUN bash -c 'update-locale LANG=en_US.UTF-8 >& this.log || (cat this.log 1>&2 && exit 1)'
+LABEL uk.ac.sanger.cgp.version="2.1.0"
+
+RUN apt-get update -yq && \
+apt-get install -qy --no-install-recommends lsb-release
+
+RUN echo "deb http://cran.rstudio.com/bin/linux/ubuntu `lsb_release -cs`/" >> /etc/apt/sources.list && \
+gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9 && \
+gpg -a --export E084DAB9 | apt-key add -
+
+RUN apt-get update -yq && \
+apt-get install -yq --no-install-recommends \
+apt-transport-https \
+locales \
+curl \
+ca-certificates \
+libperlio-gzip-perl \
+libssh2-1 \
+bzip2 \
+psmisc \
+time \
+zlib1g \
+liblzma5 \
+libncurses5 \
+libcairo2 \
+gfortran \
+r-base \
+exonerate \
+libboost-iostreams-dev \
+p11-kit \
+unattended-upgrades && \
+unattended-upgrade -d -v && \
+apt-get remove -yq unattended-upgrades && \
+apt-get autoremove -yq
+
+RUN locale-gen en_US.UTF-8 && \
+update-locale LANG=en_US.UTF-8
 
 ENV OPT /opt/wtsi-cgp
 ENV PATH $OPT/bin:$OPT/biobambam2/bin:$PATH

README.md

@@ -12,21 +12,17 @@ the [Dockstore.org](https://dockstore.org/) framework.
 | --------------------------------------------- | ----------------------------------------------- |
 | [![Master Badge][travis-master]][travis-base] | [![Develop Badge][travis-develop]][travis-base] |
 
-<!-- TOC depthFrom:2 depthTo:6 withLinks:1 updateOnSave:1 orderedList:0 -->
-
-- [Usage](#usage)
-	- [Usable Cores](#usable-cores)
-- [Other uses](#other-uses)
-	- [Native docker](#native-docker)
-	- [Singularity](#singularity)
-- [Verifying your deployment](#verifying-your-deployment)
-- [Example data](#example-data)
-- [Diagram of internals](#diagram-of-internals)
-- [Development environment](#development-environment)
-- [Release process](#release-process)
-- [LICENCE](#licence)
-
-<!-- /TOC -->
+* [Usage](#usage)
+  * [Usable Cores](#usable-cores)
+* [Other uses](#other-uses)
+  * [Native docker](#native-docker)
+  * [Singularity](#singularity)
+* [Verifying your deployment](#verifying-your-deployment)
+* [Example data](#example-data)
+* [Diagram of internals](#diagram-of-internals)
+* [Development environment](#development-environment)
+* [Release process](#release-process)
+* [LICENCE](#licence)
 
 ## Usage
 
@@ -40,14 +36,14 @@ You should see the usage for the `ds-cgpwgs.pl` script for all parameters (or th
 Required input files are
 
 1. Tumour BAM file
-1. Normal BAM file
-1. Core reference archive (e.g. [core_ref_GRCh37d5.tar.gz][ftp-ref])
-1. WXS reference archive (e.g. [SNV_INDEL_ref_GRCh37d5.tar.gz][ftp-ref])
-1. WGS reference archive (e.g. [CNV_SV_ref_GRCh37d5_brass6+.tar.gz][ftp-ref])
-1. VAGrENT (annotation) reference archive (e.g. [VAGrENT_ref_GRCh37d5_ensembl_75.tar.gz][ftp-ref])
-1. Subclonal reference archive ([SUBCL_ref_GRCh37d5.tar.gz][ftp-ref])
+2. Normal BAM file
+3. Core reference archive (e.g. [core_ref_GRCh37d5.tar.gz][ftp-ref])
+4. WXS reference archive (e.g. [SNV_INDEL_ref_GRCh37d5.tar.gz][ftp-ref])
+5. WGS reference archive (e.g. [CNV_SV_ref_GRCh37d5_brass6+.tar.gz][ftp-ref])
+6. VAGrENT (annotation) reference archive (e.g. [VAGrENT_ref_GRCh37d5_ensembl_75.tar.gz][ftp-ref])
+7. Subclonal reference archive ([SUBCL_ref_GRCh37d5.tar.gz][ftp-ref])
   * Only needed if `skipbb` is `false`
-1. QC reference archive (e.g. [qcGenotype_GRCh37d5.tar.gz][ftp-ref])
+8. QC reference archive (e.g. [qcGenotype_GRCh37d5.tar.gz][ftp-ref])
 
 Inputs 1&2 are expected to have been mapped using [dockstore-cgpmap][dockstore-cgpmap]
 
@@ -56,12 +52,12 @@ your own reference files.  Much of this information is available on the individu
 pages (or the subsequently linked protocols papers).
 
 * Specific to dockstore-cgpwgs:
-	* [BRASS][brass-wiki]
-	* [ascatNgs][ascatngs-wiki]
+  * [BRASS][brass-wiki]
+  * [ascatNgs][ascatngs-wiki]
 * Inherited from dockstore-cgpwxs:
-	* [cgpCaVEManWrapper][caveman-wiki]
-	* [cgpPindel][cgppindel-wiki]
-	* [VAGrENT][vagrent-wiki]
+  * [cgpCaVEManWrapper][caveman-wiki]
+  * [cgpPindel][cgppindel-wiki]
+  * [VAGrENT][vagrent-wiki]
 
 
 ### Usable Cores
@@ -80,54 +76,18 @@ If not set detects available cores on system.
 All of the tools installed as part of [dockstore-cgpmap][dockstore-cgpmap] and
 [dockstore-cgpwxs][dockstore-cgpwxs] and the above packages are available for direct use.
 
-```
-export CGPWGS_VER=X.X.X
-docker pull quay.io/wtsicgp/dockstore-cgpwgs:$CGPWGS_VER
-docker --rm -ti [--volume ...] quay.io/wtsicgp/dockstore-cgpwgs:$CGPWGS_VER bash
-```
+See the [docker guide in the wiki][cgpmap-wiki-docker] for more details.
 
 ### Singularity
 
-The resulting docker container has been tested with Singularity.  The command to exec is:
+The resulting docker container has been tested with Singularity.
 
-```
-ds-cgpwgs.pl -h
-```
-
-Expected use would be along the lines of:
-
-```
-export CGPWGS_VER=X.X.X
-singularity pull docker://quay.io/wtsicgp/dockstore-cgpwgs:$CGPWGS_VER
-
-singularity exec\
- --workdir /.../workspace  \
- --home /.../workspace:/home  \
- --bind /.../ref/human:/var/spool/ref:ro  \
- --bind /.../example_data/cgpwgs:/var/spool/data:ro  \
- dockstore-cgpwgs-${CGPWGS_VER}.simg  \
-  ds-cgpwgs.pl \
-   -reference /var/spool/ref/core_ref_GRCh37d5.tar.gz \
-   -annot /var/spool/ref/VAGrENT_ref_GRCh37d5_ensembl_75.tar.gz \
-   -snv_indel /var/spool/ref/SNV_INDEL_ref_GRCh37d5-fragment.tar.gz \
-   -tumour /var/spool/data/COLO-829_ex.cram \
-   ...
-```
-
-For a system automatically attaching _all local mount points_ (not default singularity behaviour)
-you need not specify any `exec` params (workdir, home, bind) but you should specify the `-outdir`
-option for `ds-cgpwgs.pl` to prevent data being written to your home directory.
-
-By default results are written to the home directory of the container so ensure you bind
-a large volume and set the `-home` variable.  As indicated above the location can be overridden
-via the options of `ds-cgpwgs.pl`
+See the [docker guide in the wiki][cgpmap-wiki-singularity] for more details.
 
 ## Verifying your deployment
 
 The `examples/` tree contains test json files populated with data that can be used to verify the
-tool.  More details can be found [here](examples/README.md).
-
-The `expected/` tree contains the expected output for each tool.  More details can be found [here](expected/README.md).
+tool.
 
 ## Example data
 
@@ -148,7 +108,7 @@ will need to be manually cleaned to keep the repo size down.
 
 Activate the hooks with
 
-```
+```bash
 git config core.hooksPath git-hooks
 ```
 
@@ -157,21 +117,19 @@ git config core.hooksPath git-hooks
 This project is maintained using HubFlow.
 
 1. Make appropriate changes
-1. Build image locally
-1. Run all example inputs and verify any changes are acceptable
-1. Update `expected/` tree with output of last step
-1. Bump version in `Dockerfile` and `Dockstore.cwl`
-1. Push changes
-1. Check state on Travis
-1. Update expected datasets
-1. Generate the release (add notes to GitHub)
-1. Confirm that image has been built on [quay.io][quay-builds]
-1. Update the [dockstore][dockstore-cgpwgs] entry, see [their docs][dockstore-get-started].
+2. Build image locally
+3. Run all example inputs and verify any changes are acceptable
+4. Bump version in `Dockerfile` and `Dockstore.cwl`
+5. Push changes
+6. Check state on Travis
+7. Generate the release (add notes to GitHub)
+8. Confirm that image has been built on [quay.io][quay-builds]
+9. Update the [dockstore][dockstore-cgpwgs] entry, see [their docs][dockstore-get-started].
 
 ## LICENCE
 
-```
-Copyright (c) 2017-2018 Genome Research Ltd.
+```none
+Copyright (c) 2017-2019 Genome Research Ltd.
 
 Author: Cancer Genome Project <[email protected]>
 
@@ -208,6 +166,8 @@ identical to a statement that reads ‘Copyright (c) 2005, 2006, 2007, 2008,
 [caveman-wiki]: https://github.com/cancerit/cgpCaVEManWrapper/wiki
 [ascatngs-wiki]: https://github.com/cancerit/ascatNgs/wiki
 [vagrent-wiki]: https://github.com/cancerit/VAGrENT/wiki
+[cgpmap-wiki-docker]: https://github.com/cancerit/dockstore-cgpwgs/wiki/Running-under-docker
+[cgpmap-wiki-singularity]: https://github.com/cancerit/dockstore-cgpwgs/wiki/Running-under-singularity
 
 <!-- Travis -->
 [travis-base]: https://travis-ci.org/cancerit/dockstore-cgpwgs

build/opt-build.sh

@@ -9,7 +9,7 @@ fi
 set -u
 
 ### alleleCount
-VER_ALLELECOUNT="v4.0.0"
+VER_ALLELECOUNT="v4.0.2"
 
 ### cgpNgsQc
 VER_CGPNGSQC="v1.5.1"
@@ -23,13 +23,10 @@ SRC_ASCAT="https://raw.githubusercontent.com/Crick-CancerGenomics/ascat/v2.5.1/A
 VER_GRASS="v2.1.1"
 
 ### BRASS
-VER_BRASS="v6.1.2"
+VER_BRASS="v6.2.1"
 SOURCE_BLAT="http://users.soe.ucsc.edu/~kent/src/blatSrc35.zip"
 SRC_FASTA36="https://github.com/wrpearson/fasta36/archive/fasta-v36.3.8g.tar.gz"
 
-### cgpBattenberg
-VER_CGPBB="v3.3.0"
-
 if [ "$#" -lt "1" ] ; then
   echo "Please provide an installation path such as /opt/ICGC"
   exit 1
@@ -220,21 +217,6 @@ if [ ! -e $SETUP_DIR/BRASS.success ]; then
   touch $SETUP_DIR/BRASS.success
 fi
 
-### cgpBattenberg
-if [ ! -e $SETUP_DIR/cgpBB.success ]; then
-  curl -sSL --retry 10 https://github.com/cancerit/cgpBattenberg/archive/${VER_CGPBB}.tar.gz > distro.tar.gz
-  rm -rf distro/*
-  tar --strip-components 1 -C distro -xzf distro.tar.gz
-  cd distro/Rsupport
-  ./setupR.sh $INST_PATH
-  cd $SETUP_DIR/distro/perl
-  cpanm --no-interactive --notest --mirror http://cpan.metacpan.org --notest -l $INST_PATH --installdeps .
-  cpanm -v --no-interactive --mirror http://cpan.metacpan.org -l $INST_PATH .
-  cd $SETUP_DIR
-  rm -rf distro.* distro/*
-  touch $SETUP_DIR/cgpBB.success
-fi
-
 cd $HOME
 rm -rf $SETUP_DIR
 

build/rlib-build.R

@@ -28,12 +28,4 @@ ipak(c("poweRlaw"))
 ipak(c("zlibbioc"))
 ipak(c("RColorBrewer"))
 
-# add for BB
-ipak(c("stringi"))
-ipak(c("readr"))
-ipak(c("doParallel"))
-ipak(c("ggplot2"))
-ipak(c("gridExtra"))
-ipak(c("gtools"))
-
-install_github("sb43/copynumber", ref="f1688edc154f1a0e3aacf7781090afe02882f623")
+install_github("Irrationone/copynumber", ref="87d2663fe6b11c03cf6006b4ee9ed70450eacb5a")

cwls/cgpwgs.cwl

@@ -25,7 +25,7 @@ doc: |
 
 requirements:
   - class: DockerRequirement
-    dockerPull: "quay.io/wtsicgp/dockstore-cgpwgs:2.0.2"
+    dockerPull: "quay.io/wtsicgp/dockstore-cgpwgs:2.1.0"
 
 hints:
   - class: ResourceRequirement
@@ -62,13 +62,6 @@ inputs:
       prefix: -cnv_sv
       separate: true
 
-  subcl:
-    type: File
-    doc: "Supporting files for allele counts used by Battenberg Subclonal CNV analysis"
-    inputBinding:
-      prefix: -subcl
-      separate: true
-
   qcset:
     type: File
     doc: "Supporting files for QC tools"
@@ -134,13 +127,6 @@ inputs:
       separate: true
       shellQuote: true
 
-  skipbb:
-    type: boolean?
-    doc: "Skip Battenberg allele counts"
-    inputBinding:
-      prefix: -skipbb
-      separate: true
-
   skipqc:
     type: boolean?
     doc: "Disable genotype and verifyBamID steps"