Error validating output record. the `vrn_file_region` field is not valid because value is a list, expected File or null #1

[leipzig]

Hi was testing this with cwltool with the somatic workflow. Does this error look familiar?
double-posted here:
FarahZKhan/bcbio_test_cwlprov#1 (comment)

...
[workflow variantcall_3] starting step variantcall_batch_region_3
[step variantcall_batch_region_3] start
[job variantcall_batch_region_3] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.
[job variantcall_batch_region_3] /private/tmp/docker_tmp5v1bm9xz$ docker \
    run \
    -i \
    --volume=/private/tmp/docker_tmp5v1bm9xz:/iwvxRN:rw \
    --volume=/private/var/folders/b3/pgs49r2x3flcsw92j5yzcbh00000gn/T/tmp0p0evo6f:/tmp:rw \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/seq/hg19.fa:/var/lib/cwl/stgecc680e8-8865-470f-b496-152f39326831/hg19.fa:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/seq/hg19.fa.fai:/var/lib/cwl/stgecc680e8-8865-470f-b496-152f39326831/hg19.fa.fai:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/seq/hg19.dict:/var/lib/cwl/stgecc680e8-8865-470f-b496-152f39326831/hg19.dict:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/config/vcfanno/gemini.conf:/var/lib/cwl/stgd1b85f48-1d9d-4f24-a78a-254a269335dd/gemini.conf:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/config/vcfanno/gemini.lua:/var/lib/cwl/stgc9bbbe5b-0a73-4376-8306-5d5a38cb6408/gemini.lua:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/config/vcfanno/somatic.conf:/var/lib/cwl/stgdeddc7ac-bed2-4761-8b48-3df0c4f0458b/somatic.conf:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/1000g.vcf.gz:/var/lib/cwl/stg7c7935bb-dd79-4577-aa5b-87a46a34ae97/1000g.vcf.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/1000g.vcf.gz.tbi:/var/lib/cwl/stg7c7935bb-dd79-4577-aa5b-87a46a34ae97/1000g.vcf.gz.tbi:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz:/var/lib/cwl/stg9b57a8a7-3d54-4716-8c04-1ec700d11302/hapmap_3.3.vcf.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/hapmap_3.3.vcf.gz.tbi:/var/lib/cwl/stg9b57a8a7-3d54-4716-8c04-1ec700d11302/hapmap_3.3.vcf.gz.tbi:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/clinvar.vcf.gz:/var/lib/cwl/stg66c47d20-6618-49b0-9aa3-914ff50ca08f/clinvar.vcf.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/clinvar.vcf.gz.tbi:/var/lib/cwl/stg66c47d20-6618-49b0-9aa3-914ff50ca08f/clinvar.vcf.gz.tbi:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/esp.vcf.gz:/var/lib/cwl/stgcd4b954f-ebf9-406a-bbe9-a71e6adfdaa0/esp.vcf.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/esp.vcf.gz.tbi:/var/lib/cwl/stgcd4b954f-ebf9-406a-bbe9-a71e6adfdaa0/esp.vcf.gz.tbi:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/ucsc/hg19.2bit:/var/lib/cwl/stge19179f7-d3e0-45e9-9a1d-624092878a07/hg19.2bit:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz:/var/lib/cwl/stg289a7ba6-7591-4477-9546-d9d75578c9f1/test.bed.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test.bed.gz.tbi:/var/lib/cwl/stg289a7ba6-7591-4477-9546-d9d75578c9f1/test.bed.gz.tbi:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz:/var/lib/cwl/stg4225f899-2988-48aa-adca-e2a57eb26477/test2.bed.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/coverage/problem_regions/GA4GH/test2.bed.gz.tbi:/var/lib/cwl/stg4225f899-2988-48aa-adca-e2a57eb26477/test2.bed.gz.tbi:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/reference_material/7_100326_FC6107FAAXX-grade.vcf:/var/lib/cwl/stg7cf31441-b08d-457d-b251-2ec013b0ed77/7_100326_FC6107FAAXX-grade.vcf:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/snpeff--hg19-wf.tar.gz:/var/lib/cwl/stg284ab5e7-71be-4dc4-b5c3-bf078d3965d7/snpeff--hg19-wf.tar.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/automated/variant_regions-bam.bed:/var/lib/cwl/stg395525f0-0871-4bf8-bae0-aad54537818c/variant_regions-bam.bed:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/exac.vcf.gz:/var/lib/cwl/stgb3e4fb2d-d5d3-4cfc-aea9-2dc4a4d611e8/exac.vcf.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/exac.vcf.gz.tbi:/var/lib/cwl/stgb3e4fb2d-d5d3-4cfc-aea9-2dc4a4d611e8/exac.vcf.gz.tbi:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/dbsnp.vcf.gz:/var/lib/cwl/stg330f182e-4245-43f4-a2af-60b653703899/dbsnp.vcf.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/dbsnp.vcf.gz.tbi:/var/lib/cwl/stg330f182e-4245-43f4-a2af-60b653703899/dbsnp.vcf.gz.tbi:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/cosmic.vcf.gz:/var/lib/cwl/stg6881fb08-46b9-417a-9e47-dd5cb48b8fbb/cosmic.vcf.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/cosmic.vcf.gz.tbi:/var/lib/cwl/stg6881fb08-46b9-417a-9e47-dd5cb48b8fbb/cosmic.vcf.gz.tbi:ro \
    --volume=/private/tmp/docker_tmpjx1x0p6f/bedprep/cleaned-variant_regions-bam.bed:/var/lib/cwl/stg1b4158d0-4de7-4c94-8de9-b57b5d53780d/cleaned-variant_regions-bam.bed:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/rtg--hg19.sdf-wf.tar.gz:/var/lib/cwl/stgf9b390e4-b1d9-47de-bc8b-cb4cc3ed1a05/rtg--hg19.sdf-wf.tar.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz:/var/lib/cwl/stgf4fadcf5-0613-4864-808a-ded7300154c5/Mills_Devine_2hit.indels.vcf.gz:ro \
    --volume=/Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/testdata/genomes/hg19/variation/Mills_Devine_2hit.indels.vcf.gz.tbi:/var/lib/cwl/stgf4fadcf5-0613-4864-808a-ded7300154c5/Mills_Devine_2hit.indels.vcf.gz.tbi:ro \
    --volume=/private/tmp/docker_tmp7pt0tz1z/align/Test2/Test2-sort.bam:/var/lib/cwl/stg9f0e8229-267d-47e1-a82e-b52b3b567aae/Test2-sort.bam:ro \
    --volume=/private/tmp/docker_tmp7pt0tz1z/align/Test2/Test2-sort.bam.bai:/var/lib/cwl/stg9f0e8229-267d-47e1-a82e-b52b3b567aae/Test2-sort.bam.bai:ro \
    --volume=/private/tmp/docker_tmpjx1x0p6f/bedprep/cleaned-variant_regions-bam-merged.bed:/var/lib/cwl/stg5cd05388-584a-44bc-b349-aa502628ab71/cleaned-variant_regions-bam-merged.bed:ro \
    --volume=/private/tmp/docker_tmpxmroulsf/align/Test2/Test2-sort-callable_sample.bed:/var/lib/cwl/stga0c9f9b7-e920-4a08-930d-57320d06edef/Test2-sort-callable_sample.bed:ro \
    --volume=/private/tmp/docker_tmpzs42xzqp/regions/b1-analysis_blocks.bed:/var/lib/cwl/stgda8c6829-fe29-4c1d-86b0-a55db066b8e7/b1-analysis_blocks.bed:ro \
    --volume=/private/tmp/docker_tmp5k6auq_t/prealign/Test1/Test1-prealign.bam:/var/lib/cwl/stg8956939b-98d5-4bbb-9dbe-757d712a44db/Test1-prealign.bam:ro \
    --volume=/private/tmp/docker_tmp5k6auq_t/prealign/Test1/Test1-prealign.bam.bai:/var/lib/cwl/stg8956939b-98d5-4bbb-9dbe-757d712a44db/Test1-prealign.bam.bai:ro \
    --volume=/private/tmp/docker_tmphti9h9ko/align/Test1/Test1-prealign-callable_sample.bed:/var/lib/cwl/stgb432af7a-64bb-45ec-b41c-672168731473/Test1-prealign-callable_sample.bed:ro \
    --workdir=/iwvxRN \
    --read-only=true \
    --user=501:20 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/iwvxRN \
    --env=MPLCONFIGDIR=. \
    quay.io/bcbio/bcbio-vc \
    bcbio_nextgen.py \
    runfn \
    variantcall_batch_region \
    cwl \
    sentinel_runtime=cores,2,ram,4096 \
    sentinel_parallel=batch-parallel \
    sentinel_outputs=vrn_file_region,region_block \
    sentinel_inputs=batch_rec:record,region_block:var \
    run_number=0
Traceback (most recent call last):
  File "/usr/local/bin/bcbio_nextgen.py", line 230, in <module>
    runfn.process(kwargs["args"])
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python3.6/site-packages/bcbio/distributed/runfn.py", line 58, in process
    out = fn(*fnargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python3.6/site-packages/bcbio/utils.py", line 55, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python3.6/site-packages/bcbio/distributed/multitasks.py", line 303, in variantcall_batch_region
    return genotype.variantcall_batch_region(*args)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python3.6/site-packages/bcbio/variation/genotype.py", line 457, in variantcall_batch_region
    [_region_to_coords(r) for r in region_block], out_file)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python3.6/site-packages/bcbio/variation/strelka2.py", line 30, in run
    call_file = _run_somatic(paired, ref_file, assoc_files, region, call_file, strelka_work_dir)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python3.6/site-packages/bcbio/variation/strelka2.py", line 324, in _run_somatic
    _run_workflow(paired.tumor_data, workflow_file, tx_work_dir)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python3.6/site-packages/bcbio/variation/strelka2.py", line 342, in _run_workflow
    do.run(cmd, "Run Strelka2: %s" % dd.get_sample_name(data))
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command '/usr/local/share/bcbio-nextgen/anaconda/envs/python2/bin/python /iwvxRN/bcbiotx/tmptgn0d1n5/b1-chrM_0_5077-block-work/runWorkflow.py -m local -j 2 --quiet
' returned non-zero exit status 1.
[job variantcall_batch_region_3] Job error:
Error validating output record. the `vrn_file_region` field is not valid because
  value is a list, expected File or null
 in {
    "vrn_file_region": [],
    "region_block": []
}
[job variantcall_batch_region_3] completed permanentFail
[step variantcall_batch_region_3] completed permanentFail
[workflow variantcall_3] completed permanentFail
Workflow cannot make any more progress.
Final process status is permanentFail
[provenance] Finalizing Research Object
[provenance] Research Object saved to /Users/leipzig/Documents/dev/cwlprov/bcbio_test_cwlprov/somatic/somatic-workflow/somaticwf_0.5.0_mac

[chapmanb]

Jeremy;
Apologies about the issues. This hasn't got much development work in a while and so appears to be not happy with the return value in this case where nothing is getting called in the region. cwltool is likely being stricter now and not counting empty lists as null and we should be returning an explicit none here from bcbio.

I don't want to send you too far down this path since this hasn't got much work in a while so there may be other issues. What are your goals? Very happy to figure out if we can meet those with what currently existing in bcbio.

[leipzig]

Ah I just wanted to use the cwlprov demo which happened to pick this example. I’ll try to patch in a None and move forward. Thanks!

…

On Sun, Jan 19, 2020 at 6:17 AM Brad Chapman ***@***.***> wrote: Jeremy; Apologies about the issues. This hasn't got much development work in a while and so appears to be not happy with the return value in this case where nothing is getting called in the region. cwltool is likely being stricter now and not counting empty lists as null and we should be returning an explicit none here from bcbio. I don't want to send you too far down this path since this hasn't got much work in a while so there may be other issues. What are your goals? Very happy to figure out if we can meet those with what currently existing in bcbio. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#8?email_source=notifications&email_token=AABEEFZCH3LYO5O2JETMKRDQ6QZGBA5CNFSM4KHKSRMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKPT7I#issuecomment-575994365>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEEF3HPCWKZ6OSOQNLC2DQ6QZGBANCNFSM4KHKSRMA> .