Bug: make minimal: Out of order prereqs; failing medgen-disease-extract.obo

[joeflack4]

Overview

For some reason after changing make all to make minimal temporarily in order to address several issues in #11, to get the GitHub action to work, I'm getting a strange failure. It appears that the order of goals and prereqs is the same between all and minimal, so I'm not sure what the issue is.

Errors

Short err: No such file or directory

Long log

(medgen) make minimal
./src/make_uid2cui.pl > uid2cui.tsv
./src/medgen2obo.pl > medgen.obo.tmp && mv medgen.obo.tmp medgen.obo
owltools medgen.obo --extract-subset Disease-or-Syndrome --set-ontology-id http://purl.obolibrary.org/obo/mondo/x-Disease-or-Syndrome.obo -o -f obo x-Disease-or-Syndrome.obo
beep
owltools medgen.obo --extract-subset Neoplastic-Process --set-ontology-id http://purl.obolibrary.org/obo/mondo/x-Neoplastic-Process.obo -o -f obo x-Neoplastic-Process.obo
owltools x-Disease-or-Syndrome.obo x-Neoplastic-Process.obo --merge-support-ontologies -o -f obo medgen-disease-extract.obo
adding support ont Ontology(OntologyID(OntologyIRI(<http://purl.obolibrary.org/obo/mondo/x-Neoplastic-Process.obo.owl>) VersionIRI(<null>))) [Axioms: 1269032 Logical Axioms: 709388] First 20 axioms: {AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#id> <http://purl.obolibrary.org/obo/UMLS_C1332523> "UMLS:C1332523"^^xsd:string) AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> <http://purl.obolibrary.org/obo/UMLS_C0238198> "HP:0100723"^^xsd:string) AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#id> <http://purl.obolibrary.org/obo/UMLS_C0341528> "UMLS:C0341528"^^xsd:string) AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> <http://purl.obolibrary.org/obo/MEDGENCUI_C0684752> "SCTID:94420002"^^xsd:string) AnnotationAssertion(Annotation(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> "NCI:C6338"^^xsd:string) <http://www.geneontology.org/formats/oboInOwl#hasRelatedSynonym> <http://purl.obolibrary.org/obo/UMLS_C1336903> "Corpus Uteri Adenomyoma"^^xsd:string) AnnotationAssertion(rdfs:label <http://purl.obolibrary.org/obo/UMLS_C0279986> "Childhood Leiomyosarcoma"^^xsd:string) AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> <http://purl.obolibrary.org/obo/MEDGENCUI_C0730568> "SCTID:313250007"^^xsd:string) AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#inSubset> <http://purl.obolibrary.org/obo/MEDGENCUI_C4725846> <http://purl.obolibrary.org/obo/mondo/x-Neoplastic-Process.obo#Neoplastic-Process>) AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#inSubset> <http://purl.obolibrary.org/obo/MEDGENCUI_C1511320> <http://purl.obolibrary.org/obo/mondo/x-Neoplastic-Process.obo#Neoplastic-Process>) AnnotationAssertion(Annotation(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> "NCI:C5955"^^xsd:string) <http://www.geneontology.org/formats/oboInOwl#hasRelatedSynonym> <http://purl.obolibrary.org/obo/UMLS_C1334605> "Malignant Minor Salivary Gland Mixed Neoplasm"^^xsd:string) AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> <http://purl.obolibrary.org/obo/UMLS_C5420354> "NCIT:C173716"^^xsd:string) AnnotationAssertion(Annotation(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> "SNOMEDCT_US:94592002"^^xsd:string) <http://www.geneontology.org/formats/oboInOwl#hasRelatedSynonym> <http://purl.obolibrary.org/obo/UMLS_C0684740> "Metastatic malignant neoplasm to soft tissues of perineum"^^xsd:string) AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#id> <http://purl.obolibrary.org/obo/MEDGENCUI_C5237384> "MEDGENCUI:C5237384"^^xsd:string) AnnotationAssertion(Annotation(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> "NCI:C159226"^^xsd:string) <http://www.geneontology.org/formats/oboInOwl#hasRelatedSynonym> <http://purl.obolibrary.org/obo/UMLS_C5204273> "Renal Paraganglioma"^^xsd:string) AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#id> <http://purl.obolibrary.org/obo/UMLS_C1882716> "UMLS:C1882716"^^xsd:string) AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#id> <http://purl.obolibrary.org/obo/MEDGENCUI_C1516503> "MEDGENCUI:C1516503"^^xsd:string) AnnotationAssertion(Annotation(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> "NCI:C4880"^^xsd:string) <http://www.geneontology.org/formats/oboInOwl#hasRelatedSynonym> <http://purl.obolibrary.org/obo/MEDGENCUI_C0684516> "Benign Neoplasm of Bone"^^xsd:string) AnnotationAssertion(Annotation(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> "NCI:C7571"^^xsd:string) <http://www.geneontology.org/formats/oboInOwl#hasRelatedSynonym> <http://purl.obolibrary.org/obo/MEDGENCUI_C1456781> "Benign Melanocytic Nevus of Skin"^^xsd:string) AnnotationAssertion(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> <http://purl.obolibrary.org/obo/MEDGENCUI_C0279633> "MONDO:AN1607105"^^xsd:string) AnnotationAssertion(Annotation(<http://www.geneontology.org/formats/oboInOwl#hasDbXref> "MSH:D011125"^^xsd:string) <http://www.geneontology.org/formats/oboInOwl#hasRelatedSynonym> <http://purl.obolibrary.org/obo/MEDGENCUI_C0032580> "Polyposis Coli"^^xsd:string) }
python src/mondo_robot_template.py -i ftp.ncbi.nlm.nih.gov/pub/medgen/MedGenIDMappings.txt -o medgen-xrefs.robot.template.tsv
mkdir -p output/release
mv medgen.obo output/release/
mv medgen-disease-extract.obo output/release/
owltools medgen-disease-extract.obo -o medgen-disease-extract.owl
2023-08-30 19:56:30,392 ERROR (CommandRunner:4815) could not parse:medgen-disease-extract.obo
org.semanticweb.owlapi.io.OWLOntologyCreationIOException: /Users/joeflack4/projects/medgen/medgen-disease-extract.obo (No such file or directory)
	at uk.ac.manchester.cs.owl.owlapi.OWLOntologyFactoryImpl.loadOWLOntology(OWLOntologyFactoryImpl.java:230)
	at uk.ac.manchester.cs.owl.owlapi.OWLOntologyManagerImpl.actualParse(OWLOntologyManagerImpl.java:1254)
	at uk.ac.manchester.cs.owl.owlapi.OWLOntologyManagerImpl.loadOntology(OWLOntologyManagerImpl.java:1208)
	at uk.ac.manchester.cs.owl.owlapi.OWLOntologyManagerImpl.loadOntology(OWLOntologyManagerImpl.java:1108)
	at uk.ac.manchester.cs.owl.owlapi.OWLOntologyManagerImpl.loadOntology(OWLOntologyManagerImpl.java:1064)
	at owltools.io.ParserWrapper.parseOWL(ParserWrapper.java:163)
	at owltools.io.ParserWrapper.parseOWL(ParserWrapper.java:150)
	at owltools.io.ParserWrapper.parseOBO(ParserWrapper.java:136)
	at owltools.cli.CommandRunner.runSingleIteration(CommandRunner.java:4801)
	at owltools.cli.CommandRunnerBase.run(CommandRunnerBase.java:76)
	at owltools.cli.CommandRunnerBase.run(CommandRunnerBase.java:68)
	at owltools.cli.CommandLineInterface.main(CommandLineInterface.java:12)
Caused by: java.io.FileNotFoundException: /Users/joeflack4/projects/medgen/medgen-disease-extract.obo (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
	at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
	at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:189)
	at org.semanticweb.owlapi.io.AbstractOWLParser.getInputStreamFromContentEncoding(AbstractOWLParser.java:179)
	at org.semanticweb.owlapi.io.AbstractOWLParser.getInputStream(AbstractOWLParser.java:141)
	at org.semanticweb.owlapi.io.AbstractOWLParser.getInputSource(AbstractOWLParser.java:264)
	at org.semanticweb.owlapi.rdf.rdfxml.parser.RDFXMLParser.parse(RDFXMLParser.java:72)
	at uk.ac.manchester.cs.owl.owlapi.OWLOntologyFactoryImpl.loadOWLOntology(OWLOntologyFactoryImpl.java:220)
	... 11 more
make: *** [medgen-disease-extract.owl] Error 1

Examination

I tried a few variations, but here's basically how things end up running:

./src/[make_uid2cui.pl](http://make_uid2cui.pl/) > uid2cui.tsv
[8404](https://github.com/monarch-initiative/medgen/actions/runs/6031085372/job/16364118029#step:4:8405)./src/[medgen2obo.pl](http://medgen2obo.pl/) > medgen.obo.tmp && mv medgen.obo.tmp medgen.obo
[8405](https://github.com/monarch-initiative/medgen/actions/runs/6031085372/job/16364118029#step:4:8406)owltools medgen.obo --extract-subset Disease-or-Syndrome --set-ontology-id http://purl.obolibrary.org/obo/mondo/x-Disease-or-Syndrome.obo -o -f obo x-Disease-or-Syndrome.obo
[8406](https://github.com/monarch-initiative/medgen/actions/runs/6031085372/job/16364118029#step:4:8407)owltools medgen.obo --extract-subset Neoplastic-Process --set-ontology-id http://purl.obolibrary.org/obo/mondo/x-Neoplastic-Process.obo -o -f obo x-Neoplastic-Process.obo
[8407](https://github.com/monarch-initiative/medgen/actions/runs/6031085372/job/16364118029#step:4:8408)owltools x-Disease-or-Syndrome.obo x-Neoplastic-Process.obo --merge-support-ontologies -o -f obo medgen-disease-extract.obo

gzip -d [ftp.ncbi.nlm.nih.gov/pub/medgen/MedGenIDMappings.txt.gz](http://ftp.ncbi.nlm.nih.gov/pub/medgen/MedGenIDMappings.txt.gz)
python src/[mondo_robot_template.py](http://mondo_robot_template.py/) -i [ftp.ncbi.nlm.nih.gov/pub/medgen/MedGenIDMappings.txt](http://ftp.ncbi.nlm.nih.gov/pub/medgen/MedGenIDMappings.txt) -o medgen-xrefs.robot.template.tsv

mkdir -p output/release/
mv medgen.obo output/release/
mv medgen-disease-extract.obo output/release/

owltools medgen-disease-extract.obo -o medgen-disease-extract.owl
2023-08-30 23:36:34,008 ERROR (CommandRunner:4815) could not parse:medgen-disease-extract.obo

Notice lower down how the mv commands happen before this last command to create medgen-disease-extract.owl. That's the cause of the issue. I don't know why these mv commands (which come as a result of the stage goal are happening before the rest of the pipeline.

I don't understand why stage is executing in the middle of creating the chain of artefacts in build-lite. Observe the goal dependency chain & prereq order:

minimal: build-lite stage clean

# Stage & clean
PRODUCTS=medgen.obo medgen-disease-extract.obo medgen-disease-extract.owl medgen.sssom.tsv medgen-xrefs.robot.template.tsv
stage: $(patsubst %, stage-%, $(PRODUCTS))
stage-%: % | output/release/
	mv $< output/release/
clean:
	rm medgen.obographs.json
	rm uid2cui.tsv
	rm *.obo

# Build
build-lite: medgen-disease-extract.obo medgen-xrefs.robot.template.tsv

# medgen-xrefs.robot.template.tsv doesn't depend on the other artefacts
ftp.ncbi.nlm.nih.gov/pub/medgen/MedGenIDMappings.txt: ftp.ncbi.nlm.nih.gov/
	gzip -d ftp.ncbi.nlm.nih.gov/pub/medgen/MedGenIDMappings.txt.gz
medgen-xrefs.robot.template.tsv: ftp.ncbi.nlm.nih.gov/pub/medgen/MedGenIDMappings.txt
	python src/mondo_robot_template.py -i $< -o $@

# medgen-disease-extract.obo and more
x-%.obo: medgen.obo
	owltools $< --extract-subset $* --set-ontology-id $(OBO)/mondo/$@ -o -f obo $@

medgen-disease-extract.obo: x-Disease-or-Syndrome.obo x-Neoplastic-Process.obo
	owltools $^ --merge-support-ontologies -o -f obo $@

medgen.obo: ftp.ncbi.nlm.nih.gov/ uid2cui.tsv
	./src/medgen2obo.pl > $@.tmp && mv $@.tmp $@

I also read that make prereq order can't be enforced. But I've never run into this problem before. And also strange that make all did not have the same problems, and is very similar to make minimal.

Update:
Looks like it has something to do with the fact that I updated PRODUCTS from PRODUCTS=medgen-disease-extract.obo medgen-disease-extract.owl to PRODUCTS=medgen.obo medgen-disease-extract.obo medgen-disease-extract.owl medgen.sssom.tsv medgen-, and I didn't realize the % prereq in stage-%: % | output/release/. It seems to be creating and moving things before the next step starts.