Tutorial: Converting an existing project to a Calkit project

[petebachant]

This is a discussion thread for the tutorial Converting an existing project to a Calkit project.

Questions or comments?

[rebeccamccabe]

What steps should be taken if my existing project already uses source control and already has a (very large) overleaf for the publication?
It is a matlab project https://github.com/symbiotic-engineering/MDOcean, and I currently use CI with github actions and matlab unit test framework to run my optimization and validation study and generate all the figures reproducibly. I also auto-generate some latex tables from the results, but don't have automatic latex integration set up, I copy paste the auto-generated latex code, and manually upload the auto-generated figure pdfs, into my overleaf document.

[petebachant]

The first thing you'll want to do is download your files from Overleaf and add them to the MDOcean repo. You can do that by clicking on the "menu" button in the upper left corner on Overleaf, downloading the source, and unzipping into the repo. From the README, it sounds like you have a conference paper as well, so maybe you want to have a pubs directory, and put both as subdirectories inside that.

Then you will want to git add and git commit only the text files from the Overleaf folder, not the figures or PDF, since you'll generate those with the pipeline.

Next, you can turn the project into a Calkit project with (requires Calkit v0.21.5; update with pip install --upgrade calkit-python):

calkit new project . \
    --title MDOcean \
    --name mdocean \
    --description "Multidisciplinary Design Optimization (MDO) to optimize an ocean wave energy converter (WEC)." \
    --cloud \
    --public

To use the --cloud option, you'll need to ensure you've set your Calkit token.

Then it's time to start building the pipeline. You probably want to have one stage per figure (and table), and then of course one to build the paper, which we can do with a Docker environment, created with:

calkit new docker-env --name tex --image texlive/texlive:latest-full

In dvc.yaml, start adding stages like:

# In dvc.yaml
stages:
  # A stage to make a figure
  plot-something:
    cmd: matlab -noFigureWindows -batch "run('scripts/plot_something.m');"
    deps:
      - scripts/plot_something.m
      - data/raw. # TODO: Are there data file dependencies?
    outs:
      - figures/something.png
    meta:
      calkit:
        type: figure
        title: Something
        description: A plot of something.
  # A stage to make a table
  make-results-table:
    cmd: matlab -noFigureWindows -batch "run('scripts/make_table.m');"
    deps:
      - scripts/make_table.m
      # TODO: Add more dependencies if they exist
    outs:
      - tables/my-table.tex:
          cache: false # No need to cache with DVC since this is a small text file
  # A stage to build the paper
  build-paper:
    cmd: >
      calkit xenv -n tex --
      latexmk -cd -interaction=nonstopmode -pdf pubs/article/paper.tex
    deps:
      # Set the figures and tables directories as dependencies so we rebuild on change
      - figures
      - tables
      - pubs/article/paper.tex
    outs:
      - pubs/article/paper.pdf
    meta:
      calkit:
        type: publication
        kind: journal-article
        title: This is the title of my article
        description: This is the description of my article.

To include your LaTeX tables in the manuscript so you don't need to copy/paste them, you can write the tables out to their own .tex files, e.g., in a tables subdirectory, and use the \input command in the manuscript like:

\begin{table}[htbp]
  \centering
  \caption{Summary of Experimental Results}
  \label{tab:experimental_results}
  % Use table data from ../../tables/my-table.tex
  \input(../../tables/my-table)
\end{table}

If you want to add a dev container spec to the project (helpful if you or one of your collaborators wants to edit in GitHub Codespaces), you can do so with:

calkit update devcontainer

However, I have not yet set up MATLAB in a dev container, but it looks like it's possible to add as a feature.

Finally, you can setup a GitHub Actions workflow to execute calkit run and calkit save to build and push all of your artifacts to the cloud. There's an example here, though you'll need to add the setup-matlab step. You'll also need to ensure you set a CALKIT_DVC_TOKEN secret in GitHub Actions.

This should get you into a fully-reproducible state, and if you split up your MATLAB scripts well, you will be able to skip steps in the pipeline whose cached outputs are still valid, saving computation time.

Definitely let me know if anything goes wrong with any of these!

[petebachant]

Edit: With the new Overleaf integration features released in v0.22.0, it's now possible to import and sync an Overleaf project as a Calkit publication, and the Docker environment and pipeline will be automatically updated to build the document with calkit run. Figures can tables can be configured to push up to Overleaf from the Calkit project as well. Full docs here: https://docs.calkit.org/overleaf/

[rebeccamccabe]

Woah that’s pretty cool! I will have time to look at getting it set up in a week or so and can let you know how it works

…

On Thu, Apr 17, 2025 at 2:57 PM Pete Bachant ***@***.***> wrote: Edit: With the new Overleaf integration features released in v0.22.0, it's now possible to import and sync an Overleaf project as a Calkit publication, and the Docker environment and pipeline will be automatically updated to build the document with calkit run. Figures can tables can be configured to push up to Overleaf from the Calkit project as well. Full docs here: https://docs.calkit.org/overleaf/ — Reply to this email directly, view it on GitHub <#241 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHUJPSVUAGSPG7X3ZSJAXGL2Z72QPAVCNFSM6AAAAABWRMLVK6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBXGEYDKNQ> . You are receiving this because you commented.Message ID: ***@***.***>

[rebeccamccabe]

When I try it with the cloud option, I get the following error:
Error: Posting new project to cloud failed: 403: Could not validate credentials which seems to be due to the token problem in my other issue calkit/calkit-assistant#24.

So I tried it without cloud, and got:

calkit new project . --title MDOcean --name mdocean --description "Multidisciplinary Design Optimization (MDO) to optimize an ocean wave energy converter (WEC)."
Creating project mdocean
Using title: MDOcean
Downloading https://raw.githubusercontent.com/calkit/devcontainer/refs/heads/mai
n/devcontainer.json
Writing to C:\Users\rgm222\Documents\Github\SEA-Lab\MDOcean\.devcontainer\devcon
tainer.json
Warning: README.md already exists; not modifying
Setting up Calkit Cloud DVC remote
Setting 'calkit' as a default remote.
INFO:calkit:Creating token for DVC scope
Warning: Failed to set up Calkit Cloud DVC remote: 403: Could not validate crede
ntials

Congrats on creating your new Calkit project!

Next, you'll probably want to start building your pipeline.

Check out the docs at https://docs.calkit.org.

Should I expect to see the DVC token warning if I'm not running with cloud?

[petebachant]

The --cloud flag will attempt to create a new project on calkit.io. When you don't specify the --cloud flag, Calkit will assume the project already exists in the cloud and will still attempt to set up the DVC remote, hence the warning.

Creating this project on calkit.io should also attempt to create a new organization, MDOcean. I haven't tested that thoroughly, so if you run into any issues, let me know and I will help get it all set up properly.

[rebeccamccabe]

Error when trying overleaf:

 calkit overleaf import <my-url> paper --title "Journal Renewable Energy" --kind journal-article --sync-path mdocean.tex --push-path figs
Using PDF path: mdocean.pdf
Adding '.calkit\overleaf' to .gitignore
Cloning Overleaf project
Checking that this project has a LaTeX environment
Creating TeXlive Docker environment
Checking for a build stage in the pipeline
Creating build stage 'build-paper'
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ C:\Users\rgm222\AppData\Roaming\uv\tools\calkit-python\Lib\site-packages\cal │
│ kit\cli\overleaf.py:215 in import_publication                                │
│                                                                              │
│   212 │   │   │   stage_name = f"{stage_name}-{n}"                           │
│   213 │   │   │   n += 1                                                     │
│   214 │   │   typer.echo(f"Creating build stage '{stage_name}'")             │
│ ❱ 215 │   │   new_stage(                                                     │
│   216 │   │   │   name=stage_name,                                           │
│   217 │   │   │   environment=tex_env_name,                                  │
│   218 │   │   │   kind=StageKind.latex,                                      │
│                                                                              │
│ C:\Users\rgm222\AppData\Roaming\uv\tools\calkit-python\Lib\site-packages\cal │
│ kit\cli\new.py:1482 in new_stage                                             │
│                                                                              │
│   1479 │   │   warn("No environment is specified")                           │
│   1480 │   │   cmd = ""                                                      │
│   1481 │   else:                                                             │
│ ❱ 1482 │   │   if environment not in ck_info["environments"] and not no_chec │
│   1483 │   │   │   raise_error(f"Environment '{environment}' does not exist" │
│   1484 │   │   cmd = f"calkit xenv -n {environment} -- "                     │
│   1485 │   │   # Add environment path as a dependency if applicable          │
│                                                                              │
│ C:\Users\rgm222\AppData\Roaming\uv\tools\calkit-python\Lib\site-packages\rua │
│ mel\yaml\comments.py:853 in __getitem__                                      │
│                                                                              │
│    850 │                                                                     │
│    851 │   def __getitem__(self, key: Any) -> Any:                           │
│    852 │   │   try:                                                          │
│ ❱  853 │   │   │   return ordereddict.__getitem__(self, key)                 │
│    854 │   │   except KeyError:                                              │
│    855 │   │   │   for merged in getattr(self, merge_attrib, []):            │
│    856 │   │   │   │   if key in merged[1]:                                  │
╰──────────────────────────────────────────────────────────────────────────────╯
KeyError: 'environments'

[petebachant]

Looks like some careless programming on my part. You can add environments: {} in calkit.yaml as a workaround, but I will push a fix shortly.

[rebeccamccabe]

When retrying overleaf with that change, error due to write protection of the git repo:

calkit overleaf import https://www.overleaf.com/project/<my-url> paper --title "Journal Renewable Energy" --kind journal-article --sync-path mdocean.tex --push-path figs
Using PDF path: mdocean.pdf
.calkit\overleaf is already ignored
Warning: This Overleaf project has already been cloned; removing
╭────────────────────────────── Traceback (most recent call last) ──────────────────────────────╮
│ C:\Users\rgm222\AppData\Roaming\uv\tools\calkit-python\Lib\site-packages\calkit\cli\overleaf. │
│ py:164 in import_publication                                                                  │
│                                                                                               │
│   161 │   )                                                                                   │
│   162 │   if os.path.isdir(overleaf_project_dir):                                             │
│   163 │   │   warn("This Overleaf project has already been cloned; removing")                 │
│ ❱ 164 │   │   shutil.rmtree(overleaf_project_dir)                                             │
│   165 │   # Clone the Overleaf project                                                        │
│   166 │   typer.echo("Cloning Overleaf project")                                              │
│   167 │   git.Repo.clone_from(                                                                │
│                                                                                               │
│ C:\Users\rgm222\AppData\Roaming\uv\python\cpython-3.12.9-windows-x86_64-none\Lib\shutil.py:78 │
│ 1 in rmtree                                                                                   │
│                                                                                               │
│    778 │   │   │   onexc(os.path.islink, path, err)                                           │
│    779 │   │   │   # can't continue even if onexc hook returns                                │
│    780 │   │   │   return                                                                     │
│ ❱  781 │   │   return _rmtree_unsafe(path, onexc)                                             │
│    782                                                                                        │
│    783 # Allow introspection of whether or not the hardening against symlink                  │
│    784 # attacks is supported on the current platform                                         │
│                                                                                               │
│ C:\Users\rgm222\AppData\Roaming\uv\python\cpython-3.12.9-windows-x86_64-none\Lib\shutil.py:63 │
│ 5 in _rmtree_unsafe                                                                           │
│                                                                                               │
│    632 │   │   │   try:                                                                       │
│    633 │   │   │   │   os.unlink(fullname)                                                    │
│    634 │   │   │   except OSError as err:                                                     │
│ ❱  635 │   │   │   │   onexc(os.unlink, fullname, err)                                        │
│    636 │   try:                                                                               │
│    637 │   │   os.rmdir(path)                                                                 │
│    638 │   except OSError as err:                                                             │
│                                                                                               │
│ C:\Users\rgm222\AppData\Roaming\uv\python\cpython-3.12.9-windows-x86_64-none\Lib\shutil.py:63 │
│ 3 in _rmtree_unsafe                                                                           │
│                                                                                               │
│    630 │   │   for name in filenames:                                                         │
│    631 │   │   │   fullname = os.path.join(dirpath, name)                                     │
│    632 │   │   │   try:                                                                       │
│ ❱  633 │   │   │   │   os.unlink(fullname)                                                    │
│    634 │   │   │   except OSError as err:                                                     │
│    635 │   │   │   │   onexc(os.unlink, fullname, err)                                        │
│    636 │   try:                                                                               │
╰───────────────────────────────────────────────────────────────────────────────────────────────╯
PermissionError: [WinError 5] Access is denied:
'.calkit\\overleaf\\<my-url>\\.git\\objects\\pack\\pack-06d7168e5f3e4d2b88d5b0ce3
affd1266958a848.idx'

I was able to remove it by hand with rm -r -f .calkit/overleaf and then got further, erroring on not finding the figs path:

$ calkit overleaf import https://www.overleaf.com/project/<my-url> paper --title "Journal Renewable Energy" --kind journal-article --sync-path mdocean.tex --push-path figs
Using PDF path: mdocean.pdf
.calkit\overleaf is already ignored
Cloning Overleaf project
Checking that this project has a LaTeX environment
Creating TeXlive Docker environment
Checking for a build stage in the pipeline
Creating build stage 'build-paper'
Could not create .gitignore entry in C:\Users\rgm222\Documents\Github\SEA-Lab\mdocean\paper\.giti
gnore. DVC will attempt to create .gitignore entry again when the stage is run.
Added stage 'build-paper' in 'dvc.yaml'

To track the changes with git, run:

        git add dvc.yaml

To enable auto staging, run:

        dvc config core.autostage true
Adding publication to calkit.yaml
Committing changes
Syncing paper\mdocean.pdf with Overleaf project ID <my-url>
Pulling the latest version from Overleaf
No last sync commit defined; copying all files from Overleaf project
Error: Source path paper\figs does not exist; please check your Overleaf config

It seems like none of the overleaf files or folders except the main paper were downloaded from overleaf:

$ ls paper
mdocean.tex

[rebeccamccabe]

Actually the above was my error for not including everything in overleaf > sync_paths in the yaml file. It seems to work fine after adding all the files.

[rebeccamccabe]

Is there a way to disable the automatic git push? I would prefer for it to just add/commit so I can check that the changes are what I expect. thanks.

[petebachant]

Do you mean for the Overleaf sync command? There is a --no-commit option, but Git commands would need to be run in the Overleaf repo to check that everything makes sense before committing/pushing. That might be a nice feature to add: Unified status, commit, and push commands that integrate all synced Overleaf projects.

[rebeccamccabe]

Yes for the overleaf sync. It seemed that in the earlier calkit commands I tried (setting up dvc, adding stuff to gitignores), when it made auto commits it didn't push them, but it did push them for overleaf. I am not totally clear on the difference between the overleaf git repos and the papers/ folder of the project repo, it seems like calkit duplicated those files? Unified git commands aren't super high priority for what my workflow would be, but seems generally reasonable.

[petebachant]

One assumption is that the main Git repo is the most important one, and should therefore be the primary source of truth for the paper. So, the Overleaf repo cloned under .calkit is intended to be private--only used for syncing, not for editing. The workflow might be something like:

1. Collaborator makes edits on Overleaf (mainly because they don't want to setup a local environment for editing).
2. You call calkit overleaf sync to pull their changes into the main version.
3. You make some edits to the paper locally (in paper, not .calkit/...), or maybe update some figures, which are defined as push-only, since the main repo is the only place where figures can be generated.
4. You call calkit overleaf sync again to make both versions agree with each other.

[rebeccamccabe]

when trying to compile the overleaf:

$ calkit run
Checking system-level dependencies
Running stage 'build-paper':
> calkit xenv -n tex -- latexmk -cd -interaction=nonstopmode -pdf paper\mdocean.tex
docker: error during connect: in the default daemon configuration on Windows, the docker client m
ust be run with elevated privileges to connect: Head "http://%2F%2F.%2Fpipe%2Fdocker_engine/_ping
": open //./pipe/docker_engine: The system cannot find the file specified.

Run 'docker run --help' for more information
Error: Failed to run in Docker environment
ERROR: failed to reproduce 'build-paper': failed to run: calkit xenv -n tex -- latexmk -cd -inter
action=nonstopmode -pdf paper\mdocean.tex, exited with 1
Error: DVC pipeline failed

This was solved by opening docker desktop and accepting the license terms.
Then:

$ calkit run
Checking system-level dependencies
Running stage 'build-paper':
> calkit xenv -n tex -- latexmk -cd -interaction=nonstopmode -pdf paper\mdocean.tex
docker: request returned 500 Internal Server Error for API route and version http://%2F%2F.%2Fpip
e%2FdockerDesktopLinuxEngine/_ping, check if the server supports the requested API version

Run 'docker run --help' for more information
Error: Failed to run in Docker environment
ERROR: failed to reproduce 'build-paper': failed to run: calkit xenv -n tex -- latexmk -cd -inter
action=nonstopmode -pdf paper\mdocean.tex, exited with 1
Error: DVC pipeline failed

This was solved by hitting the restart ubuntu button that appeared in docker desktop.
Then:

$ calkit run
Checking system-level dependencies
Running stage 'build-paper':
> calkit xenv -n tex -- latexmk -cd -interaction=nonstopmode -pdf paper\mdocean.tex
Unable to find image 'texlive/texlive:latest-full' locally
latest-full: Pulling from texlive/texlive
c9d3b77a4420: Pull complete
4f4fb700ef54: Pull complete
28d2ae8da647: Pull complete
b6aab21ce24a: Pull complete
992e549ab51f: Pull complete
e327f073e290: Pull complete
9a58ebf350ef: Pull complete
34ea088e77cd: Pull complete
6172b9f14e87: Pull complete
Digest: sha256:ac2a1fbc4dad1ec27c7407c24907af5a38a5f53c4baf9c6dc5973efd34b355e6
Status: Downloaded newer image for texlive/texlive:latest-full
Rc files read:
  NONE
Latexmk: This is Latexmk, John Collins, 27 Dec. 2024. Version 4.86a.
Latexmk: Changing directory to './'
Latexmk: Undoing directory change

------------
Latexmk: Could not find file 'papermdocean.tex'.
-- Use the -f option to force complete processing.
Error: Failed to run in Docker environment
ERROR: failed to reproduce 'build-paper': failed to run: calkit xenv -n tex -- latexmk -cd -inter
action=nonstopmode -pdf paper\mdocean.tex, exited with 1
Error: DVC pipeline failed

which seems like a calkit issue for windows filepaths again where the \ should be a / (should be paper/mdocean.tex).

[petebachant]

which seems like a calkit issue for windows filepaths again where the \ should be a / (should be paper/mdocean.tex).

Yeah, seems like another one of those instances where a Posix path should be enforced. As a workaround, you can change the command in dvc.yaml manually to use a forward slash.

[rebeccamccabe]

#344 attempts to address the posixpath

With manually updating dvc.yaml and calkit.yaml to have the right slash, the pdf is generated, but it does not correctly do any of the internal links (ie table of contents, list of figures/tables, citations, section/figure/table references), and cakit reports an error:

$ calkit run
Checking system-level dependencies
Running stage 'build-paper':
> calkit xenv -n tex -- latexmk -cd -interaction=nonstopmode -pdf paper/mdocean.tex
Rc files read:
  NONE
Latexmk: This is Latexmk, John Collins, 27 Dec. 2024. Version 4.86a.
Latexmk: Changing directory to 'paper/'
No existing .aux file, so I'll make a simple one, and require run of *latex.
Latexmk: applying rule 'pdflatex'...
Rule 'pdflatex':  Reasons for rerun
Category 'other':
  Rerun of 'pdflatex' forced or previously required:
    Reason or flag: 'Initial setup'

------------
Run number 1 of rule 'pdflatex'
------------
------------
Running 'pdflatex  -interaction=nonstopmode -recorder  "mdocean.tex"'
------------
This is pdfTeX, Version 3.141592653-2.6-1.40.27 (TeX Live 2025) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode
(./mdocean.tex
LaTeX2e <2024-11-01> patch level 2
L3 programming layer <2025-04-14>
<long list of latex warnings>
LaTeX Warning: Label(s) may have changed. Rerun to get cross-references right.


Package rerunfilecheck Warning: File `mdocean.out' has changed.
(rerunfilecheck)                Rerun to get outlines right
(rerunfilecheck)                or use package `bookmark'.

 )
<list of fonts>
Output written on mdocean.pdf (157 pages, 4822528 bytes).
Transcript written on mdocean.log.
Latexmk: Getting log file 'mdocean.log'
Latexmk: Examining 'mdocean.fls'
Latexmk: Examining 'mdocean.log'
Latexmk: Missing input file 'mdocean.toc' (or dependence on it) from following:
  No file mdocean.toc.
Latexmk: Missing input file 'mdocean.lof' (or dependence on it) from following:
  No file mdocean.lof.
Latexmk: Missing input file 'mdocean.lot' (or dependence on it) from following:
  No file mdocean.lot.
Latexmk: Missing input file 'mdocean.wcsum' (or dependence on it) from following:
  No file mdocean.wcsum.
Latexmk: Missing input file 'sections/model.wcdetail' (or dependence on it) from following:
  No file sections/model.wcdetail.
Latexmk: Missing input file 'sections/other-appendices.wcdetail' (or dependence on it) from following:
  No file sections/other-appendices.wcdetail.
Latexmk: Missing bbl file 'mdocean.bbl' in following:
 No file mdocean.bbl.
Latexmk: References changed.
Latexmk: References changed.
Latexmk: Log file says output to 'mdocean.pdf'
Latexmk: Found bibliography file(s):
  ./references.bib
Latexmk: applying rule 'bibtex mdocean'...
Rule 'bibtex mdocean':  Reasons for rerun
Category 'other':
  Rerun of 'bibtex mdocean' forced or previously required:
    Reason or flag: 'Initial set up of rule'

------------
Run number 1 of rule 'bibtex mdocean'
------------
------------
Running 'bibtex  "mdocean.aux"'
------------
This is BibTeX, Version 0.99d (TeX Live 2025)
The top-level auxiliary file: mdocean.aux
The style file: elsarticle-num-names.bst
Database file #1: references.bib
<long list of bibtex warnings>
(There was 1 error message)
Latexmk: Summary of warnings from last run of *latex:
  Latex failed to resolve 245 reference(s)
  =====Latex reported missing or unavailable character(s).
=====See log file for details.
  Latex failed to resolve 287 citation(s)
Latexmk: ====Undefined refs and citations with line #s in .tex file:
  Citation `akdemir_opportunities_2023' on page 4 undefined on input line 7
  Citation `bhattacharya_timing_2021' on page 4 undefined on input line 7
  Citation `pennock_temporal_2022' on page 4 undefined on input line 7
  Citation `PBE' on page 4 undefined on input line 7
  Citation `RM3' on page 4 undefined on input line 15
  Reference `fig:rm3-parts' on page 4 undefined on input line 15
  Citation `trueworthy_wave_2020' on page 5 undefined on input line 29
 And 526 more --- see log file 'mdocean.log'
Latexmk: Errors, so I did not complete making targets
Collected error summary (may duplicate other messages):
  bibtex mdocean: Bibtex errors: See file 'mdocean.blg'
Latexmk: Undoing directory change

Latexmk: Sometimes, the -f option can be used to get latexmk
  to try to force complete processing.
  But normally, you will need to correct the file(s) that caused the
  error, and then rerun latexmk.
  In some cases, it is best to clean out generated files before rerunning
  latexmk after you've corrected the files.
Error: Failed to run in Docker environment
ERROR: failed to reproduce 'build-paper': failed to run: calkit xenv -n tex -- latexmk -cd -interaction=nonstopmode -pdf paper/mdocean.tex, exited with 1
Error: DVC pipeline failed

It seems that getting the links wrong the first time is expected and it says to rerun. I had to run 3 times total, but got it working! On first glance, everything seems the same as on overleaf except my word count that I use during drafting, which isn't a huge detail, It might be good for calkit to rerun automatically though. Also, even after rerunning several times, calkit still reports a fail when it seems to me a success:

$ calkit run
<various latex warnings>
Latexmk: All targets (mdocean.pdf) are up-to-date
Collected error summary (may duplicate other messages):
  bibtex mdocean: gave an error in previous invocation of latexmk.
Latexmk: Undoing directory change

Latexmk: Sometimes, the -f option can be used to get latexmk
  to try to force complete processing.
  But normally, you will need to correct the file(s) that caused the
  error, and then rerun latexmk.
  In some cases, it is best to clean out generated files before rerunning
  latexmk after you've corrected the files.
Error: Failed to run in Docker environment
ERROR: failed to reproduce 'build-paper': failed to run: calkit xenv -n tex -- latexmk -cd -interaction=nonstopmode -pdf paper/mdocean.tex, exited with 1
Error: DVC pipeline failed

Finally, it is a little inconvenient that I can't run calkit while viewing the pdf, since that creates the following file conflict:

$ calkit run
Checking system-level dependencies
ERROR: failed to reproduce 'build-paper': [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\rgm222\\Documents\\Github\
\SEA-Lab\\MDOcean\\paper\\mdocean.pdf'
Error: DVC pipeline failed

I am glad to have gotten it working! Thanks for the support.

[petebachant]

How are you viewing the PDF? I've found it pretty nice to use VS Code with the LaTeX Workshop extension, and view the PDF with that while working on it. It's also possible to setup the LaTeX build command / play button to rerun the entire pipeline.

[rebeccamccabe]

Running the pipeline results in a bunch of untracked latex files, should I be committing these or gitignoring them?

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        paper/mdocean.aux
        paper/mdocean.bbl
        paper/mdocean.blg
        paper/mdocean.fdb_latexmk
        paper/mdocean.fls
        paper/mdocean.lof
        paper/mdocean.lot
        paper/mdocean.out
        paper/mdocean.pdf
        paper/mdocean.spl
        paper/mdocean.toc
        paper/numbers.aux
        paper/sections/discussion.aux
        paper/sections/formulation.aux
        paper/sections/introduction.aux
        paper/sections/meem-appendix.aux
        paper/sections/model.aux
        paper/sections/optimization.aux
        paper/sections/other-appendices.aux
        paper/sections/results.aux

[petebachant]

Everything except the PDF can be ignored. That should be defined as a pipeline output, and can be cached with DVC. If you have this work in progress on a branch, I'd be happy to take a look/contribute if you want to share it.

[petebachant]

I will usually put a .gitignore in the paper directory with:

*.aux
*.fdb_latexmk
*.out
*.lof
*.lot
*.bbl
*.fls
*.spl
etc...

[rebeccamccabe]

Ok thanks! I am working on this branch https://github.com/symbiotic-engineering/MDOcean/tree/calkit and was just viewing the pdf with adobe reader. Normally I used overleaf but I will check out the vscode extension.

…

-Becca

On Tue, May 27, 2025 at 10:58 AM Pete Bachant ***@***.***> wrote: How are you viewing the PDF? I've found it pretty nice to use VS Code with the LaTeX Workshop extension, and view the PDF with that while working on it. It's also possible to setup the LaTeX build command / play button to rerun the entire pipeline. — Reply to this email directly, view it on GitHub <#241 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHUJPSTADOCA5ZYCCY5Z5VD3AR4QBAVCNFSM6AAAAABWRMLVK6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMRYGUYTAMQ> . You are receiving this because you commented.Message ID: ***@***.***>