The Docker PSOCK example times out.

[wlandau]

I just retried the Docker PSOCK example on my personal desktop before work this morning, and there seems to be trouble when a target relies on input files or produces output files. @januz, to check if this is really the problem on your end, you might try building everything except the report.

library(future)
library(drake)

cl <- future::makeClusterPSOCK( # nolint
  "localhost",
  ## Launch Rscript inside Docker container
  rscript = c(
    "docker", "run", "--net=host", "rocker/r-base",
    "Rscript"
  ),
  ## Install drake
  rscript_args = c(
    "-e", shQuote("install.packages('drake')")
  )
)

future::plan(cluster, workers = cl)
load_mtcars_example()
my_plan <- my_plan[-1, ] # Skip the report
make(my_plan, parallelism = "future")

Either way, we need some way to share files with the Docker container. Ideally, we would make the working directory inside Docker match the working directory of the parent R session. I have not used Docker seriously enough to know how to do this off the top of my head, and I would gladly accept pull requests.

[januz]

@wlandau Thanks for your response!

Unfortunately, on my system, the timeout error already occurs while assigning the cl object. I get a lot of output from the Docker image being built, which even continues after the error message appears, but the cl object is never assigned. So maybe I should take this issue to the future package?!

Regarding sharing files/directories between the host and the container: I haven't worked with Docker much, but if I get the first problem sorted, I'll look into options for how to do this. From a quick search, one should be able to do so using either the --mount or -v flags with docker run (see here or here).

One last point: If you should at some point have time to look over my approach to using a packaged drake workflow for reproducible research (see the Github repo or my issue in @tiernanmartin's drakepkg), any thoughts/advice would be greatly appreciated! Thank you

[januz]

@wlandau with the help of @HenrikBengtsson my general Docker issue could be resolved (see)

I now tried out how one can mount a host's directory in the container to have the Docker drake example run successfully. The following works on my machine:

library(future)
library(drake)
library(dplyr)
library(stringr)

cl <- future::makeClusterPSOCK( # nolint
  "localhost",
  ## Launch Rscript inside Docker container
  rscript = c(
    "docker", "run", "--net=host",
    "--mount", paste("type=bind,source=", getwd(), ",target=/home/rstudio", sep = ""),
    "rocker/verse",
    "Rscript"
  ),
  rscript_args = c(
    ## Install drake
    "-e", shQuote("install.packages('drake')"),
    
    ## set working directory to bound dir
    "-e", shQuote("setwd('home/rstudio')")
  ),
  master = if (grepl("(Darwin|Windows)", Sys.info()["sysname"])) "host.docker.internal" else "localhost"
)

future::plan(cluster, workers = cl)
load_mtcars_example()

# run workflow on host
plan_host <- my_plan %>% 
  mutate(
    command = str_replace(command, "report.md", "report_host.md")
  )
make(plan_host)

# run workflow within Docker container
plan_docker <- my_plan %>% 
  mutate(
    command = str_replace(command, "report.md", "report_docker.md")
  )
make(plan_docker, parallelism = "future")

A few comments:

• To use the local directory in the Docker container, one needs to add the option --mount ... and mount the current working directory to a folder with writing permissions in the container. Furthermore, one has to setwd() to this directory
• the option --master was needed to solve my Docker problems on the Mac. I only tried it on a Mac, but I assume that it should also work on Windows and Linux machines.
• I replaced rocker/base with rocker/verse. When using rocker/base, the command timed out on my machine as installing all the dependencies for drake took too long. Using the image with more pre-installed packages solved that problem. In general it might be a good idea to provide a Docker image that has drake already installed, so users don't have to wait too long for the creation of the worker.
• To be able to see whether one runs in the container or on the host machine, I modified the report.Rmd file and made two versions of the plan to be able to compare the knitted files.

Modification to report.Rmd:

# Check whether `drake` ran in Docker container or on host

```{r}
Sys.info()
```

One problem/question:

When I just run

make(my_plan)

and afterwards

make(my_plan, parallelism = "future")

drake reports no differences and doesn't re-knit the report. My assumption was that drake would consider the chunk as outdated because it results in different outputs depending on whether it's run within the Docker container or on the host. But apparently it doesn't. The same is true when I define a new target for the Sys.info() call.

If you'd like, I can prepare a pull request after implementing your feedback.

[wlandau]

Wow, this is fantastic! Thanks so much @januz! And yes, I would really appreciate a pull request.

A few (very minor) comments:

• For this drake-examples repository, I think we can just go with make(plan, parallelism = "future") rather than separate plan_host and plan_docker runs. These separate plans were useful in your original comment, however.
• In the next release of drake, future will be moved to "Suggests:" rather than "Imports:", so I think we need to install it separately.
• Could we move the if() statement outside makeClusterPSOCK()?
• I really like having Sys.info() in the report. Maybe we could append it in a code chunk before make().
• In the PR, please feel free to acknowledge yourself as a contributor here, either in a new file with the authors or in the existing README.md. Your work really helps.

Does this work for you? I will also try on Ubuntu 16.04 when I get a chance later.

library(drake)
library(future)
library(stringr)

platform <- "localhost"
if (grepl("(Darwin|Windows)", Sys.info()["sysname"])) {
  platform <- "host.docker.internal"
}

cl <- future::makeClusterPSOCK( # nolint
  "localhost",
  ## Launch Rscript inside Docker container
  rscript = c(
    "docker", "run", "--net=host",
    "--mount", paste("type=bind,source=", getwd(), ",target=/home/rstudio", sep = ""),
    "rocker/verse",
    "Rscript"
  ),
  rscript_args = c(
    ## Install drake
    "-e", shQuote("install.packages('drake')"),

    ## Install future
    "-e", shQuote("install.packages('drake')"),

    ## set working directory to bound dir
    "-e", shQuote("setwd('home/rstudio')")
  ),
  master = platform
)

load_mtcars_example()

# Add a code chunk in `report.Rmd` to verify that
# we are really running it in a Docker container.
write("\n```{r info}\nSys.info()\n```", "report.Rmd", append = TRUE)

make(my_plan, parallelism = "future")

[wlandau]

Also, thanks for switching to an image with more pre-installed packages.

[januz]

@wlandau Thanks for your response! I just submitted the PR.

What are your thoughts regarding the "problem" I lined out above, with drake not recognizing changes in objects that are dependent on whether the workflow is run inside the container or on the host?

[wlandau]

@wlandau Thanks for your response! I just submitted the PR.

Awesome!

What are your thoughts regarding the "problem" I lined out above, with drake not recognizing changes in objects that are dependent on whether the workflow is run inside the container or on the host?

I think it is the correct default behavior. drake would be too brittle if projects run on one machine did not stay up to date when transferred to another machine or another mode of parallel computing. One of the goals is to show tangible evidence that the output matches the code and data it came from. So we really do want to be able to send the project to someone else, have them run outdated() or make(), and see for themselves that everything is up to date.

That said, you could probably set up a custom trigger to detect the presence of the image and invalidate targets that really should depend on it.

docker_sys_info <- function(){
  ...
}

drake_plan(
  ...,
  report_with_sys.info = target(
    command = render(knitr_in("report.Rmd"), ...),
    trigger = trigger(change = docker_sys_info())
  )
)

[januz]

I think it is the correct default behavior. drake would be too brittle if projects run on one machine did not stay up to date when transferred to another machine or another mode of parallel computing.

Alright, that makes sense!