Using qgis_process instead of **RQGIS3** #41
Comments
|
Just a quick note that it might be worth looking at how WhiteboxTools approaches this...I believe it's also wrapping a command-line application and I imagine that a QGIS tools wrapper would benefit from aligning syntax. (I haven't used yet...just stumbled across the other day!) Also a note that processx is probably the way to go (rather than system)...more options for status updates and cancelling a process that's taking too long. |
|
Minor update on this: there is now what I think is a reproducible environment to test/develop the interface proposed above with Docker. Reproducible example: Unfortunately that fails currently with the following message: Any ideas how to overcome that issue (perhaps running the command with arguments will mean it doesn't need QT but I'm worried about QT issues after rstudio/rstudio#4606 ) @paleolimbot (good idea on I have just sent a new image with the nightly build to DockerHub which should be available in the next hour or so that should be able to be run as follows: |
|
@Robinlovelace I haven't tried, but maybe you'll have more luck with |
|
Update, the qgis-dev version shows the same output: Great news @florisvdh I tried your suggestion and it worked 🎉 docker run -e PASSWORD=pw --rm -ti geocompr/geocompr:qgis-dev /bin/bash
# ...
env QT_QPA_PLATFORM='offscreen' qgis_process
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
../../src/core/qgsproviderregistry.cpp:245 : (init) [338ms] Loaded 22 providers (DB2;OAPIF;WFS;arcgisfeatureserver;arcgismapserver;delimitedtext;gdal;geonode;gpx;mdal;memory;mesh_memory;mssql;ogr;ows;postgres;postgresraster;spatialite;vectortile;virtual;wcs;wms)
QGIS Processing Executor - 3.15.0-Master 'Master' (3.15.0-Master)
Usage: /usr/bin/qgis_process.bin [command] [algorithm id or path to model file] [parameters]
Available commands:
plugins list available and active plugins
list list all available processing algorithms
help show help for an algorithm. The algorithm id or a path to a model file must be specified.
run runs an algorithm. The algorithm id or a path to a model file and parameter values must be specified. Parameter values are specified via the --PARAMETER=VALUE syntax.
If required, the ellipsoid to use for distance and area calculations can be specified via the "--ELLIPSOID=name" argument.
If required, an existing QGIS project to use during the algorithm execution can be specified via the "--PROJECT_PATH=path" argument. |
|
BTW see Dockerfiles defining the |
|
If you want to run scripts in the Docker container that are meant to be examples for users working in a GUI environment, the |
|
It is possible to set env vars in the Dockerfile using the ENV command, and I am pretty sure that this would work, when running the container with /bin/bash. However, I doubt that the env var will be exported when running the RStudio server version. At least, I didn't achieve this when I tried to do so but I vaguely remember that this is a known issue with rocker. |
|
I imagine that the env var should be set with every call (e.g., using I'll work up something in the next week or so with what I had in mind for wrapping the calls with processx...basically, it gives you some options for cancelling a long-runninng process and capturing output on stdout/stderr that might be useful for communicating progress in a way that |
|
Ok! My two cents are here: https://github.com/paleolimbot/qgisprocess Feel free to rewrite, fork, modify, do whatever! I provided a skeleton but haven't really tested. Basic idea is
Still don't know how to deal with the output values, because they're printed to a crazy format that probably aren't even parseable using regexes. If anybody has opened QGIS issues before, it would be really helpful to have a |
|
I have just tried your README example on Arch, and it is working perfectly! Excellent work! Pls let me know if I can be of any help! |
|
I have tested successfully a raster example as well: library("qgisprocess")
#> Using 'qgis_process' at 'qgis_process'.
#> Run `qgis_configure()` for details.
library("raster")
#> Loading required package: sp
data(dem, package = "RQGIS3")
projection(dem) = sf::st_crs(32717)$proj4string
pcurv = file.path(tempdir(), "pcurv.tif")
vcurv = file.path(tempdir(), "vcurv.tif")
info = qgis_run_algorithm(
alg = "grass7:r.slope.aspect",
elevation = dem,
pcurvature = pcurv,
tcurvature = vcurv,
.quiet = TRUE
)
out = stack(c(pcurv, vcurv))
plot(out)Created on 2020-10-08 by the reprex package (v0.3.0) Two things to note:
|
|
Just a heads up that you should reinstall! (I just did some major improvements/bugfixes). I'm not sure the default values are available with |
|
Ok, thanks for the heads-up! As I said, if multiple options are available, then the first option (coded as 0) is afaik always the default in QGIS. In RQGIS(3) we simply collected all arguments, and if more than one option was available and the user did not specify anything, we simply used the first option by setting it to 0. Additionally, we issued a message telling the user, which default options were chosen. Not sure, if the same approach can be used here, since it might be difficult to extract all the arguments and potential options from |
|
There's a bit of grepping involved with Some reprex-like thoughts: qgisprocess::qgis_arguments("native:simplifygeometries")
#> # A tibble: 4 x 3
#> name description qgis_type
#> <chr> <chr> <chr>
#> 1 INPUT Input layer source
#> 2 METHOD Simplification method enum
#> 3 TOLERANCE Tolerance distance
#> 4 OUTPUT Simplified sink
qgisprocess::qgis_outputs("native:simplifygeometries")
#> # A tibble: 1 x 3
#> name description qgis_output_type
#> <chr> <chr> <chr>
#> 1 OUTPUT Simplified outputVector
# not provided by user
qgisprocess::as_qgis_argument(rlang::zap(), "enum")
#> Error: Don't know how to convert object of type 'rlang_zap' to QGIS type 'enum'
# provided as number
qgisprocess::as_qgis_argument(1, "enum")
#> [1] "1"
# provided as string (should return the index of the value)
qgisprocess::as_qgis_argument("some valid value", "enum")
#> [1] "some valid value"Created on 2020-10-08 by the reprex package (v0.3.0) |
@jannes-m re-assigning the CRS should work using other CRS representations as well (EPSG-code, WKT), but it needs Reprex + session infolibrary(raster)
#> Loading required package: sp
rgdal::rgdal_extSoftVersion()
#> GDAL GDAL_with_GEOS PROJ sp
#> "3.1.3" "TRUE" "7.1.1" "1.4-4"
data(dem, package = "RQGIS3")
wkt(dem) # empty
#> Warning in wkt(dem): no wkt comment
#> [1] ""
wkt(crs(dem)) # same
#> Warning in wkt(crs(dem)): CRS object has no comment
#> NULL
crs(dem)
#> CRS arguments:
#> +proj=utm +zone=17 +south +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m
#> +no_defs
sf::st_crs(32717)$proj4string
#> [1] "+proj=utm +zone=17 +south +datum=WGS84 +units=m +no_defs"
crs(dem) <- 32717 # needs development version of raster;
# wkt can also be assigned
cat(wkt(dem))
#> PROJCRS["WGS 84 / UTM zone 17S",
#> BASEGEOGCRS["WGS 84",
#> DATUM["World Geodetic System 1984",
#> ELLIPSOID["WGS 84",6378137,298.257223563,
#> LENGTHUNIT["metre",1]]],
#> PRIMEM["Greenwich",0,
#> ANGLEUNIT["degree",0.0174532925199433]],
#> ID["EPSG",4326]],
#> CONVERSION["UTM zone 17S",
#> METHOD["Transverse Mercator",
#> ID["EPSG",9807]],
#> PARAMETER["Latitude of natural origin",0,
#> ANGLEUNIT["degree",0.0174532925199433],
#> ID["EPSG",8801]],
#> PARAMETER["Longitude of natural origin",-81,
#> ANGLEUNIT["degree",0.0174532925199433],
#> ID["EPSG",8802]],
#> PARAMETER["Scale factor at natural origin",0.9996,
#> SCALEUNIT["unity",1],
#> ID["EPSG",8805]],
#> PARAMETER["False easting",500000,
#> LENGTHUNIT["metre",1],
#> ID["EPSG",8806]],
#> PARAMETER["False northing",10000000,
#> LENGTHUNIT["metre",1],
#> ID["EPSG",8807]]],
#> CS[Cartesian,2],
#> AXIS["(E)",east,
#> ORDER[1],
#> LENGTHUNIT["metre",1]],
#> AXIS["(N)",north,
#> ORDER[2],
#> LENGTHUNIT["metre",1]],
#> USAGE[
#> SCOPE["unknown"],
#> AREA["World - S hemisphere - 84°W to 78°W - by country"],
#> BBOX[-80,-84,0,-78]],
#> ID["EPSG",32717]]Created on 2020-10-09 by the reprex package (v0.3.0) Session infodevtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.0.2 (2020-06-22)
#> os Linux Mint 20
#> system x86_64, linux-gnu
#> ui X11
#> language nl_BE:nl
#> collate nl_BE.UTF-8
#> ctype nl_BE.UTF-8
#> tz Europe/Brussels
#> date 2020-10-09
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date lib source
#> assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.0.2)
#> backports 1.1.10 2020-09-15 [1] CRAN (R 4.0.2)
#> callr 3.4.4 2020-09-07 [1] CRAN (R 4.0.2)
#> class 7.3-17 2020-04-26 [4] CRAN (R 4.0.0)
#> classInt 0.4-3 2020-04-07 [1] CRAN (R 4.0.2)
#> cli 2.0.2 2020-02-28 [1] CRAN (R 4.0.2)
#> codetools 0.2-16 2018-12-24 [4] CRAN (R 4.0.0)
#> crayon 1.3.4 2017-09-16 [1] CRAN (R 4.0.2)
#> DBI 1.1.0 2019-12-15 [1] CRAN (R 4.0.2)
#> desc 1.2.0 2018-05-01 [1] CRAN (R 4.0.2)
#> devtools 2.3.2 2020-09-18 [1] CRAN (R 4.0.2)
#> digest 0.6.25 2020-02-23 [1] CRAN (R 4.0.2)
#> dplyr 1.0.2 2020-08-18 [1] CRAN (R 4.0.2)
#> e1071 1.7-3 2019-11-26 [1] CRAN (R 4.0.2)
#> ellipsis 0.3.1 2020-05-15 [1] CRAN (R 4.0.2)
#> evaluate 0.14 2019-05-28 [1] CRAN (R 4.0.2)
#> fansi 0.4.1 2020-01-08 [1] CRAN (R 4.0.2)
#> fs 1.5.0 2020-07-31 [1] CRAN (R 4.0.2)
#> generics 0.0.2 2018-11-29 [1] CRAN (R 4.0.2)
#> glue 1.4.2 2020-08-27 [1] CRAN (R 4.0.2)
#> highr 0.8 2019-03-20 [1] CRAN (R 4.0.2)
#> htmltools 0.5.0 2020-06-16 [1] CRAN (R 4.0.2)
#> KernSmooth 2.23-17 2020-04-26 [4] CRAN (R 4.0.0)
#> knitr 1.30 2020-09-22 [1] CRAN (R 4.0.2)
#> lattice 0.20-41 2020-04-02 [4] CRAN (R 4.0.0)
#> lifecycle 0.2.0 2020-03-06 [1] CRAN (R 4.0.2)
#> magrittr 1.5 2014-11-22 [1] CRAN (R 4.0.2)
#> memoise 1.1.0 2017-04-21 [1] CRAN (R 4.0.2)
#> pillar 1.4.6 2020-07-10 [1] CRAN (R 4.0.2)
#> pkgbuild 1.1.0 2020-07-13 [1] CRAN (R 4.0.2)
#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.0.2)
#> pkgload 1.1.0 2020-05-29 [1] CRAN (R 4.0.2)
#> prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.0.2)
#> processx 3.4.4 2020-09-03 [1] CRAN (R 4.0.2)
#> ps 1.3.4 2020-08-11 [1] CRAN (R 4.0.2)
#> purrr 0.3.4 2020-04-17 [1] CRAN (R 4.0.2)
#> R6 2.4.1 2019-11-12 [1] CRAN (R 4.0.2)
#> raster * 3.3-16 2020-09-28 [1] Github (rspatial/raster@caf4148)
#> Rcpp 1.0.5 2020-07-06 [1] CRAN (R 4.0.2)
#> remotes 2.2.0 2020-07-21 [1] CRAN (R 4.0.2)
#> reprex 0.3.0 2019-05-16 [1] CRAN (R 4.0.2)
#> rgdal 1.5-17 2020-10-08 [1] CRAN (R 4.0.2)
#> rlang 0.4.7 2020-07-09 [1] CRAN (R 4.0.2)
#> rmarkdown 2.4 2020-09-30 [1] CRAN (R 4.0.2)
#> rprojroot 1.3-2 2018-01-03 [1] CRAN (R 4.0.2)
#> sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 4.0.2)
#> sf 0.9-6 2020-09-13 [1] CRAN (R 4.0.2)
#> sp * 1.4-4 2020-10-07 [1] Github (rsbivand/sp@333db94)
#> stringi 1.5.3 2020-09-09 [1] CRAN (R 4.0.2)
#> stringr 1.4.0 2019-02-10 [1] CRAN (R 4.0.2)
#> testthat 2.3.2 2020-03-02 [1] CRAN (R 4.0.2)
#> tibble 3.0.3 2020-07-10 [1] CRAN (R 4.0.2)
#> tidyselect 1.1.0 2020-05-11 [1] CRAN (R 4.0.2)
#> units 0.6-7 2020-06-13 [1] CRAN (R 4.0.2)
#> usethis 1.6.3 2020-09-17 [1] CRAN (R 4.0.2)
#> vctrs 0.3.4 2020-08-29 [1] CRAN (R 4.0.2)
#> withr 2.3.0 2020-09-22 [1] CRAN (R 4.0.2)
#> xfun 0.18 2020-09-29 [1] CRAN (R 4.0.2)
#> yaml 2.2.1 2020-02-01 [1] CRAN (R 4.0.2)
#>
#> [1] /home/floris/lib/R/library
#> [2] /usr/local/lib/R/site-library
#> [3] /usr/lib/R/site-library
#> [4] /usr/lib/R/library |
|
Great to see this discussion progressing, fantastic to see people chiming in and rallying-round to support solutions. In terms of the direction of travel, I suggest we use this thread as a metadiscussion around the approach in support of getting |
|
@florisvdh Thanks, note that sorting out CRS representation may need to see what QGIS (version?) and the underlying GRASS, SAGA etc. are doing. Possibly the versions need checking and aligning. |
|
@rsbivand I imagine that is a challenge, thanks for pointing out. |
|
See also #43 (comment) wrt. axis order, etc. |
|
From the qgisprocess package standpoint, it "just" writes a tempfile for the R objects it encounters...I'm not familiar enough with this stuff to know if I can choose a file format with the fewest restrictions by default (right now I'm also not sure this package would be a good fit for CRAN, which will never have a QGIS install (nor should it). There's also no compiled code that users would benefit from having pre-built or rigorously checked...but maybe I'm missing something obvious here. |
|
From spgrass6 and rgrass7, CRAN has not worried that example code conditioned on having access to external software; the assumption has been that the maintainer is responsible for ensuring that the package works as expected when the software is present. So distribution through CRAN should not be a problem for loosely linked integration (temporary files, use of |
From now on, we discourage the usage of RQGIS3 since it is no longer actively maintained and due to various unresolvable issues. However, the good news is that there is a new standalone executable available from QGIS 3.14 onwards (thanks to @paleolimbot for pointing me to it several months ago). This means, one can now run QGIS processing algorithms from the command line with ease. The tool is easy to use, takes care of setting the environmental variables to run QGIS and the user no longer has to use Python. In short, this seems to be the perfect replacement for RQGIS3 . For a short description how to use
qgis_processfrom within R, please have a look at https://github.com/r-spatial/RQGIS3/Of course, we should wait a bit to see how and if
qgis_processmatures but if it does it would be relatively easy to write an R wrapper package around it. I and @pat-s will probably not have the time to do so (unfortunately) but we would support anybody in doing so as best as we can. So please let us know if anybody has the time and the passion to write a new RQGIS3 package. If not, one could still access all QGIS geoprocessing viasystem()calls. Of course, this is not as user-friendly as a dedicated R package but still relatively easily managable. I also plan to update the RQGIS3 section in Geocomputation with R by explaining a bit more in detail how to useqgis_process(@Robinlovelace, @Nowosad).The text was updated successfully, but these errors were encountered: