Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

R Objects saved from model_parts() do not find their methods #569

Closed
RaymondBalise opened this issue Aug 17, 2024 · 3 comments
Closed

R Objects saved from model_parts() do not find their methods #569

RaymondBalise opened this issue Aug 17, 2024 · 3 comments

Comments

@RaymondBalise
Copy link

Hello DALEX and DALEXtra folks,
My team is incredibly grateful for your work but we have hit a bug (using DALEX 2.4.3 and DALEXtra 2.3.0 under R 4.4.1).

I can use the model_parts() function to make an object and it prints and plots as it should (it looks like a VIP plot from your book):

  vip_lasso <-
    the_explainer |>
    model_parts(N = NULL, B = 20, type = "difference")

When I save and reload the object and packages in another/new R session like this:

load("../ctn0094modeling/data/vip_lasso.RData")

library(tidyverse)
library(tidymodels)
library(DALEX)
library(DALEXtra)

the classes on the saved object look right:

class(vip_lasso)
[1] "model_parts"                  "feature_importance_explainer" "data.frame"

but when I plot the object with plot(vip_lasso) it no longer displays as a VIP plot. Instead it shows this:

SnagitHelper2024 2024-08-17 10 30 04

and when I print it, it shows all the permutation results rather than the average of the repetitions.

I think that means it is not finding the plot.model_parts method and the print.model_parts methods,

If I make any vip plot (even with different data), for example calling variable_importance()

load(glue::glue("../ctn0094modeling/data/a_train.RData"))
load(glue::glue("../ctn0094modeling/data/rf_final_fit.RData"))

the_explainer <-
  explain_tidymodels(
    rf_final_fit,
    data = a_train %>% select(-did_relapse),
    y = as.numeric(a_train$did_relapse),
    #label = as.character(call_info[[2]]),
    verbose = FALSE
  )

the_vi <- variable_importance(the_explainer)

the problematic plot displays properly if I try to plot it again. That is, the original vip_lasso object now plots and prints correctly. I checked the loaded packages before and after running variable_importance() and no other libraries have been loaded as a side effect. The only thing that I can see that changes is, a couple functions in DALEX are no longer promises.

Can you think of a reason why the plot and print methods for "model_parts" objects are not being used after the packages are loaded but they are available after making a plot?

The data is large, complex and confidential. So making a REPREX is going to be rough but I can do it if you can't suggest a way to fix this.
@pbiecek
Copy link
Member

pbiecek commented Aug 17, 2024

Would you share devtools::session_info() for both scenarios?
It looks like some required package is not loaded (lazy loading)
likely it is about iBreakDown

@RaymondBalise
Copy link
Author

Thank you so much for the fast response!

After loading these packages:
library(tidyverse)
library(tidymodels)
library(DALEX)
library(DALEXtra)

> devtools::session_info()
─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.4.1 (2024-06-14)
 os       macOS Sonoma 14.6.1
 system   aarch64, darwin20
 ui       RStudio
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       America/New_York
 date     2024-08-18
 rstudio  2024.04.2+764 Chocolate Cosmos (desktop)
 pandoc   NA

─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────
 package      * version    date (UTC) lib source
 backports      1.5.0      2024-05-23 [1] CRAN (R 4.4.0)
 broom        * 1.0.6      2024-05-17 [1] CRAN (R 4.4.0)
 cachem         1.1.0      2024-05-16 [1] CRAN (R 4.4.0)
 class          7.3-22     2023-05-03 [1] CRAN (R 4.4.1)
 cli            3.6.3      2024-06-21 [1] CRAN (R 4.4.0)
 codetools      0.2-20     2024-03-31 [1] CRAN (R 4.4.1)
 colorspace     2.1-1      2024-07-26 [1] CRAN (R 4.4.0)
 DALEX        * 2.4.3      2023-01-15 [1] CRAN (R 4.4.0)
 DALEXtra     * 2.3.0      2023-05-26 [1] CRAN (R 4.4.0)
 data.table     1.15.4     2024-03-30 [1] CRAN (R 4.4.0)
 devtools     * 2.4.5      2022-10-11 [1] CRAN (R 4.4.0)
 dials        * 1.2.1      2024-02-22 [1] CRAN (R 4.4.0)
 DiceDesign     1.10       2023-12-07 [1] CRAN (R 4.4.0)
 digest         0.6.36     2024-06-23 [1] CRAN (R 4.4.0)
 dplyr        * 1.1.4      2023-11-17 [1] CRAN (R 4.4.0)
 ellipsis       0.3.2      2021-04-29 [1] CRAN (R 4.4.0)
 fansi          1.0.6      2023-12-08 [1] CRAN (R 4.4.0)
 fastmap        1.2.0      2024-05-15 [1] CRAN (R 4.4.0)
 forcats      * 1.0.0      2023-01-29 [1] CRAN (R 4.4.0)
 foreach        1.5.2      2022-02-02 [1] CRAN (R 4.4.0)
 fs             1.6.4      2024-04-25 [1] CRAN (R 4.4.0)
 furrr          0.3.1      2022-08-15 [1] CRAN (R 4.4.0)
 future         1.33.2     2024-03-26 [1] CRAN (R 4.4.0)
 future.apply   1.11.2     2024-03-28 [1] CRAN (R 4.4.0)
 generics       0.1.3      2022-07-05 [1] CRAN (R 4.4.0)
 ggplot2      * 3.5.1      2024-04-23 [1] CRAN (R 4.4.0)
 ggthemes       5.1.0      2024-02-10 [1] CRAN (R 4.4.0)
 globals        0.16.3     2024-03-08 [1] CRAN (R 4.4.0)
 glue           1.7.0      2024-01-09 [1] CRAN (R 4.4.0)
 gower          1.0.1      2022-12-22 [1] CRAN (R 4.4.0)
 GPfit          1.0-8      2019-02-08 [1] CRAN (R 4.4.0)
 gtable         0.3.5      2024-04-22 [1] CRAN (R 4.4.0)
 hardhat        1.4.0      2024-06-02 [1] CRAN (R 4.4.0)
 hms            1.1.3      2023-03-21 [1] CRAN (R 4.4.0)
 htmltools      0.5.8.1    2024-04-04 [1] CRAN (R 4.4.0)
 htmlwidgets    1.6.4      2023-12-06 [1] CRAN (R 4.4.0)
 httpuv         1.6.15     2024-03-26 [1] CRAN (R 4.4.0)
 infer        * 1.0.7      2024-03-25 [1] CRAN (R 4.4.0)
 ipred          0.9-15     2024-07-18 [1] CRAN (R 4.4.0)
 iterators      1.0.14     2022-02-05 [1] CRAN (R 4.4.0)
 later          1.3.2      2023-12-06 [1] CRAN (R 4.4.0)
 lattice        0.22-6     2024-03-20 [1] CRAN (R 4.4.1)
 lava           1.8.0      2024-03-05 [1] CRAN (R 4.4.0)
 lhs            1.2.0      2024-06-30 [1] CRAN (R 4.4.0)
 lifecycle      1.0.4      2023-11-07 [1] CRAN (R 4.4.0)
 listenv        0.9.1      2024-01-29 [1] CRAN (R 4.4.0)
 lubridate    * 1.9.3      2023-09-27 [1] CRAN (R 4.4.0)
 magrittr       2.0.3      2022-03-30 [1] CRAN (R 4.4.0)
 MASS           7.3-61     2024-06-13 [1] CRAN (R 4.4.0)
 Matrix         1.7-0      2024-04-26 [1] CRAN (R 4.4.1)
 memoise        2.0.1      2021-11-26 [1] CRAN (R 4.4.0)
 mime           0.12       2021-09-28 [1] CRAN (R 4.4.0)
 miniUI         0.1.1.1    2018-05-18 [1] CRAN (R 4.4.0)
 modeldata    * 1.4.0      2024-06-19 [1] CRAN (R 4.4.0)
 munsell        0.5.1      2024-04-01 [1] CRAN (R 4.4.0)
 nnet           7.3-19     2023-05-03 [1] CRAN (R 4.4.1)
 parallelly     1.38.0     2024-07-27 [1] CRAN (R 4.4.0)
 parsnip      * 1.2.1      2024-03-22 [1] CRAN (R 4.4.0)
 pillar         1.9.0      2023-03-22 [1] CRAN (R 4.4.0)
 pkgbuild       1.4.4      2024-03-17 [1] CRAN (R 4.4.0)
 pkgconfig      2.0.3      2019-09-22 [1] CRAN (R 4.4.0)
 pkgload        1.4.0      2024-06-28 [1] CRAN (R 4.4.0)
 prodlim        2024.06.25 2024-06-24 [1] CRAN (R 4.4.0)
 profvis        0.3.8      2023-05-02 [1] CRAN (R 4.4.0)
 promises       1.3.0      2024-04-05 [1] CRAN (R 4.4.0)
 purrr        * 1.0.2      2023-08-10 [1] CRAN (R 4.4.0)
 R6             2.5.1      2021-08-19 [1] CRAN (R 4.4.0)
 ranger         0.16.0     2023-11-12 [1] CRAN (R 4.4.0)
 Rcpp           1.0.13     2024-07-17 [1] CRAN (R 4.4.0)
 readr        * 2.1.5      2024-01-10 [1] CRAN (R 4.4.0)
 recipes      * 1.1.0      2024-07-04 [1] CRAN (R 4.4.0)
 remotes        2.5.0      2024-03-17 [1] CRAN (R 4.4.0)
 reprex       * 2.1.1      2024-07-06 [1] CRAN (R 4.4.0)
 rlang          1.1.4      2024-06-04 [1] CRAN (R 4.4.0)
 rpart          4.1.23     2023-12-05 [1] CRAN (R 4.4.1)
 rsample      * 1.2.1      2024-03-25 [1] CRAN (R 4.4.0)
 rstudioapi     0.16.0     2024-03-24 [1] CRAN (R 4.4.0)
 scales       * 1.3.0      2023-11-28 [1] CRAN (R 4.4.0)
 sessioninfo    1.2.2      2021-12-06 [1] CRAN (R 4.4.0)
 shiny          1.8.1.1    2024-04-02 [1] CRAN (R 4.4.0)
 stringi        1.8.4      2024-05-06 [1] CRAN (R 4.4.0)
 stringr      * 1.5.1      2023-11-14 [1] CRAN (R 4.4.0)
 survival       3.7-0      2024-06-05 [1] CRAN (R 4.4.0)
 tibble       * 3.2.1      2023-03-20 [1] CRAN (R 4.4.0)
 tidymodels   * 1.2.0      2024-03-25 [1] CRAN (R 4.4.0)
 tidyr        * 1.3.1      2024-01-24 [1] CRAN (R 4.4.0)
 tidyselect     1.2.1      2024-03-11 [1] CRAN (R 4.4.0)
 tidyverse    * 2.0.0      2023-02-22 [1] CRAN (R 4.4.0)
 timechange     0.3.0      2024-01-18 [1] CRAN (R 4.4.0)
 timeDate       4032.109   2023-12-14 [1] CRAN (R 4.4.0)
 tune         * 1.2.1      2024-04-18 [1] CRAN (R 4.4.0)
 tzdb           0.4.0      2023-05-12 [1] CRAN (R 4.4.0)
 urlchecker     1.0.1      2021-11-30 [1] CRAN (R 4.4.0)
 usethis      * 2.2.3      2024-02-19 [1] CRAN (R 4.4.0)
 utf8           1.2.4      2023-10-22 [1] CRAN (R 4.4.0)
 vctrs          0.6.5      2023-12-01 [1] CRAN (R 4.4.0)
 withr          3.0.0      2024-01-16 [1] CRAN (R 4.4.0)
 workflows    * 1.1.4      2024-02-19 [1] CRAN (R 4.4.0)
 workflowsets * 1.1.0      2024-03-21 [1] CRAN (R 4.4.0)
 xtable         1.8-4      2019-04-21 [1] CRAN (R 4.4.0)
 yardstick    * 1.3.1      2024-03-21 [1] CRAN (R 4.4.0)

 [1] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library

... the model_parts object does not plot/print properly.

After running this code:

HR_ranger_model_HR <- 
  ranger::ranger(status~., data = HR, num.trees = 50, probability = TRUE)
explainer_ranger_HR  <- 
  explain(
    HR_ranger_model_HR, data = HR[,-6], y = HR$status, 
    label = "Ranger HR", verbose = FALSE
  )
model_parts_ranger_HR <- model_parts(explainer_ranger_HR, type = "raw")

... it does.

> devtools::session_info()
─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.4.1 (2024-06-14)
 os       macOS Sonoma 14.6.1
 system   aarch64, darwin20
 ui       RStudio
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       America/New_York
 date     2024-08-18
 rstudio  2024.04.2+764 Chocolate Cosmos (desktop)
 pandoc   NA

─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────
 package      * version    date (UTC) lib source
 backports      1.5.0      2024-05-23 [1] CRAN (R 4.4.0)
 broom        * 1.0.6      2024-05-17 [1] CRAN (R 4.4.0)
 cachem         1.1.0      2024-05-16 [1] CRAN (R 4.4.0)
 class          7.3-22     2023-05-03 [1] CRAN (R 4.4.1)
 cli            3.6.3      2024-06-21 [1] CRAN (R 4.4.0)
 codetools      0.2-20     2024-03-31 [1] CRAN (R 4.4.1)
 colorspace     2.1-1      2024-07-26 [1] CRAN (R 4.4.0)
 DALEX        * 2.4.3      2023-01-15 [1] CRAN (R 4.4.0)
 DALEXtra     * 2.3.0      2023-05-26 [1] CRAN (R 4.4.0)
 data.table     1.15.4     2024-03-30 [1] CRAN (R 4.4.0)
 devtools     * 2.4.5      2022-10-11 [1] CRAN (R 4.4.0)
 dials        * 1.2.1      2024-02-22 [1] CRAN (R 4.4.0)
 DiceDesign     1.10       2023-12-07 [1] CRAN (R 4.4.0)
 digest         0.6.36     2024-06-23 [1] CRAN (R 4.4.0)
 dplyr        * 1.1.4      2023-11-17 [1] CRAN (R 4.4.0)
 ellipsis       0.3.2      2021-04-29 [1] CRAN (R 4.4.0)
 fansi          1.0.6      2023-12-08 [1] CRAN (R 4.4.0)
 farver         2.1.2      2024-05-13 [1] CRAN (R 4.4.0)
 fastmap        1.2.0      2024-05-15 [1] CRAN (R 4.4.0)
 forcats      * 1.0.0      2023-01-29 [1] CRAN (R 4.4.0)
 foreach        1.5.2      2022-02-02 [1] CRAN (R 4.4.0)
 fs             1.6.4      2024-04-25 [1] CRAN (R 4.4.0)
 furrr          0.3.1      2022-08-15 [1] CRAN (R 4.4.0)
 future         1.33.2     2024-03-26 [1] CRAN (R 4.4.0)
 future.apply   1.11.2     2024-03-28 [1] CRAN (R 4.4.0)
 generics       0.1.3      2022-07-05 [1] CRAN (R 4.4.0)
 ggplot2      * 3.5.1      2024-04-23 [1] CRAN (R 4.4.0)
 ggthemes       5.1.0      2024-02-10 [1] CRAN (R 4.4.0)
 globals        0.16.3     2024-03-08 [1] CRAN (R 4.4.0)
 glue           1.7.0      2024-01-09 [1] CRAN (R 4.4.0)
 gower          1.0.1      2022-12-22 [1] CRAN (R 4.4.0)
 GPfit          1.0-8      2019-02-08 [1] CRAN (R 4.4.0)
 gtable         0.3.5      2024-04-22 [1] CRAN (R 4.4.0)
 hardhat        1.4.0      2024-06-02 [1] CRAN (R 4.4.0)
 hms            1.1.3      2023-03-21 [1] CRAN (R 4.4.0)
 htmltools      0.5.8.1    2024-04-04 [1] CRAN (R 4.4.0)
 htmlwidgets    1.6.4      2023-12-06 [1] CRAN (R 4.4.0)
 httpuv         1.6.15     2024-03-26 [1] CRAN (R 4.4.0)
 infer        * 1.0.7      2024-03-25 [1] CRAN (R 4.4.0)
 ingredients    2.3.0      2023-01-15 [1] CRAN (R 4.4.0)
 ipred          0.9-15     2024-07-18 [1] CRAN (R 4.4.0)
 iterators      1.0.14     2022-02-05 [1] CRAN (R 4.4.0)
 labeling       0.4.3      2023-08-29 [1] CRAN (R 4.4.0)
 later          1.3.2      2023-12-06 [1] CRAN (R 4.4.0)
 lattice        0.22-6     2024-03-20 [1] CRAN (R 4.4.1)
 lava           1.8.0      2024-03-05 [1] CRAN (R 4.4.0)
 lhs            1.2.0      2024-06-30 [1] CRAN (R 4.4.0)
 lifecycle      1.0.4      2023-11-07 [1] CRAN (R 4.4.0)
 listenv        0.9.1      2024-01-29 [1] CRAN (R 4.4.0)
 lubridate    * 1.9.3      2023-09-27 [1] CRAN (R 4.4.0)
 magrittr       2.0.3      2022-03-30 [1] CRAN (R 4.4.0)
 MASS           7.3-61     2024-06-13 [1] CRAN (R 4.4.0)
 Matrix         1.7-0      2024-04-26 [1] CRAN (R 4.4.1)
 memoise        2.0.1      2021-11-26 [1] CRAN (R 4.4.0)
 mime           0.12       2021-09-28 [1] CRAN (R 4.4.0)
 miniUI         0.1.1.1    2018-05-18 [1] CRAN (R 4.4.0)
 modeldata    * 1.4.0      2024-06-19 [1] CRAN (R 4.4.0)
 munsell        0.5.1      2024-04-01 [1] CRAN (R 4.4.0)
 nnet           7.3-19     2023-05-03 [1] CRAN (R 4.4.1)
 parallelly     1.38.0     2024-07-27 [1] CRAN (R 4.4.0)
 parsnip      * 1.2.1      2024-03-22 [1] CRAN (R 4.4.0)
 pillar         1.9.0      2023-03-22 [1] CRAN (R 4.4.0)
 pkgbuild       1.4.4      2024-03-17 [1] CRAN (R 4.4.0)
 pkgconfig      2.0.3      2019-09-22 [1] CRAN (R 4.4.0)
 pkgload        1.4.0      2024-06-28 [1] CRAN (R 4.4.0)
 prodlim        2024.06.25 2024-06-24 [1] CRAN (R 4.4.0)
 profvis        0.3.8      2023-05-02 [1] CRAN (R 4.4.0)
 promises       1.3.0      2024-04-05 [1] CRAN (R 4.4.0)
 purrr        * 1.0.2      2023-08-10 [1] CRAN (R 4.4.0)
 R6             2.5.1      2021-08-19 [1] CRAN (R 4.4.0)
 ranger         0.16.0     2023-11-12 [1] CRAN (R 4.4.0)
 Rcpp           1.0.13     2024-07-17 [1] CRAN (R 4.4.0)
 readr        * 2.1.5      2024-01-10 [1] CRAN (R 4.4.0)
 recipes      * 1.1.0      2024-07-04 [1] CRAN (R 4.4.0)
 remotes        2.5.0      2024-03-17 [1] CRAN (R 4.4.0)
 reprex       * 2.1.1      2024-07-06 [1] CRAN (R 4.4.0)
 rlang          1.1.4      2024-06-04 [1] CRAN (R 4.4.0)
 rpart          4.1.23     2023-12-05 [1] CRAN (R 4.4.1)
 rsample      * 1.2.1      2024-03-25 [1] CRAN (R 4.4.0)
 rstudioapi     0.16.0     2024-03-24 [1] CRAN (R 4.4.0)
 scales       * 1.3.0      2023-11-28 [1] CRAN (R 4.4.0)
 sessioninfo    1.2.2      2021-12-06 [1] CRAN (R 4.4.0)
 shiny          1.8.1.1    2024-04-02 [1] CRAN (R 4.4.0)
 stringi        1.8.4      2024-05-06 [1] CRAN (R 4.4.0)
 stringr      * 1.5.1      2023-11-14 [1] CRAN (R 4.4.0)
 survival       3.7-0      2024-06-05 [1] CRAN (R 4.4.0)
 tibble       * 3.2.1      2023-03-20 [1] CRAN (R 4.4.0)
 tidymodels   * 1.2.0      2024-03-25 [1] CRAN (R 4.4.0)
 tidyr        * 1.3.1      2024-01-24 [1] CRAN (R 4.4.0)
 tidyselect     1.2.1      2024-03-11 [1] CRAN (R 4.4.0)
 tidyverse    * 2.0.0      2023-02-22 [1] CRAN (R 4.4.0)
 timechange     0.3.0      2024-01-18 [1] CRAN (R 4.4.0)
 timeDate       4032.109   2023-12-14 [1] CRAN (R 4.4.0)
 tune         * 1.2.1      2024-04-18 [1] CRAN (R 4.4.0)
 tzdb           0.4.0      2023-05-12 [1] CRAN (R 4.4.0)
 urlchecker     1.0.1      2021-11-30 [1] CRAN (R 4.4.0)
 usethis      * 2.2.3      2024-02-19 [1] CRAN (R 4.4.0)
 utf8           1.2.4      2023-10-22 [1] CRAN (R 4.4.0)
 vctrs          0.6.5      2023-12-01 [1] CRAN (R 4.4.0)
 withr          3.0.0      2024-01-16 [1] CRAN (R 4.4.0)
 workflows    * 1.1.4      2024-02-19 [1] CRAN (R 4.4.0)
 workflowsets * 1.1.0      2024-03-21 [1] CRAN (R 4.4.0)
 xtable         1.8-4      2019-04-21 [1] CRAN (R 4.4.0)
 yardstick    * 1.3.1      2024-03-21 [1] CRAN (R 4.4.0)

 [1] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library

I compared the two session_info() reports and the difference is that after running the code these things are listed:

 farver         2.1.2      2024-05-13 [1] CRAN (R 4.4.0)
 ingredients    2.3.0      2023-01-15 [1] CRAN (R 4.4.0)
 labeling       0.4.3      2023-08-29 [1] CRAN (R 4.4.0)

I loaded those three packages:

library(tidyverse)
library(tidymodels)
library(DALEX)
library(DALEXtra)
library(farver)
library(ingredients)
library(labeling)

... and it fixed the problem!

So I am set (and very happy).

I have never run into an issue like this with the packages I have built. So, I am very curious to hear what went wrong and how it can be fixed. Do you mind explaining the issue?

Again, thank you so much for your wonderful work. You make my job possible.

@pbiecek
Copy link
Member

pbiecek commented Aug 18, 2024

Excellent, glad it works.

@pbiecek pbiecek closed this as completed Aug 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pbiecek @RaymondBalise