filter(.by = ) leads to incorrect results when NAs are present

[markfairbanks]

library(dtplyr)
library(dplyr)

df <- tibble(x = c(1, 2, NA), y = c("a", "a", "b"))

lazy_dt(df) %>%
  filter(x != 2, .by = y)
#> Source: local data table [2 x 2]
#> Call:   `_DT1`[`_DT1`[, .I[x != 2], by = .(y)]$V1]
#> 
#>       x y    
#>   <dbl> <chr>
#> 1     1 a    
#> 2    NA <NA> 
#> 
#> # Use as.data.table()/as.data.frame()/as_tibble() to access results

df %>%
  filter(x != 2, .by = y)
#> # A tibble: 1 × 2
#>       x y    
#>   <dbl> <chr>
#> 1     1 a

[markfairbanks]

Looks like this is happening because we use the .I trick when filtering by group. .I returns a vector with NAs, and slicing using NAs causes you to make empty rows in data.table.

library(dtplyr)
library(dplyr)

example_df <- data.table(id = c(1, 1, 1, 1), value = c(NA, NA, 0, 1))

example_df %>%
  lazy_dt() %>%
  filter(value == 0, .by = id)
#> Source: local data table [3 x 2]
#> Call:   `_DT1`[`_DT1`[, .I[value == 0], by = .(id)]$V1]
#> 
#>      id value
#>   <dbl> <dbl>
#> 1    NA    NA
#> 2    NA    NA
#> 3     1     0
#> 
#> # Use as.data.table()/as.data.frame()/as_tibble() to access results

example_df[c(NA, NA, 3)]
#>       id value
#>    <num> <num>
#> 1:    NA    NA
#> 2:    NA    NA
#> 3:     1     0