Inconsistent replacement of list column element with NULL in table with 1 row

[tdhock]

In base R data.frame we can replace an element of a list column with NULL via:

> DF1=data.frame(L=I(list("A")),i=1)
> DF1$L=list(NULL)
> DF1
     L i
1 NULL 1

However in data.table, doing that results in deleting the list column entirely:

> DT1=data.table(L=list("A"),i=1)
> DT1$L=list(NULL)
> DT1
       i
   <num>
1:     1

Request: can we make the above code do a replacement (like base R data.frame) instead of deleting the column?

The above issue is confusing because it is inconsistent with how data table handles replacement in tables with more than one row:

> DT2=data.table(L=list("B","C"),i=1)
> DT2$L <- list(NULL,NULL)
> DT2
        L     i
   <list> <num>
1:            1
2:            1
> DF2=data.frame(L=I(list("B","C")),i=1)
> DF2$L <- list(NULL,NULL)
> DF2
     L i
1 NULL 1
2 NULL 1

Actually, we can do the replacement (inconsistently) in data.table via

> DT1=data.table(L=list("A"),i=1)
> DT1$L=list(list(NULL))
> DT1
        L     i
   <list> <num>
1:            1

[avimallu]

I can do this:

library(data.table)
DT1=data.table(L=list("A"),i=1)
DT1[, `:=`(L = list(NULL))]

that works, but oddly not

DT1[, L := list(NULL)]

which should be identical per data.table documentation.

[MichaelChirico]

The fix here, #6167, caused a regression: #6740 / markfairbanks/tidytable#831.

In line with the GOVERNANCE rules I think we should revert #6167, at least for now and at least by default:

data.table/GOVERNANCE.md

Line 17 in 3a7ec2d

* Stable code base (strong preference for user-friendly back-compatibility with data.table itself and with old versions of R)

If we decide the new approach landed on in #6167 is preferable, or that it unlocks a behavior that was previously impossible, we should re-introduce #6167 gradually with a deprecation cycle. Here's my proposal:

1. 1.17.0: Leave Consistent Replacement of List Column with NULL #6167, but hide it behind if (!isTRUE(getOption("datatable.old.list.assignment", FALSE))) { ... }, document option and intention to change defaults from the next release
2. 1.18.0: Change option default to TRUE
3. 1.19.0: Throw a warning for FALSE option
4. 1.20.0: Throw an error for FALSE option
5. 1.21.0: Ignore option, throw warning if it is set
6. 1.22.0: Ignore option, throw error if it is set
7. 1.23.0: Remove option totally

[tdhock]

related: #6732

[tdhock]

sure let's revert.
and reopen this issue.

[MichaelChirico]

sure let's revert.

FYI, I'll plan to do a cherry-pick style revert instead of a simple clean one -- I think the test suite in the PR is really great, and of utility even if the exact behavior tested needs to change slightly.