setDF should drop indices

[OfekShilon]

ff2.zip

Attached is a small zip archive of a saved data.table that demonstrates a problem where sometimes the [ operator gives misleading results.

load("./ff2")   # from the path you extracted the archive to
dim(ff[symbol == "ZM" & date == 20210128])
#  [1] 0 2     <---- wrong results, this subset is not empty

One way to show the correct result:

> dim(ff[symbol == "ZM"][date == 20210128])
# [1] 98  2

Many seemingly no-ops fix the data.table state. One example:

ff[, symbol:=symbol]
dim(ff[symbol == "ZM" & date == 20210128])
# [1] 98  2

Tested on both Windows and Ubuntu, and data.table versions 1.13.6 and 1.11.8 correspondingly. (the sessionInfo below is of the windows machine)

# Output of sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19042)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] data.table_1.13.6

loaded via a namespace (and not attached):
[1] bit_4.0.4 compiler_3.6.1 tools_3.6.1 bit64_4.0.5

[tdeenes]

I think the problem is with setDF and not with the assignment. From the manual of setDF:

All data.table attributes including any keys of the input data.table are stripped off.

The correct behaviour (and the adjusted doc) should read as:

All data.table attributes including any keys and indices of the input data.table are stripped off.

d <- data.table(x = 1)
setindex(d, "x")
indices(d)
# [1] "x"

setDF(d)
indices(d)
# [1] "x" -> should have been stripped off

[OfekShilon]

I agree with @tdeenes, raised this in the 2nd part of my comment above:

(2)
setDF clears a few DT-only attributes, but not index. That's very risky:

dt <- data.table(***)
# ... do stuff that creates indices
setDF(dt)
# ... do stuff that changes data, breaking the index
setDT(dt)
#  index attribute is still there, but the data doesn't match.

The 1st half still worries me too, but it might be a different bug.

Also @jangorecki note that the update that uses <- is performed on a data.frame, not data.table.

[OfekShilon]

Self contained repro (without loading anything, suitable for unit tests):

d <- data.table(a=1:100, b=1:100)
invisible(d[a == 50])   

setDF(d)
d[1:50, "a"] <- d[51:100, "a"]
setDT(d)

print(d[a == 99])    # one line, wrong result

# manual fix:
setindex(d,NULL)
print(d[a == 99])    # two lines, right result

[OfekShilon]

I created a single line PR for this