Use Rf_charIsASCII for IS_ASCII instead of testing LEVELS on R >= 4.5 #6422
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Since data.table now depends on R >= 3.3, the backports are no longer needed. Moreover, MAYBE_SHARED is currently a function, while MAYBE_REFERENCED expands to !NO_REFERENCES (which is a function). In debugging output, show MAYBE_REFERENCED (NAMED > 0) instead of NAMED.
getCharCE appeared in R-2.7, making it possible to check for strings _marked_ as UTF-8 or Latin-1. There is no marking as ASCII, so fixing IS_ASCII will have to wait for R >= 4.5.
There's no explicit encoding code for ASCII, so use charIsASCII()
("eapi", expected to appear in R-4.5.0).
|
Generated via commit b409a40 Download link for the artifact containing the test results: ↓ atime-results.zip Time taken to finish the standard R installation steps: 11 minutes and 31 seconds Time taken to run |
Don't mind that, we maintainers will take care of it :) |
|
set #6420 as the target of this PR to make the chain clearer |
|
Good catch! It's a bit worse: LEVELS(x) & 64 returns 0 or 64.
Here are the only users of IS_ASCII:
data.table.h:
#define NEED2UTF8(s) !(IS_ASCII(s) || (s)==NA_STRING || IS_UTF8(s))
fwriteR.c:
#define TO_NATIVE(s) (native && (s)!=NA_STRING && !IS_ASCII(s))
forder.c:
if (!anynotutf8 && /*...*/ !IS_ASCII(s)) {
They all use the result of the macro in boolean context (if / || / &&),
so any non-zero result is interpreted as true.
|
MichaelChirico
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This part is split from #6420 (but includes #6420; can rebase if needed) because the R version containing
Rf_isCharASCIIis not currently released. The overall issue is #6180. Not yet sure how to changeNEWS.mdin case we're expectingdata.tablereleases in the meantime.