[python] Update ASCII storage for dataframes #273
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Status note: the CI failure is puzzling me as unit tests are passing for me locally 👀 |
johnkerl
force-pushed
the
kerl/ascii
branch
2 times, most recently
from
3edca86
to
ad10f3e
Compare
johnkerl
force-pushed
the
kerl/ascii
branch
from
ad10f3e
to
25927cc
Compare
johnkerl
force-pushed
the
kerl/ascii
branch
3 times, most recently
from
1b487b5
to
f3a0d62
Compare
|
Sorry for the delay. @Shelnutt2 @nguyenv this is ready for review. Failure of Windows |
johnkerl
force-pushed
the
kerl/ascii
branch
9 times, most recently
from
a75f33e
to
4da886a
Compare
|
@nguyenv @ihnorton @Shelnutt2 ping 🙏 |
nguyenv
reviewed
Sep 19, 2022
Comment on lines
+78
to
+90
|
|
||
| # TileDB string dims are ASCII not UTF-8. Decode them so they readback not like | ||
| # `b"AKR1C3"` but rather like `"AKR1C3"`. Update as of | ||
| # https://github.com/TileDB-Inc/TileDB-Py/pull/1304 these dims will read back OK. | ||
| retval = A.query(attrs=[], dims=[self.dim_name])[:][self.dim_name].tolist() | ||
| return [e.decode() for e in retval] | ||
|
|
||
| retval = [e.decode() for e in retval] | ||
|
|
||
| if len(retval) > 0 and isinstance(retval[0], bytes): | ||
| return [e.decode() for e in retval] | ||
| else: | ||
| # list(...) is there to appease the linter which thinks we're returning `Any` | ||
| return list(retval) |
There was a problem hiding this comment.
Could something like this work?
retval = A.query(attrs=[], dims=[self.dim_name])[:][self.dim_name]
return np.frombuffer(retval, dtype="U").tolist()
There was a problem hiding this comment.
no sorry @nguyenv !
return np.frombuffer(retval, dtype="U").tolist()
ValueError: itemsize cannot be zero in type
There was a problem hiding this comment.
But regardless all I'm doing is putting an if around known-good code, invoking it when needed for older arrays which predate this PR!
There was a problem hiding this comment.
@nguyenv what works remains to accept or reject this PR?
johnkerl
force-pushed
the
kerl/ascii
branch
from
4da886a
to
eda0866
Compare
nguyenv
approved these changes
Sep 23, 2022
|
Thanks @nguyenv ! :) |
This was referenced Oct 2, 2022
johnkerl
changed the title
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On collaboration with @nguyenv on TileDB-Inc/TileDB-Py#1304 -- new/upated feedback (thanks @nguyenv !) is that all along we can obtain reduce type-conversion at readback by a one-line change in this package.
Note this means that while before this PR we can store (encode/decode) user-supplied Unicode data in dataframes, as of this PR we no longer can. That ability will be restored in a future core release -- perhaps early 2023. Meanwhile unit tests on this PR that used to validate the ability to encode/decode user-supplied Unicode data in dataframes have been commented out, to accommodate the feature loss.
Summary:
"ascii"per se, notbytesand notstr"ascii", conditionally retain the decode-on-readback logic which is essential since otherwiseb"foo" != "foo"