Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Tools should be able to recognize dataType aliases #1492

Closed
amahussein opened this issue Jan 9, 2025 · 0 comments · Fixed by #1506
Closed

[BUG] Tools should be able to recognize dataType aliases #1492

amahussein opened this issue Jan 9, 2025 · 0 comments · Fixed by #1506
Assignees
@amahussein
Labels
bug Something isn't working core_tools Scope the core module (scala)

Comments

@amahussein
Copy link
Collaborator

Describe the bug

The SQL spark define aliases for some datatypes. The tools should be able to recognize the mapping in order to accurately get accurate estimation

Data Type Aliases
Numeric Types
ByteType TINYINT
ShortType SMALLINT
IntegerType INT, INTEGER
LongType BIGINT
FloatType FLOAT, REAL
DoubleType DOUBLE
DecimalType DECIMAL, NUMERIC
String and Binary
StringType STRING, VARCHAR, CHAR
BinaryType BINARY
Date and Time
DateType DATE
TimestampType TIMESTAMP
TimestampNTZType TIMESTAMP WITHOUT TIME ZONE
Boolean
BooleanType BOOLEAN
Complex Types
ArrayType ARRAY
MapType MAP
StructType STRUCT
NullType NULL
Other Types
BinaryType BLOB
IntervalType INTERVAL
@amahussein amahussein added ? - Needs Triage bug Something isn't working core_tools Scope the core module (scala) labels Jan 9, 2025
@amahussein amahussein mentioned this issue Jan 9, 2025
Merged
@amahussein amahussein self-assigned this Jan 22, 2025
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Jan 22, 2025
@amahussein
Add string and binary to the list of dataType map
ebaf7c4 
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Fixes NVIDIA#1492

Adds 2 new entries to the dataTypes map defined in the PluginTypeChecker
- string: varchar, car
- binary: blob
@amahussein amahussein mentioned this issue Jan 22, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amahussein amahussein

Labels
bug Something isn't working core_tools Scope the core module (scala)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add stringType and binaryType to the list of dataType map amahussein/spark-rapids-tools
1 participant
@amahussein