fix(expr-common): Coerce to Decimal(20, 0) when combining UInt64 with signed integers #14223
Conversation
… signed integers Previously, when combining UInt64 with any signed integer, the resulting type would be Int64, which would result in lost information. Now, combining UInt64 with a signed integer results in a Decimal(20, 0), which is able to encode all (64-bit) integer types.
| // accommodates all values of both types. Note that to avoid information | ||
| // loss when combining UInt64 with signed integers we use Decimal128(20, 0). | ||
| (Decimal128(20, 0), _) | ||
| | (_, Decimal128(20, 0)) |
There was a problem hiding this comment.
This looks not correct. For example, combining Decimal128(20, 0) with Decimal128(30, 0) should not result in Decimal128(20, 0)
There was a problem hiding this comment.
I think when both types are decimal they are handled above before this match, when calling the decimal_coercion function.
There was a problem hiding this comment.
I rechecked it, and decimal_coercion already covers them in
datafusion/datafusion/expr-common/src/type_coercion/binary.rs
Lines 932 to 940 in 2f28327
Although it doesn't handle unsigned integer types, we can supplement it there, maybe as a follow-up PR.
There was a problem hiding this comment.
So I think we shouldn't combine decimal with integer types here because decimal_coercion has already handled it.
There was a problem hiding this comment.
I tried that initially but then it would not handle UInt64 and Decimal128(20, 0):
Cannot infer common argument type for comparison operation UInt64 = Decimal128(20, 0)
So maybe it would be best to add new arms to the coerce_numeric_type_to_decimal to include unsigned integers as well?
match numeric_type {
Int8 => Some(Decimal128(3, 0)),
Int16 => Some(Decimal128(5, 0)),
Int32 => Some(Decimal128(10, 0)),
Int64 => Some(Decimal128(20, 0)),
Float32 => Some(Decimal128(14, 7)),
Float64 => Some(Decimal128(30, 15)),
_ => None,
}There was a problem hiding this comment.
So maybe it would be best to add new arms to the
coerce_numeric_type_to_decimalto include unsigned integers as well?
I think so.
|
Perhaps we should add a sqllogictest test for #14208. |
Done. |
| | (UInt32, Int8) | ||
| | (Int8, UInt32) | ||
| | (UInt32, Int16) | ||
| | (Int16, UInt32) | ||
| | (UInt32, Int32) | ||
| | (Int32, UInt32) => Some(Int64), | ||
| (UInt64, _) | (_, UInt64) => Some(UInt64), |
There was a problem hiding this comment.
We need to keep this arm because it can coerce UInt64 vs UInt8/16/32. The following query returns an incorrect result on this PR.
DataFusion CLI v44.0.0
> select arrow_typeof(column1) from values(arrow_cast(1, 'UInt8')), (arrow_cast(1, 'UInt64'));
+-----------------------+
| arrow_typeof(column1) |
+-----------------------+
| UInt8 |
| UInt8 |
+-----------------------+
2 row(s) fetched.
Elapsed 0.007 seconds.
> select arrow_typeof(column1) from values(arrow_cast(1, 'UInt8')), (arrow_cast(1000, 'UInt64'));
Arrow error: Cast error: Can't cast value 1000 to type UInt8There was a problem hiding this comment.
Done. I also added a new test to check that.
Previously, when combining
UInt64with any signed integer, the resulting type would beInt64, which would result in lost information. Now, combiningUInt64with a signed integer results in aDecimal(20, 0), which is able to encode all (64-bit) integer types. Thanks @jonahgao for the pointers.The function
bitwise_coercionremains the same, since it's probably not a good idea to introduce decimals when performing bitwise operations. In this case, it converts(UInt64 | _)toUInt64.Which issue does this PR close?
Closes #14208.
What changes are included in this PR?
binary_numeric_coercioninexpr-common/type_coercion/binary.rs.expr-common/type_coercion/binary.rs.Are these changes tested?
Yes.
Are there any user-facing changes?
No.