Skip to content

Commit 30d0b01

Browse files
krm95MaxGekk
authored andcommitted
[SPARK-50364][SQL] Implement serialization for LocalDateTime type in Row.jsonValue
### What changes were proposed in this pull request? In this Pull request, I propose to add a LocalDateTime serializer to the Row.jsonValue method ```scala case (d: LocalDateTime, _) => JString(timestampFormatter.format(d)) ``` In order to enable JSON serialization of _TimestampNTZType_ columns ### Why are the changes needed? Currently trying to serialize a Row containing a _TimestampNTZType_ column results in an error: ```java [FAILED_ROW_TO_JSON] Failed to convert the row value '2018-05-14T12:13' of the class class java.time.LocalDateTime to the target SQL type "TIMESTAMPNTZTYPE" in the JSON format. SQLSTATE: 2203G org.apache.spark.SparkIllegalArgumentException: [FAILED_ROW_TO_JSON] Failed to convert the row value '2018-05-14T12:13' of the class class java.time.LocalDateTime to the target SQL type "TIMESTAMPNTZTYPE" in the JSON format. SQLSTATE: 2203G at org.apache.spark.sql.Row.toJson$1(Row.scala:663) at org.apache.spark.sql.Row.toJson$1(Row.scala:651) at org.apache.spark.sql.Row.jsonValue(Row.scala:665) at org.apache.spark.sql.Row.jsonValue$(Row.scala:598) at org.apache.spark.sql.catalyst.expressions.GenericRow.jsonValue(rows.scala:28) at org.apache.spark.sql.RowJsonSuite.$anonfun$testJson$1(RowJsonSuite.scala:41) ``` How to reproduce the issue: ```scala import org.apache.spark.sql.Row import java.time.LocalDateTime val r = Row.fromSeq(LocalDateTime.of(2024,8,10,12,33) :: Nil) r.json r.prettyJson ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tests were added to the already existing RowJsonSuite.scala class ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#48905 from karim-ramadan/bugfix/LocalDateTime-support-in-Row-json. Authored-by: Karim Ramadan <[email protected]> Signed-off-by: Max Gekk <[email protected]>
1 parent 6ee53da commit 30d0b01

File tree

2 files changed

+4
-2
lines changed

2 files changed

+4
-2
lines changed

sql/api/src/main/scala/org/apache/spark/sql/Row.scala

+2-1
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
package org.apache.spark.sql
1919

2020
import java.sql.{Date, Timestamp}
21-
import java.time.{Instant, LocalDate}
21+
import java.time.{Instant, LocalDate, LocalDateTime}
2222
import java.util.Base64
2323

2424
import scala.collection.mutable
@@ -627,6 +627,7 @@ trait Row extends Serializable {
627627
case (d: Date, _) => JString(dateFormatter.format(d))
628628
case (i: Instant, _) => JString(timestampFormatter.format(i))
629629
case (t: Timestamp, _) => JString(timestampFormatter.format(t))
630+
case (d: LocalDateTime, _) => JString(timestampFormatter.format(d))
630631
case (i: CalendarInterval, _) => JString(i.toString)
631632
case (a: Array[_], ArrayType(elementType, _)) =>
632633
iteratorToJsonArray(a.iterator, elementType)

sql/catalyst/src/test/scala/org/apache/spark/sql/RowJsonSuite.scala

+2-1
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
package org.apache.spark.sql
1818

1919
import java.sql.{Date, Timestamp}
20-
import java.time.LocalDate
20+
import java.time.{LocalDate, LocalDateTime}
2121

2222
import org.json4s.JsonAST.{JArray, JBool, JDecimal, JDouble, JLong, JNull, JObject, JString, JValue}
2323

@@ -84,6 +84,7 @@ class RowJsonSuite extends SparkFunSuite {
8484
Timestamp.valueOf("2017-05-30 10:22:03.00").toInstant,
8585
TimestampType,
8686
JString("2017-05-30 10:22:03"))
87+
testJson(LocalDateTime.of(2018, 5, 14, 12, 13), TimestampNTZType, JString("2018-05-14 12:13:00"))
8788

8889
// Complex types
8990
testJson(

0 commit comments

Comments
 (0)