Changes

Summary

  1. [SPARK-33306][SQL] Timezone is needed when cast date to string (commit: 69c27f4) (details)
Commit 69c27f49acf2fe6fbc8335bde2aac4afd4188678 by dhyun
[SPARK-33306][SQL] Timezone is needed when cast date to string
### What changes were proposed in this pull request? When
`spark.sql.legacy.typeCoercion.datetimeToString.enabled` is enabled,
spark will cast date to string when compare date with string. In Spark3,
timezone is needed when casting date to string as
https://github.com/apache/spark/blob/72ad9dcd5d484a8dd64c08889de85ef9de2a6077/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L309.
Howerver, the timezone may not be set because `CastBase.needsTimeZone`
returns false for this kind of casting.
A simple way to reproduce this is
``` spark-shell --conf
spark.sql.legacy.typeCoercion.datetimeToString.enabled=true
``` when we execute the following sql,
``` select a.d1 from
(select to_date(concat('2000-01-0', id)) as d1 from range(1, 2)) a join
(select concat('2000-01-0', id) as d2 from range(1, 2)) b on a.d1 = b.d2
``` it will throw
``` java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:529)
at scala.None$.get(Option.scala:527)
at
org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId(datetimeExpressions.scala:56)
at
org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId$(datetimeExpressions.scala:56)
at
org.apache.spark.sql.catalyst.expressions.CastBase.zoneId$lzycompute(Cast.scala:253)
at
org.apache.spark.sql.catalyst.expressions.CastBase.zoneId(Cast.scala:253)
at
org.apache.spark.sql.catalyst.expressions.CastBase.dateFormatter$lzycompute(Cast.scala:287)
at
org.apache.spark.sql.catalyst.expressions.CastBase.dateFormatter(Cast.scala:287)
```
### Why are the changes needed? As described above, it's a bug here.
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? Add more UT
Closes #30213 from WangGuangxin/SPARK-33306.
Authored-by: wangguangxin.cn <wangguangxin.cn@gmail.com> Signed-off-by:
Dongjoon Hyun <dhyun@apple.com>
(commit: 69c27f4)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala (diff)