Changes

Summary

  1. [SPARK-33272][SQL] prune the attributes mapping in (commit: 2639ad4) (details)
  2. [SQL][MINOR] Update from_unixtime doc (commit: c592ae6) (details)
  3. [SPARK-33270][SQL] Return SQL schema instead of Catalog string from the (commit: 9d5e48e) (details)
Commit 2639ad43cb8357db235e7fc9ce24930cca7f2525 by yamamuro
[SPARK-33272][SQL] prune the attributes mapping in
QueryPlan.transformUpWithNewOutput
### What changes were proposed in this pull request?
For complex query plans, `QueryPlan.transformUpWithNewOutput` will keep
accumulating the attributes mapping to be propagated, which may hurt
performance. This PR prunes the attributes mapping before propagating.
### Why are the changes needed?
A simple perf improvement.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
existing tests
Closes #30173 from cloud-fan/bug.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Takeshi
Yamamuro <yamamuro@apache.org>
(commit: 2639ad4)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/AnalysisHelper.scala (diff)
Commit c592ae6ed81cf381512749c43ed688411ef1b431 by gurwls223
[SQL][MINOR] Update from_unixtime doc
### What changes were proposed in this pull request? This PR fixes
from_unixtime documentation to show that fmt is optional parameter.
### Does this PR introduce _any_ user-facing change? Yes, documentation
update.
**Before change:**
![image](https://user-images.githubusercontent.com/4176173/97497659-18c6cc80-1928-11eb-93d8-453ef627ac7c.png)
**After change:**
![image](https://user-images.githubusercontent.com/4176173/97496153-c5537f00-1925-11eb-8102-457e85e019d5.png)
### How was this patch tested? Style check using: ./dev/run-tests Manual
check and screenshotting with: ./sql/create-docs.sh Manual verification
of behavior with latest spark-sql binary.
Closes #30176 from Obbay2/from_unixtime_doc.
Authored-by: Nathan Wreggit <obbay2@hotmail.com> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
(commit: c592ae6)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala (diff)
Commit 9d5e48ea95d1c3017a51ff69584f32a18901b2b5 by gurwls223
[SPARK-33270][SQL] Return SQL schema instead of Catalog string from the
`SchemaOfJson` expression
### What changes were proposed in this pull request? Return schema in
SQL format instead of Catalog string from the `SchemaOfJson` expression.
### Why are the changes needed? In some cases, `from_json()` cannot
parse schemas returned by `schema_of_json`, for instance, when JSON
fields have spaces (gaps). Such fields will be quoted after the changes,
and can be parsed by `from_json()`.
Here is the example:
```scala val in = Seq("""{"a b": 1}""").toDS()
in.select(from_json('value, schema_of_json("""{"a b": 100}""")) as
"parsed")
``` raises the exception:
```
== SQL == struct<a b:bigint>
------^^^
at
org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:263)
at
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:130)
at
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseTableSchema(ParseDriver.scala:76)
at org.apache.spark.sql.types.DataType$.fromDDL(DataType.scala:131)
at
org.apache.spark.sql.catalyst.expressions.ExprUtils$.evalTypeExpr(ExprUtils.scala:33)
at
org.apache.spark.sql.catalyst.expressions.JsonToStructs.<init>(jsonExpressions.scala:537)
at org.apache.spark.sql.functions$.from_json(functions.scala:4141)
```
### Does this PR introduce _any_ user-facing change? Yes. For example,
`schema_of_json` for the input `{"col":0}`.
Before: `struct<col:bigint>` After: `STRUCT<`col`: BIGINT>`
### How was this patch tested? By existing test suites
`JsonFunctionsSuite` and `JsonExpressionsSuite`.
Closes #30172 from MaxGekk/schema_of_json-sql-schema.
Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(commit: 9d5e48e)
The file was modifieddocs/sql-migration-guide.md (diff)
The file was modifiedpython/pyspark/sql/functions.py (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/json-functions.sql.out (diff)
The file was modifiedR/pkg/tests/fulltests/test_sparkSQL.R (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala (diff)