Changes

Summary

  1. [SPARK-32791][SQL] Non-partitioned table metric should not have dynamic (commit: 0b3bb45) (details)
  2. [SPARK-32800][SQL] Remove ExpressionSet from the 2.13 branch (commit: f556946) (details)
  3. [SPARK-32548][SQL] - Add Application attemptId support to SQL Rest API (commit: f5360e7) (details)
Commit 0b3bb45b893005d73afb15e4904ac6440c87811c by yumwang
[SPARK-32791][SQL] Non-partitioned table metric should not have dynamic
partition pruning time
### What changes were proposed in this pull request?
This pr make non-partitioned table metric should not have dynamic
partition pruning time.
### Why are the changes needed?
It is useless for non-partitioned table.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test
Before this pr:
![image](https://user-images.githubusercontent.com/5399861/92141803-87fed380-ee45-11ea-9784-09625b246fea.png)
After this pr:
![image](https://user-images.githubusercontent.com/5399861/92141774-7c131180-ee45-11ea-8a9e-6775c592f496.png)
Closes #29641 from wangyum/SPARK-32791.
Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Yuming Wang
<yumwang@ebay.com>
(commit: 0b3bb45)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala (diff)
Commit f55694638d45f34ab91f6f6ec2066cbf7631f4af by gurwls223
[SPARK-32800][SQL] Remove ExpressionSet from the 2.13 branch
### What changes were proposed in this pull request? This PR is a
followup on #29598 and removes the `ExpressionSet` class from the 2.13
branch.
### Why are the changes needed?
`ExpressionSet` does not extend Scala `Set` anymore and this class is no
longer needed in the 2.13 branch.
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? Passes existing tests
Closes #29648 from dbaliafroozeh/RemoveExpressionSetFrom2.13Branch.
Authored-by: Ali Afroozeh <ali.afroozeh@databricks.com> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
(commit: f556946)
The file was addedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExpressionSet.scala
The file was removedsql/catalyst/src/main/scala-2.13/org/apache/spark/sql/catalyst/expressions/ExpressionSet.scala
The file was removedsql/catalyst/src/main/scala-2.12/org/apache/spark/sql/catalyst/expressions/ExpressionSet.scala
Commit f5360e761ef161f7e04526b59a4baf53f1cf8cd5 by gengliang.wang
[SPARK-32548][SQL] - Add Application attemptId support to SQL Rest API
### What changes were proposed in this pull request? Currently, Spark
Public Rest APIs support Application attemptId except SQL API. This
causes `no such app: application_X` issue when the application has
`attemptId` (e.g: YARN cluster mode).
Please find existing and supported Rest endpoints with attemptId.
```
// Existing Rest Endpoints applications/{appId}/sql
applications/{appId}/sql/{executionId}
// Rest Endpoints required support applications/{appId}/{attemptId}/sql
applications/{appId}/{attemptId}/sql/{executionId}
``` Also fixing following compile warning on `SqlResourceSuite`:
```
[WARNING] [Warn]
~/spark/sql/core/src/test/scala/org/apache/spark/status/api/v1/sql/SqlResourceSuite.scala:67:
Reference to uninitialized value edges
```
### Why are the changes needed? This causes `no such app: application_X`
issue when the application has `attemptId`.
### Does this PR introduce _any_ user-facing change? Not yet because SQL
Rest API is being planned to release with `Spark 3.1`.
### How was this patch tested? 1. New Unit tests are added for existing
Rest endpoints. `attemptId` seems not coming in `local-mode` and coming
in `YARN cluster mode` so could not be added for `attemptId` case
(Suggestions are welcome). 2. Also, patch has been tested manually
through both Spark Core and History Server Rest APIs.
Closes #29364 from erenavsarogullari/SPARK-32548.
Authored-by: Eren Avsarogullari <erenavsarogullari@gmail.com>
Signed-off-by: Gengliang Wang <gengliang.wang@databricks.com>
(commit: f5360e7)
The file was modifiedsql/core/src/main/scala/org/apache/spark/status/api/v1/sql/ApiSqlRootResource.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/status/api/v1/sql/SqlResourceSuite.scala (diff)
The file was addedsql/core/src/test/scala/org/apache/spark/status/api/v1/sql/SqlResourceWithActualMetricsSuite.scala