Changes

Summary

  1. [SPARK-32919][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Driver side changes (commit: 8218b48) (details)
  2. [SPARK-33441][BUILD][FOLLOWUP] Make unused-imports check for SBT (commit: 2289389) (details)
  3. [SPARK-32512][SQL][TESTS][FOLLOWUP] Remove duplicate tests for ALTER (commit: 870d409) (details)
  4. [SPARK-33422][DOC] Fix the correct display of left menu item (commit: cbc8be2) (details)
  5. [SPARK-33468][SQL] ParseUrl  in ANSI mode should fail if input string is (commit: 3384bda) (details)
Commit 8218b488035049434271dc9e3bd5af45ffadf0fd by mridulatgmail.com
[SPARK-32919][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Driver side changes
for coordinating push based shuffle by selecting external shuffle
services for merging partitions
### What changes were proposed in this pull request? Driver side changes
for coordinating push based shuffle by selecting external shuffle
services for merging partitions.
This PR includes changes related to `ShuffleMapStage` preparation which
is selection of merger locations and initializing them as part of
`ShuffleDependency`.
Currently this code is not used as some of the changes would come
subsequently as part of
https://issues.apache.org/jira/browse/SPARK-32917 (shuffle blocks push
as part of `ShuffleMapTask`),
https://issues.apache.org/jira/browse/SPARK-32918 (support for finalize
API) and https://issues.apache.org/jira/browse/SPARK-32920 (finalization
of push/merge phase). This is why the tests here are also partial, once
these above mentioned changes are raised as PR we will have enough tests
for DAGScheduler piece of code as well.
### Why are the changes needed? Added a new API in `SchedulerBackend` to
get merger locations for push based shuffle. This is currently
implemented for Yarn and other cluster managers can have separate
implementations which is why a new API is introduced.
### Does this PR introduce _any_ user-facing change? Yes, user facing
config to enable push based shuffle is introduced
### How was this patch tested? Added unit tests partially and some of
the changes in DAGScheduler depends on future changes, DAGScheduler
tests will be added along with those changes.
Lead-authored-by: Venkata krishnan Sowrirajan vsowrirajanlinkedin.com
Co-authored-by: Min Shen mshenlinkedin.com
Closes #30164 from venkata91/upstream-SPARK-32919.
Lead-authored-by: Venkata krishnan Sowrirajan <vsowrirajan@linkedin.com>
Co-authored-by: Min Shen <mshen@linkedin.com> Signed-off-by: Mridul
Muralidharan <mridul<at>gmail.com>
(commit: 8218b48)
The file was modifiedcore/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/storage/BlockManagerId.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/internal/config/package.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala (diff)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/Dependency.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/util/UtilsSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/util/Utils.scala (diff)
Commit 2289389821a23e5b5badabfb4e62c427de2554a5 by gurwls223
[SPARK-33441][BUILD][FOLLOWUP] Make unused-imports check for SBT
specific
### What changes were proposed in this pull request? Move
"unused-imports" check config to `SparkBuild.scala` and make it SBT
specific.
### Why are the changes needed? Make unused-imports check for SBT
specific.
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? Pass the Jenkins or GitHub Action
Closes #30441 from LuciferYang/SPARK-33441-FOLLOWUP.
Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(commit: 2289389)
The file was modifiedpom.xml (diff)
The file was modifiedproject/SparkBuild.scala (diff)
Commit 870d4095336f29f5bef77b9232d6cb9d025987dd by wenchen
[SPARK-32512][SQL][TESTS][FOLLOWUP] Remove duplicate tests for ALTER
TABLE .. PARTITIONS from DataSourceV2SQLSuite
### What changes were proposed in this pull request? Remove tests from
`DataSourceV2SQLSuite` that were copied to
`AlterTablePartitionV2SQLSuite` by
https://github.com/apache/spark/pull/29339.
### Why are the changes needed?
- To reduce tests execution time
- To improve test maintenance
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? By running the modified tests:
```
$ build/sbt "test:testOnly *DataSourceV2SQLSuite"
$ build/sbt "test:testOnly *AlterTablePartitionV2SQLSuite"
```
Closes #30444 from MaxGekk/dedup-tests-AlterTablePartitionV2SQLSuite.
Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(commit: 870d409)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala (diff)
Commit cbc8be24c896ed25be63ef9a111ff015af4fabec by gurwls223
[SPARK-33422][DOC] Fix the correct display of left menu item
### What changes were proposed in this pull request? Limit the height of
the menu area on the left to display vertical scroll bar
### Why are the changes needed?
The bottom menu item cannot be displayed when the left menu tree is long
### Does this PR introduce any user-facing change?
Yes, if the menu item shows more, you'll see it by pulling down the
vertical scroll bar
before:
![image](https://user-images.githubusercontent.com/28332082/98805115-16995d80-2452-11eb-933a-3b72c14bea78.png)
after:
![image](https://user-images.githubusercontent.com/28332082/98805418-7e4fa880-2452-11eb-9a9b-8d265078297c.png)
### How was this patch tested? NA
Closes #30335 from liucht-inspur/master.
Authored-by: liucht <liucht@inspur.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(commit: cbc8be2)
The file was modifieddocs/css/main.css (diff)
Commit 3384bda453d0e728be311ce458e00d70d2484973 by wenchen
[SPARK-33468][SQL] ParseUrl  in ANSI mode should fail if input string is
not a valid url
### What changes were proposed in this pull request?
With `ParseUrl`, instead of return null we throw exception if input
string is not a vaild url.
### Why are the changes needed?
For ANSI mode.
### Does this PR introduce _any_ user-facing change?
Yes, user will get exception if `set spark.sql.ansi.enabled=true`.
### How was this patch tested?
Add test.
Closes #30399 from ulysses-you/SPARK-33468.
Lead-authored-by: ulysses <youxiduo@weidian.com> Co-authored-by:
ulysses-you <youxiduo@weidian.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(commit: 3384bda)
The file was modifieddocs/sql-ref-ansi-compliance.md (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala (diff)