Changes

Summary

  1. [SPARK-32741][SQL] Check if the same ExprId refers to the unique (commit: 3a299aa) (details)
  2. [SPARK-33006][K8S][DOCS] Add dynamic PVC usage example into K8s doc (commit: ece8d8e) (details)
  3. [SPARK-31753][SQL][DOCS][FOLLOW-UP] Add missing keywords in the SQL docs (commit: 3bdbb55) (details)
  4. [SPARK-33012][BUILD][K8S] Upgrade fabric8 to 4.10.3 (commit: d75222d) (details)
  5. [SPARK-33023][CORE] Judge  path of Windows need  add condition (commit: 0b5a379) (details)
  6. [SPARK-32723][WEBUI] Upgrade to jQuery 3.5.1 (commit: 28ed3a5) (details)
  7. [SPARK-32992][SQL] Map Oracle's ROWID type to StringType in read via (commit: 5651284) (details)
  8. [SQL][DOC][MINOR] Corrects input table names in the examples of CREATE (commit: d3dbe1a) (details)
  9. [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13 (commit: 0963fcd) (details)
Commit 3a299aa6480ac22501512cd0310d31a441d7dfdc by yamamuro
[SPARK-32741][SQL] Check if the same ExprId refers to the unique
attribute in logical plans
### What changes were proposed in this pull request?
Some plan transformations (e.g., `RemoveNoopOperators`) implicitly
assume the same `ExprId` refers to the unique attribute. But,
`RuleExecutor` does not check this integrity between logical plan
transformations. So, this PR intends to add this check in
`isPlanIntegral` of `Analyzer`/`Optimizer`.
This PR comes from the talk with cloud-fan viirya in
https://github.com/apache/spark/pull/29485#discussion_r475346278
### Why are the changes needed?
For better logical plan integrity checking.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
Closes #29585 from maropu/PlanIntegrityTest.
Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by:
Takeshi Yamamuro <yamamuro@apache.org>
(commit: 3a299aa)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FoldablePropagationSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEOptimizer.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala (diff)
The file was addedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlanIntegritySuite.scala
Commit ece8d8e22cf7e3924e44c16f58028c323dc54356 by dhyun
[SPARK-33006][K8S][DOCS] Add dynamic PVC usage example into K8s doc
### What changes were proposed in this pull request?
This updates K8s document to describe new dynamic PVC features.
### Why are the changes needed?
This will help the user use the new features easily.
### Does this PR introduce _any_ user-facing change?
Yes, but it's a doc updates.
### How was this patch tested?
Manual.
<img width="847" alt="Screen Shot 2020-09-28 at 3 54 53 PM"
src="https://user-images.githubusercontent.com/9700541/94494923-3ed04400-01a5-11eb-81f9-127db42d4256.png">
<img width="779" alt="Screen Shot 2020-09-28 at 3 55 07 PM"
src="https://user-images.githubusercontent.com/9700541/94494930-4394f800-01a5-11eb-9387-50ebc14af477.png">
Closes #29897 from dongjoon-hyun/SPARK-33006.
Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
(commit: ece8d8e)
The file was modifieddocs/running-on-kubernetes.md (diff)
Commit 3bdbb5546d2517dda6f71613927cc1783c87f319 by yamamuro
[SPARK-31753][SQL][DOCS][FOLLOW-UP] Add missing keywords in the SQL docs
### What changes were proposed in this pull request? update sql-ref
docs, the following key words will be added in this PR.
CLUSTERED BY SORTED BY INTO num_buckets BUCKETS
### Why are the changes needed? let more users know the sql key words
usage
### Does this PR introduce _any_ user-facing change? No
![image](https://user-images.githubusercontent.com/46367746/94428281-0a6b8080-01c3-11eb-9ff3-899f8da602ca.png)
![image](https://user-images.githubusercontent.com/46367746/94428285-0d667100-01c3-11eb-8a54-90e7641d917b.png)
![image](https://user-images.githubusercontent.com/46367746/94428288-0f303480-01c3-11eb-9e1d-023538aa6e2d.png)
### How was this patch tested? generate html test
Closes #29883 from GuoPhilipse/add-sql-missing-keywords.
Lead-authored-by: GuoPhilipse
<46367746+GuoPhilipse@users.noreply.github.com> Co-authored-by:
GuoPhilipse <guofei_ok@126.com> Signed-off-by: Takeshi Yamamuro
<yamamuro@apache.org>
(commit: 3bdbb55)
The file was modifieddocs/sql-ref-syntax-ddl-create-table-datasource.md (diff)
The file was modifieddocs/sql-ref-syntax-ddl-create-table-hiveformat.md (diff)
Commit d75222dd1b0cdaaa7c22964e974117923fd069bb by dhyun
[SPARK-33012][BUILD][K8S] Upgrade fabric8 to 4.10.3
### What changes were proposed in this pull request?
This PR aims to upgrade `kubernetes-client` library to track fabric8's
declared compatibility for k8s 1.18.0:
https://github.com/fabric8io/kubernetes-client#compatibility-matrix
### Why are the changes needed? According to fabric8, 4.9.2 is
incompatible with k8s 1.18.0.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Not tested yet.
Closes #29888 from laflechejonathan/jlf/fabric8Ugprade.
Authored-by: jlafleche <jlafleche@palantir.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
(commit: d75222d)
The file was modifiedresource-managers/kubernetes/integration-tests/pom.xml (diff)
The file was modifiedresource-managers/kubernetes/core/pom.xml (diff)
The file was modifieddev/deps/spark-deps-hadoop-2.7-hive-2.3 (diff)
The file was modifieddev/deps/spark-deps-hadoop-2.7-hive-1.2 (diff)
The file was modifieddev/deps/spark-deps-hadoop-3.2-hive-2.3 (diff)
Commit 0b5a379c1fb87aa536ebe9433e501dcf4f80ea60 by dhyun
[SPARK-33023][CORE] Judge  path of Windows need  add condition
`Utils.isWindows`
### What changes were proposed in this pull request? according to
https://github.com/apache/spark/pull/29881#discussion_r496648397 we need
add condition `Utils.isWindows`
### Why are the changes needed? add strict condition of judging path is
window path
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? No
Closes #29909 from AngersZhuuuu/SPARK-33023.
Authored-by: angerszhu <angers.zhu@gmail.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
(commit: 0b5a379)
The file was modifiedcore/src/main/scala/org/apache/spark/SparkContext.scala (diff)
Commit 28ed3a512ac6fcaafa885eb8092a68fe9e8f5c26 by dhyun
[SPARK-32723][WEBUI] Upgrade to jQuery 3.5.1
### What changes were proposed in this pull request? Upgrade to the
latest available version of jQuery (3.5.1).
### Why are the changes needed? There are some CVE-s reported
(CVE-2020-11022, CVE-2020-11023) affecting older versions of jQuery.
Although Spark UI is read-only and those CVEs doesn't seem to affect
Spark, using the latest version of this library can help to handle
vulnerability reports of security scans.
### Does this PR introduce _any_ user-facing change? No.
### How was this patch tested? Manual tests and checked the jQuery 3.5
upgrade guide.
Closes #29902 from peter-toth/SPARK-32723-upgrade-to-jquery-3.5.1.
Authored-by: Peter Toth <peter.toth@gmail.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
(commit: 28ed3a5)
The file was modifiedcore/src/main/scala/org/apache/spark/ui/UIUtils.scala (diff)
The file was modifieddocs/_layouts/global.html (diff)
The file was removedcore/src/main/resources/org/apache/spark/ui/static/jquery-3.4.1.min.js
The file was addedcore/src/main/resources/org/apache/spark/ui/static/jquery-3.5.1.min.js
The file was modifieddev/.rat-excludes (diff)
The file was addeddocs/js/vendor/jquery-3.5.1.min.js
The file was removeddocs/js/vendor/jquery-3.4.1.min.js
Commit 5651284c3b2f3d5cf3ced1301ab24e3f4d04c09e by yamamuro
[SPARK-32992][SQL] Map Oracle's ROWID type to StringType in read via
JDBC
### What changes were proposed in this pull request? Convert the `ROWID`
type in the Oracle JDBC dialect to Catalyst's `StringType`. The doc for
Oracle 19c says explicitly that the type must be string:
https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/Data-Types.html#GUID-AEF1FE4C-2DE5-4BE7-BB53-83AD8F1E34EF
### Why are the changes needed? To avoid the exception showed in
https://stackoverflow.com/questions/52244492/spark-jdbc-dataframereader-fails-to-read-oracle-table-with-datatype-as-rowid
### Does this PR introduce _any_ user-facing change? Yes
### How was this patch tested? N/A
Closes #29884 from MaxGekk/jdbc-oracle-rowid-string.
Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Takeshi
Yamamuro <yamamuro@apache.org>
(commit: 5651284)
The file was modifiedexternal/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/jdbc/OracleDialect.scala (diff)
Commit d3dbe1a9076c8a76be0590ca071bfbec6114813b by yamamuro
[SQL][DOC][MINOR] Corrects input table names in the examples of CREATE
FUNCTION doc
### What changes were proposed in this pull request? Fix Typo
### Why are the changes needed? To maintain consistency. Correct table
name should be used for SELECT command.
### Does this PR introduce _any_ user-facing change? Yes. Now CREATE
FUNCTION doc will show the correct name of table.
### How was this patch tested? Manually. Doc changes.
Closes #29920 from iRakson/fixTypo.
Authored-by: iRakson <raksonrakesh@gmail.com> Signed-off-by: Takeshi
Yamamuro <yamamuro@apache.org>
(commit: d3dbe1a)
The file was modifieddocs/sql-ref-syntax-ddl-create-function.md (diff)
Commit 0963fcd848f62b4f2231dfcf67f9beabf927c21e by srowen
[SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
### What changes were proposed in this pull request? After `SPARK-32851`
set `CODEGEN_FACTORY_MODE` to `CODEGEN_ONLY` of `sparkConf` in
`SharedSparkSessionBase`  to construction `SparkSession`  in test, the
test suite `SPARK-32459: UDF should not fail on WrappedArray` in
s.sql.UDFSuite exposed a codegen fallback issue in Scala 2.13 as follow:
```
- SPARK-32459: UDF should not fail on WrappedArray *** FAILED *** Caused
by: org.codehaus.commons.compiler.CompileException: File
'generated.java', Line 47, Column 99: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java',
Line 47, Column 99: No applicable constructor/method found for zero
actual parameters; candidates are: "public
scala.collection.mutable.Builder
scala.collection.mutable.ArraySeq$.newBuilder(java.lang.Object)",
"public scala.collection.mutable.Builder
scala.collection.mutable.ArraySeq$.newBuilder(scala.reflect.ClassTag)",
"public abstract scala.collection.mutable.Builder
scala.collection.EvidenceIterableFactory.newBuilder(java.lang.Object)"
```
The root cause is `WrappedArray` represent `mutable.ArraySeq`  in Scala
2.13 and has a different constructor of `newBuilder` method.
The main change of is pr is add Scala 2.13 only code part to deal with
`case match WrappedArray` in Scala 2.13.
### Why are the changes needed? We need to support a Scala 2.13 build
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested?
- Scala 2.12: Pass the Jenkins or GitHub Action
- Scala 2.13: All tests passed.
Do the following:
``` dev/change-scala-version.sh 2.13 mvn clean install -DskipTests  -pl
sql/core -Pscala-2.13 -am mvn test -pl sql/core -Pscala-2.13
```
**Before**
``` Tests: succeeded 8540, failed 1, canceled 1, ignored 52, pending 0
*** 1 TEST FAILED ***
```
**After**
``` Tests: succeeded 8541, failed 0, canceled 1, ignored 52, pending 0
All tests passed.
```
Closes #29903 from LuciferYang/fix-udfsuite.
Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Sean Owen
<srowen@gmail.com>
(commit: 0963fcd)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala (diff)